Fix the bugs found by the nu-conformance open questions (#19271)

## Description

The [nu-conformance](https://github.com/Fikoko/nu-conformance) suite
(#19206) recorded 35 behaviors as open questions. This PR fixes the ones
that are bugs, plus five that needed a decision (T2, T4, T8, T12 and
T13). The IDs are the question IDs in that repo's
[`open-questions.md`](https://github.com/Fikoko/nu-conformance/blob/master/open-questions.md).

### Before and after

| | Code | main | branch |
|---|---|---|---|
| S2 | `"\xFF"` | error `nu::parser::parse_mismatch`: expected any shape
| error `nu::parser::error`: invalid UTF-8 from `\x` escapes in string |
| S3 | `r#'abc` | error `nu::parser::unexpected_eof`: expected closing
'# | error `nu::parser::unclosed_delimiter`: expected `'#` here |
| T1 | `inf == 1` | `true` | `false` |
| T1 | `inf > 1` | `false` | `true` |
| T1 | `[1.0 inf] \| math max` | `1.0` | `inf` |
| T2 | `let x = 5; -$x` | `-$x` | `-5` |
| T2 | `let x = 5; 1 + -$x` | error
`nu::parser::operator_incompatible_types`: string | `-4` |
| T2 | `let x = 5sec; -$x` | `-$x` | `-5sec` |
| T2 | `-(2 + 3) \| describe` | `string` | `int` |
| T2 | `-(1)px` | `-1px` | `-1px` |
| T3 | `-0x10` | `-0x10` | `-16` |
| T3 | `+0b11` | error `nu::shell::external_command`: Command `+0b11`
not found | `3` |
| T4 | `9223372036854775808` | `9223372036854775808.0` | error
`nu::parser::error`: number too large (add `.0` for a float) in int |
| T4 | `9223372036854775808.0` | `9223372036854775808.0` |
`9223372036854775808.0` |
| T6 | `[INF Nan Infinity] \| to nuon` | `[inf, NaN, inf]` | `["INF",
"Nan", "Infinity"]` |
| T6 | `[inf nan NaN] \| to nuon` | `[inf, NaN, NaN]` | `[inf, NaN,
NaN]` |
| T7 | `9223372036854775808ns` | `15250wk 1day 23hr 47min 16sec 854ms
775µs 807ns` | error `nu::parser::literal_too_large`: duration too large
|
| T7 | `8EiB` | `9.2 EB` | error `nu::parser::literal_too_large`:
filesize too large |
| T8 | `[2023-02-29] \| to nuon` | `["2023-02-29"]` | error
`nu::parser::error`: not a valid date in datetime |
| T8 | `[2023-02-29.log] \| to nuon` | `["2023-02-29.log"]` |
`["2023-02-29.log"]` |
| T10 | `+5sec` | error `nu::shell::external_command`: Command `+5sec`
not found | `5sec` |
| T11 | `[1 2]] \| to nuon` | `[1, "2]"]` | error
`nu::parser::unbalanced_delimiter`: unexpected `]` (unbalanced with `[`)
|
| T12 | `{true: 1, null: 2} \| to nuon` | error
`nu::parser::parse_mismatch_with_full_string_msg`: expected string |
`{"true": 1, "null": 2}` |
| T12 | `[[1 2]; [3 4]] \| to nuon` | error `nu::parser::error`: must be
a string | `[["1", "2"]; [3, 4]]` |
| T13 | `print before; {a: 1, a: 2}` | error
`nu::shell::column_defined_twice`: field first defined here | error
`nu::parser::column_defined_twice`: field redefined here |
| O1 | `not "a" == "b"` | error `nu::shell::cant_convert`: can't convert
string to boolean | `true` |
| O1 | `let x = 1; not $x == 2` | error `nu::shell::cant_convert`: can't
convert int to boolean | `true` |
| O3 | `(-9223372036854775808) mod -1` | error
`nu::shell::operator_overflow`: modulo operation overflowed | `0` |
| O4 | `1 in [1.0]` | error `nu::parser::operator_incompatible_types`:
list<float> | `true` |
| D1 | `const x = 1; unlet $x; $x` | `1` | error
`nu::compile::invalid_literal`: constants cannot be deleted |
| D2 | `unlet $in` | `nothing` | error `nu::compile::invalid_literal`:
'$in' is a built-in variable and cannot be deleted |
| D3 | `mut x = 0; [1 2 3] \| where {\|v\| $x += 1; true } \| ignore;
$x` | `0` | error `nu::parser::expected_keyword`: capture of mutable
variable |
| F2 | `while false { } \| describe` | `nothing` | error
`nu::parser::unexpected_keyword`: not allowed in pipeline |
| F3 | `if 1 { }` | error `nu::shell::cant_convert`: can't convert int
to boolean | error `nu::shell::type_mismatch`: expected bool |
| F4 | `2.0 in 1.0..2.0` | `false` | `true` |
| F4 | `2.0 in 1.0..<2.0` | `true` | `false` |
| C1 | `def f [...r, a] { {r: $r, a: $a} }; f 1 2 3 \| to nuon` | `{r:
[2, 3], a: 1}` | error `nu::parser::positional_after_rest`: parameter a
after the rest parameter |
| C2 | `def f [a, a] { $a }; f 1 2` | `2` | error
`nu::parser::duplicate_parameter`: `a` uses a name that is already taken
|
| C2 | `def f [--x, --x] { $x }; f --x` | `false` | error
`nu::parser::duplicate_parameter`: `--x` uses a name that is already
taken |
| C2 | `def f [--help] { 'mine' }; f --help \| lines \| first` |
`Usage:` | error `nu::parser::error`: every command already has `--help`
|

C3, a script that calls two deprecated commands:

```nushell
$ nu two_deprecated.nu    # main
Warning: nu::parser::deprecated
   : `-- g is deprecated and will be removed in a future release.
```
```nushell
$ nu two_deprecated.nu    # branch
Warning: nu::parser::deprecated
   : `-- g is deprecated and will be removed in a future release.
Warning: nu::parser::deprecated
   : `-- k is deprecated and will be removed in a future release.
```
## User-facing changes (Release notes)

### Breaking changes

- `not` now has the precedence `help operators` lists: `not $a == $b`
means `not ($a == $b)`. Before, `not` applied to the next value only.
- `-$x` and `-(expr)` negate their value, also as a flag's value or an
argument after `--`. Before, `-$x` was the string `"-$x"` and `-(2 + 3)`
the string `"-5"`. Arguments to external commands are unchanged.
- An integer literal too large for 64 bits, such as
`9223372036854775808`, is an error. Add `.0` to write a float.
- Filesize and duration literals that don't fit are a parse error
(`nu::parser::literal_too_large`) instead of saturating to the largest
value.
- A word written like a date that isn't a valid date, such as
`2023-02-29`, is an error.
- `INF`, `Nan`, `Infinity` and other spellings of special floats are
strings. Use `inf`, `nan` or `NaN`. `from nuon` reads them the same way.
- A record or table literal that names a field twice is a parse error.
- `[1 2]]` is an error instead of `[1, "2]"]`.
- In a custom command, repeated parameter or flag names, repeated short
flags, a parameter after `...rest`, and a `--help` flag are errors.
- `while` and `loop` can't be piped, like `for`.
- `unlet` errors on constants and on `$in`.
- Assigning to a `mut` variable inside a `where` condition is an error;
the assignment never took effect.

### Fixes

- `inf` compares as larger than every finite number, and `[1.0 inf] |
math max` is `inf`.
- `2.0 in 1.0..2.0` is `true`, and `2.0 in 1.0..<2.0` is `false`.
- `-0x10`, `+0b11`, `+5sec` and `+1kB` are numbers.
- `1 in [1.0]` is `true`.
- `{true: 1}` and `[[1 2]; [3 4]]` parse, with the keys `"true"`, `"1"`
and `"2"`.
- `(-9223372036854775808) mod -1` is `0`.
- `if 1 { }` reports the same type mismatch as `while 1 { }`, and an
`if` or `while` condition that fails while it runs reports its own
error.
- `9223372036854775807ns` and `9223372036854775807b` are exact, and
integer filesize and duration literals no longer lose precision.
- A script or `nu -c` command shows every parse warning once, not only
the first.
- Clearer errors for invalid UTF-8 from `\x` escapes and for unclosed
raw strings.

## Additional notes
This commit is contained in:
Darren Schroeder authored and GitHub committed 2026-10-17 14:54:47 -05:00
1 parent de4d33d410
commit 973c590918
52 files changed
+1777 -431

No files matched your search

+3 -1
View File
@@ -255,7 +255,9 @@ fn is_operator_lhs(expr: &Expr) -> bool {
| Expr::Nothing
| Expr::Subexpression(_)
| Expr::Block(_)
| Expr::Closure(_) => true,
| Expr::Closure(_)
// `-$x` is one operand that binds tighter than every operator, like `-5`.
| Expr::UnaryMinus(_) => true,
Expr::AttributeBlock(_)
| Expr::VarDecl(_)
| Expr::Call(_)
+5 -5
View File
@@ -7,11 +7,11 @@ use nu_protocol::{
engine::{EngineState, Stack, StateWorkingSet},
process::check_exit_status_future,
report_error::report_compile_error,
report_parse_error, report_parse_warning,
report_parse_error,
shell_error::generic::GenericError,
};
use crate::util::print_pipeline;
use crate::util::{print_pipeline, report_parse_warnings};
#[derive(Default)]
pub struct EvaluateCommandsOpts {
@@ -64,9 +64,7 @@ pub fn evaluate_commands(
let mut working_set = StateWorkingSet::new(engine_state);
let check_errors = |working_set: &StateWorkingSet| {
if let Some(warning) = working_set.parse_warnings.first() {
report_parse_warning(Some(stack), working_set, warning);
}
report_parse_warnings(Some(stack), working_set);
if let Some(err) = working_set.parse_errors.first() {
report_parse_error(Some(stack), working_set, err);
@@ -83,6 +81,8 @@ pub fn evaluate_commands(
check_errors(&working_set);
if find_main_block_id_in_script(&working_set, &output).is_some() {
// The script's warnings are reported, so only the `main` call's are left to report.
working_set.parse_warnings.clear();
// The CLI parser has already escaped script arguments via `args_to_script`, so we must not
// escape them again here or we would double-quote values like `"arg 2"`.
output = parse(
+3 -5
View File
@@ -1,4 +1,4 @@
use crate::util::{eval_parsed_block_source, eval_source, print_pipeline};
use crate::util::{eval_parsed_block_source, eval_source, print_pipeline, report_parse_warnings};
use log::{info, trace};
use nu_engine::{eval_block, exit::kill_jobs_and_exit};
use nu_parser::parse;
@@ -8,7 +8,7 @@ use nu_protocol::{
debugger::WithoutDebug,
engine::{EngineState, Stack, StateWorkingSet},
report_error::report_compile_error,
report_parse_error, report_parse_warning,
report_parse_error,
shell_error::io::*,
};
use std::{path::PathBuf, sync::Arc};
@@ -98,9 +98,7 @@ pub fn evaluate_file(
trace!("parsing file: {file_path_str}");
let block = parse(&mut working_set, Some(file_path_str), &file, false);
if let Some(warning) = working_set.parse_warnings.first() {
report_parse_warning(None, &working_set, warning);
}
report_parse_warnings(None, &working_set);
// If any parse errors were found, report the first error and exit.
if let Some(err) = working_set.parse_errors.first() {
+25 -8
View File
@@ -541,7 +541,6 @@ fn find_matching_block_end_in_expr<'a>(
Expr::VarDecl(_) => None,
Expr::ExternalCall(..) => None,
Expr::Operator(_) => None,
Expr::UnaryNot(_) => None,
Expr::Keyword(..) => None,
Expr::ValueWithUnit(..) => None,
Expr::DateTime(_) => None,
@@ -672,13 +671,15 @@ fn find_matching_block_end_in_expr<'a>(
)
}),
Expr::Collect(_, expr) => find_matching_block_end_in_expr(
line,
get_block,
expr,
global_span_offset,
global_cursor_offset,
),
Expr::UnaryNot(expr) | Expr::UnaryMinus(expr) | Expr::Collect(_, expr) => {
find_matching_block_end_in_expr(
line,
get_block,
expr,
global_span_offset,
global_cursor_offset,
)
}
Expr::Block(block_id)
| Expr::Closure(block_id)
@@ -775,6 +776,22 @@ mod tests {
assert_eq!(on_open.raw_string(), line);
}
/// A bracket inside a negated operand still finds its match, with the cursor after it.
#[rstest]
#[case::unary_minus("-(1 + 2)")]
#[case::not_comparison("let a = true; not $a == (1 == 2)")]
fn matching_brackets_inside_negation(#[case] line: &str) {
let h = make_highlighter();
let styled = h.highlight(line, line.len());
let underlined: String = styled
.buffer
.iter()
.filter(|(style, _)| style.is_underline)
.map(|(_, text)| text.as_str())
.collect();
assert_eq!(underlined, "()", "matching brackets in {line:?}");
}
#[test]
fn highlight_slices_cover_the_whole_line() {
let h = make_highlighter();
+15 -3
View File
@@ -327,6 +327,20 @@ fn finish_eval_source(
exit_code
}
/// Report every parse warning in `working_set`, each once. The parser can try a piece of code more
/// than once and leave the same warning behind each time, so a repeat at the same place is
/// skipped. [`report_parse_warning`] itself drops repeats of `--report first` warnings.
pub(crate) fn report_parse_warnings(stack: Option<&Stack>, working_set: &StateWorkingSet) {
let mut reported: Vec<(Span, String)> = vec![];
for warning in &working_set.parse_warnings {
let key = (warning.span(), warning.to_string());
if !reported.contains(&key) {
report_parse_warning(stack, working_set, warning);
reported.push(key);
}
}
}
pub(crate) fn evaluate_source(
engine_state: &mut EngineState,
stack: &mut Stack,
@@ -343,9 +357,7 @@ pub(crate) fn evaluate_source(
source,
false,
);
if let Some(warning) = working_set.parse_warnings.first() {
report_parse_warning(Some(stack), &working_set, warning);
}
report_parse_warnings(Some(stack), &working_set);
if let Some(err) = working_set.parse_errors.first() {
report_parse_error(Some(stack), &working_set, err);
+3
View File
@@ -4086,6 +4086,9 @@ fn operator_completions(mut custom_completer: NuCompleter) {
let suggestions = custom_completer.complete_blocking("1 bit-s", 7);
let expected: Vec<_> = vec!["bit-shl", "bit-shr"];
match_suggestions(&expected, &suggestions);
// A negated int takes the operators of an int.
let suggestions = custom_completer.complete_blocking("-(1) ", 5);
assert_eq!(20, suggestions.len());
let suggestions = custom_completer.complete_blocking("'str' ", 6);
// == != > < >= <= in not-in
@@ -3,7 +3,9 @@ use itertools::Itertools;
use nu_cmd_base::input_handler::{CmdArgument, operate};
use nu_engine::command_prelude::*;
use nu_parser::{DURATION_UNIT_GROUPS, parse_unit_value};
use nu_protocol::{SUPPORTED_DURATION_UNITS, Unit, ast::Expr};
use nu_protocol::{
ParseError, SUPPORTED_DURATION_UNITS, Unit, ast::Expr, shell_error::generic::GenericError,
};
use std::str::FromStr;
use std::sync::LazyLock;
@@ -354,16 +356,28 @@ fn parse_clock_duration(s: &str, span: Span) -> Result<Option<Value>, ShellError
}
fn string_to_duration(s: &str, span: Span) -> Result<DurationPart, ShellError> {
if let Some(Ok(expression)) = parse_unit_value(
match parse_unit_value(
s.as_bytes(),
span,
DURATION_UNIT_GROUPS,
Type::Duration,
|x| x,
) && let Expr::ValueWithUnit(value) = expression.expr
&& let Expr::Int(x) = value.expr.expr
{
return Ok((value.unit.item, x, span));
) {
Some(Ok(expression)) => {
if let Expr::ValueWithUnit(value) = expression.expr
&& let Expr::Int(x) = value.expr.expr
{
return Ok((value.unit.item, x, span));
}
}
Some(Err(mk_err)) if matches!(mk_err("duration"), ParseError::LiteralTooLarge(..)) => {
return Err(ShellError::Generic(GenericError::new(
"duration too large",
"duration too large",
span,
)));
}
_ => {}
}
Err(ShellError::InvalidUnit {
+2 -2
View File
@@ -1,5 +1,5 @@
use nu_engine::command_prelude::*;
use nu_protocol::ast::{Assignment, Bits, Boolean, Comparison, Math, Operator};
use nu_protocol::ast::{Assignment, Bits, Boolean, Comparison, Math, NOT_PRECEDENCE, Operator};
use strum::IntoEnumIterator;
#[derive(Clone)]
@@ -79,7 +79,7 @@ impl Command for HelpOperators {
"operator" => Value::string("not", head),
"name" => Value::string("Not", head),
"description" => Value::string("Negates a value or expression.", head),
"precedence" => Value::int(55, head),
"precedence" => Value::int(NOT_PRECEDENCE.into(), head),
},
head,
));
+31
View File
@@ -16,3 +16,34 @@ fn if_with_non_block_body_errors_without_panic(#[case] src: &str) -> Result {
assert_matches!(err, CompileError::InvalidKeywordCall { .. });
Ok(())
}
/// A non-bool condition is the same type mismatch in `if` and `while`.
#[rstest]
#[case::int("if 1 { }")]
#[case::string("if 'a' { } else { }")]
fn if_non_bool_condition(#[case] src: &str) -> Result {
test()
.run(src)
.expect_error_code_eq("nu::shell::type_mismatch")
}
/// An `if` evaluated as a constant reports a non-bool condition the same way.
#[test]
fn const_if_non_bool_condition() -> Result {
let err = test()
.run("const x = if 1 { 2 } else { 3 }")
.expect_parse_error()?
.to_string();
assert_contains("type_mismatch", err);
Ok(())
}
/// A condition that fails while it runs reports its own error, not that it isn't a bool.
#[rstest]
#[case::if_("if ([1] | each { error make {msg: boom} }) { 1 }")]
#[case::while_("while ([1] | each { error make {msg: boom} }) { break }")]
fn failing_condition_reports_its_error(#[case] src: &str) -> Result {
test()
.run(format!("try {{ {src} }} catch {{|e| $e.msg }}"))
.expect_value_eq("boom")
}
+11
View File
@@ -57,3 +57,14 @@ fn failed_loop_should_break_running() -> Result {
.run(code)
.expect_error_code_eq("nu::shell::non_zero_exit_code")
}
/// `loop` is a statement without output, so like `for` it can't be part of a pipeline, also
/// after an environment shorthand.
#[rstest::rstest]
#[case::bare("loop { break } | describe")]
#[case::env_shorthand("FOO=1 loop { break } | describe")]
fn loop_in_pipeline_is_an_error(#[case] code: &str) -> Result {
test()
.run(code)
.expect_error_code_eq("nu::parser::unexpected_keyword")
}
+2 -3
View File
@@ -240,10 +240,9 @@ fn floor_div_mod_overflow() -> Result {
let outcome = test().run(&code).expect_shell_error()?;
assert!(matches!(outcome, ShellError::OperatorOverflow { .. }));
// Any number mod -1 is 0, even when the matching division overflows.
let code = format!("{} mod -1", i64::MIN);
let outcome = test().run(&code).expect_shell_error()?;
assert!(matches!(outcome, ShellError::OperatorOverflow { .. }));
Ok(())
test().run(&code).expect_value_eq(0)
}
#[test]
@@ -316,7 +316,7 @@ fn table_collapse_hearts() -> Result {
fn table_collapse_does_wrapping_for_long_strings() -> Result {
test()
.run("
[[a]; [11111111111111111111111111111111111111111111111111111111111111111111111111111111]]
[[a]; [11111111111111111111111111111111111111111111111111111111111111111111111111111111.0]]
| table --width=80 --collapse
")
.expect_value_eq(indoc! {"
+23
View File
@@ -69,3 +69,26 @@ fn unlet_multiple_deletes_both() -> Result {
assert_matches!(err, ShellError::VariableNotFoundAtRuntime { .. });
Ok(())
}
/// In a pipeline the parser replaces `$in` with a fresh variable named `$in`, which must still be
/// treated as the built-in.
#[test]
fn unlet_builtin_in() -> Result {
let err = test().run("unlet $in").expect_compile_error()?;
assert_matches!(
err,
CompileError::InvalidLiteral { msg, .. } if msg.contains("'$in' is a built-in variable")
);
Ok(())
}
/// A constant can't be deleted: reading it afterwards would still give its value.
#[test]
fn unlet_constant() -> Result {
let err = test().run("const x = 1; unlet $x").expect_compile_error()?;
assert_matches!(
err,
CompileError::InvalidLiteral { msg, .. } if msg.contains("constants cannot be deleted")
);
Ok(())
}
@@ -223,3 +223,22 @@ fn nested_stored_condition() -> Result {
.run("let nested = {cond: { $in mod 2 == 0 }}; 1..10 | where $nested.cond")
.expect_value_eq([2, 4, 6, 8, 10])
}
/// A `where` closure is a row condition, so like `where $it > $x` it can read a `mut` variable.
#[test]
fn closure_reads_mutable_variable() -> Result {
test()
.run("mut x = 1; [1 2 3] | where {|v| $v > $x }")
.expect_value_eq([2, 3])
}
/// A row condition runs over copies of the variables it uses, so assigning to a `mut` variable
/// in one is an error instead of an update that is silently lost.
#[rstest::rstest]
#[case::closure("mut x = 0; [1 2 3] | where {|v| $x += 1; true }")]
#[case::block("mut x = 0; [1 2 3] | where { $x = 5; true }")]
fn assigning_mutable_variable_is_an_error(#[case] code: &str) -> Result {
let err = test().run(code).expect_parse_error()?;
pretty_assertions::assert_matches!(err, ParseError::CaptureOfMutableVar(_));
Ok(())
}
@@ -60,3 +60,22 @@ fn failed_while_should_break_running() -> Result {
.run(code)
.expect_error_code_eq("nu::shell::non_zero_exit_code")
}
/// `while` is a statement without output, so like `for` it can't be part of a pipeline, also
/// after an environment shorthand.
#[rstest::rstest]
#[case::bare("while false { } | describe")]
#[case::env_shorthand("FOO=1 while false { } | describe")]
fn while_in_pipeline_is_an_error(#[case] code: &str) -> Result {
test()
.run(code)
.expect_error_code_eq("nu::parser::unexpected_keyword")
}
/// A non-bool condition is the same type mismatch in `while` and `if`.
#[test]
fn while_non_bool_condition() -> Result {
test()
.run("while 1 { }")
.expect_error_code_eq("nu::shell::type_mismatch")
}
+26 -22
View File
@@ -476,7 +476,7 @@ fn bare_var_id(expr: &Expression) -> Option<nu_protocol::VarId> {
}
pub(crate) fn compile_unlet(
_working_set: &StateWorkingSet,
working_set: &StateWorkingSet,
builder: &mut BlockBuilder,
call: &Call,
io_reg: RegId,
@@ -494,28 +494,32 @@ pub(crate) fn compile_unlet(
match var_id {
Some(var_id) => {
// Prevent deletion of built-in variables that are essential for nushell operation
if var_id == NU_VARIABLE_ID
|| var_id == ENV_VARIABLE_ID
|| var_id == IN_VARIABLE_ID
|| var_id == nu_protocol::LAST_VARIABLE_ID
{
// Determine the variable name for the error message
let var_name = match var_id {
NU_VARIABLE_ID => "nu",
ENV_VARIABLE_ID => "env",
IN_VARIABLE_ID => "in",
_ if var_id == nu_protocol::LAST_VARIABLE_ID => {
nu_protocol::LAST_RESULT_VAR_NAME
}
_ => "unknown", // This should never happen due to the check above
};
let variable = working_set.get_variable(var_id);
// Prevent deletion of built-in variables that are essential for nushell
// operation. In a pipeline the parser replaces `$in` with a fresh variable that
// it names `$in` (see `wrap_expr_with_collect`), which is the built-in too.
let builtin_name = match var_id {
NU_VARIABLE_ID => Some("nu"),
ENV_VARIABLE_ID => Some("env"),
IN_VARIABLE_ID => Some("in"),
nu_protocol::LAST_VARIABLE_ID => Some(nu_protocol::LAST_RESULT_VAR_NAME),
_ if variable.name.as_deref() == Some(b"$in") => Some("in"),
_ => None,
};
if let Some(var_name) = builtin_name {
return Err(CompileError::InvalidLiteral {
msg: format!(
"'${}' is a built-in variable and cannot be deleted",
var_name
),
msg: format!("'${var_name}' is a built-in variable and cannot be deleted"),
span: arg.span,
});
}
// A constant lives in the engine state, not on the stack, and reading it after
// a `DropVariable` falls back to its value. It was also already inlined where
// the parser evaluated constants, so deleting it can't work. (`$nu` is a constant
// too, which is why the built-ins are checked first.)
if variable.const_val.is_some() {
return Err(CompileError::InvalidLiteral {
msg: "constants cannot be deleted".into(),
span: arg.span,
});
}
+24 -1
View File
@@ -5,7 +5,7 @@ use super::{
use nu_protocol::{
ENV_VARIABLE_ID, IN_VARIABLE_ID, IntoSpanned, RegId, Value,
ast::{CellPath, Expr, Expression, ListItem, RecordItem, ValueWithUnit},
ast::{CellPath, Expr, Expression, ListItem, Math, Operator, RecordItem, ValueWithUnit},
engine::StateWorkingSet,
ir::{DataSlice, Instruction, Literal},
};
@@ -197,6 +197,29 @@ pub(crate) fn compile_expression(
builder.push(Instruction::Not { src_dst: out_reg }.into_spanned(expr.span))?;
Ok(())
}
Expr::UnaryMinus(operand) => {
drop_input(builder)?;
compile_expression(
working_set,
builder,
operand,
RedirectModes::value(operand.span),
None,
out_reg,
)?;
// Negate by multiplying by -1, which works for every type that can be negated
// (ints, floats, durations and filesizes).
let minus_one = builder.literal(Literal::Int(-1).into_spanned(expr.span))?;
builder.push(
Instruction::BinaryOp {
lhs_dst: out_reg,
op: Operator::Math(Math::Multiply),
rhs: minus_one,
}
.into_spanned(expr.span),
)?;
Ok(())
}
Expr::BinaryOp(lhs, op, rhs) => {
if let Expr::Operator(operator) = op.expr {
// Don't drop input if the expression uses $in
+29 -26
View File
@@ -19,12 +19,17 @@ pub(crate) fn compile_if(
// Pseudocode:
//
// %io_reg <- <condition>
// not %io_reg
// branch-if %io_reg, FALSE
// collect %io_reg
// branch-if %io_reg, TRUE
// jump FALSE
// TRUE: ...<true_block>...
// jump END
// FALSE: ...<else_expr>... OR drop %io_reg
// END:
//
// Branching on the condition itself, as `while` does, makes a non-bool condition the same
// "expected bool" type mismatch in both keywords. Collecting it first runs a stream to its
// end, so a failing command in the condition reports its own error.
let invalid = || CompileError::InvalidKeywordCall {
keyword: "if".into(),
span: call.head,
@@ -41,31 +46,27 @@ pub(crate) fn compile_if(
let false_label = builder.label(None);
let end_label = builder.label(None);
let not_condition_reg = {
// Compile the condition first
let condition_reg = builder.next_register()?;
compile_expression(
working_set,
builder,
condition,
RedirectModes::value(condition.span),
None,
condition_reg,
)?;
// Compile the condition first
let condition_reg = builder.next_register()?;
compile_expression(
working_set,
builder,
condition,
RedirectModes::value(condition.span),
None,
condition_reg,
)?;
// Negate the condition - we basically only want to jump if the condition is false
builder.push(
Instruction::Not {
src_dst: condition_reg,
}
.into_spanned(call.head),
)?;
condition_reg
};
// Set up a branch if the condition is false.
builder.branch_if(not_condition_reg, false_label, call.head)?;
// Branch to the true case, and jump to the false case otherwise.
builder.push(
Instruction::Collect {
src_dst: condition_reg,
}
.into_spanned(condition.span),
)?;
builder.branch_if(condition_reg, true_label, call.head)?;
builder.add_comment("if true");
builder.jump(false_label, call.head)?;
builder.add_comment("if false");
// Compile the true case
@@ -894,6 +895,8 @@ pub(crate) fn compile_while(
io_reg,
)?;
// Collect the condition first, as `if` does, so a failing command in it reports its own error.
builder.push(Instruction::Collect { src_dst: io_reg }.into_spanned(cond_arg.span))?;
builder.branch_if(io_reg, true_label, call.head)?;
builder.add_comment("while");
builder.jump(loop_.break_label, call.head)?;
@@ -6,6 +6,7 @@
use super::{BlockKind, OpenFrame};
use nu_protocol::{ParseError, Span};
use std::borrow::Cow;
pub(super) fn closing_delimiter_str(block: BlockKind) -> &'static str {
match block {
@@ -177,7 +178,7 @@ fn trim_ascii_end(bytes: &[u8]) -> &[u8] {
pub(super) fn unclosed_from_open(
input: &[u8],
span_offset: usize,
delimiter: &'static str,
delimiter: impl Into<Cow<'static, str>>,
open_span: Span,
end_span: Span,
) -> ParseError {
+7
View File
@@ -245,6 +245,13 @@ fn flatten_expression_into(
));
flatten_expression_into(working_set, not, output);
}
Expr::UnaryMinus(operand) => {
output.push((
Span::new(expr.span.start, expr.span.start + 1),
FlatShape::Operator,
));
flatten_expression_into(working_set, operand, output);
}
Expr::Collect(_, expr) => {
flatten_expression_into(working_set, expr, output);
}
+62 -17
View File
@@ -246,13 +246,26 @@ const NESTED_FAST_CLASS: [u8; 256] = {
table
};
/// What the lexer is reading, for the few rules that depend on it.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub(crate) enum LexMode {
/// Code.
Code,
/// A signature, where `<` and `>` nest like brackets.
Signature,
/// The inside of a list literal, without its own brackets. A `]` that closes no `[` there
/// would close the list's `[` before the list ends (`[1 2]]`), so it is an error instead of
/// part of a bare word.
ListItems,
}
fn lex_item<B: Brackets>(
input: &[u8],
curr_offset: &mut usize,
span_offset: usize,
additional_whitespace: &[u8],
special_tokens: &[u8],
in_signature: bool,
mode: LexMode,
brackets: B,
) -> (Token, Option<ParseError>) {
let token_start = *curr_offset;
@@ -262,7 +275,7 @@ fn lex_item<B: Brackets>(
span_offset,
additional_whitespace,
special_tokens,
in_signature,
mode,
brackets,
);
if B::JUMPS && err.is_some() {
@@ -278,7 +291,7 @@ fn lex_item<B: Brackets>(
span_offset,
additional_whitespace,
special_tokens,
in_signature,
mode,
Rescan,
);
}
@@ -292,7 +305,7 @@ fn lex_item_scan<B: Brackets>(
span_offset: usize,
additional_whitespace: &[u8],
special_tokens: &[u8],
in_signature: bool,
mode: LexMode,
brackets: B,
) -> (Token, Option<ParseError>) {
// Tracks the opening quote character and its span while inside a string.
@@ -569,7 +582,7 @@ fn lex_item_scan<B: Brackets>(
last_sig_char = Some(c);
at_line_start = false;
} else if B::JUMPS
&& !in_signature
&& mode != LexMode::Signature
&& (c == b'[' || c == b'{' || c == b'(')
&& let Some(close) = brackets.close_of(span_offset + *curr_offset)
&& close - span_offset < input.len()
@@ -592,7 +605,7 @@ fn lex_item_scan<B: Brackets>(
});
last_sig_char = Some(c);
at_line_start = false;
} else if c == b'<' && in_signature {
} else if c == b'<' && mode == LexMode::Signature {
let open_span = Span::new(span_offset + *curr_offset, span_offset + *curr_offset + 1);
block_level.push(OpenFrame {
kind: BlockKind::AngleBracket,
@@ -600,7 +613,7 @@ fn lex_item_scan<B: Brackets>(
});
last_sig_char = Some(c);
at_line_start = false;
} else if c == b'>' && in_signature {
} else if c == b'>' && mode == LexMode::Signature {
if let Some(OpenFrame {
kind: BlockKind::AngleBracket,
..
@@ -611,12 +624,14 @@ fn lex_item_scan<B: Brackets>(
last_sig_char = Some(c);
at_line_start = false;
} else if c == b']' {
// Closing `]` — pop matching `[`, else real mismatch if another opener is open.
// Closing `]` — pop matching `[`, else real mismatch if another opener is open. A
// stray `]` in a bare word (`a]`, the glob `[]]*`) is allowed, except inside a list,
// where it closes the list early: `[1 2]]` would otherwise parse as `[1, "2]"]`.
if let Some(frame) =
block_level.pop_if(|frame| matches!(frame.kind, BlockKind::SquareBracket))
{
brackets.record(frame.open_span.start, span_offset + *curr_offset);
} else if !block_level.is_empty() {
} else if !block_level.is_empty() || mode == LexMode::ListItems {
*curr_offset += 1;
let span = Span::new(span_offset + token_start, span_offset + *curr_offset);
let close_span = Span::new(span.end - 1, span.end);
@@ -977,11 +992,21 @@ fn lex_raw_string(
*curr_offset += 1
}
if !matches {
// Report it like any other unclosed string, with the closer this raw string needs.
let mut expected = '\''.to_string();
expected.push_str(&"#".repeat(prefix_sharp_cnt));
return Err(ParseError::UnexpectedEof(
let opener = Span::new(
span_offset + start,
span_offset + start + prefix_sharp_cnt + 2,
);
// Label the last byte, as the other unclosed strings do.
let end = span_offset + *curr_offset;
return Err(unclosed_from_open(
input,
span_offset,
expected,
Span::new(span_offset + *curr_offset - 1, span_offset + *curr_offset),
opener,
Span::new(end - 1, end),
));
}
Ok(())
@@ -1005,7 +1030,7 @@ pub fn lex_signature(
additional_whitespace,
special_tokens,
skip_comment,
true,
LexMode::Signature,
None,
NoBrackets,
);
@@ -1057,7 +1082,7 @@ pub(crate) fn lex_n_tokens_with<B: Brackets>(
additional_whitespace,
special_tokens,
skip_comment,
false,
LexMode::Code,
Some(max_tokens),
brackets,
);
@@ -1097,6 +1122,27 @@ pub(crate) fn lex_with<B: Brackets>(
special_tokens: &[u8],
skip_comment: bool,
brackets: B,
) -> (Vec<Token>, Option<ParseError>) {
lex_in_mode(
input,
span_offset,
additional_whitespace,
special_tokens,
skip_comment,
LexMode::Code,
brackets,
)
}
/// [`lex_with`], reading `input` as `mode` says.
pub(crate) fn lex_in_mode<B: Brackets>(
input: &[u8],
span_offset: usize,
additional_whitespace: &[u8],
special_tokens: &[u8],
skip_comment: bool,
mode: LexMode,
brackets: B,
) -> (Vec<Token>, Option<ParseError>) {
let mut state = LexState {
input,
@@ -1111,7 +1157,7 @@ pub(crate) fn lex_with<B: Brackets>(
additional_whitespace,
special_tokens,
skip_comment,
false,
mode,
None,
brackets,
);
@@ -1123,8 +1169,7 @@ fn lex_internal<B: Brackets>(
additional_whitespace: &[u8],
special_tokens: &[u8],
skip_comment: bool,
// within signatures we want to treat `<` and `>` specially
in_signature: bool,
mode: LexMode,
max_tokens: Option<usize>,
brackets: B,
) {
@@ -1266,7 +1311,7 @@ fn lex_internal<B: Brackets>(
state.span_offset,
additional_whitespace,
special_tokens,
in_signature,
mode,
brackets,
);
if state.error.is_none() {
+36 -3
View File
@@ -14,7 +14,9 @@
//! the same opening bracket inside a group. The exception is signature lexing, where `<` and `>`
//! nest too, so it neither records nor jumps.
use crate::lex::{LexState, NoBrackets, RecordBrackets, Token, lex_n_tokens_with, lex_with};
use crate::lex::{
LexMode, LexState, NoBrackets, RecordBrackets, Token, lex_in_mode, lex_n_tokens_with, lex_with,
};
use nu_protocol::{
ParseError, Span,
engine::{BracketTable, StateWorkingSet},
@@ -106,23 +108,54 @@ pub(crate) fn lex_span(
additional_whitespace: &[u8],
special_tokens: &[u8],
skip_comment: bool,
) -> (Vec<Token>, Option<ParseError>) {
lex_span_in_mode(
working_set,
span,
additional_whitespace,
special_tokens,
skip_comment,
LexMode::Code,
)
}
/// Lex the inside of a list literal (`span`, without its brackets) as [`lex_span`] does, with the
/// items separated by whitespace, newlines or commas, and a `]` that closes no `[` an error (see
/// [`LexMode::ListItems`]).
pub(crate) fn lex_list_items(
working_set: &StateWorkingSet,
span: Span,
) -> (Vec<Token>, Option<ParseError>) {
lex_span_in_mode(working_set, span, b"\n\r,", &[], true, LexMode::ListItems)
}
/// [`lex_span`], reading the span as `mode` says.
fn lex_span_in_mode(
working_set: &StateWorkingSet,
span: Span,
additional_whitespace: &[u8],
special_tokens: &[u8],
skip_comment: bool,
mode: LexMode,
) -> (Vec<Token>, Option<ParseError>) {
let input = working_set.get_span_contents(span);
match find_bracket_table(working_set, span) {
Some(table) => lex_with(
Some(table) => lex_in_mode(
input,
span.start,
additional_whitespace,
special_tokens,
skip_comment,
mode,
table,
),
None => lex_with(
None => lex_in_mode(
input,
span.start,
additional_whitespace,
special_tokens,
skip_comment,
mode,
NoBrackets,
),
}
+7
View File
@@ -119,6 +119,13 @@ fn parse_unknown_arg(
.map(|arg| arg.shape.clone())
.unwrap_or(SyntaxShape::Any);
// The argument goes to another program, which reads a leading `-` as a flag, so `-$x` and
// `-(expr)` stay the strings they spell, as they do for `^cmd`, instead of negating.
let shape = match (shape, working_set.get_span_contents(span)) {
(SyntaxShape::Any, [b'-', b'$' | b'(', ..]) => SyntaxShape::String,
(shape, _) => shape,
};
crate::parser::parse_value(working_set, span, &shape, None)
}
+43 -1
View File
@@ -10,6 +10,7 @@ use nu_protocol::{
use rustc_hash::FxBuildHasher;
use std::{
collections::{HashMap, HashSet},
ops::ControlFlow,
sync::Arc,
};
@@ -253,6 +254,38 @@ pub fn discover_captures_in_pattern(pattern: &MatchPattern, seen: &mut SeenVars)
}
}
/// The span of the target of an assignment in `block` (`$x = …`, `$x.a += …`) to a `mut`
/// variable among `captures`.
fn assigned_mutable_capture(
working_set: &StateWorkingSet,
block: &Block,
captures: &[(VarId, Span)],
) -> Option<Span> {
block.find_map(working_set, &|expr: &Expression| {
let Expr::BinaryOp(lhs, op, _) = &expr.expr else {
return ControlFlow::Continue(());
};
let target = match &lhs.expr {
Expr::Var(var_id) => Some(*var_id),
Expr::FullCellPath(path) => match path.head.expr {
Expr::Var(var_id) => Some(var_id),
_ => None,
},
_ => None,
};
match target {
Some(var_id)
if matches!(op.expr, Expr::Operator(Operator::Assignment(_)))
&& captures.iter().any(|(captured, _)| *captured == var_id)
&& working_set.get_variable(var_id).mutable =>
{
ControlFlow::Break(Some(lhs.span))
}
_ => ControlFlow::Continue(()),
}
})
}
pub fn discover_captures_in_expr(
working_set: &StateWorkingSet,
expr: &Expression,
@@ -268,7 +301,7 @@ pub fn discover_captures_in_expr(
discover_captures_in_expr(working_set, lhs, seen, seen_blocks, output)?;
discover_captures_in_expr(working_set, rhs, seen, seen_blocks, output)?;
}
Expr::UnaryNot(expr) => {
Expr::UnaryNot(expr) | Expr::UnaryMinus(expr) => {
discover_captures_in_expr(working_set, expr, seen, seen_blocks, output)?;
}
Expr::Closure(block_id) => {
@@ -492,6 +525,15 @@ pub fn discover_captures_in_expr(
results
};
// A row condition (`where {|v| ...}` included) runs as a closure over copies of what
// it captures. Reading a `mut` variable there is fine (`where size > $limit`), but an
// assignment would only change the copy, so it is the same error as in a closure.
if matches!(expr.expr, Expr::RowCondition(_))
&& let Some(span) = assigned_mutable_capture(working_set, block, &results)
{
return Err(ParseError::CaptureOfMutableVar(span));
}
for &(var_id, span) in &results {
if !seen.contains(&var_id) {
output.push((var_id, span))
+13
View File
@@ -660,6 +660,19 @@ rest.name
if let Some(decl_id) = working_set.find_predecl(name.as_bytes()) {
signature.name.clone_from(&name);
if !has_wrapped {
// `--help` always shows the command's help before the body runs, so a flag of
// that name could never be set. (`-h` stays usable as another flag's short form.)
if let Some(flag) = signature.get_long_flag("help") {
let span = flag.var_id.map_or(call_span, |var_id| {
working_set.get_variable(var_id).declaration_span
});
working_set.error(ParseError::LabeledErrorWithHelp {
error: "The `--help` flag is reserved".into(),
label: "every command already has `--help`".into(),
help: "Rename this flag. `--help` always shows the command's help.".into(),
span,
});
}
*signature = signature.add_help();
}
signature.description = desc;
+268 -122
View File
@@ -3,11 +3,11 @@
use crate::{
Token, TokenContents,
lex::{LexState, is_assignment_operator},
lex_once::{find_bracket_table, lex_n_tokens_in, lex_span},
lex_once::{find_bracket_table, lex_list_items, lex_n_tokens_in, lex_span},
lite_parser::{LiteCommand, lite_parse},
parse_helpers::{
PERCENT_FORCED_BUILTIN_PARSER_INFO, extract_spread_list, extract_spread_record, garbage,
garbage_pipeline,
PERCENT_FORCED_BUILTIN_PARSER_INFO, earlier_use, extract_spread_list,
extract_spread_record, garbage, garbage_pipeline,
},
parse_keywords::{
is_unaliasable_parser_keyword, parse_alias, parse_attribute_block, parse_const, parse_def,
@@ -16,7 +16,7 @@ use crate::{
parse_overlay_use, parse_run, parse_run_expr, parse_source, parse_type_alias, parse_use,
parse_where, parse_where_expr,
},
parse_literals::has_range_operator,
parse_literals::{has_range_operator, is_unmatched_paren},
parse_patterns::parse_pattern,
parse_pipelines::{parse_block, parse_pipeline_element, redirecting_builtin_error},
parser::{
@@ -27,7 +27,7 @@ use crate::{
parse_raw_string, parse_regular_external_arg, parse_signature, parse_signature_helper,
parse_simple_cell_path, parse_string, parse_string_strict,
},
type_check::math_result_type,
type_check::{math_result_type, negation_type},
};
use itertools::Itertools;
use log::trace;
@@ -49,31 +49,27 @@ pub fn is_math_expression_like(working_set: &mut StateWorkingSet, span: Span) ->
let starting_error_count = working_set.parse_errors.len();
// Number
parse_number(working_set, span);
if working_set.parse_errors.len() == starting_error_count {
return true;
// A token that a literal parser accepts is a value. So is one it rejects as an invalid
// literal of its kind (`9223372036854775808`, `2023-02-29`, `9223372036854775808ns`): parsing
// it as a value reports that error instead of running the token as an external command.
// Other errors mean the token isn't that kind of literal (`42m_b` and `0xdump` are commands).
let is_literal_error = |error: Option<&ParseError>| {
matches!(
error,
Some(ParseError::InvalidLiteral(..) | ParseError::LiteralTooLarge(..))
)
};
let literal_parsers: [fn(&mut StateWorkingSet, Span) -> Expression; 4] =
[parse_number, parse_filesize, parse_duration, parse_datetime];
for parse_literal in literal_parsers {
parse_literal(working_set, span);
let first_error = working_set.parse_errors.get(starting_error_count);
let is_literal = first_error.is_none() || is_literal_error(first_error);
working_set.parse_errors.truncate(starting_error_count);
if is_literal {
return true;
}
}
working_set.parse_errors.truncate(starting_error_count);
// Filesize
parse_filesize(working_set, span);
if working_set.parse_errors.len() == starting_error_count {
return true;
}
working_set.parse_errors.truncate(starting_error_count);
parse_duration(working_set, span);
if working_set.parse_errors.len() == starting_error_count {
return true;
}
working_set.parse_errors.truncate(starting_error_count);
parse_datetime(working_set, span);
if working_set.parse_errors.len() == starting_error_count {
return true;
}
working_set.parse_errors.truncate(starting_error_count);
parse_binary(working_set, span);
// We need an additional negate match to check if the last error was unexpected
@@ -90,13 +86,16 @@ pub fn is_math_expression_like(working_set: &mut StateWorkingSet, span: Span) ->
}
working_set.parse_errors.truncate(starting_error_count);
// A bound that is an invalid literal (`1..99999999999999999999`) makes a range too, as above.
let is_range = has_range_operator(working_set.get_span_contents(span))
&& parse_range(working_set, span).is_some();
&& (parse_range(working_set, span).is_some()
|| is_literal_error(working_set.parse_errors.get(starting_error_count)));
working_set.parse_errors.truncate(starting_error_count);
is_range
}
fn is_env_variable_name(bytes: &[u8]) -> bool {
/// Whether `bytes` is a name that a `NAME=value` environment shorthand can set.
pub(crate) fn is_env_variable_name(bytes: &[u8]) -> bool {
match bytes {
[first, rest @ ..] if first == &b'_' || first.is_ascii_alphabetic() => {
rest.iter().all(|&b| b.is_ascii_alphanumeric() || b == b'_')
@@ -110,7 +109,17 @@ pub fn parse_list_expression(
span: Span,
element_shape: &SyntaxShape,
) -> Expression {
parse_list_expression_lexed(working_set, span, element_shape, None)
parse_list_expression_lexed(working_set, span, ListItems::Values(element_shape), None)
}
/// What the items of a list literal are.
#[derive(Clone, Copy)]
enum ListItems<'a> {
/// Values of a shape.
Values(&'a SyntaxShape),
/// The column names of a table's header row, each parsed like a record key (see
/// [`parse_field_name`]).
Names,
}
/// Lex the inside of a `[ ... ]` literal (`inner_span`, without the brackets), whose items are
@@ -120,7 +129,7 @@ fn lex_list_inner(
working_set: &StateWorkingSet,
inner_span: Span,
) -> (Vec<Token>, Option<ParseError>) {
lex_span(working_set, inner_span, &[b'\n', b'\r', b','], &[], true)
lex_list_items(working_set, inner_span)
}
/// [`parse_list_expression`], given the tokens of the list's inside (see [`lex_list_inner`]) when
@@ -129,7 +138,7 @@ fn lex_list_inner(
fn parse_list_expression_lexed(
working_set: &mut StateWorkingSet,
span: Span,
element_shape: &SyntaxShape,
items: ListItems,
tokens: Option<Vec<Token>>,
) -> Expression {
let bytes = working_set.get_span_contents(span);
@@ -196,11 +205,15 @@ fn parse_list_expression_lexed(
// Parse the spread operator
// Remove "..." before parsing argument to spread operator
command.parts[spans_idx] = trimmed_span;
let element_shape = match items {
ListItems::Values(shape) => shape.clone(),
ListItems::Names => SyntaxShape::String,
};
let spread_arg = parse_multispan_value(
working_set,
&command.parts,
&mut spans_idx,
&SyntaxShape::List(Box::new(element_shape.clone())),
&SyntaxShape::List(Box::new(element_shape)),
None,
);
let elem_ty = match &spread_arg.ty {
@@ -210,13 +223,16 @@ fn parse_list_expression_lexed(
let span = Span::new(curr_span.start, curr_span.start + 3);
(ListItem::Spread(span, spread_arg), elem_ty)
} else {
let arg = parse_multispan_value(
working_set,
&command.parts,
&mut spans_idx,
element_shape,
None,
);
let arg = match items {
ListItems::Values(element_shape) => parse_multispan_value(
working_set,
&command.parts,
&mut spans_idx,
element_shape,
None,
),
ListItems::Names => parse_field_name(working_set, curr_span),
};
let ty = arg.ty.clone();
(ListItem::Item(arg), ty)
};
@@ -248,8 +264,9 @@ fn parse_list_expression_lexed(
fn parse_table_row(
working_set: &mut StateWorkingSet,
span: Span,
items: ListItems,
) -> Result<(Vec<Expression>, Span), Span> {
let list = parse_list_expression(working_set, span, &SyntaxShape::Any);
let list = parse_list_expression_lexed(working_set, span, items, None);
let Expression {
expr: Expr::List(list),
span,
@@ -303,22 +320,28 @@ pub(crate) fn parse_table_expression(
// Check that we have all arguments first, before trying to parse the first
// in order to avoid exponential parsing time
let list_items = ListItems::Values(list_element_shape);
let [first, second, rest @ ..] = &tokens[..] else {
return parse_list_expression_lexed(working_set, span, list_element_shape, Some(tokens));
return parse_list_expression_lexed(working_set, span, list_items, Some(tokens));
};
if !working_set.get_span_contents(first.span).starts_with(b"[")
|| second.contents != TokenContents::Semicolon
{
return parse_list_expression_lexed(working_set, span, list_element_shape, Some(tokens));
return parse_list_expression_lexed(working_set, span, list_items, Some(tokens));
}
let head = parse_table_row(working_set, first.span);
let head = parse_table_row(working_set, first.span, ListItems::Names);
let errors = working_set.parse_errors.len();
let (head, rows) = match head {
Ok((head, _)) => {
let mut names = HashMap::new();
for column in &head {
check_duplicate_name(working_set, &mut names, column);
}
if rest.is_empty() {
working_set.error(ParseError::Expected(
"table row",
@@ -343,7 +366,11 @@ pub(crate) fn parse_table_expression(
working_set.error(err);
None
}
_ => match parse_table_row(working_set, it.span) {
_ => match parse_table_row(
working_set,
it.span,
ListItems::Values(&SyntaxShape::Any),
) {
Ok((list, span)) => {
match list.len().cmp(&head.len()) {
Ordering::Less => {
@@ -406,6 +433,37 @@ pub(crate) fn parse_table_expression(
Expression::new(working_set, Expr::Table(table), span, ty)
}
/// Parse a record key or a table column name. A bare word is the string it spells, even one that
/// would be a keyword, number or date as a value (`true`, `1`, `2023-02-29`). Variables,
/// subexpressions and quoted strings are parsed as they are anywhere else.
fn parse_field_name(working_set: &mut StateWorkingSet, span: Span) -> Expression {
match working_set.get_span_contents(span) {
// The `string` shape rejects these keywords for command arguments, but a name is a name.
b"true" | b"false" | b"null" => parse_string(working_set, span),
_ => parse_value(working_set, span, &SyntaxShape::String, None),
}
}
/// Report a record field or table column that a literal names twice. `names` holds the names
/// seen so far, with where each was first used. Names only known at runtime (`$k`, spreads) are
/// checked when the literal runs.
fn check_duplicate_name(
working_set: &mut StateWorkingSet,
names: &mut HashMap<String, Span>,
name: &Expression,
) {
let (Expr::String(name_str) | Expr::RawString(name_str)) = &name.expr else {
return;
};
if let Some(first) = earlier_use(names, name_str.clone(), name.span) {
working_set.error(ParseError::ColumnDefinedTwice(
name_str.clone(),
name.span,
first,
));
}
}
fn table_type(head: &[Expression], rows: &[Vec<Expression>]) -> (Type, Vec<ParseError>) {
let mut errors = vec![];
let mut rows: Vec<_> = rows.iter().map(|row| row.iter()).collect();
@@ -881,6 +939,57 @@ pub fn parse_closure_expression(
Expression::new(working_set, Expr::Closure(block_id), span, Type::Closure)
}
/// Parse `-$x` or `-(expr)` as the negation of a number, duration or filesize, with the
/// operand's own errors (`-$x+1` is as wrong as `$x+1`). Every other word gives `None`, with
/// nothing parsed, so it keeps its meaning: the range `-$x..5`, and the bare string
/// interpolations `-(1)px` and `-(1)+(2)`, whose `(` closes before the end of the word.
///
/// Values negate wherever they are parsed, a flag's value and an argument after `--` included.
/// A command's other arguments don't get here as `-$x`, which is read as a flag first, and an
/// argument passed through to an external command stays the string it spells (see
/// `parse_unknown_arg`).
fn parse_negation(
working_set: &mut StateWorkingSet,
span: Span,
input_type: Option<&Type>,
) -> Option<Expression> {
let operand_span = Span::new(span.start + 1, span.end);
let operand = match working_set.get_span_contents(span) {
bytes @ [b'-', b'$', _, ..] if !has_range_operator(bytes) => {
parse_dollar_expr(working_set, operand_span, &SyntaxShape::Any, input_type)
}
[b'-', b'(', .., b')'] => {
// Lexing the inside shows whether the `(` closes at the end of the word without
// parsing it: the inside of `-(1)+(2)` has an extra `)`. Deciding before parsing
// keeps each word parsed once, however often its shapes are tried.
let inside = Span::new(span.start + 2, span.end - 1);
let (_, err) = lex_span(working_set, inside, &[b'\n', b'\r'], &[], true);
if err.as_ref().is_some_and(is_unmatched_paren) {
return None;
}
parse_paren_expr(working_set, operand_span, &SyntaxShape::Any)
}
_ => return None,
};
let ty = negation_type(&operand.ty).unwrap_or_else(|| {
working_set.error(ParseError::OperatorUnsupportedType {
op: "-",
unsupported: operand.ty.clone(),
op_span: Span::new(span.start, span.start + 1),
unsupported_span: operand.span,
help: Some("Only numbers, durations and filesizes can be negated."),
});
Type::Any
});
Some(Expression::new(
working_set,
Expr::UnaryMinus(Box::new(operand)),
span,
ty,
))
}
pub fn parse_value(
working_set: &mut StateWorkingSet,
span: Span,
@@ -895,6 +1004,7 @@ pub fn parse_value(
working_set.error(ParseError::IncompleteParser(span));
return garbage(working_set, span);
}
let starts_with_minus = bytes[0] == b'-';
if let SyntaxShape::OneOf(possible_shapes) = shape {
return parse_oneof(
@@ -907,6 +1017,22 @@ pub fn parse_value(
);
}
if starts_with_minus
&& matches!(
shape,
SyntaxShape::Any
| SyntaxShape::Number
| SyntaxShape::Int
| SyntaxShape::Float
| SyntaxShape::Duration
| SyntaxShape::Filesize
)
&& let Some(negation) = parse_negation(working_set, span, input_type)
{
return negation;
}
let bytes = working_set.get_span_contents(span);
match bytes[0] {
b'$' => return parse_dollar_expr(working_set, span, shape, input_type),
b'(' => return parse_paren_expr(working_set, span, shape),
@@ -1179,10 +1305,10 @@ pub fn parse_assignment_expression(
)
}
pub fn parse_operator(working_set: &mut StateWorkingSet, span: Span) -> Expression {
let contents = working_set.get_span_contents(span);
let operator = match contents {
/// The binary operator spelled by `bytes`, if it is one. Unlike [`parse_operator`], this reports
/// nothing, so it can look ahead at tokens.
fn operator_from_bytes(bytes: &[u8]) -> Option<Operator> {
Some(match bytes {
b"==" => Operator::Comparison(Comparison::Equal),
b"!=" => Operator::Comparison(Comparison::NotEqual),
b"<" => Operator::Comparison(Comparison::LessThan),
@@ -1215,6 +1341,15 @@ pub fn parse_operator(working_set: &mut StateWorkingSet, span: Span) -> Expressi
b"or" => Operator::Boolean(Boolean::Or),
b"xor" => Operator::Boolean(Boolean::Xor),
b"and" => Operator::Boolean(Boolean::And),
_ => return None,
})
}
pub fn parse_operator(working_set: &mut StateWorkingSet, span: Span) -> Expression {
let contents = working_set.get_span_contents(span);
let operator = match contents {
contents if let Some(operator) = operator_from_bytes(contents) => operator,
// WARNING: not actual operators below! Error handling only
pow @ (b"^" | b"pow") => {
working_set.error(ParseError::UnknownOperator(
@@ -1333,6 +1468,75 @@ fn expand_row_condition_operands(
}
}
/// Parse the operand of a math expression at `spans[*idx]` and move `idx` past it.
///
/// An operand can start with `not`. `not` has precedence [`NOT_PRECEDENCE`], so it negates
/// everything up to the next operator that binds looser than it (`and`, `xor`, `or`): `not a ==
/// b` is `not (a == b)`, and `not a and b` is `(not a) and b`. Returns `None` after reporting an
/// error when a `not` has nothing to negate.
fn parse_math_operand(
working_set: &mut StateWorkingSet,
spans: &[Span],
idx: &mut usize,
lhs_row_var_id: Option<VarId>,
input_type: Option<&Type>,
) -> Option<Expression> {
let is_not =
|working_set: &StateWorkingSet, span: Span| working_set.get_span_contents(span) == b"not";
let first_not = *idx;
while *idx < spans.len() && is_not(working_set, spans[*idx]) {
*idx += 1;
}
if *idx == first_not {
let operand = parse_value(working_set, spans[*idx], &SyntaxShape::Any, input_type);
*idx += 1;
return Some(operand);
}
if *idx == spans.len() {
let end = spans[*idx - 1].end;
working_set.error(ParseError::Expected("expression", Span::new(end, end)));
return None;
}
// Find where the negated operand ends: at the first operator that binds looser than `not`,
// or anything else that isn't an operator. Operands in between may start with `not` too.
let start = *idx;
let mut end = start + 1;
while let Some(&op_span) = spans.get(end)
&& operator_from_bytes(working_set.get_span_contents(op_span))
.is_some_and(|op| op.precedence() > NOT_PRECEDENCE)
{
end += 1;
while end < spans.len() && is_not(working_set, spans[end]) {
end += 1;
}
// `if` and `match` take the rest of the expression, as in `parse_math_expression`.
end = match spans
.get(end)
.map(|span| working_set.get_span_contents(*span))
{
Some(b"if" | b"match") | None => spans.len(),
Some(_) => end + 1,
};
}
let mut operand =
parse_math_expression(working_set, &spans[start..end], lhs_row_var_id, input_type);
for not_span in spans[first_not..start].iter().rev() {
operand = Expression::new(
working_set,
Expr::UnaryNot(Box::new(operand)),
Span::new(not_span.start, spans[end - 1].end),
Type::Bool,
);
}
*idx = end;
Some(operand)
}
pub fn parse_math_expression(
working_set: &mut StateWorkingSet,
spans: &[Span],
@@ -1359,8 +1563,6 @@ pub fn parse_math_expression(
let first_span = working_set.get_span_contents(spans[0]);
let mut not_start_spans = vec![];
if first_span == b"if" || first_span == b"match" {
// If expression
if spans.len() > 1 {
@@ -1372,42 +1574,13 @@ pub fn parse_math_expression(
));
return garbage(working_set, spans[0]);
}
} else if first_span == b"not" {
not_start_spans.push(spans[idx].start);
idx += 1;
while idx < spans.len() {
let next_value = working_set.get_span_contents(spans[idx]);
if next_value == b"not" {
not_start_spans.push(spans[idx].start);
idx += 1;
} else {
break;
}
}
if idx == spans.len() {
working_set.error(ParseError::Expected(
"expression",
Span::new(spans[idx - 1].end, spans[idx - 1].end),
));
return garbage(working_set, spans[idx - 1]);
}
}
let mut lhs = parse_value(working_set, spans[idx], &SyntaxShape::Any, input_type);
for not_start_span in not_start_spans.iter().rev() {
lhs = Expression::new(
working_set,
Expr::UnaryNot(Box::new(lhs)),
Span::new(*not_start_span, spans[idx].end),
Type::Bool,
);
}
not_start_spans.clear();
idx += 1;
let Some(mut lhs) =
parse_math_operand(working_set, spans, &mut idx, lhs_row_var_id, input_type)
else {
return garbage(working_set, spans[spans.len() - 1]);
};
if idx >= spans.len() {
// We already found the one part of our expression, so let's expand
@@ -1444,39 +1617,12 @@ pub fn parse_math_expression(
expr_stack.push(op);
expr_stack.push(rhs);
break;
} else if content == b"not" {
not_start_spans.push(spans[idx].start);
idx += 1;
while idx < spans.len() {
let next_value = working_set.get_span_contents(spans[idx]);
if next_value == b"not" {
not_start_spans.push(spans[idx].start);
idx += 1;
} else {
break;
}
}
if idx == spans.len() {
working_set.error(ParseError::Expected(
"expression",
Span::new(spans[idx - 1].end, spans[idx - 1].end),
));
return garbage(working_set, spans[idx - 1]);
}
}
let mut rhs = parse_value(working_set, spans[idx], &SyntaxShape::Any, input_type);
for not_start_span in not_start_spans.iter().rev() {
rhs = Expression::new(
working_set,
Expr::UnaryNot(Box::new(rhs)),
Span::new(*not_start_span, spans[idx].end),
Type::Bool,
);
}
not_start_spans.clear();
let Some(rhs) =
parse_math_operand(working_set, spans, &mut idx, lhs_row_var_id, input_type)
else {
return garbage(working_set, spans[spans.len() - 1]);
};
// Parsing power must be right-associative unlike most operations which are left
// Hence, we should not collapse if the last and current operations are both power
@@ -1526,8 +1672,6 @@ pub fn parse_math_expression(
expr_stack.push(rhs);
last_prec = op_prec;
idx += 1;
}
while expr_stack.len() != 1 {
@@ -2087,6 +2231,7 @@ pub fn parse_record(working_set: &mut StateWorkingSet, span: Span) -> Expression
let mut idx = 0;
let mut field_types = Some(vec![]);
let mut field_names = HashMap::new();
while idx < tokens.len() {
let curr_span = tokens[idx].span;
let curr_tok = working_set.get_span_contents(curr_span);
@@ -2128,11 +2273,12 @@ pub fn parse_record(working_set: &mut StateWorkingSet, span: Span) -> Expression
});
garbage(working_set, curr_span)
} else {
let field = parse_value(working_set, curr_span, &SyntaxShape::String, None);
let field = parse_field_name(working_set, curr_span);
if let Some(error) = check_record_key_or_value(working_set, &field, "key") {
working_set.error(error);
garbage(working_set, field.span)
} else {
check_duplicate_name(working_set, &mut field_names, &field);
field
}
};
+20
View File
@@ -1,7 +1,27 @@
use nu_protocol::{Span, Spanned, ast::*, engine::StateWorkingSet};
use std::{
collections::{HashMap, hash_map::Entry},
hash::Hash,
};
pub(crate) const PERCENT_FORCED_BUILTIN_PARSER_INFO: &str = "percent_forced_builtin";
/// Record that `name` is used at `span`, and give the span of its first use if it was used
/// before. Callers report a repeated record field, table column or parameter with their own error.
pub(crate) fn earlier_use<T: Eq + Hash>(
uses: &mut HashMap<T, Span>,
name: T,
span: Span,
) -> Option<Span> {
match uses.entry(name) {
Entry::Occupied(first) => Some(*first.get()),
Entry::Vacant(entry) => {
entry.insert(span);
None
}
}
}
/// three dots b"..."
pub(crate) const SPREAD_OPERATOR: &[u8; 3] = b"...";
+288 -66
View File
@@ -16,7 +16,7 @@ use nu_protocol::{
DidYouMean, FilesizeUnit, IntoSpanned, ParseError, Span, Spanned, SyntaxShape, Type, Unit,
VarId, ast::*, casing::Casing, engine::StateWorkingSet,
};
use std::{borrow::Cow, sync::Arc};
use std::{borrow::Cow, num::IntErrorKind, sync::Arc};
use crate::parse_expressions::{
parse_block_expression, parse_closure_expression, parse_match_block_expression, parse_record,
@@ -151,7 +151,13 @@ fn first_non_separator(token: &[u8]) -> Option<u8> {
enum IntLiteral {
Value(i64),
Empty,
InvalidDigits { radix: u32 },
InvalidDigits {
radix: u32,
},
/// Valid digits for a number that doesn't fit in an `i64`.
OutOfRange {
radix: u32,
},
NotInt,
}
@@ -173,25 +179,54 @@ fn read_int(token: &[u8]) -> IntLiteral {
fn decode_int(token: &[u8]) -> IntLiteral {
let token = strip_underscores(token);
// Parse as a u64, then cast to i64, otherwise, for numbers like "0xffffffffffffffef",
// you'll get `Error parsing hex string: number too large to fit in target type`.
// A sign can come before a radix prefix too, as in `-0x10`.
let (sign, unsigned) = match token.as_bytes().first() {
Some(&sign @ (b'-' | b'+')) => (Some(sign), &token[1..]),
_ => (None, &token[..]),
};
// Parse as a u64, so that an unsigned 64-bit pattern such as "0xffffffffffffffef" wraps to a
// negative int instead of being too large. A signed literal is a number that has to fit:
// `-0x8000000000000000` is `i64::MIN`, and `+0x8000000000000000` is too large.
// Rust reports an overflow as soon as the digits read so far overflow, before it reaches a
// later non-digit, so a token is only out of range if it is all digits.
let all_digits = |digits: &str, radix: u32| digits.chars().all(|c| c.is_digit(radix));
let extract_int = |digits: &str, radix: u32| match u64::from_str_radix(digits, radix) {
Ok(num) => IntLiteral::Value(num as i64),
Ok(num) => {
let value = match sign {
Some(b'-') => 0i64.checked_sub_unsigned(num),
Some(_) => i64::try_from(num).ok(),
None => Some(num as i64),
};
value.map_or(IntLiteral::OutOfRange { radix }, IntLiteral::Value)
}
Err(err) if *err.kind() == IntErrorKind::PosOverflow && all_digits(digits, radix) => {
IntLiteral::OutOfRange { radix }
}
Err(_) => IntLiteral::InvalidDigits { radix },
};
if token.is_empty() {
IntLiteral::Empty
} else if let Some(num) = token.strip_prefix("0b") {
} else if let Some(num) = unsigned.strip_prefix("0b") {
extract_int(num, 2)
} else if let Some(num) = token.strip_prefix("0o") {
} else if let Some(num) = unsigned.strip_prefix("0o") {
extract_int(num, 8)
} else if let Some(num) = token.strip_prefix("0x") {
} else if let Some(num) = unsigned.strip_prefix("0x") {
extract_int(num, 16)
} else if let Ok(num) = token.parse::<i64>() {
IntLiteral::Value(num)
} else {
IntLiteral::NotInt
match token.parse::<i64>() {
Ok(num) => IntLiteral::Value(num),
Err(err)
if matches!(
err.kind(),
IntErrorKind::PosOverflow | IntErrorKind::NegOverflow
) && all_digits(unsigned, 10) =>
{
IntLiteral::OutOfRange { radix: 10 }
}
Err(_) => IntLiteral::NotInt,
}
}
}
@@ -210,19 +245,49 @@ pub fn parse_int(working_set: &mut StateWorkingSet, span: Span) -> Expression {
));
garbage(working_set, span)
}
// An int literal never silently becomes a float that has lost precision. Only a decimal
// number can be written as a float instead: `0x10000000000000000.0` isn't one.
IntLiteral::OutOfRange { radix } => {
let msg = if radix == 10 {
"number too large (add `.0` for a float)"
} else {
"number too large for 64 bits"
};
working_set.error(ParseError::InvalidLiteral(msg.into(), "int".into(), span));
garbage(working_set, span)
}
}
}
/// The names `f64::from_str` takes for special floats that are also Nushell's: `inf` and `nan`,
/// plus `NaN`, the spelling Nushell prints (so NUON output reads back in). Rust also accepts
/// `infinity` and any letter case, which would make bare words like `Nan` floats.
const SPECIAL_FLOAT_NAMES: [&str; 3] = ["inf", "nan", "NaN"];
/// Read `token` as a float literal, with `_` digit separators.
fn read_float(token: &[u8]) -> Option<f64> {
// `f64::from_str` only accepts a sign, a digit, `.`, or `inf`/`infinity`/`nan` in any case
// at the start, so reject everything else before decoding it (see `read_int`).
// `f64::from_str` only accepts a sign, a digit, `.`, or a special name at the start, so
// reject everything else before decoding it (see `read_int`).
let first = first_non_separator(token)?;
if !(first.is_ascii_digit() || matches!(first, b'+' | b'-' | b'.' | b'i' | b'I' | b'n' | b'N'))
if !(first.is_ascii_digit() || matches!(first, b'+' | b'-' | b'.' | b'i' | b'n' | b'N')) {
return None;
}
let text = strip_underscores(token);
let unsigned = text.strip_prefix(['+', '-']).unwrap_or(&text);
// A special float is spelled exactly as one of the names, with no `_` in it (`i_n_f` is a
// bare word).
let written = match token {
[b'+' | b'-', rest @ ..] => rest,
_ => token,
};
if unsigned.starts_with(|c: char| c.is_ascii_alphabetic())
&& !SPECIAL_FLOAT_NAMES
.iter()
.any(|name| name.as_bytes() == written)
{
return None;
}
strip_underscores(token).parse::<f64>().ok()
text.parse::<f64>().ok()
}
pub fn parse_float(working_set: &mut StateWorkingSet, span: Span) -> Expression {
@@ -239,11 +304,18 @@ pub fn parse_number(working_set: &mut StateWorkingSet, span: Span) -> Expression
let starting_error_count = working_set.parse_errors.len();
let result = parse_int(working_set, span);
if starting_error_count == working_set.parse_errors.len() {
// An int too large for 64 bits isn't retried as a float, which would lose precision.
if working_set.parse_errors.len() == starting_error_count
|| matches!(
read_int(working_set.get_span_contents(span)),
IntLiteral::OutOfRange { .. }
)
{
return result;
} else if let Some(ParseError::Expected(_, _)) = working_set.parse_errors.last() {
working_set.parse_errors.truncate(starting_error_count);
}
// Any other token may still be a float. One with bad radix digits isn't, and ends as
// "expected number", so that a word like `0xdump` can still name a command.
working_set.parse_errors.truncate(starting_error_count);
let result = parse_float(working_set, span);
@@ -539,6 +611,17 @@ pub fn parse_raw_string(working_set: &mut StateWorkingSet, span: Span) -> Expres
}
}
/// Whether `error` is an unmatched `(` or `)`. As the first error of a word that starts with `(`,
/// it shows that the word isn't one parenthesized expression (`(1)+(2)`), so it is read as a
/// string interpolation instead.
pub(crate) fn is_unmatched_paren(error: &ParseError) -> bool {
match error {
ParseError::Unclosed(right, ..) => *right == ")",
ParseError::Unbalanced(left, right, ..) => *left == "(" && *right == ")",
_ => false,
}
}
pub fn parse_paren_expr(
working_set: &mut StateWorkingSet,
span: Span,
@@ -568,11 +651,7 @@ pub fn parse_paren_expr(
if fcp_error_count > starting_error_count {
let malformed_subexpr = working_set.parse_errors[starting_error_count..]
.first()
.is_some_and(|e| match e {
ParseError::Unclosed(right, ..) if (*right == ")") => true,
ParseError::Unbalanced(left, right, ..) if *left == "(" && *right == ")" => true,
_ => false,
});
.is_some_and(is_unmatched_paren);
if malformed_subexpr {
working_set.parse_errors.truncate(starting_error_count);
while working_set.delta.scope.len() > starting_scope_count {
@@ -1350,6 +1429,35 @@ pub fn parse_filepath(working_set: &mut StateWorkingSet, span: Span) -> Expressi
parse_path_like(working_set, span, PathLikeKind::Filepath)
}
/// Whether `bytes` is written like a date literal but names a day that doesn't exist
/// (`2023-02-29`, `2023-13-01`). A date literal is `YYYY-MM-DD`, alone or followed by `T` or `t`
/// and a time that starts `HH:MM`. Any other text after the date (`2023-02-29.log`,
/// `2024-01-15T10-30-00.log`) makes the word a bare string, and so does a real day with a time
/// that isn't a valid one (`2024-01-15T10:00`).
fn is_invalid_date_literal(bytes: &[u8]) -> bool {
let digits = |range: std::ops::Range<usize>| {
bytes
.get(range)
.is_some_and(|digits| digits.iter().all(u8::is_ascii_digit))
};
let date_shape = digits(0..4)
&& bytes.get(4) == Some(&b'-')
&& digits(5..7)
&& bytes.get(7) == Some(&b'-')
&& digits(8..10);
let time_follows = match bytes.get(10) {
None => true,
Some(b'T' | b't') => digits(11..13) && bytes.get(13) == Some(&b':') && digits(14..16),
Some(_) => false,
};
date_shape
&& time_follows
&& str::from_utf8(&bytes[..10])
.ok()
.and_then(|date| chrono::NaiveDate::parse_from_str(date, "%Y-%m-%d").ok())
.is_none()
}
pub fn parse_datetime(working_set: &mut StateWorkingSet, span: Span) -> Expression {
trace!("parsing: datetime");
@@ -1384,6 +1492,17 @@ pub fn parse_datetime(working_set: &mut StateWorkingSet, span: Span) -> Expressi
return Expression::new(working_set, Expr::DateTime(datetime), span, Type::Date);
}
// A token written like a date that isn't a real day (`2023-02-29`) is a typo, not a bare
// string, the same way `0xZZ` is a bad hex literal rather than a string.
if is_invalid_date_literal(working_set.get_span_contents(span)) {
working_set.error(ParseError::InvalidLiteral(
"not a valid date".into(),
"datetime".into(),
span,
));
return garbage(working_set, span);
}
working_set.error(ParseError::Expected("datetime", span));
garbage(working_set, span)
@@ -1417,8 +1536,8 @@ pub fn parse_filesize(working_set: &mut StateWorkingSet, span: Span) -> Expressi
let bytes = working_set.get_span_contents(span);
// the hex digit `b` might be mistaken for the unit `b`, so check that first
if bytes.starts_with(b"0x") {
// The hex digit `b` might be mistaken for the unit `b` (`0x1b`, `-0x1b`), so rule hex out first.
if matches!(bytes, [b'0', b'x', ..] | [b'-' | b'+', b'0', b'x', ..]) {
working_set.error(ParseError::Expected("filesize with valid units", span));
return garbage(working_set, span);
}
@@ -1451,10 +1570,10 @@ pub fn parse_unit_value<'res>(
ty: Type,
transform: fn(String) -> String,
) -> Option<ParseUnitResult<'res>> {
// A unit value starts like a number: a digit, `.digit`, or a sign and a digit.
if bytes.len() < 2
|| !(bytes[0].is_ascii_digit()
|| (bytes[0] == b'.' && bytes[1].is_ascii_digit())
|| (bytes[0] == b'-' && bytes[1].is_ascii_digit()))
|| (matches!(bytes[0], b'.' | b'-' | b'+') && bytes[1].is_ascii_digit()))
{
return None;
}
@@ -1487,49 +1606,51 @@ pub fn parse_unit_value<'res>(
}
});
let mut unit = match convert {
let unit = match convert {
Some(convert_to) => convert_to.0,
None => *unit,
};
let num_float = match convert {
Some(convert_to) => {
(number_part * convert_to.1 as f64) + (decimal_part * convert_to.1 as f64)
// Store every filesize in bytes and every duration in nanoseconds, to minimize loss of
// precision.
let (base_unit, factor) = if ty == Type::Filesize {
(Unit::Filesize(FilesizeUnit::B), unit_to_byte_factor(&unit)?)
} else {
(Unit::Nanosecond, unit_to_ns_factor(&unit)?)
};
// An integer is scaled exactly, so the largest value fits (`9223372036854775807ns`) and the
// next one doesn't. A fraction is scaled as an `f64`, where `i64::MAX as f64` rounds up to
// 2^63, which doesn't fit, so its upper bound is exclusive.
let num = match lhs.parse::<i128>() {
Ok(int) => {
let scale =
i128::from(convert.map_or(1, |convert_to| convert_to.1)) * factor as i128;
int.checked_mul(scale)
.and_then(|num| i64::try_from(num).ok())
}
None => number_part,
};
// Convert all durations to nanoseconds, and filesizes to bytes,
// to minimize loss of precision
let factor = match ty {
Type::Filesize => unit_to_byte_factor(&unit),
Type::Duration => unit_to_ns_factor(&unit),
_ => None,
};
let num = match factor {
Some(factor) => {
let num_base = num_float * factor;
if i64::MIN as f64 <= num_base && num_base <= i64::MAX as f64 {
unit = if ty == Type::Filesize {
Unit::Filesize(FilesizeUnit::B)
} else {
Unit::Nanosecond
};
num_base as i64
} else {
// not safe to convert, because of the overflow
num_float as i64
}
Err(_) => {
let num_float = match convert {
Some(convert_to) => {
(number_part * convert_to.1 as f64) + (decimal_part * convert_to.1 as f64)
}
None => number_part,
} * factor;
(i64::MIN as f64..i64::MAX as f64)
.contains(&num_float)
.then_some(num_float as i64)
}
None => num_float as i64,
};
let Some(num) = num else {
let mk_err = move |name: &str| ParseError::LiteralTooLarge(name.into(), span);
return Some(Err(Box::new(mk_err)));
};
trace!("-- found {num} {unit:?}");
trace!("-- found {num} {base_unit:?}");
let value = ValueWithUnit {
expr: Expression::new_unknown(Expr::Int(num), lhs_span, Type::Number),
unit: Spanned {
item: unit,
item: base_unit,
span: unit_span,
},
};
@@ -1947,14 +2068,25 @@ pub fn unescape_string(bytes: &[u8], span: Span) -> (Vec<u8>, Option<ParseError>
pub fn unescape_unquote_string(bytes: &[u8], span: Span) -> (String, Option<ParseError>) {
if bytes.starts_with(b"\"") {
// Needs unescaping
let bytes = trim_quotes(bytes);
let quoted = trim_quotes(bytes);
let (bytes, err) = unescape_string(bytes, span);
let (unescaped, err) = unescape_string(quoted, span);
if let Ok(token) = String::from_utf8(bytes) {
(token, err)
} else {
(String::new(), Some(ParseError::Expected("string", span)))
match String::from_utf8(unescaped) {
Ok(token) => (token, err),
// The source text is UTF-8, so the bytes that aren't came from `\x` escapes.
Err(_) if err.is_none() && str::from_utf8(quoted).is_ok() => (
String::new(),
Some(ParseError::InvalidLiteral(
"invalid UTF-8 from `\\x` escapes".into(),
"string".into(),
span,
)),
),
Err(_) => (
String::new(),
err.or(Some(ParseError::Expected("string", span))),
),
}
} else {
let bytes = trim_quotes(bytes);
@@ -1967,6 +2099,11 @@ pub fn unescape_unquote_string(bytes: &[u8], span: Span) -> (String, Option<Pars
}
}
/// Reject a quoted string with text after its closing quote, such as `'a'b` or `'a'"b"`.
///
/// A quoted word runs from its opening quote to the last quote of the same kind in the word, and
/// quotes of that kind in between are part of the string: `"[{a: "b"}]"` is `[{a: "b"}]` and
/// `'a''b'` is `a''b`. Scripts rely on this for text such as JSON and HTML templates.
fn check_string_no_trailing_tokens(
bytes: &[u8],
span: Span,
@@ -2158,7 +2295,10 @@ mod number_tests {
#[test]
fn floats_read_as_before() {
for token in TOKENS {
// Special float names other than `inf`, `nan` and `NaN` are rejected on purpose; see
// `special_float_names` below.
let renamed = [&b"-Infinity"[..], b"INFINITY"];
for token in TOKENS.iter().filter(|token| !renamed.contains(token)) {
let before = strip_underscores(token).parse::<f64>().ok();
let after = read_float(token);
// Compare bit patterns so that NaN equals NaN.
@@ -2170,4 +2310,86 @@ mod number_tests {
);
}
}
#[test]
fn special_float_names() {
for token in ["inf", "+inf", "-inf", "nan", "NaN", "-nan", "+NaN"] {
assert!(read_float(token.as_bytes()).is_some(), "{token}");
}
for token in [
"INF",
"Inf",
"NAN",
"Nan",
"infinity",
"Infinity",
"-INFINITY",
// `_` separators belong in numbers, not in names.
"i_n_f",
"n_an",
"_inf",
"-_inf",
"inf_",
] {
assert_eq!(read_float(token.as_bytes()), None, "{token}");
}
}
#[test]
fn signed_radix_ints() {
assert_eq!(decode_int(b"-0x10"), IntLiteral::Value(-16));
assert_eq!(decode_int(b"+0b11"), IntLiteral::Value(3));
assert_eq!(decode_int(b"-0o17"), IntLiteral::Value(-15));
assert_eq!(
decode_int(b"-0x8000000000000000"),
IntLiteral::Value(i64::MIN)
);
assert_eq!(
decode_int(b"-0x8000000000000001"),
IntLiteral::OutOfRange { radix: 16 }
);
assert_eq!(
decode_int(b"+0x7fffffffffffffff"),
IntLiteral::Value(i64::MAX)
);
// A `+` makes the literal a positive number, which has to fit.
assert_eq!(
decode_int(b"+0x8000000000000000"),
IntLiteral::OutOfRange { radix: 16 }
);
// Without a sign, 64-bit patterns still wrap to negative ints.
assert_eq!(decode_int(b"0xffffffffffffffff"), IntLiteral::Value(-1));
}
#[test]
fn out_of_range_ints() {
assert_eq!(
decode_int(b"9223372036854775807"),
IntLiteral::Value(i64::MAX)
);
assert_eq!(
decode_int(b"-9223372036854775808"),
IntLiteral::Value(i64::MIN)
);
assert_eq!(
decode_int(b"9223372036854775808"),
IntLiteral::OutOfRange { radix: 10 }
);
assert_eq!(
decode_int(b"-9223372036854775809"),
IntLiteral::OutOfRange { radix: 10 }
);
assert_eq!(
decode_int(b"0x10000000000000000"),
IntLiteral::OutOfRange { radix: 16 }
);
assert_eq!(decode_int(b"0xZZ"), IntLiteral::InvalidDigits { radix: 16 });
// Overflowing digits before a non-digit are not an out-of-range int.
assert_eq!(decode_int(b"9223372036854775808.0"), IntLiteral::NotInt);
assert_eq!(decode_int(b"99999999999999999999e3"), IntLiteral::NotInt);
assert_eq!(
decode_int(b"0x10000000000000000g"),
IntLiteral::InvalidDigits { radix: 16 }
);
}
}
+20
View File
@@ -5,6 +5,7 @@ use std::sync::Arc;
use crate::{
Token,
lite_parser::{LiteCommand, LitePipeline, LiteRedirection, LiteRedirectionTarget, lite_parse},
parse_expressions::is_env_variable_name,
parse_keywords::parse_def_predecl,
parser::{
parse_builtin_commands, parse_expression, parse_value, wrap_element_with_collect,
@@ -93,6 +94,25 @@ pub fn parse_pipeline(
[] => unreachable!("at this point the pipeline must have at least one element"),
[single] => parse_builtin_commands(working_set, single, input_type),
[first, rest @ ..] => {
// Loops are statements without output, so like `for` (rejected in
// `parse_expression`), `while` and `loop` can't be part of a pipeline. The head is
// the first part after any `NAME=value` environment shorthands, as there.
for command in &pipeline.commands {
let is_env_shorthand = |part: &&Span| {
let bytes = working_set.get_span_contents(**part);
bytes
.iter()
.position(|&b| b == b'=')
.is_some_and(|equals| is_env_variable_name(&bytes[..equals]))
};
if let Some(&head) = command.parts.iter().find(|part| !is_env_shorthand(part))
&& let keyword @ (b"while" | b"loop") = working_set.get_span_contents(head)
{
let keyword = String::from_utf8_lossy(keyword).into_owned();
working_set.error(ParseError::BuiltinCommandInPipeline(keyword, head));
}
}
let mut current_pipeline_type = input_type.cloned().unwrap_or(Type::Any);
let mut elements = Vec::new();
+63 -2
View File
@@ -6,7 +6,7 @@ use crate::parse_literals::parse_full_cell_path;
use crate::{
Token, TokenContents,
lex::lex_signature,
parse_helpers::{SPREAD_OPERATOR, garbage, is_variable},
parse_helpers::{SPREAD_OPERATOR, earlier_use, garbage, is_variable},
parse_shape_specs::{parse_completer, parse_shape_name, parse_type},
type_check::type_compatible,
};
@@ -15,7 +15,10 @@ use nu_protocol::{
Flag, ParseError, PositionalArg, Signature, Span, SyntaxShape, Type, TypeSet, Value, VarId,
ast::*, engine::StateWorkingSet, eval_const::eval_constant,
};
use std::{collections::HashSet, sync::Arc};
use std::{
collections::{HashMap, HashSet},
sync::Arc,
};
pub fn parse_import_pattern<'a>(
working_set: &mut StateWorkingSet,
@@ -1213,7 +1216,65 @@ pub fn parse_signature_helper(
let mut sig = Signature::new(String::new());
// Parameters and flags share one variable namespace (`--foo-bar` binds `$foo_bar`, as do
// `foo_bar` and `$foo_bar`), and each short flag needs its own letter. A repeat would silently
// shadow or swallow the earlier one. Only `_` may repeat, to ignore several arguments.
let mut variable_names = HashMap::new();
let mut short_flags = HashMap::new();
let declaration_span = |working_set: &StateWorkingSet, var_id: Option<VarId>| {
var_id.map_or(span, |var_id| {
working_set.get_variable(var_id).declaration_span
})
};
for arg in args {
// An `extern` only describes another program's arguments for completions and binds no
// variables, so it is not checked: generated completions often list a flag twice, and
// a real program's last argument may follow a list.
if !is_external {
match &arg {
Arg::Positional {
arg: positional, ..
}
| Arg::RestPositional(positional) => {
let at = declaration_span(working_set, positional.var_id);
let name = positional.name.trim_start_matches('$');
let is_rest = matches!(arg, Arg::RestPositional(_));
if name != "_"
&& let Some(first) = earlier_use(&mut variable_names, name.to_string(), at)
{
let prefix = if is_rest { "..." } else { "" };
let display = format!("{prefix}{}", positional.name);
working_set.error(ParseError::DuplicateParameter(display, at, first));
}
if !is_rest && sig.rest_positional.is_some() {
working_set
.error(ParseError::PositionalAfterRest(positional.name.clone(), at));
}
}
Arg::Flag { flag, .. } => {
let at = declaration_span(working_set, flag.var_id);
// A long flag binds a variable named after it, with `-` replaced by `_`; a
// flag with only a short form binds one named after its letter.
let (name, display) = match flag.short {
Some(short) if flag.long.is_empty() => {
(short.to_string(), format!("-{short}"))
}
_ => (flag.long.replace('-', "_"), format!("--{}", flag.long)),
};
if let Some(first) = earlier_use(&mut variable_names, name, at) {
working_set.error(ParseError::DuplicateParameter(display, at, first));
}
if let Some(short) = flag.short
&& let Some(first) = earlier_use(&mut short_flags, short, at)
{
let display = format!("-{short}");
working_set.error(ParseError::DuplicateParameter(display, at, first));
}
}
}
}
match arg {
Arg::Positional {
arg: positional,
+16 -1
View File
@@ -310,10 +310,25 @@ fn str_cmp_op(lhs: &Type, rhs: &Type) -> Option<Type> {
operator_check(op, None, lhs, rhs)
}
/// The type of `-x` for an operand of type `ty`, or `None` if values of that type can't be
/// negated. Negation multiplies by -1, so it is typed like that multiplication.
pub fn negation_type(ty: &Type) -> Option<Type> {
math::multiply(ty, &Type::Int)
}
fn in_op(lhs: &Type, rhs: &Type) -> Option<Type> {
/// Whether a value of type `t` can be equal to an element of type `u`. Ints and floats
/// compare by value at runtime (`1 == 1.0`), so they are compatible here too.
fn element_compatible(t: &Type, u: &Type) -> bool {
match u {
Type::OneOf(members) => members.iter().any(|u| element_compatible(t, u)),
u => type_compatible(t, u) || number_types(t, u).is_some(),
}
}
fn op(lhs: &Type, rhs: &Type) -> Option<Type> {
Some(match (lhs, rhs) {
(t, Type::List(u)) if type_compatible(t, u) => Type::Bool,
(t, Type::List(u)) if element_compatible(t, u) => Type::Bool,
(Type::Int | Type::Float | Type::Number, Type::Range) => Type::Bool,
(Type::String, Type::String) => Type::Bool,
(Type::String, Type::Record(_)) => Type::Bool,
+2 -2
View File
@@ -659,7 +659,7 @@ fn parse_unclosed_quotes_still_report() {
// Quote failures may surface as Unclosed("\"") or similar string errors.
let msg = format!("{err:?}");
assert!(
matches!(err, ParseError::Unclosed(d, ..) if d.contains('"') || d.contains('\''))
matches!(&err, ParseError::Unclosed(d, ..) if d.contains('"') || d.contains('\''))
|| msg.to_lowercase().contains("quote")
|| msg.contains('\"'),
"expected quote-related error, got {err:?}"
@@ -675,7 +675,7 @@ fn parse_unclosed_quotes_multiline_still_report(#[case] file: &[u8]) {
let err = parse_first_error(file);
let msg = format!("{err:?}");
assert!(
matches!(err, ParseError::Unclosed(d, ..) if d.contains('"') || d.contains('\''))
matches!(&err, ParseError::Unclosed(d, ..) if d.contains('"') || d.contains('\''))
|| msg.to_lowercase().contains("quote")
|| msg.contains('\"')
|| msg.contains('\''),
+4
View File
@@ -25,6 +25,8 @@ pub enum Expr {
Operator(Operator),
RowCondition(BlockId),
UnaryNot(Box<Expression>),
/// Negation of a number, duration or filesize, written `-$x` or `-(expr)`.
UnaryMinus(Box<Expression>),
BinaryOp(Box<Expression>, Box<Expression>, Box<Expression>), //lhs, op, rhs
Collect(VarId, Box<Expression>),
Subexpression(BlockId),
@@ -82,6 +84,7 @@ impl Expr {
Expr::Operator(_) => "an operator",
Expr::RowCondition(_) => "a row condition",
Expr::UnaryNot(_) => "a negation",
Expr::UnaryMinus(_) => "a numeric negation",
Expr::BinaryOp(_, _, _) => "a binary operation",
Expr::Collect(_, _) => "a collect expression",
Expr::Subexpression(_) => "a subexpression",
@@ -134,6 +137,7 @@ impl Expr {
| Expr::Range(_)
| Expr::Var(_)
| Expr::UnaryNot(_)
| Expr::UnaryMinus(_)
| Expr::BinaryOp(_, _, _)
| Expr::Closure(_) // piping into a closure value, not into a closure call
| Expr::List(_)
+3 -3
View File
@@ -113,7 +113,7 @@ impl Expression {
Expr::BinaryOp(left, _, right) => {
left.has_in_variable(working_set) || right.has_in_variable(working_set)
}
Expr::UnaryNot(expr) => expr.has_in_variable(working_set),
Expr::UnaryNot(expr) | Expr::UnaryMinus(expr) => expr.has_in_variable(working_set),
Expr::Block(block_id) | Expr::Closure(block_id) => {
let block = working_set.get_block(*block_id);
block
@@ -294,7 +294,7 @@ impl Expression {
left.replace_span(working_set, replaced, new_span);
right.replace_span(working_set, replaced, new_span);
}
Expr::UnaryNot(expr) => {
Expr::UnaryNot(expr) | Expr::UnaryMinus(expr) => {
expr.replace_span(working_set, replaced, new_span);
}
Expr::Block(block_id) => {
@@ -507,7 +507,7 @@ impl Expression {
*working_set.get_block_mut(*block_id) = block;
}
}
Expr::UnaryNot(expr) => {
Expr::UnaryNot(expr) | Expr::UnaryMinus(expr) => {
expr.replace_in_variable(working_set, new_var_id);
}
Expr::BinaryOp(lhs, op, rhs) => {
+5
View File
@@ -242,6 +242,11 @@ pub enum Operator {
Assignment(Assignment),
}
/// Precedence of the prefix `not`: it binds looser than comparisons and bitwise operators and
/// tighter than `and`, `xor` and `or`, so `not a == b` is `not (a == b)` and `not a and b` is
/// `(not a) and b`. `not` is not an [`Operator`], since it takes one operand.
pub const NOT_PRECEDENCE: u8 = 55;
impl Operator {
pub const fn as_str(&self) -> &'static str {
match self {
+6 -2
View File
@@ -124,7 +124,9 @@ impl Traverse for Expression {
recur(arg.expr());
}
}
Expr::UnaryNot(expr) | Expr::Collect(_, expr) => recur(expr.as_ref()),
Expr::UnaryNot(expr) | Expr::UnaryMinus(expr) | Expr::Collect(_, expr) => {
recur(expr.as_ref())
}
Expr::BinaryOp(lhs, op, rhs) => {
recur(lhs);
recur(op);
@@ -211,7 +213,9 @@ impl Traverse for Expression {
Expr::ExternalCall(head, args) => {
recur(head.as_ref()).or(args.iter().find_map(|arg| recur(arg.expr())))
}
Expr::UnaryNot(expr) | Expr::Collect(_, expr) => recur(expr.as_ref()),
Expr::UnaryNot(expr) | Expr::UnaryMinus(expr) | Expr::Collect(_, expr) => {
recur(expr.as_ref())
}
Expr::BinaryOp(lhs, op, rhs) => recur(lhs).or(recur(op)).or(recur(rhs)),
Expr::MatchBlock(matches) => matches.iter().find_map(|(pattern, expr)| {
pattern.find_map(working_set, f).or(recur(expr))
@@ -357,6 +357,7 @@ fn expr_to_string(engine_state: &EngineState, expr: &Expr) -> String {
Expr::Subexpression(_) => "subexpression".to_string(),
Expr::Table(_) => "table".to_string(),
Expr::UnaryNot(_) => "unary not".to_string(),
Expr::UnaryMinus(_) => "unary minus".to_string(),
Expr::ValueWithUnit(_) => "value with unit".to_string(),
Expr::Var(_) => "var".to_string(),
Expr::VarDecl(_) => "var decl".to_string(),
+66 -16
View File
@@ -3,6 +3,7 @@ use crate::{Span, Type, ast::RedirectionSource, did_you_mean};
use miette::Diagnostic;
use serde::{Deserialize, Serialize};
use std::{
borrow::Cow,
fmt::Display,
str::{Utf8Error, from_utf8},
};
@@ -40,13 +41,45 @@ pub enum ParseError {
#[label = "required parameter {0} after optional parameter"] Span,
),
/// A positional parameter follows the rest parameter, which takes every remaining
/// positional argument.
#[error("Positional parameter after the rest parameter.")]
#[diagnostic(
code(nu::parser::positional_after_rest),
help(
"The rest parameter takes all remaining positional arguments, so it must be the last positional parameter. Flags can still follow it."
)
)]
PositionalAfterRest(
String,
#[label = "parameter {0} after the rest parameter"] Span,
),
/// Two parameters or flags of a signature share a name, or two flags share a short flag.
///
/// - `0`: the repeated name as written (`a`, `--name`, `-n`)
/// - `1`: span of the repeat
/// - `2`: span of the first use
#[error("Duplicate parameter or flag.")]
#[diagnostic(
code(nu::parser::duplicate_parameter),
help(
"Each parameter and flag needs its own name, and each short flag its own letter. A long flag's variable replaces `-` with `_`, so `--foo-bar` and `foo_bar` clash."
)
)]
DuplicateParameter(
String,
#[label(primary, "`{0}` uses a name that is already taken")] Span,
#[label("first used here")] Span,
),
#[error("Unexpected end of code.")]
#[diagnostic(code(nu::parser::unexpected_eof))]
UnexpectedEof(String, #[label("expected closing {0}")] Span),
/// A delimiter was opened but never closed.
///
/// - `0`: expected closer (e.g. `"}"`, `"]"`, `")"`)
/// - `0`: expected closer (e.g. `"}"`, `"]"`, `")"`, or a raw string's `'##`)
/// - `1`: span of the **opening** delimiter
/// - `2`: span where the closer was expected (primary — often mid-file or EOF)
/// - `3`: help text (generic or heuristic structure hint)
@@ -56,7 +89,7 @@ pub enum ParseError {
#[error("Unclosed delimiter.")]
#[diagnostic(code(nu::parser::unclosed_delimiter), help("{3}"))]
Unclosed(
&'static str,
Cow<'static, str>,
#[label("unclosed — opens here (need `{0}`)")] Span,
#[label(primary, "expected `{0}` here")] Span,
String,
@@ -194,6 +227,16 @@ pub enum ParseError {
#[diagnostic(code(nu::parser::expected_keyword))]
CaptureOfMutableVar(#[label("capture of mutable variable")] Span),
/// A record or table literal names the same field or column twice. The parse-time
/// counterpart of `ShellError::ColumnDefinedTwice`, for keys written as literals.
#[error("Record field or table column used twice: {0}")]
#[diagnostic(code(nu::parser::column_defined_twice))]
ColumnDefinedTwice(
String,
#[label(primary, "field redefined here")] Span,
#[label("field first defined here")] Span,
),
#[error("Expected keyword.")]
#[diagnostic(code(nu::parser::expected_keyword))]
ExpectedKeyword(String, #[label("expected {0}")] Span),
@@ -662,6 +705,13 @@ pub enum ParseError {
#[diagnostic()]
InvalidLiteral(String, String, #[label("{0} in {1}")] Span),
/// A filesize or duration literal is outside the 64-bit range of its type.
///
/// - `0`: the type, `filesize` or `duration`
#[error("Invalid literal")]
#[diagnostic(code(nu::parser::literal_too_large))]
LiteralTooLarge(String, #[label("{0} too large")] Span),
#[error("{0}")]
#[diagnostic()]
LabeledError(String, String, #[label("{1}")] Span),
@@ -734,29 +784,25 @@ impl ParseError {
}
/// Build an [`Unclosed`](ParseError::Unclosed) with default help text.
pub fn unclosed(delimiter: &'static str, open_span: Span, end_span: Span) -> Self {
Self::Unclosed(
delimiter,
open_span,
end_span,
default_unclosed_help(delimiter, None),
)
pub fn unclosed(
delimiter: impl Into<Cow<'static, str>>,
open_span: Span,
end_span: Span,
) -> Self {
Self::unclosed_with_hint(delimiter, open_span, end_span, None)
}
/// Build an [`Unclosed`](ParseError::Unclosed) with an optional structure hint
/// (e.g. `record field ls`, `def foo`). Empty/`None` uses generic help.
pub fn unclosed_with_hint(
delimiter: &'static str,
delimiter: impl Into<Cow<'static, str>>,
open_span: Span,
end_span: Span,
structure_hint: Option<&str>,
) -> Self {
Self::Unclosed(
delimiter,
open_span,
end_span,
default_unclosed_help(delimiter, structure_hint),
)
let delimiter = delimiter.into();
let help = default_unclosed_help(&delimiter, structure_hint);
Self::Unclosed(delimiter, open_span, end_span, help)
}
/// Build an [`Unbalanced`](ParseError::Unbalanced) with default help text.
@@ -792,6 +838,7 @@ impl ParseError {
ParseError::NameIsBuiltinVar(_, s) => *s,
ParseError::NameIsKeyword(_, _, s) => *s,
ParseError::CaptureOfMutableVar(s) => *s,
ParseError::ColumnDefinedTwice(_, s, _) => *s,
ParseError::IncorrectValue(_, s, _) => *s,
ParseError::InvalidBinaryString(s, _) => *s,
ParseError::MultipleRestParams(s) => *s,
@@ -817,6 +864,8 @@ impl ParseError {
ParseError::NonUtf8(s) => *s,
ParseError::UnknownFlag(_, _, s, _) => *s,
ParseError::RequiredAfterOptional(_, s) => *s,
ParseError::PositionalAfterRest(_, s) => *s,
ParseError::DuplicateParameter(_, s, _) => *s,
ParseError::UnknownType(s) => *s,
ParseError::TypeAliasUsedBeforeDefinition(_, s, _) => *s,
ParseError::TypeAliasSelfReference(_, s) => *s,
@@ -861,6 +910,7 @@ impl ParseError {
ParseError::UnexpectedRedirection { span } => *span,
ParseError::UnknownOperator(_, _, s) => *s,
ParseError::InvalidLiteral(_, _, s) => *s,
ParseError::LiteralTooLarge(_, s) => *s,
ParseError::LabeledErrorWithHelp { span: s, .. } => *s,
ParseError::RedirectingBuiltinCommand(_, s, _) => *s,
ParseError::UnexpectedSpreadArg(_, s) => *s,
+5
View File
@@ -222,6 +222,11 @@ pub trait Eval {
}),
}
}
Expr::UnaryMinus(operand) => {
// Negation is multiplication by -1, as in the IR compiler.
let value = Self::eval::<D>(state, mut_state, operand)?;
value.mul(expr_span, &Value::int(-1, expr_span), expr_span)
}
Expr::BinaryOp(lhs, op, rhs) => {
let op_span = op.span(&state);
let op = eval_operator(op)?;
+11 -1
View File
@@ -378,7 +378,17 @@ fn eval_const_if(
span: then_expr.span,
})?;
if eval_constant(working_set, cond)?.as_bool()? {
// A non-bool condition is the same type mismatch as when `if` runs.
let condition = match eval_constant(working_set, cond)? {
Value::Bool { val, .. } => val,
other => {
return Err(ShellError::TypeMismatch {
err_message: "expected bool".into(),
span: other.span(),
});
}
};
if condition {
let block = working_set.get_block(then_block);
eval_const_subexpression(working_set, block, input, block.span.unwrap_or(call.head))
} else if let Some(else_case) = else_case {
+15 -38
View File
@@ -2436,11 +2436,12 @@ impl PartialOrd for Value {
// scaled as the value being compared increases in magnitude (using relative epsilon-based
// tolerance). Implementation is similar to python's `math.isclose()` function:
// https://docs.python.org/3/library/math.html#math.isclose. Fallback to the default strict
// float comparison if the difference exceeds the error epsilon.
// float comparison if the difference exceeds the error epsilon. Infinities skip the
// tolerance: with an infinite operand it would be infinite and match every number.
fn compare_floats(val: f64, other: f64) -> Option<Ordering> {
let prec = f64::EPSILON.max(val.abs().max(other.abs()) * f64::EPSILON);
if (other - val).abs() <= prec {
if val.is_finite() && other.is_finite() && (other - val).abs() <= prec {
return Some(Ordering::Equal);
}
@@ -3558,8 +3559,14 @@ impl Value {
}
pub fn modulo(&self, op: Span, rhs: &Value, span: Span) -> Result<Value, ShellError> {
// Based off the unstable `div_floor` function in the std library.
// Based off the unstable `div_floor` function in the std library. Returns `None` only for
// a zero divisor.
fn checked_mod_i64(dividend: i64, divisor: i64) -> Option<i64> {
// `checked_rem` also fails for `i64::MIN % -1`, because the matching division
// overflows, but any number mod -1 is 0. Only a zero divisor is an error.
if divisor == -1 {
return Some(0);
}
let remainder = dividend.checked_rem(divisor)?;
if (remainder > 0 && divisor < 0) || (remainder < 0 && divisor > 0) {
// Note that `remainder + divisor` cannot overflow, because `remainder` and
@@ -3587,14 +3594,8 @@ impl Value {
(Value::Int { val: lhs, .. }, Value::Int { val: rhs, .. }) => {
if let Some(val) = checked_mod_i64(*lhs, *rhs) {
Ok(Value::int(val, span))
} else if *rhs == 0 {
Err(ShellError::DivisionByZero { span: op })
} else {
Err(ShellError::OperatorOverflow {
msg: "modulo operation overflowed".into(),
span,
help: None,
})
Err(ShellError::DivisionByZero { span: op })
}
}
(Value::Int { val: lhs, .. }, Value::Float { val: rhs, .. }) => {
@@ -3621,27 +3622,15 @@ impl Value {
(Value::Filesize { val: lhs, .. }, Value::Filesize { val: rhs, .. }) => {
if let Some(val) = checked_mod_i64(lhs.get(), rhs.get()) {
Ok(Value::filesize(val, span))
} else if *rhs == Filesize::ZERO {
Err(ShellError::DivisionByZero { span: op })
} else {
Err(ShellError::OperatorOverflow {
msg: "modulo operation overflowed".into(),
span,
help: None,
})
Err(ShellError::DivisionByZero { span: op })
}
}
(Value::Filesize { val: lhs, .. }, Value::Int { val: rhs, .. }) => {
if let Some(val) = checked_mod_i64(lhs.get(), *rhs) {
Ok(Value::filesize(val, span))
} else if *rhs == 0 {
Err(ShellError::DivisionByZero { span: op })
} else {
Err(ShellError::OperatorOverflow {
msg: "modulo operation overflowed".into(),
span,
help: None,
})
Err(ShellError::DivisionByZero { span: op })
}
}
(Value::Filesize { val: lhs, .. }, Value::Float { val: rhs, .. }) => {
@@ -3662,27 +3651,15 @@ impl Value {
(Value::Duration { val: lhs, .. }, Value::Duration { val: rhs, .. }) => {
if let Some(val) = checked_mod_i64(*lhs, *rhs) {
Ok(Value::duration(val, span))
} else if *rhs == 0 {
Err(ShellError::DivisionByZero { span: op })
} else {
Err(ShellError::OperatorOverflow {
msg: "division operation overflowed".into(),
span,
help: None,
})
Err(ShellError::DivisionByZero { span: op })
}
}
(Value::Duration { val: lhs, .. }, Value::Int { val: rhs, .. }) => {
if let Some(val) = checked_mod_i64(*lhs, *rhs) {
Ok(Value::duration(val, span))
} else if *rhs == 0 {
Err(ShellError::DivisionByZero { span: op })
} else {
Err(ShellError::OperatorOverflow {
msg: "division operation overflowed".into(),
span,
help: None,
})
Err(ShellError::DivisionByZero { span: op })
}
}
(Value::Duration { val: lhs, .. }, Value::Float { val: rhs, .. }) => {
+4 -4
View File
@@ -451,8 +451,8 @@ mod float_range {
return false;
}
match self.end {
Bound::Included(end) if value <= end => return false,
Bound::Excluded(end) if value < end => return false,
Bound::Included(end) if value < end => return false,
Bound::Excluded(end) if value <= end => return false,
_ => {}
};
} else {
@@ -461,8 +461,8 @@ mod float_range {
return false;
}
match self.end {
Bound::Included(end) if value >= end => return false,
Bound::Excluded(end) if value > end => return false,
Bound::Included(end) if value > end => return false,
Bound::Excluded(end) if value >= end => return false,
_ => {}
};
}
+43
View File
@@ -3,6 +3,7 @@ use nu_protocol::{
engine::{EngineState, Stack},
};
use rstest::rstest;
use std::cmp::Ordering;
#[test]
fn test_comparison_nothing() {
@@ -103,6 +104,48 @@ fn test_float_equality_comparison() {
}
}
/// The equality tolerance scales with the operands, so an infinite operand must skip it.
/// Otherwise `inf == 1` and `inf == -inf` would be true.
#[rstest]
#[case::inf_vs_int(
Value::test_float(f64::INFINITY),
Value::test_int(1),
Ordering::Greater
)]
#[case::inf_vs_float(
Value::test_float(f64::INFINITY),
Value::test_float(1.0),
Ordering::Greater
)]
#[case::neg_inf_vs_zero(
Value::test_float(f64::NEG_INFINITY),
Value::test_int(0),
Ordering::Less
)]
#[case::inf_vs_neg_inf(
Value::test_float(f64::INFINITY),
Value::test_float(f64::NEG_INFINITY),
Ordering::Greater
)]
#[case::inf_vs_inf(
Value::test_float(f64::INFINITY),
Value::test_float(f64::INFINITY),
Ordering::Equal
)]
#[case::max_vs_inf(
Value::test_float(f64::MAX),
Value::test_float(f64::INFINITY),
Ordering::Less
)]
fn test_float_infinity_comparison(
#[case] lhs: Value,
#[case] rhs: Value,
#[case] expected: Ordering,
) {
assert_eq!(lhs.partial_cmp(&rhs), Some(expected));
assert_eq!(rhs.partial_cmp(&lhs), Some(expected.reverse()));
}
#[rstest]
#[case(365 * 24 * 3600 * 1_000_000_000, "52wk 1day")]
#[case( (((((((7 + 2) * 24 + 3) * 60 + 4) * 60) + 5) * 1000 + 6) * 1000 + 7) * 1000 + 8,
+55 -63
View File
@@ -1,5 +1,6 @@
use nu_protocol::{
DEFAULT_ERROR_CONTEXT, Filesize, IntoValue, Range, Record, ShellError, Span, Type, Unit, Value,
DEFAULT_ERROR_CONTEXT, Filesize, IntoValue, ParseError, Range, Record, ShellError, Span, Type,
Unit, Value,
ast::{Expr, Expression, ListItem, RecordItem},
engine::{EngineState, StateWorkingSet},
shell_error::generic::GenericError,
@@ -17,6 +18,22 @@ fn truncated_nuon_error(src: &str, err_span: Span, msg: impl Into<String>) -> Sh
}
}
/// The error for a filesize or duration (`ty`) at `err_span` that doesn't fit in 64 bits.
fn too_large_error(src: &str, err_span: Span, ty: &str) -> ShellError {
let msg = format!("{ty} too large");
if ty == "filesize" {
let (src, span) = truncated_source_window(src, err_span, DEFAULT_ERROR_CONTEXT);
ShellError::OutsideSpannedLabeledError {
src,
error: msg.clone(),
msg,
span,
}
} else {
truncated_nuon_error(src, err_span, msg)
}
}
/// convert a raw string representation of NUON data to an actual Nushell [`Value`]
///
// WARNING: please leave the following two trailing spaces, they matter for the documentation
@@ -115,6 +132,22 @@ pub fn from_nuon(input: &str, span: Option<Span>) -> Result<Value, ShellError> {
}
};
// The parser rejects a repeated key and a filesize or duration too large for 64 bits; report
// them as the errors they are rather than as a general parse failure.
match working_set.parse_errors.first() {
Some(ParseError::ColumnDefinedTwice(col_name, second_use, first_use)) => {
return Err(ShellError::ColumnDefinedTwice {
col_name: col_name.clone(),
second_use: *second_use,
first_use: *first_use,
});
}
Some(ParseError::LiteralTooLarge(ty, span)) => {
return Err(too_large_error(input, *span, ty));
}
_ => {}
}
if let Some(err) = working_set.parse_errors.first() {
let (src, label_span) = truncated_source_window(input, err.span(), DEFAULT_ERROR_CONTEXT);
return Err(ShellError::Generic(
@@ -164,7 +197,7 @@ fn convert_to_value(
expr.span,
"binary operators not supported in nuon",
)),
Expr::UnaryNot(..) => Err(truncated_nuon_error(
Expr::UnaryNot(..) | Expr::UnaryMinus(..) => Err(truncated_nuon_error(
original_text,
expr.span,
"unary operators not supported in nuon",
@@ -290,8 +323,8 @@ fn convert_to_value(
}
Expr::Record(key_vals) => {
let mut record = Record::with_capacity(key_vals.len());
let mut key_spans = Vec::with_capacity(key_vals.len());
// The parser has already rejected a key used twice (see `from_nuon`).
for key_val in key_vals {
match key_val {
RecordItem::Pair(key, val) => {
@@ -306,16 +339,7 @@ fn convert_to_value(
}
};
if let Some(i) = record.index_of(&key_str) {
return Err(ShellError::ColumnDefinedTwice {
col_name: key_str,
second_use: key.span,
first_use: key_spans[i],
});
} else {
key_spans.push(key.span);
record.push(key_str, convert_to_value(val, span, original_text)?);
}
record.push(key_str, convert_to_value(val, span, original_text)?);
}
RecordItem::Spread(_, inner) => {
return Err(truncated_nuon_error(
@@ -365,6 +389,7 @@ fn convert_to_value(
let mut output = vec![];
// The parser has already rejected a column named twice (see `from_nuon`).
for key in table.columns.as_mut() {
let key_str = match &mut key.expr {
Expr::String(key_str) => key_str,
@@ -377,15 +402,7 @@ fn convert_to_value(
}
};
if let Some(idx) = cols.iter().position(|existing| existing == key_str) {
return Err(ShellError::ColumnDefinedTwice {
col_name: key_str.clone(),
second_use: key.span,
first_use: table.columns[idx].span,
});
} else {
cols.push(std::mem::take(key_str));
}
cols.push(std::mem::take(key_str));
}
for row in table.rows.into_vec() {
@@ -422,34 +439,22 @@ fn convert_to_value(
}
};
// The parser stores every filesize in bytes and every duration in nanoseconds, and
// rejects one too large for 64 bits, so parsed text always fits here. A value with
// a larger unit still has to be checked against the 64-bit range.
match value.unit.item {
Unit::Filesize(unit) => match Filesize::from_unit(size, unit) {
Some(val) => Ok(val.into_value(span)),
None => {
let (src, label_span) = truncated_source_window(
original_text,
expr.span,
DEFAULT_ERROR_CONTEXT,
);
Err(ShellError::OutsideSpannedLabeledError {
src,
error: "filesize too large".into(),
msg: "filesize too large".into(),
span: label_span,
})
}
None => Err(too_large_error(original_text, expr.span, "filesize")),
},
// Every duration unit goes through `Unit::build_value`, which
// already checks the multiplication against the 64-bit range.
// A plain `size * factor` would panic in debug builds and wrap
// to a negative duration in release ones, and the nuon spec
// requires an error. The unit is not named in the message: the
// parser has already folded the one that was written, so
// `1e30sec` arrives as a millisecond count.
unit => unit.build_value(size, span).map_err(|_| {
truncated_nuon_error(original_text, expr.span, "duration too large")
}),
// Every duration unit goes through `Unit::build_value`, which checks the
// multiplication against the 64-bit range. A plain `size * factor` would panic
// in debug builds and wrap to a negative duration in release ones, and the nuon
// spec requires an error.
unit => unit
.build_value(size, span)
.map_err(|_| too_large_error(original_text, expr.span, "duration")),
}
}
Expr::Var(..) => Err(truncated_nuon_error(
@@ -481,21 +486,10 @@ mod tests {
// says: "overflow is an **error**; it must not wrap or saturate. a positive
// literal must never read back as a negative duration."
//
// The parser clamps an out-of-range head to i64::MAX and keeps the unit it
// folded to, so `[1e30sec]` reaches this function as Int(i64::MAX) plus
// Unit::Millisecond. `size * 1000 * 1000` then overflowed: a panic in debug
// builds and -1000000ns in release ones. Unit::Day and Unit::Week already
// used checked_mul; the units that real `ms`/`sec`/`min`/`hr`/`day` literals
// fold to did not.
//
// `ns` and `us` fold to Unit::Nanosecond, which multiplies by nothing, so
// they are not covered here.
//
// A negative head is specified to be allowed (the spec shows `-1_000ns` as
// valid), so `[-1sec]` has to keep decoding — but the clamped negative head
// is the same i64::MIN the positive one is, and the raw multiplies it used
// to reach turned it into exactly 0ns, which is neither the number written
// nor an error. Both signs go through the same arms, so both are pinned.
// The parser rejects a duration too large for 64 bits in nanoseconds, of
// either sign, and `from_nuon` reports that as the error below. Both signs
// are pinned, since a negative head is allowed (the spec shows `-1_000ns`
// as valid) and has to keep decoding when it fits.
#[test]
fn duration_overflow_is_rejected() {
for unit in ["ms", "sec", "min", "hr", "day", "wk"] {
@@ -509,9 +503,7 @@ mod tests {
panic!("`{input}` should report a labeled duration overflow, got {other:?}")
}
};
// The unit is deliberately not named: the parser folds the one that
// was written, so `1e30sec` reaches the reader as a millisecond
// count and `1e30wk` as a day count.
// The message names the type, not the unit that was written.
assert_eq!(
msg, "duration too large",
"unexpected message for `{input}`"
+1 -1
View File
@@ -7,7 +7,7 @@ use rstest::rstest;
fn record_with_redefined_key() -> Result {
test()
.run("{x: 1, x: 2}")
.expect_error_code_eq("nu::shell::column_defined_twice")
.expect_error_code_eq("nu::parser::column_defined_twice")
}
#[test]
+27
View File
@@ -1,4 +1,6 @@
use crate::repl::tests::{TestResult, fail_test, run_test};
use nu_test_support::prelude::*;
use rstest::rstest;
#[test]
fn add_simple() -> TestResult {
@@ -25,6 +27,31 @@ fn modulo2() -> TestResult {
run_test("5.25 mod 2", "1.25")
}
/// Infinity orders above every finite number and is equal only to itself.
#[rstest]
#[case::eq_int("inf == 1", false)]
#[case::eq_neg_inf("inf == -inf", false)]
#[case::eq_self("inf == inf", true)]
#[case::gt_int("inf > 1", true)]
#[case::lt_inf("1 < inf", true)]
#[case::neg_inf_lt_zero("-inf < 0", true)]
#[case::max("[1.0 inf] | math max | $in == inf", true)]
fn infinity_comparison(#[case] code: &str, #[case] expected: bool) -> Result {
test().run(code).expect_value_eq(expected)
}
/// `in` accepts an int in a list of floats (and the reverse), like `==` does.
#[rstest]
#[case::int_in_floats("1 in [1.0]", true)]
#[case::float_in_ints("1.0 in [1]", true)]
#[case::int_in_mixed("1 in [1.0 'a']", true)]
#[case::string_in_mixed("'a' in [1 'a']", true)]
#[case::not_in("2 not-in [1.0]", true)]
#[case::has("[1.0] has 1", true)]
fn in_operator_int_float(#[case] code: &str, #[case] expected: bool) -> Result {
test().run(code).expect_value_eq(expected)
}
#[test]
fn bit_shr() -> TestResult {
run_test("16 bit-shr 1", "8")
+245
View File
@@ -58,6 +58,62 @@ fn hex_ints_with_underscores() -> Result {
test().run("0x68__9d__6a").expect_value_eq(6_856_042)
}
/// A sign may come before a radix prefix, as it may before a decimal number.
#[rstest]
#[case::negative_hex("-0x10", -16)]
#[case::positive_binary("+0b11", 3)]
#[case::negative_octal("-0o17", -15)]
#[case::smallest_int("-0x8000000000000000", i64::MIN)]
// Hex digits that spell a filesize unit (`b`, `eb`) are still hex.
#[case::negative_hex_ending_in_b("-0x1b", -27)]
#[case::positive_hex_ending_in_eb("+0xeb", 235)]
fn signed_radix_ints(#[case] code: &str, #[case] expected: i64) -> Result {
test().run(code).expect_value_eq(expected)
}
/// An int literal that doesn't fit in 64 bits is an error, not a float that lost precision.
#[rstest]
#[case::decimal("9223372036854775808")]
#[case::negative("-9223372036854775809")]
#[case::in_list("[9223372036854775808]")]
#[case::hex("0x10000000000000000")]
#[case::positive_hex("+0x8000000000000000")]
#[case::range_end("let r = 1..99999999999999999999")]
fn out_of_range_int_literal(#[case] code: &str) -> Result {
let err = test().run(code).expect_parse_error()?;
assert_matches!(err, ParseError::InvalidLiteral(msg, ..) if msg.contains("too large"));
Ok(())
}
/// A word with bad radix digits isn't a number literal, so it can name a command.
#[test]
fn radix_like_command_name() -> Result {
test()
.run(r#"def 0xdump [] { "ran" }; 0xdump"#)
.expect_value_eq("ran")
}
/// Written as a float, a number past the int range is still a float.
#[test]
fn large_float_literal() -> Result {
test()
.run("9223372036854775808.0")
.expect_value_eq(2f64.powi(63))
}
/// The special floats are spelled `inf`, `nan` or `NaN` (what `to nuon` writes). Other
/// spellings that Rust would accept are bare words.
#[rstest]
#[case::inf("[inf -inf] | describe", "list<float>")]
#[case::nan("[nan NaN] | describe", "list<float>")]
#[case::upper_case("[INF NAN] | describe", "list<string>")]
#[case::infinity("[Infinity] | describe", "list<string>")]
#[case::underscores("[i_n_f n_an] | describe", "list<string>")]
#[case::name("{name: Nan} | get name | describe", "string")]
fn special_float_names(#[case] code: &str, #[case] expected: &str) -> Result {
test().run(code).expect_value_eq(expected)
}
#[test]
fn alias_2() -> Result {
test()
@@ -418,6 +474,62 @@ fn record_quotes_with_equals(#[case] code: &str, #[case] expected: &str) -> Resu
test().run(code).expect_value_eq(expected)
}
/// A bare word record key or table column name is the string it spells, keywords and numbers
/// included.
#[test]
fn bare_word_names() -> Result {
test()
.run("{true: 1, null: 2, 1: 3} | columns")
.expect_value_eq(["true", "null", "1"])?;
test()
.run("[[1 true 1kb]; [a b c]] | columns")
.expect_value_eq(["1", "true", "1kb"])?;
// Not a valid date, a negation or a number too large to be a value: still just names.
test()
.run("let x = 5; [[2023-02-29 -(1) -$x 99999999999999999999]; [a b c d]] | columns")
.expect_value_eq(["2023-02-29", "-1", "-$x", "99999999999999999999"])
}
/// A field or column a literal names twice is a parse error, before anything runs.
#[rstest]
#[case::record("print before; {a: 1, a: 2}")]
#[case::table("[[a a]; [1 2]]")]
#[case::bare_and_quoted(r#"{a: 1, "a": 2}"#)]
#[case::raw_string("print before; {a: 1, r#'a'#: 2}")]
fn duplicate_literal_names(#[case] code: &str) -> Result {
test()
.run(code)
.expect_error_code_eq("nu::parser::column_defined_twice")
}
/// A key only known at runtime is still checked when the record is built.
#[test]
fn duplicate_runtime_name() -> Result {
test()
.run("let k = 'a'; {a: 1, $k: 2}")
.expect_error_code_eq("nu::shell::column_defined_twice")
}
/// An extra `]` after a list is unbalanced, as an extra `}` or `)` is.
#[rstest]
#[case::list("[1 2]]")]
#[case::table("[[a]; [1]]]")]
#[case::nested("[1 [2]]]")]
fn list_extra_closing_bracket(#[case] code: &str) -> Result {
test()
.run(code)
.expect_error_code_eq("nu::parser::unbalanced_delimiter")
}
/// A word that isn't a list may have a `]` after its first bracket group closes, as glob and
/// regex classes do when they include `]` (`[]]*`, `[^]]+`).
#[rstest]
#[case::regex_class(r#""a]b[c]d" | str replace -r -a [^]]+ x"#, "x]x]x")]
#[case::string_argument("def f [x: string] { $x }; f [a]]", "[a]]")]
fn bracket_word_extra_closing_bracket(#[case] code: &str, #[case] expected: &str) -> Result {
test().run(code).expect_value_eq(expected)
}
#[test]
fn recursive_parse() -> Result {
test()
@@ -1004,6 +1116,79 @@ fn unary_not_6() -> Result {
.expect_value_eq("def")
}
/// `not` binds looser than comparisons and tighter than `and`, `xor` and `or` (precedence 55 in
/// `help operators`), so `not a == b` is `not (a == b)`.
#[rstest]
#[case::strings(r#"not "a" == "b""#, true)]
#[case::variable("let x = 1; not $x == 2", true)]
#[case::bools("let a = true; let b = false; not $a == $b", true)]
#[case::membership("not 3 in [1 2]", true)]
#[case::and_binds_looser("not true and false", false)]
#[case::or_binds_looser("not false or false", true)]
#[case::double("not not true == false", false)]
#[case::after_operator("true and not 1 == 2", true)]
fn not_precedence(#[case] code: &str, #[case] expected: bool) -> Result {
test().run(code).expect_value_eq(expected)
}
#[test]
fn not_precedence_in_row_condition() -> Result {
test()
.run("[[a b]; [1 2] [3 4]] | where not a == 1 | get a")
.expect_value_eq([3])
}
/// `-$x` and `-(expr)` negate a number, duration or filesize.
#[rstest]
#[case::variable("let x = 5; -$x", -5)]
#[case::subexpression("-(2 + 3)", -5)]
#[case::operand("let x = 5; 1 + -$x", -4)]
#[case::list_item("let x = 5; [-$x] | first", -5)]
#[case::record_value("let x = 5; {a: -$x} | get a", -5)]
#[case::constant("const y = -(2 + 3); $y", -5)]
#[case::duration("let x = 5sec; -$x | into int", -5_000_000_000)]
#[case::filesize("let x = 2kB; -$x | into int", -2000)]
#[case::range_start("let x = 1; -$x..1 | first", -1)]
fn unary_minus(#[case] code: &str, #[case] expected: i64) -> Result {
test().run(code).expect_value_eq(expected)
}
#[test]
fn unary_minus_float() -> Result {
test().run("let x = 1.5; -$x").expect_value_eq(-1.5)
}
/// An argument passed through to an external command stays the string it spells, as for
/// `^cmd`, even where the `extern` declares untyped rest arguments.
#[test]
fn extern_argument_is_not_negated() -> Result {
test()
.run(r#"extern ext [...args]; let s = "tf"; let f = {|| ext -($s) -$s }; "parsed""#)
.expect_value_eq("parsed")
}
/// Words that start with `-(` but aren't one subexpression stay bare string interpolations.
#[rstest]
#[case::suffix("-(1)px", "-1px")]
#[case::two_groups("-(1)+(2)", "-1+2")]
fn minus_bare_word_interpolation(#[case] code: &str, #[case] expected: &str) -> Result {
test().run(code).expect_value_eq(expected)
}
#[test]
fn unary_minus_type_error() -> Result {
let err = test().run("let x = 'a'; -$x").expect_parse_error()?;
assert_matches!(
err,
ParseError::OperatorUnsupportedType {
op: "-",
unsupported: Type::String,
..
}
);
Ok(())
}
#[test]
fn comment_in_multiple_pipelines() -> Result {
let code = "
@@ -1022,6 +1207,31 @@ fn date_literal() -> Result {
.expect_value_eq(10)
}
/// A word written like a date that isn't a valid date is an error, not a bare string or a
/// command name.
#[rstest]
#[case::in_list("[2023-02-29]")]
#[case::command_position("2023-02-29")]
#[case::with_time("[2023-02-29T10:00:00]")]
fn invalid_date_literal(#[case] code: &str) -> Result {
let err = test().run(code).expect_parse_error()?;
assert_matches!(err, ParseError::InvalidLiteral(msg, ..) if msg == "not a valid date");
Ok(())
}
/// Anything after the date other than a time keeps a word a bare string, and so does a real day
/// with a time that isn't one a literal can have.
#[rstest]
#[case::extension("2023-02-29.log")]
#[case::dashed_time("2024-01-15T10-30-00.log")]
#[case::word_after_t("2024-01-15test.txt")]
#[case::no_seconds("2024-01-15T10:00")]
fn date_like_file_name(#[case] word: &str) -> Result {
test()
.run(format!("[{word}] | first"))
.expect_value_eq(word)
}
#[test]
fn and_and_or() -> Result {
test().run("true and false or true").expect_value_eq(true)
@@ -1256,6 +1466,41 @@ fn filesize_is_not_hex() -> Result {
test().run("0x42b").expect_value_eq(1067)
}
/// A duration or filesize that reaches 2^63 in its base unit is out of range, not saturated to
/// the largest value, whatever unit it is written in.
#[rstest]
#[case::nanoseconds("9223372036854775808ns")]
#[case::bytes("9223372036854775808b")]
#[case::exbibytes("8EiB")]
#[case::days("106752day")]
#[case::below_smallest("-9223372036854775809ns")]
fn unit_literal_out_of_range(#[case] code: &str) -> Result {
let err = test().run(code).expect_parse_error()?;
assert_matches!(err, ParseError::LiteralTooLarge(..));
Ok(())
}
/// The largest and smallest values fit exactly, also when NUON writes them and reads them back.
#[rstest]
#[case::largest_duration("9223372036854775807ns | into int", i64::MAX)]
#[case::largest_filesize("9223372036854775807b | into int", i64::MAX)]
#[case::smallest_duration("-9223372036854775808ns | into int", i64::MIN)]
#[case::nuon_round_trip(
"9223372036854775807 | into duration | to nuon | from nuon | into int",
i64::MAX
)]
fn unit_literal_limits(#[case] code: &str, #[case] expected: i64) -> Result {
test().run(code).expect_value_eq(expected)
}
/// A `+` sign works on unit literals as it does on plain numbers.
#[rstest]
#[case::duration("+5sec == 5sec")]
#[case::filesize("+1kB == 1kB")]
fn unit_literal_plus_sign(#[case] code: &str) -> Result {
test().run(code).expect_value_eq(true)
}
#[test]
fn let_variable_type_mismatch() -> Result {
let err = test().run(r#"let x: int = "foo""#).expect_parse_error()?;
+15
View File
@@ -1,4 +1,5 @@
use crate::repl::tests::{TestResult, fail_test, run_test};
use nu_test_support::prelude::*;
use rstest::rstest;
#[test]
@@ -77,6 +78,20 @@ fn float_not_in_unbounded_stepped_range() -> TestResult {
run_test("2.1 in 1.2..3..", "false")
}
/// A float range includes its end with `..` and excludes it with `..<`, like an int range.
#[rstest]
#[case::inclusive_end("2.0 in 1.0..2.0", true)]
#[case::exclusive_end("2.0 in 1.0..<2.0", false)]
#[case::int_in_float_range("2 in 1.0..2.0", true)]
#[case::float_in_int_range("2.0 in 1..2", true)]
#[case::float_past_end("2.5 in 1.0..2.0", false)]
#[case::decreasing_inclusive_end("0.0 in 2.0..1.0..0.0", true)]
#[case::decreasing_exclusive_end("0.0 in 2.0..1.0..<0.0", false)]
#[case::match_pattern("match 2.0 { 1..2 => true, _ => false }", true)]
fn float_range_end_bound(#[case] code: &str, #[case] expected: bool) -> Result {
test().run(code).expect_value_eq(expected)
}
#[rstest]
#[case("1..=3..", "expected number")]
#[case("..=3..=15", "expected number")]
+57
View File
@@ -1,5 +1,6 @@
use nu_protocol::Type;
use nu_test_support::prelude::*;
use pretty_assertions::assert_matches;
use crate::repl::tests::{TestResult, fail_test, run_test};
use rstest::rstest;
@@ -466,3 +467,59 @@ fn oneof_annotations_with_extra_characters() -> TestResult {
fn closure_param_list_not_terminated(#[case] input: &str) -> TestResult {
fail_test(input, "expected `|`")
}
/// The rest parameter takes every remaining positional argument, so no positional parameter can
/// follow it. Flags can.
#[rstest]
#[case::required("def f [...r, a] { }")]
#[case::optional("def f [...r, a?] { }")]
fn positional_after_rest(#[case] code: &str) -> Result {
test()
.run(code)
.expect_error_code_eq("nu::parser::positional_after_rest")
}
#[test]
fn flag_after_rest() -> Result {
test()
.run("def f [...r, --x] { [$r $x] }; f 1 --x")
.expect_value_eq(test_value!([[1], true]))
}
/// Parameters and flags share one variable namespace, and short flags need distinct letters.
#[rstest]
#[case::positionals("def f [a, a] { }")]
#[case::long_flags("def f [--x, --x] { }")]
#[case::short_flags("def f [--x (-x), --y (-x)] { }")]
#[case::positional_and_flag("def f [foo_bar, --foo-bar] { }")]
#[case::closure_params("{|a, a| $a}")]
// `$a` and `a` bind the same variable.
#[case::dollar_spelling("def f [$a, a] { }")]
#[case::dollar_and_flag("def f [$a, --a] { }")]
fn duplicate_parameters(#[case] code: &str) -> Result {
test()
.run(code)
.expect_error_code_eq("nu::parser::duplicate_parameter")
}
/// `_` may name several parameters, to ignore those arguments.
#[rstest]
#[case::closure("[1 2 3] | reduce {|_, _| 7 }", 7)]
#[case::def("def f [_, _] { 1 }; f 2 3", 1)]
fn repeated_underscore_parameters(#[case] code: &str, #[case] expected: i64) -> Result {
test().run(code).expect_value_eq(expected)
}
/// Every command has `--help`, so a custom `--help` flag could never be set. `-h` can still be
/// another flag's short form.
#[test]
fn help_flag_is_reserved() -> Result {
let err = test().run("def f [--help] { }").expect_parse_error()?;
assert_matches!(
err,
ParseError::LabeledErrorWithHelp { error, .. } if error.contains("`--help` flag is reserved")
);
test()
.run("def f [--host (-h)] { $host }; f -h")
.expect_value_eq(true)
}
+50 -1
View File
@@ -1,4 +1,7 @@
use crate::repl::tests::{TestResult, fail_test, run_test};
use nu_test_support::prelude::*;
use pretty_assertions::assert_matches;
use rstest::rstest;
#[test]
fn cjk_in_substrings() -> TestResult {
@@ -169,7 +172,7 @@ fn raw_string_inside_closure() -> TestResult {
#[test]
fn incomplete_string() -> TestResult {
fail_test("r#abc", "expected '")?;
fail_test("r#'bc", "expected closing '#")?;
fail_test("r#'bc", "expected `'#`")?;
fail_test("'ab\"", "expected `'`")?;
fail_test("\"ab'", "expected `\"`")?;
fail_test(
@@ -182,3 +185,49 @@ fn incomplete_string() -> TestResult {
"expected `\"`",
)
}
/// Text after a string's last quote is an error.
#[rstest]
#[case::single_then_bare("'a'b")]
#[case::single_then_double(r#"'a'"b""#)]
fn text_after_closing_quote(#[case] code: &str) -> Result {
test()
.run(code)
.expect_error_code_eq("nu::parser::extra_token_after_closing_delimiter")
}
/// A quoted word runs to its last quote of the same kind, and the quotes of that kind inside it
/// are part of the string. Scripts rely on this for JSON and HTML text.
#[rstest]
#[case::single("'a''b'", "a''b")]
#[case::double(r#""[{a: "b"}]""#, r#"[{a: "b"}]"#)]
#[case::interpolation(r#"let c = 'x'; $"<p class="($c)">""#, r#"<p class="x">"#)]
fn same_quotes_inside_a_string(#[case] code: &str, #[case] expected: &str) -> Result {
test().run(code).expect_value_eq(expected)
}
/// An escaped quote and a quote inside an interpolated subexpression don't close the string.
#[rstest]
#[case::escaped_quote(r#""a\"b""#, "a\"b")]
#[case::subexpression_quote(r#"$"a(")")b""#, "a)b")]
fn quotes_that_do_not_close(#[case] code: &str, #[case] expected: &str) -> Result {
test().run(code).expect_value_eq(expected)
}
/// `\x` escapes add raw bytes, which must form valid UTF-8.
#[test]
fn hex_escapes_invalid_utf8() -> Result {
let err = test().run(r#""\xFF""#).expect_parse_error()?;
assert_matches!(err, ParseError::InvalidLiteral(msg, ..) if msg.contains("UTF-8"));
test().run(r#""\xC3\xA9""#).expect_value_eq("é")
}
/// An unclosed raw string reports the closer it needs, like any other unclosed string.
#[rstest]
#[case::one_hash("r#'abc", "'#")]
#[case::two_hashes("r##'abc", "'##")]
fn unclosed_raw_string(#[case] code: &str, #[case] closer: &str) -> Result {
let err = test().run(code).expect_parse_error()?;
assert_matches!(err, ParseError::Unclosed(delimiter, ..) if delimiter == closer);
Ok(())
}
+52
View File
@@ -259,6 +259,58 @@ fn run_in_noninteractive_mode() {
assert!(child_output.stderr.is_empty());
}
/// `nu -c` reports every parse warning, not only the first. `@deprecated --report every` warns
/// at each call, and the default `--report first` warns once per deprecated command.
#[rstest]
#[case::report_every("@deprecated --report every\ndef g [] { 1 }\ng; g; g", 3)]
#[case::report_first("@deprecated\ndef g [] { 1 }\ng; g; g", 1)]
#[case::two_commands("@deprecated\ndef g [] { 1 }\n@deprecated\ndef k [] { 2 }\ng; k", 2)]
// The `main` call is parsed after the script, whose warnings are not reported again.
#[case::with_main(
"@deprecated --report every\ndef g [] { 1 }\ndef main [] { g; g; g }",
3
)]
// Code the parser tries more than once still warns once per call.
#[case::retried_parse("@deprecated --report every\ndef g [] { 1 }\n[-(g)(2)] | ignore", 1)]
#[case::range_bound("@deprecated --report every\ndef g [] { 1 }\n1..(g) | ignore", 1)]
#[nu_test_support::test]
#[deps(NU)]
fn commands_report_every_parse_warning(#[case] code: &str, #[case] expected: usize) -> Result {
let output = std::process::Command::new(NU.path())
.args(["-n", "-c", code])
.output()?;
let stderr = String::from_utf8_lossy(&output.stderr);
assert_eq!(
stderr.matches("nu::parser::deprecated").count(),
expected,
"{stderr}"
);
Ok(())
}
/// A script file reports every parse warning too.
#[test]
#[deps(NU)]
fn script_reports_every_parse_warning() -> Result {
Playground::setup("script_parse_warnings", |dirs, sandbox| -> Result {
sandbox.with_files(&[FileWithContent(
"deprecated.nu",
"@deprecated\ndef g [] { 1 }\n@deprecated\ndef k [] { 2 }\ng; k\n",
)]);
let output = std::process::Command::new(NU.path())
.args(["-n", "deprecated.nu"])
.current_dir(dirs.test())
.output()?;
let stderr = String::from_utf8_lossy(&output.stderr);
assert_eq!(
stderr.matches("nu::parser::deprecated").count(),
2,
"{stderr}"
);
Ok(())
})
}
// `$nu.startup-time` must be a real duration wherever it can be read, also before startup has
// finished and in engines that never ran nushell's `main`.
#[test]
+4 -4
View File
@@ -715,8 +715,8 @@ fn table_with_commas() -> Result {
fn duration_overflow() -> Result {
let err = test()
.run("ls | get modified | each { |it| $it + 10000000000000000day }")
.expect_compile_error()?;
assert_matches!(err, CompileError::InvalidLiteral { msg, .. } if msg == "duration too large");
.expect_parse_error()?;
assert_matches!(err, ParseError::LiteralTooLarge(..));
Ok(())
}
@@ -724,8 +724,8 @@ fn duration_overflow() -> Result {
fn date_and_duration_overflow() -> Result {
let err = test()
.run("ls | get modified | each { |it| $it + 1000000000day }")
.expect_compile_error()?;
assert_matches!(err, CompileError::InvalidLiteral { msg, .. } if msg == "duration too large");
.expect_parse_error()?;
assert_matches!(err, ParseError::LiteralTooLarge(..));
Ok(())
}