Added an evaluation of "Strict YAML" to replace "PyYAML".
Conclusion: not worth the effort
This commit is contained in:
parent
19023ef4e6
commit
734d2b9c35
|
|
@ -0,0 +1,193 @@
|
||||||
|
# Strict YAML
|
||||||
|
|
||||||
|
This is an experiment. The idea was: why not let the YAML parser validate the syntax? Something like a DTD.
|
||||||
|
|
||||||
|
Also: give better information for the error, like position in the file.
|
||||||
|
|
||||||
|
Looking for a solution I found [Strict YAML](https://hitchdev.com/strictyaml/). The main idea is really cool, but the implementation isn't good for users having to write a configuration file.
|
||||||
|
|
||||||
|
## Background
|
||||||
|
|
||||||
|
* I'm comparing with the current [PyYAML](https://pypi.org/project/PyYAML/) parser.
|
||||||
|
* I downloaded the sources from [PyPI](https://pypi.org/project/strictyaml/) v1.0.6.
|
||||||
|
* I then tried the code from [GitHub](https://github.com/crdoconnor/strictyaml) commit 63ceb9ba28e1e6829d0ea597ab8707863f2da1ee
|
||||||
|
* I tried to parse KiPlot's configuration files.
|
||||||
|
* The library depends on [ruamel YAML](https://pypi.org/project/ruamel.yaml/)
|
||||||
|
|
||||||
|
Test code:
|
||||||
|
|
||||||
|
|
||||||
|
```
|
||||||
|
schema_ver = MapPattern(Str(), Any())
|
||||||
|
fname = 'test.yaml'
|
||||||
|
with open(fname) as f:
|
||||||
|
s = f.read()
|
||||||
|
try:
|
||||||
|
parsed = load(s, schema_ver, label=fname)
|
||||||
|
except InconsistentIndentationDisallowed as e:
|
||||||
|
print('Use the same indentation across the file')
|
||||||
|
print(e)
|
||||||
|
sys.exit(1)
|
||||||
|
except YAMLError as e:
|
||||||
|
print('YAML parsing error:')
|
||||||
|
print(e)
|
||||||
|
sys.exit(1)
|
||||||
|
|
||||||
|
schema = Map({"kiplot":
|
||||||
|
Map({"version": Int()}),
|
||||||
|
Optional("preflight"): Map({
|
||||||
|
Optional("run_drc"): Bool(),
|
||||||
|
Optional("run_erc"): Bool(),
|
||||||
|
Optional("update_xml"): Bool(),
|
||||||
|
Optional("check_zone_fills"): Bool(),
|
||||||
|
Optional("ignore_unconnected"): Bool(),
|
||||||
|
}),
|
||||||
|
Optional("outputs"): Seq(Any())})
|
||||||
|
|
||||||
|
try:
|
||||||
|
parsed = load(s, schema, label=fname)
|
||||||
|
except YAMLError as e:
|
||||||
|
print('YAML parsing error:')
|
||||||
|
print(e)
|
||||||
|
sys.exit(1)
|
||||||
|
|
||||||
|
print(repr(parsed))
|
||||||
|
print(parsed['kiplot']['version'])
|
||||||
|
|
||||||
|
```
|
||||||
|
|
||||||
|
|
||||||
|
## What I found
|
||||||
|
|
||||||
|
### Strict indentation forced, can't be disabled
|
||||||
|
|
||||||
|
I understand that detecting inconsistent indentation is a good thing. But looks too strict for my case and can't be disabled:
|
||||||
|
|
||||||
|
```
|
||||||
|
kiplot:
|
||||||
|
version: 1
|
||||||
|
|
||||||
|
preflight:
|
||||||
|
run_drc: true
|
||||||
|
|
||||||
|
```
|
||||||
|
|
||||||
|
Gives:
|
||||||
|
|
||||||
|
```
|
||||||
|
While parsing
|
||||||
|
in "indent.yaml", line 5, column 4:
|
||||||
|
run_drc: true
|
||||||
|
^ (line: 5)
|
||||||
|
Found mapping with indentation inconsistent with previous mapping
|
||||||
|
in "indent.yaml", line 7, column 1:
|
||||||
|
|
||||||
|
^ (line: 7)
|
||||||
|
Nuevamente en el editor
|
||||||
|
|
||||||
|
```
|
||||||
|
|
||||||
|
Here:
|
||||||
|
- The exception can't be masked
|
||||||
|
- The message is hard to understand, line 7 doesn't even exist. The message should point to line 2.
|
||||||
|
- The `InconsistentIndentationDisallowed` is hard to import, needs:
|
||||||
|
|
||||||
|
```
|
||||||
|
from strictyaml.exceptions import InconsistentIndentationDisallowed
|
||||||
|
```
|
||||||
|
|
||||||
|
### Errors seems to be descriptive, but they are missleading
|
||||||
|
|
||||||
|
Schema:
|
||||||
|
|
||||||
|
```
|
||||||
|
schema = Map({"kiplot":
|
||||||
|
Map({"version": Int()}),
|
||||||
|
Optional("preflight"): Map({
|
||||||
|
Optional("run_drc"): Bool(),
|
||||||
|
Optional("run_erc"): Bool(),
|
||||||
|
Optional("update_xml"): Bool(),
|
||||||
|
Optional("check_zone_fills"): Bool(),
|
||||||
|
Optional("ignore_unconnected"): Bool(),
|
||||||
|
}),
|
||||||
|
Optional("outputs"): Seq(Any())})
|
||||||
|
```
|
||||||
|
|
||||||
|
YAML:
|
||||||
|
|
||||||
|
```
|
||||||
|
# Example KiPlot config file
|
||||||
|
kiplot:
|
||||||
|
version: 1
|
||||||
|
|
||||||
|
preflight:
|
||||||
|
run_drc: true
|
||||||
|
run_erc: true
|
||||||
|
update_xml: true
|
||||||
|
check_zone_fills: true
|
||||||
|
ignore_unconnected: false
|
||||||
|
|
||||||
|
outputs:
|
||||||
|
- name: 'gerbers'
|
||||||
|
|
||||||
|
cualquiera:
|
||||||
|
- name: 'gerbers'
|
||||||
|
|
||||||
|
```
|
||||||
|
|
||||||
|
We get:
|
||||||
|
|
||||||
|
```
|
||||||
|
YAML parsing error:
|
||||||
|
while parsing a mapping
|
||||||
|
in "test.yaml", line 14, column 1:
|
||||||
|
|
||||||
|
^ (line: 14)
|
||||||
|
unexpected key not in schema 'cualquiera'
|
||||||
|
in "test.yaml", line 16, column 1:
|
||||||
|
- name: gerbers
|
||||||
|
^ (line: 16)
|
||||||
|
```
|
||||||
|
|
||||||
|
While parsing an empty line? Found 'cualquiera' in line 16?
|
||||||
|
|
||||||
|
It doesn't help more than what we have:
|
||||||
|
|
||||||
|
```
|
||||||
|
Unknown section `cualquiera` in config.
|
||||||
|
```
|
||||||
|
|
||||||
|
### Parser errors are as fuzzy as the ones in PyYAML
|
||||||
|
|
||||||
|
Some errors from ruamel YAML go directly to our application:
|
||||||
|
|
||||||
|
```
|
||||||
|
preflight:
|
||||||
|
check_zone_fills: true
|
||||||
|
ignore_unconnected: false
|
||||||
|
```
|
||||||
|
|
||||||
|
Gives:
|
||||||
|
|
||||||
|
```
|
||||||
|
ruamel.yaml.scanner.ScannerError: mapping values are not allowed here
|
||||||
|
in "test1.yaml", line 3, column 23:
|
||||||
|
ignore_unconnected: false
|
||||||
|
^ (line: 3)
|
||||||
|
```
|
||||||
|
|
||||||
|
This doesn't help more than:
|
||||||
|
|
||||||
|
```
|
||||||
|
mapping values are not allowed here
|
||||||
|
in "scanner_error.yaml", line 3, column 23
|
||||||
|
```
|
||||||
|
|
||||||
|
## Conclusion
|
||||||
|
|
||||||
|
Using it involves a lot of adaptations in the code and we:
|
||||||
|
|
||||||
|
- Don't gain better messages. They point to lines and columns, but not the right ones.
|
||||||
|
- Need to patch the library in order to make indentation checks optional.
|
||||||
|
|
||||||
|
|
||||||
|
|
@ -0,0 +1,6 @@
|
||||||
|
kiplot:
|
||||||
|
version: 1
|
||||||
|
|
||||||
|
preflight:
|
||||||
|
run_drc: true
|
||||||
|
|
||||||
|
|
@ -0,0 +1,4 @@
|
||||||
|
preflight:
|
||||||
|
check_zone_fills: true
|
||||||
|
ignore_unconnected: false
|
||||||
|
|
||||||
|
|
@ -0,0 +1,49 @@
|
||||||
|
#!/usr/bin/python3
|
||||||
|
import os
|
||||||
|
import sys
|
||||||
|
cur_dir = os.path.dirname(os.path.abspath(__file__))
|
||||||
|
# subdir = 'strictyaml-1.0.6'
|
||||||
|
subdir = 'strictyaml'
|
||||||
|
sys.path.append(os.path.join(cur_dir, subdir))
|
||||||
|
# Depende de ruamel: python3-ruamel.yaml
|
||||||
|
# Depende de dateutil: python3-dateutil
|
||||||
|
from strictyaml import (load, Map, Str, Int, Seq, Any, Bool, Optional, MapPattern, YAMLError)
|
||||||
|
from strictyaml.exceptions import InconsistentIndentationDisallowed
|
||||||
|
|
||||||
|
schema_ver = MapPattern(Str(), Any())
|
||||||
|
# fname = 'scanner_error.yaml'
|
||||||
|
# fname = 'indent.yaml'
|
||||||
|
fname = 'test.yaml'
|
||||||
|
with open(fname) as f:
|
||||||
|
s = f.read()
|
||||||
|
try:
|
||||||
|
parsed = load(s, schema_ver, label=fname)
|
||||||
|
except InconsistentIndentationDisallowed as e:
|
||||||
|
print('Use the same indentation across the file')
|
||||||
|
print(e)
|
||||||
|
sys.exit(1)
|
||||||
|
except YAMLError as e:
|
||||||
|
print('YAML parsing error:')
|
||||||
|
print(e)
|
||||||
|
sys.exit(1)
|
||||||
|
|
||||||
|
schema = Map({"kiplot":
|
||||||
|
Map({"version": Int()}),
|
||||||
|
Optional("preflight"): Map({
|
||||||
|
Optional("run_drc"): Bool(),
|
||||||
|
Optional("run_erc"): Bool(),
|
||||||
|
Optional("update_xml"): Bool(),
|
||||||
|
Optional("check_zone_fills"): Bool(),
|
||||||
|
Optional("ignore_unconnected"): Bool(),
|
||||||
|
}),
|
||||||
|
Optional("outputs"): Seq(Any())}) # noqa: E127
|
||||||
|
|
||||||
|
try:
|
||||||
|
parsed = load(s, schema, label=fname)
|
||||||
|
except YAMLError as e:
|
||||||
|
print('YAML parsing error:')
|
||||||
|
print(e)
|
||||||
|
sys.exit(1)
|
||||||
|
|
||||||
|
print(repr(parsed))
|
||||||
|
print(parsed['kiplot']['version'])
|
||||||
|
|
@ -0,0 +1,17 @@
|
||||||
|
# Example KiPlot config file
|
||||||
|
kiplot:
|
||||||
|
version: 1
|
||||||
|
|
||||||
|
preflight:
|
||||||
|
run_drc: true
|
||||||
|
run_erc: true
|
||||||
|
update_xml: true
|
||||||
|
check_zone_fills: true
|
||||||
|
ignore_unconnected: false
|
||||||
|
|
||||||
|
outputs:
|
||||||
|
- name: 'gerbers'
|
||||||
|
|
||||||
|
cualquiera:
|
||||||
|
- name: 'gerbers'
|
||||||
|
|
||||||
|
|
@ -0,0 +1,14 @@
|
||||||
|
# Example KiPlot config file
|
||||||
|
kiplot:
|
||||||
|
version: 1
|
||||||
|
|
||||||
|
preflight:
|
||||||
|
run_drc: true
|
||||||
|
run_erc: true
|
||||||
|
update_xml: true
|
||||||
|
check_zone_fills: true
|
||||||
|
ignore_unconnected: false
|
||||||
|
|
||||||
|
cualquiera:
|
||||||
|
- name: 'gerbers'
|
||||||
|
|
||||||
Loading…
Reference in New Issue