templates
/
KiBot
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

[Added] Support for `groups` of `outputs`

This commit is contained in:
Salvador E. Tropea 2023-01-04 08:38:21 -03:00
parent 7be6a5a7f4
commit 17aacf8daf
9 changed files with 704 additions and 117 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

3
CHANGELOG.md
View File

@ -6,6 +6,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [1.5.2] - Unreleased
### Added
- Support for `groups` of `outputs`
- New output:
- `vrml` export the 3D model in Virtual Reality Modeling Language (#349)
- Plot related outputs and PCB_Print:
@ -22,7 +23,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
Now all 2D variant stuff is applied before calling iBoM (#350)
- Copy_Files:
- Problems on KiCad 5 (no 3rd party dir) (#357)
ç
## [1.5.1] - 2022-12-16
### Fixed
- System level resources look-up

386
README.md
View File

File diff suppressed because it is too large Load Diff

99
docs/README.in
View File

@ -64,6 +64,7 @@
* [Consolidating BoMs](#consolidating-boms)
* [Importing outputs from another file](#importing-outputs-from-another-file)
* [Using other output as base for a new one](#using-other-output-as-base-for-a-new-one)
* [Grouping outputs](#grouping-outputs)
* [Importing filters and variants from another file](#importing-filters-and-variants-from-another-file)
* [Doing YAML substitution or preprocessing](#doing-yaml-substitution-or-preprocessing)
* [Usage](#usage)
@ -265,12 +266,14 @@ The file is divided in various sections. Some of them are optional.
The order in which they are declared is not relevant, they are interpreted in the following order:
- `kiplot`/`kibot` see [The header](#the-header)
- `import` see [Importing outputs from another file](#importing-outputs-from-another-file)
- `import` see [Importing outputs from another file](#importing-outputs-from-another-file) and
[Importing filters and variants from another file](#importing-filters-and-variants-from-another-file)
- `global` see [Default global options](#default-global-options)
- `filters` see [Filters and variants](#filters-and-variants)
- `variants` see [Filters and variants](#filters-and-variants)
- `preflight` see [The *preflight* section](#the-preflight-section)
- `outputs` see [The *outputs* section](#the-outputs-section)
- `groups` see [Grouping outputs](#grouping-outputs)
### The header
@ -1180,12 +1183,97 @@ If you need to define an output that is similar to another, and you want to avoi
To achieve it just specify the name of the base output in the `extends` value.
Note that this will use the `options` of the other output as base, not other data as the comment.
Also note that you can use YAML anchors, but this won't work if you are importing the base output from other file.
Also note that you can use [YAML anchors](https://www.educative.io/blog/advanced-yaml-syntax-cheatsheet#anchors), but this won't work if you are
importing the base output from other file.
Additionally you must be aware that extending an output doesn't disable the base output.
If you need to disable the original output use `disable_run_by_default` option.
#### Grouping outputs
Sometimes you want to generate various outputs together. An example could be the fabrication files, or the documentation for the project.
To explain it we will use an example where you have six outputs.
Three are used for fabrication: `gerbers`, `excellon_drill` and `position`.
Another three are used for documentation: `SVG`, `PcbDraw` and `PcbDraw2`.
The YAML config containing this example can be found [here](tests/yaml_samples/groups_1.kibot.yaml).
If you need to generate the fabrication outputs you must run:
```
kibot gerbers excellon_drill position
```
One mechanism to group the outputs is to create a `compress` output that just includes the outputs you want to group.
Here is one example:
```yaml
- name: compress_fab
comment: "Generates a ZIP file with all the fab outputs"
type: compress
run_by_default: false
options:
files:
- from_output: gerbers
- from_output: excellon_drill
- from_output: position
```
The `compress_fab` output will generate the `gerbers`, `excellon_drill` and `position` outputs.
Then it will create a ZIP file containing the files generated by these outputs.
The command line invocation for this is:
```
kibot compress_fab
```
Using this mechanism you are forced to create a compressed output.
To avoid it you can use `groups`.
The `groups` section is used to create groups of outputs.
Here is the example for fabrication files:
```yaml
groups:
- name: fab
outputs:
- gerbers
- excellon_drill
- position
```
So now you can just run:
```
kibot fab
```
The `gerbers`, `excellon_drill` and `position` outputs will be generated without the need to generate an extra file.
Groups can be nested, here is an example:
```yaml
groups:
- name: fab
outputs:
- gerbers
- excellon_drill
- position
- name: plot
outputs:
- SVG
- PcbDraw
- PcbDraw2
- name: fab_svg
outputs:
- fab
- SVG
```
Here the `fab_svg` group will contain `gerbers`, `excellon_drill`, `position` and `SVG`.
Groups can be imported from another YAML file.
Avoid naming groups using `_` as first character. These names are reserved for KiBot.
#### Importing filters and variants from another file
This is a more complex case of the previous [Importing outputs from another file](#importing-outputs-from-another-file).
@ -1199,9 +1287,11 @@ import:
filters: LIST_OF_FILTERS
variants: LIST_OF_VARIANTS
global: LIST_OF_GLOBALS
groups: LIST_OF_GROUPS
```
This syntax is flexible. If you don't define which `outputs`, `filters`, `variants` and/or `global` all will be imported. So you can just omit them, like this:
This syntax is flexible. If you don't define which `outputs`, `preflights`, `filters`, `variants`, `global` and/or `groups` all will be imported.
So you can just omit them, like this:
```yaml
import:
@ -1218,6 +1308,7 @@ import:
```
This will import the `one_name` output and the `name1` and `name2` filters. As `variants` is omitted, all variants will be imported.
The same applies to other things like globals and groups.
You can also use the `all` and `none` special names, like this:
```yaml
@ -1231,7 +1322,7 @@ import:
This will import all outputs and filters, but not variants or globals.
Also note that imported globals has more precedence than the ones defined in the same file.
If you want give more priority to the local values use:
If you want to give more priority to the local values use:
```
kibot:

75
kibot/config_reader.py
View File

@ -1,6 +1,6 @@
# -*- coding: utf-8 -*-
# Copyright (c) 2020-2022 Salvador E. Tropea
# Copyright (c) 2020-2022 Instituto Nacional de Tecnología Industrial
# Copyright (c) 2020-2023 Salvador E. Tropea
# Copyright (c) 2020-2023 Instituto Nacional de Tecnología Industrial
# Copyright (c) 2018 John Beard
# License: GPL-3.0
# Project: KiBot (formerly KiPlot)
@ -18,7 +18,7 @@ from collections import OrderedDict
from .error import KiPlotConfigurationError
from .misc import (NO_YAML_MODULE, EXIT_BAD_ARGS, EXAMPLE_CFG, WONT_OVERWRITE, W_NOOUTPUTS, W_UNKOUT, W_NOFILTERS,
W_NOVARIANTS, W_NOGLOBALS, TRY_INSTALL_CHECK, W_NOPREFLIGHTS)
W_NOVARIANTS, W_NOGLOBALS, TRY_INSTALL_CHECK, W_NOPREFLIGHTS, W_NOGROUPS)
from .gs import GS
from .registrable import RegOutput, RegVariant, RegFilter, RegDependency
from .pre_base import BasePreFlight
@ -37,7 +37,7 @@ PYPI_LOGO = ('![PyPi dependency]('+GITHUB_RAW+'PyPI_logo_simplified-22x22.png)')
PY_LOGO = ('![Python module]('+GITHUB_RAW+'Python-logo-notext-22x22.png)')
TOOL_LOGO = '![Tool]('+GITHUB_RAW+'llave-inglesa-22x22.png)'
AUTO_DOWN = '![Auto-download]('+GITHUB_RAW+'auto_download-22x22.png)'
VALID_SECTIONS = {'kiplot', 'kibot', 'import', 'global', 'filters', 'variants', 'preflight', 'outputs'}
VALID_SECTIONS = {'kiplot', 'kibot', 'import', 'global', 'filters', 'variants', 'preflight', 'outputs', 'groups'}
VALID_KIBOT_SEC = {'version', 'imported_global_has_less_priority'}
@ -69,6 +69,7 @@ class CollectedImports(object):
self.variants = {}
self.globals = {}
self.preflights = []
self.groups = {}
self.imported_global_has_less_priority = False
@ -160,6 +161,37 @@ class CfgYamlReader(object):
raise KiPlotConfigurationError("`outputs` must be a list")
return outputs
def _parse_group(self, tree, groups):
try:
name = str(tree['name'])
if not name:
raise KeyError
except KeyError:
raise KiPlotConfigurationError("Group needs a name in: "+str(tree))
try:
outs = tree['outputs']
if not outs:
raise KeyError
except KeyError:
raise KiPlotConfigurationError("Group `"+name+"` must contain outputs")
if not isinstance(outs, list):
raise KiPlotConfigurationError("'outputs' in group `"+name+"` must be a list (not {})".format(type(outs)))
for v in outs:
if not isinstance(v, str):
raise KiPlotConfigurationError("In group `"+name+"`: all outputs must be strings (not {})".format(type(v)))
if name in groups:
raise KiPlotConfigurationError("Duplicated group `{}`".format(name))
groups[name] = outs
def _parse_groups(self, v):
groups = {}
if isinstance(v, list):
for o in v:
self._parse_group(o, groups)
else:
raise KiPlotConfigurationError("`groups` must be a list")
return groups
def _parse_variant_or_filter(self, o_tree, kind, reg_class):
kind_f = kind[0].upper()+kind[1:]
try:
@ -340,6 +372,29 @@ class CfgYamlReader(object):
logger.warning(W_NOFILTERS+"No filters found in `{}`".format(fn_rel))
return sel_fils
def _parse_import_groups(self, groups, explicit_grps, fn_rel, data, imported):
sel_grps = {}
if groups is None or len(groups) > 0:
if 'groups' in data:
imported.groups.update(self._parse_groups(data['groups']))
i_grps = imported.groups
if groups is not None:
for f in groups:
if f in i_grps:
sel_grps[f] = i_grps[f]
else:
logger.warning(W_UNKOUT+"can't import `{}` group from `{}` (missing)".format(f, fn_rel))
else:
sel_grps = i_grps
if len(sel_grps) == 0:
if explicit_grps:
logger.warning(W_NOGROUPS+"No groups found in `{}`".format(fn_rel))
else:
logger.debug('groups loaded from `{}`: {}'.format(fn_rel, sel_grps.keys()))
if groups is None and explicit_grps and 'groups' not in data:
logger.warning(W_NOGROUPS+"No groups found in `{}`".format(fn_rel))
return sel_grps
def _parse_import_variants(self, vars, explicit_vars, fn_rel, data, imported):
sel_vars = {}
if vars is None or len(vars) > 0:
@ -416,11 +471,13 @@ class CfgYamlReader(object):
vars = []
globals = []
pre = []
groups = []
explicit_outs = True
explicit_fils = False
explicit_vars = False
explicit_globals = False
explicit_pres = False
explicit_groups = False
elif isinstance(entry, dict):
fn = outs = filters = vars = globals = pre = None
explicit_outs = explicit_fils = explicit_vars = explicit_globals = explicit_pres = False
@ -444,6 +501,9 @@ class CfgYamlReader(object):
elif k in ['global', 'globals']:
globals = self._parse_import_items(k, fn, v)
explicit_globals = True
elif k == 'groups':
vars = self._parse_import_items(k, fn, v)
explicit_groups = True
else:
self._config_error_import(fn, "unknown import entry `{}`".format(str(v)))
if fn is None:
@ -474,6 +534,8 @@ class CfgYamlReader(object):
all_collected.variants.update(self._parse_import_variants(vars, explicit_vars, fn_rel, data, imported))
# Globals
update_dict(all_collected.globals, self._parse_import_globals(globals, explicit_globals, fn_rel, data, imported))
# Groups
all_collected.groups.update(self._parse_import_groups(groups, explicit_groups, fn_rel, data, imported))
if apply:
# This is the main import (not a recursive one) apply the results
RegOutput.add_filters(all_collected.filters)
@ -482,6 +544,7 @@ class CfgYamlReader(object):
self.imported_globals = all_collected.globals
BasePreFlight.add_preflights(all_collected.preflights)
RegOutput.add_outputs(all_collected.outputs, fn_rel)
RegOutput.add_groups(all_collected.groups, fn_rel)
return all_collected
def load_yaml(self, fstream):
@ -562,6 +625,10 @@ class CfgYamlReader(object):
v1 = data.get('outputs', None)
if v1:
RegOutput.add_outputs(self._parse_outputs(v1))
# Look for groups
v1 = data.get('groups', None)
if v1:
RegOutput.add_groups(self._parse_groups(v1))
# Report invalid sections (the first we find)
defined_sections = set(data.keys())
invalid_sections = defined_sections-VALID_SECTIONS

17
kibot/kiplot.py
View File

@ -428,15 +428,18 @@ def _generate_outputs(outputs, targets, invert, skip_pre, cli_order, no_priority
targets = [out for out in RegOutput.get_outputs() if out.run_by_default]
else:
# Check we got a valid list of outputs
for name in targets:
out = RegOutput.get_output(name)
if out is None:
logger.error('Unknown output `{}`'.format(name))
unknown = next(filter(lambda x: not RegOutput.is_output_or_group(x), targets), None)
if unknown:
logger.error('Unknown output/group `{}`'.format(unknown))
exit(EXIT_BAD_ARGS)
# Check for CLI+invert inconsistency
if cli_order and invert:
logger.error("CLI order and invert options can't be used simultaneously")
exit(EXIT_BAD_ARGS)
# Expand groups
logger.debug('Outputs before groups expansion: {}'.format(targets))
targets = RegOutput.solve_groups(targets)
logger.debug('Outputs after groups expansion: {}'.format(targets))
# Now convert the list of names into a list of output objects
if cli_order:
# Add them in the same order found at the command line
@ -459,14 +462,14 @@ def _generate_outputs(outputs, targets, invert, skip_pre, cli_order, no_priority
else:
logger.debug('Skipping `{}` output'.format(out.name))
targets = new_targets
logger.debug('Outputs before preflights: {}'.format(targets))
logger.debug('Outputs before preflights: {}'.format([t.name for t in targets]))
# Run the preflights
preflight_checks(skip_pre, targets)
logger.debug('Outputs after preflights: {}'.format(targets))
logger.debug('Outputs after preflights: {}'.format([t.name for t in targets]))
if not cli_order and not no_priority:
# Sort by priority
targets = sorted(targets, key=lambda o: o.priority, reverse=True)
logger.debug('Outputs after sorting: {}'.format(targets))
logger.debug('Outputs after sorting: {}'.format([t.name for t in targets]))
# Configure and run the outputs
for out in targets:
if config_output(out, dont_stop=dont_stop):

1
kibot/misc.py
View File

@ -245,6 +245,7 @@ W_ONWIN = '(W106) '
W_AUTONONE = '(W106) '
W_AUTOPROB = '(W107) '
W_MORERES = '(W108) '
W_NOGROUPS = '(W109) '
# Somehow arbitrary, the colors are real, but can be different
PCB_MAT_COLORS = {'fr1': "937042", 'fr2': "949d70", 'fr3': "adacb4", 'fr4': "332B16", 'fr5': "6cc290"}
PCB_FINISH_COLORS = {'hal': "8b898c", 'hasl': "8b898c", 'imag': "8b898c", 'enig': "cfb96e", 'enepig': "cfb96e",

11
kibot/out_base.py
View File

@ -58,16 +58,18 @@ class BaseOutput(RegOutput):
super().__init__()
with document:
self.name = ''
""" *Used to identify this particular output definition """
""" *Used to identify this particular output definition.
Avoid using `_` as first character. These names are reserved for KiBot """
self.type = ''
""" *Type of output """
self.dir = './'
""" *Output directory for the generated files.
If it starts with `+` the rest is concatenated to the default dir """
self.comment = ''
""" *A comment for documentation purposes """
""" *A comment for documentation purposes. It helps to identify the output """
self.extends = ''
""" Copy the `options` section from the indicated output """
""" Copy the `options` section from the indicated output.
Used to inherit options from another output of the same type """
self.run_by_default = True
""" When enabled this output will be created when no specific outputs are requested """
self.disable_run_by_default = ''
@ -78,7 +80,8 @@ class BaseOutput(RegOutput):
""" Text to use for the %I expansion content. To differentiate variations of this output """
self.category = Optionable
""" [string|list(string)=''] The category for this output. If not specified an internally defined category is used.
Categories looks like file system paths, i.e. PCB/fabrication/gerber """
Categories looks like file system paths, i.e. **PCB/fabrication/gerber**.
The categories are currently used for `navigate_results` """
self.priority = 50
""" [0,100] Priority for this output. High priority outputs are created first.
Internally we use 10 for low priority, 90 for high priority and 50 for most outputs """

64
kibot/registrable.py
View File

@ -1,6 +1,6 @@
# -*- coding: utf-8 -*-
# Copyright (c) 2020-2022 Salvador E. Tropea
# Copyright (c) 2020-2022 Instituto Nacional de Tecnología Industrial
# Copyright (c) 2020-2023 Salvador E. Tropea
# Copyright (c) 2020-2023 Instituto Nacional de Tecnología Industrial
# License: GPL-3.0
# Project: KiBot (formerly KiPlot)
from collections import OrderedDict
@ -13,6 +13,12 @@ from . import log
logger = log.get_logger()
def fname(file):
if file:
return ", while importing from `{}`".format(file)
return ""
class Registrable(object):
""" This class adds the mechanism to register plug-ins """
def __init__(self):
@ -43,10 +49,12 @@ class RegOutput(Optionable, Registrable):
Used by BaseOutput.
Here because it doesn't need macros. """
_registered = {}
# List of defined filters
# Defined filters
_def_filters = {}
# List of defined variants
# Defined variants
_def_variants = {}
# Defined groups
_def_groups = {}
# List of defined outputs
_def_outputs = OrderedDict()
@ -55,10 +63,12 @@ class RegOutput(Optionable, Registrable):
@staticmethod
def reset():
# List of defined filters
# Defined filters
RegOutput._def_filters = {}
# List of defined variants
# Defined variants
RegOutput._def_variants = {}
# Defined groups
RegOutput._def_groups = {}
# List of defined outputs
RegOutput._def_outputs = OrderedDict()
@ -107,10 +117,9 @@ class RegOutput(Optionable, Registrable):
@staticmethod
def add_output(obj, file=None):
if obj.name in RegOutput._def_outputs:
msg = "Output name `{}` already defined".format(obj.name)
if file:
msg += ", while importing from `{}`".format(file)
raise KiPlotConfigurationError(msg)
raise KiPlotConfigurationError("Output name `{}` already defined".format(obj.name)+fname(file))
if obj.name in RegOutput._def_groups:
raise KiPlotConfigurationError("Output name `{}` already defined as group".format(obj.name)+fname(file))
RegOutput._def_outputs[obj.name] = obj
@staticmethod
@ -118,6 +127,20 @@ class RegOutput(Optionable, Registrable):
for o in objs:
RegOutput.add_output(o, file)
@staticmethod
def add_group(name, lst, file=None):
if name in RegOutput._def_groups:
raise KiPlotConfigurationError("Group name `{}` already defined".format(name)+fname(file))
if name in RegOutput._def_outputs:
raise KiPlotConfigurationError("Group name `{}` already defined as output".format(name)+fname(file))
RegOutput._def_groups[name] = lst
@staticmethod
def add_groups(objs, file=None):
logger.debug('Adding groups: '+str(objs))
for n, lst in objs.items():
RegOutput.add_group(n, lst, file)
@staticmethod
def get_outputs():
return RegOutput._def_outputs.values()
@ -126,6 +149,10 @@ class RegOutput(Optionable, Registrable):
def get_output(name):
return RegOutput._def_outputs.get(name, None)
@staticmethod
def is_output_or_group(name):
return name in RegOutput._def_outputs or name in RegOutput._def_groups
@staticmethod
def check_variant(variant):
if variant:
@ -136,6 +163,23 @@ class RegOutput(Optionable, Registrable):
return RegOutput.get_variant(variant)
return None
@staticmethod
def solve_groups(targets, level=0):
""" Replaces any group by its members.
Returns a new list.
Assumes the outputs and groups are valid. """
new_targets = []
level += 1
if level > 20:
raise KiPlotConfigurationError("More than 20 levels of nested groups, possible loop")
for t in targets:
if t in RegOutput._def_outputs:
new_targets.append(t)
else:
# Recursive expand
new_targets.extend(RegOutput.solve_groups(RegOutput._def_groups[t], level))
return new_targets
class RegVariant(Optionable, Registrable):
""" An optionable that is also registrable.

163
tests/yaml_samples/groups_1.kibot.yaml Normal file
View File

@ -0,0 +1,163 @@
# Groups test case and example
kibot:
version: 1
groups:
- name: fab
outputs:
- gerbers
- excellon_drill
- position
- name: plot
outputs:
- SVG
- PcbDraw
- PcbDraw2
- name: fab_svg
outputs:
- fab
- SVG
outputs:
- name: 'gerbers'
comment: "Gerbers for the Gerber god"
type: gerber
dir: gerberdir
layers: copper
- name: excellon_drill
comment: "Excellon drill files"
type: excellon
dir: Drill
options:
metric_units: true
pth_and_npth_single_file: false
use_aux_axis_as_origin: false
minimal_header: false
mirror_y_axis: false
report: '%f-%i.%x'
map: 'pdf'
- name: 'position'
comment: "Pick and place file"
type: position
dir: positiondir
options:
format: ASCII # CSV or ASCII format
units: millimeters # millimeters or inches
separate_files_for_front_and_back: true
only_smd: true
- name: SVG
comment: "SVG files"
type: svg
dir: SVG
options:
exclude_edge_layer: false
exclude_pads_from_silkscreen: false
use_aux_axis_as_origin: false
plot_sheet_reference: false
plot_footprint_refs: true
plot_footprint_values: true
force_plot_invisible_refs_vals: false
tent_vias: true
# SVG options
line_width: 0.25
drill_marks: full
mirror_plot: true
negative_plot: true
layers:
- layer: F.Cu
suffix: F_Cu
- layer: F.Fab
suffix: F_Fab
- name: PcbDraw
comment: "PcbDraw test top"
type: pcbdraw
dir: PcbDraw
options: &pcb_draw_ops
format: svg
style:
board: "#1b1f44"
copper: "#00406a"
silk: "#d5dce4"
pads: "#cfb96e"
clad: "#72786c"
outline: "#000000"
vcut: "#bf2600"
highlight_on_top: false
highlight_style: "stroke:none;fill:#ff0000;opacity:0.5;"
highlight_padding: 1.5
libs:
- default
- eagle-default
remap:
L_G1: "LEDs:LED-5MM_green"
L_B1: "LEDs:LED-5MM_blue"
L_Y1: "LEDs:LED-5MM_yellow"
'REF**': "dummy:dummy"
G***: "dummy:dummy"
svg2mod: "dummy:dummy"
JP1: "dummy:dummy"
JP2: "dummy:dummy"
JP3: "dummy:dummy"
JP4: "dummy:dummy"
remap_components:
- ref: PHOTO1
lib: yaqwsx
comp: R_PHOTO_7mm
- reference: J8
library: yaqwsx
component: Pin_Header_Straight_1x02_circle
no_drillholes: false
mirror: false
highlight:
- L_G1
- L_B1
- R10
- RV1
show_components: all
vcuts: true
warnings: visible
dpi: 600
# margin:
# left: 5
# right: 1
# top: 0
# bottom: 6
# outline_width: 3
# show_solderpaste: false
resistor_remap:
- ref: R1
val: 10K
- ref: R2
val: 4k7
resistor_flip: "R2"
size_detection: svg_paths
# size_detection: kicad_all
# size_detection: kicad_edge
- name: PcbDraw2
comment: "PcbDraw test bottom"
type: pcbdraw
dir: PcbDraw
options:
<<: *pcb_draw_ops
style: set-red-enig
bottom: true
show_components:
- L_G1
- L_B1
remap:
- name: compress_fab
comment: "Generates a ZIP file with all the fab outputs"
type: compress
run_by_default: false
options:
files:
- from_output: gerbers
- from_output: excellon_drill
- from_output: position