templates
/
KiBot
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

New KiCost variant style. 

- New internal filters `_var_rename_kicost` and `_kicost_dnp`.
- New `skip_if_no_field` and `invert` options to the regex used in the
  generic filter.
This commit is contained in:
Salvador E. Tropea 2021-03-30 09:37:36 -03:00
parent afe80052b4
commit 1290bb6995
15 changed files with 298 additions and 16 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

3
CHANGELOG.md
View File

@ -13,6 +13,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- `suparts`: Adds support for KiCost's subparts feature.
- `field_rename`: Used to rename schematic fields.
- `var_rename_kicost`: Like `var_rename` but using KiCost mechanism.
- New KiCost variant style.
- `skip_if_no_field` and `invert` options to the regex used in the generic
filter.
### Changed
- Errors and warnings from KiAuto now are printed as errors and warnings.

7
README.md
View File

@ -290,6 +290,8 @@ Currently the only type available is `generic`.
This filter is based on regular expressions.
It also provides some shortcuts for common situations.
Note that matches aren't case sensitive and spaces at the beggining and the end are removed.
The internal `_mechanical` filter emulates the KiBoM behavior for default exclusions.
The internal `_kicost_dnp` filter emulates KiCost's `dnp` field.
* Valid keys:
- `comment`: [string=''] A comment for documentation purposes.
- `config_field`: [string='Config'] Name of the field used to clasify components.
@ -301,8 +303,10 @@ Currently the only type available is `generic`.
* Valid keys:
- `column`: [string=''] Name of the column to apply the regular expression.
- *field*: Alias for column.
- `invert`: [boolean=false] Invert the regex match result.
- `regex`: [string=''] Regular expression to match.
- *regexp*: Alias for regex.
- `skip_if_no_field`: [boolean=false] Skip this test if the field doesn't exist.
- `exclude_config`: [boolean=false] Exclude components containing a key value in the config field.
Separators are applied.
- `exclude_empty_val`: [boolean=false] Exclude components with empty 'Value'.
@ -320,8 +324,10 @@ Currently the only type available is `generic`.
* Valid keys:
- `column`: [string=''] Name of the column to apply the regular expression.
- *field*: Alias for column.
- `invert`: [boolean=false] Invert the regex match result.
- `regex`: [string=''] Regular expression to match.
- *regexp*: Alias for regex.
- `skip_if_no_field`: [boolean=false] Skip this test if the field doesn't exist.
- `invert`: [boolean=false] Invert the result of the filter.
- `keys`: [string|list(string)=dnf_list] [dnc_list,dnf_list] List of keys to match.
The `dnf_list` and `dnc_list` internal lists can be specified as strings.
@ -369,6 +375,7 @@ Currently the only type available is `generic`.
- var_rename_kicost: Var_Rename_KiCost
This filter implements the kicost.VARIANT:FIELD=VALUE renamer to get FIELD=VALUE when VARIANT is in use.
It applies the KiCost concept of variants (a regex to match the VARIANT).
The internal `_var_rename_kicost` filter emulates the KiCost behavior.
* Valid keys:
- `comment`: [string=''] A comment for documentation purposes.
- `name`: [string=''] Used to identify this particular filter definition.

29
kibot/fil_base.py
View File

@ -4,7 +4,8 @@
# License: GPL-3.0
# Project: KiBot (formerly KiPlot)
from .registrable import RegFilter, Registrable, RegOutput
from .misc import IFILT_MECHANICAL, IFILT_VAR_RENAME, IFILT_ROT_FOOTPRINT, IFILT_KICOST_RENAME, DISTRIBUTORS
from .misc import (IFILT_MECHANICAL, IFILT_VAR_RENAME, IFILT_ROT_FOOTPRINT, IFILT_KICOST_RENAME, DISTRIBUTORS,
IFILT_VAR_RENAME_KICOST, IFILT_KICOST_DNP)
from .error import KiPlotConfigurationError
from .bom.columnlist import ColumnList
from .macros import macros, document # noqa: F401
@ -222,6 +223,14 @@ class BaseFilter(RegFilter):
logger.debug('Creating internal filter: '+str(o_tree))
return o_tree
@staticmethod
def _create_var_rename_kicost(name):
o_tree = {'name': name}
o_tree['type'] = 'var_rename_kicost'
o_tree['comment'] = 'Internal default variant field renamer filter (KiCost style)'
logger.debug('Creating internal filter: '+str(o_tree))
return o_tree
@staticmethod
def _create_rot_footprint(name):
o_tree = {'name': name}
@ -268,6 +277,15 @@ class BaseFilter(RegFilter):
logger.debug('Creating internal filter: '+str(o_tree))
return o_tree
@staticmethod
def _create_kicost_dnp(name):
o_tree = {'name': name}
o_tree['type'] = 'generic'
o_tree['comment'] = 'Internal filter for KiCost `dnp` field'
# dnp = 0 is included, other dnp values are excluded
o_tree['exclude_any'] = [{'column': 'dnp', 'regex': r'^\s*0(\.0*)?\s*$', 'invert': True, 'skip_if_no_field': True}]
return o_tree
@staticmethod
def _create_internal_filter(name):
if name == IFILT_MECHANICAL:
@ -280,6 +298,10 @@ class BaseFilter(RegFilter):
tree = BaseFilter._create_rot_footprint(name)
elif name == IFILT_KICOST_RENAME:
tree = BaseFilter._create_kicost_rename(name)
elif name == IFILT_VAR_RENAME_KICOST:
tree = BaseFilter._create_var_rename_kicost(name)
elif name == IFILT_KICOST_DNP:
tree = BaseFilter._create_kicost_dnp(name)
else:
return None
filter = RegFilter.get_class_for(tree['type'])()
@ -299,7 +321,10 @@ class BaseFilter(RegFilter):
# Nothing specified, use the default
if default is None:
return None
names = [default]
if isinstance(default, list):
names = default
else:
names = [default]
elif isinstance(names, str):
# User provided, but only one, make a list
if names == '_none':

20
kibot/fil_generic.py
View File

@ -30,7 +30,9 @@ class Generic(BaseFilter): # noqa: F821
""" Generic filter
This filter is based on regular expressions.
It also provides some shortcuts for common situations.
Note that matches aren't case sensitive and spaces at the beggining and the end are removed """
Note that matches aren't case sensitive and spaces at the beggining and the end are removed.
The internal `_mechanical` filter emulates the KiBoM behavior for default exclusions.
The internal `_kicost_dnp` filter emulates KiCost's `dnp` field """
def __init__(self):
super().__init__()
with document:
@ -119,8 +121,14 @@ class Generic(BaseFilter): # noqa: F821
if not self.include_only: # Nothing to match against, means include all
return True
for reg in self.include_only:
if reg.skip_if_no_field and not c.is_field(reg.column):
# Skip the check if the field doesn't exist
continue
field_value = c.get_field_value(reg.column)
if reg.regex.search(field_value):
res = reg.regex.search(field_value)
if reg.invert:
res = not res
if res:
if GS.debug_level > 1:
logger.debug("Including '{ref}': Field '{field}' ({value}) matched '{re}'".format(
ref=c.ref, field=reg.column, value=field_value, re=reg.regex))
@ -134,8 +142,14 @@ class Generic(BaseFilter): # noqa: F821
if not self.exclude_any: # Nothing to match against, means don't exclude any
return False
for reg in self.exclude_any:
if reg.skip_if_no_field and not c.is_field(reg.column):
# Skip the check if the field doesn't exist
continue
field_value = c.get_field_value(reg.column)
if reg.regex.search(field_value):
res = reg.regex.search(field_value)
if reg.invert:
res = not res
if res:
if GS.debug_level > 1:
logger.debug("Excluding '{ref}': Field '{field}' ({value}) matched '{re}'".format(
ref=c.ref, field=reg.column, value=field_value, re=reg.regex))

9
kibot/fil_var_rename_kicost.py
View File

@ -20,7 +20,8 @@ logger = log.get_logger(__name__)
class Var_Rename_KiCost(BaseFilter): # noqa: F821
""" Var_Rename_KiCost
This filter implements the kicost.VARIANT:FIELD=VALUE renamer to get FIELD=VALUE when VARIANT is in use.
It applies the KiCost concept of variants (a regex to match the VARIANT) """
It applies the KiCost concept of variants (a regex to match the VARIANT).
The internal `_var_rename_kicost` filter emulates the KiCost behavior """
def __init__(self):
super().__init__()
self._is_transform = True
@ -53,7 +54,7 @@ class Var_Rename_KiCost(BaseFilter): # noqa: F821
variant = GS.variant[0]
else:
variant = '('+'|'.join(GS.variant)+')'
var = re.compile(variant, re.IGNORECASE)
var_re = re.compile(variant, re.IGNORECASE)
for name, value in comp.get_user_fields():
name = name.strip().lower()
# Remove the prefix
@ -69,13 +70,13 @@ class Var_Rename_KiCost(BaseFilter): # noqa: F821
# Successfully separated
f_variant = res[0].lower()
f_field = res[1].lower()
if var.match(f_variant):
if var_re.match(f_variant):
# Variant matched
if GS.debug_level > 2:
logger.debug('ref: {} {}: {} -> {}'.
format(comp.ref, f_field, comp.get_field_value(f_field), value))
comp.set_field(f_field, value)
elif self.variant_to_value and var.match(name):
elif self.variant_to_value and var_re.match(name):
# The field matches the variant and the user wants to change the value
if GS.debug_level > 2:
logger.debug('ref: {} value: {} -> {}'.format(comp.ref, comp.value, value))

3
kibot/kicad/v5_sch.py
View File

@ -835,6 +835,9 @@ class SchematicComponent(object):
return self.dfields[field].value
return ''
def is_field(self, field):
return field in self.dfields
def get_free_field_number(self):
""" Looks for a field number that isn't currently in use """
max_num = -1

2
kibot/misc.py
View File

@ -79,8 +79,10 @@ KICAD_VERSION_5_99 = 5099000
# Internal filter names
IFILT_MECHANICAL = '_mechanical'
IFILT_VAR_RENAME = '_var_rename'
IFILT_VAR_RENAME_KICOST = '_var_rename_kicost'
IFILT_ROT_FOOTPRINT = '_rot_footprint'
IFILT_KICOST_RENAME = '_kicost_rename'
IFILT_KICOST_DNP = '_kicost_dnp'
# KiCad 5 GUI values for the attribute
UI_THT = 0 # 1 for KiCad 6
UI_SMD = 1 # 2 for KiCad 6

4
kibot/out_base.py
View File

@ -102,6 +102,10 @@ class BoMRegex(Optionable):
""" {column} """
self.regexp = None
""" {regex} """
self.skip_if_no_field = False
""" Skip this test if the field doesn't exist """
self.invert = False
""" Invert the regex match result """
class VariantOptions(BaseOptions):

13
kibot/var_base.py
View File

@ -5,7 +5,7 @@
# Project: KiBot (formerly KiPlot)
from .registrable import RegVariant
from .optionable import Optionable
from .fil_base import apply_exclude_filter, apply_fitted_filter, apply_fixed_filter, BaseFilter, apply_pre_transform
from .fil_base import apply_exclude_filter, apply_fitted_filter, apply_fixed_filter, apply_pre_transform
from .macros import macros, document # noqa: F401
@ -25,21 +25,20 @@ class BaseVariant(RegVariant):
# * Filters
self.pre_transform = Optionable
""" [string|list(string)=''] Name of the filter to transform fields before applying other filters.
Use '_var_rename' to transform VARIANT:FIELD fields """
Use '_var_rename' to transform VARIANT:FIELD fields.
Use '_var_rename_kicost' to transform kicost.VARIANT:FIELD fields.
Use '_kicost_rename' to apply KiCost field rename rules """
self.exclude_filter = Optionable
""" [string|list(string)=''] Name of the filter to exclude components from BoM processing.
Use '_mechanical' for the default KiBoM behavior """
self.dnf_filter = Optionable
""" [string|list(string)=''] Name of the filter to mark components as 'Do Not Fit'.
Use '_kibom_dnf' for the default KiBoM behavior """
Use '_kibom_dnf' for the default KiBoM behavior.
Use '_kicost_dnp'' for the default KiCost behavior """
self.dnc_filter = Optionable
""" [string|list(string)=''] Name of the filter to mark components as 'Do Not Change'.
Use '_kibom_dnc' for the default KiBoM behavior """
def config(self, parent):
super().config(parent)
self.pre_transform = BaseFilter.solve_filter(self.pre_transform, 'pre_transform', is_transform=True)
def filter(self, comps):
# Apply all the filters
comps = apply_pre_transform(comps, self.pre_transform)

1
kibot/var_ibom.py
View File

@ -48,6 +48,7 @@ class IBoM(BaseVariant): # noqa: F821
def config(self, parent):
super().config(parent)
self.pre_transform = BaseFilter.solve_filter(self.pre_transform, 'pre_transform', is_transform=True)
self.exclude_filter = BaseFilter.solve_filter(self.exclude_filter, 'exclude_filter', IFILT_MECHANICAL)
self.dnf_filter = BaseFilter.solve_filter(self.dnf_filter, 'dnf_filter')
self.dnc_filter = BaseFilter.solve_filter(self.dnc_filter, 'dnc_filter')

1
kibot/var_kibom.py
View File

@ -51,6 +51,7 @@ class KiBoM(BaseVariant): # noqa: F821
else:
self.variant = []
self.variant = [v.lower() for v in self.variant]
self.pre_transform = BaseFilter.solve_filter(self.pre_transform, 'pre_transform', is_transform=True)
# Filters priority:
# 1) Defined here
# 2) Delegated from the output format

77
kibot/var_kicost.py Normal file
View File

@ -0,0 +1,77 @@
# -*- coding: utf-8 -*-
# Copyright (c) 2020-2021 Salvador E. Tropea
# Copyright (c) 2020-2021 Instituto Nacional de Tecnología Industrial
# License: GPL-3.0
# Project: KiBot (formerly KiPlot)
"""
Implements the KiCost variants mechanism.
"""
import re
from .gs import GS
from .misc import IFILT_VAR_RENAME_KICOST, IFILT_KICOST_RENAME, IFILT_KICOST_DNP
from .fil_base import BaseFilter
from .macros import macros, document, variant_class # noqa: F401
from . import log
logger = log.get_logger(__name__)
@variant_class
class KiCost(BaseVariant): # noqa: F821
""" KiCost variant style
The `variant` field (configurable) contains one or more values.
If any of these values matches the variant regex the component is included.
By default a pre-transform filter is applied to support kicost.VARIANT:FIELD and
field name aliases used by KiCost.
Also a default `dnf_filter` implements the KiCost DNP mechanism """
def __init__(self):
super().__init__()
with document:
self.variant = ''
""" Variants to match (regex) """
self.variant_field = 'variant'
""" Name of the field that stores board variant/s for component """
self.separators = ',;/ '
""" Valid separators for variants in the variant field.
Each character is a valid separator """
def config(self, parent):
super().config(parent)
self.pre_transform = BaseFilter.solve_filter(self.pre_transform, 'pre_transform',
[IFILT_VAR_RENAME_KICOST, IFILT_KICOST_RENAME], is_transform=True)
self.exclude_filter = BaseFilter.solve_filter(self.exclude_filter, 'exclude_filter')
self.dnf_filter = BaseFilter.solve_filter(self.dnf_filter, 'dnf_filter', IFILT_KICOST_DNP)
self.dnc_filter = BaseFilter.solve_filter(self.dnc_filter, 'dnc_filter')
if not self.separators:
self.separators = ' '
else:
self.separators = '['+self.separators+']'
def filter(self, comps):
GS.variant = [self.variant]
comps = super().filter(comps)
logger.debug("Applying KiCost style variant `{}`".format(self.name))
if not self.variant_field or not self.variant:
# No variant field or not variant regex
# Just skip the process
return comps
# Apply to all the components
var_re = re.compile(self.variant, flags=re.IGNORECASE)
for c in comps:
logger.debug("{} {} {}".format(c.ref, c.fitted, c.included))
if not (c.fitted and c.included):
# Don't check if we already discarded it
continue
variants = c.get_field_value(self.variant_field)
if variants:
# The component belong to one or more variant
for v in re.split(self.separators, variants):
if var_re.match(v):
# Matched, remains
break
else:
# None of the variants matched
c.fitted = False
if GS.debug_level > 2:
logger.debug('ref: {} value: {} -> False'.format(c.ref, c.value))
return comps

70
tests/board_samples/kicad_5/kibom-variant_kicost.sch Normal file
View File

@ -0,0 +1,70 @@
EESchema Schematic File Version 4
EELAYER 30 0
EELAYER END
$Descr A4 11693 8268
encoding utf-8
Sheet 1 1
Title "KiBom Test Schematic"
Date "2020-03-12"
Rev "A"
Comp "https://github.com/SchrodingersGat/KiBom"
Comment1 ""
Comment2 ""
Comment3 ""
Comment4 ""
$EndDescr
Text Notes 510 730 0 79 ~ 0
This schematic serves as a test-file for the KiBot export script.\nThis is the KiCost variants style test.
Text Notes 5950 2600 0 118 ~ 0
The test tests the following \nvariants matrix:\n production test default\nC1 X\nC2 X X\nR1 X X X\nR2 X X\n
$Comp
L Device:C C1
U 1 1 5F43BEC2
P 1000 1700
F 0 "C1" H 1115 1746 50 0000 L CNN
F 1 "1nF" H 1115 1655 50 0000 L CNN
F 2 "" H 1038 1550 50 0001 C CNN
F 3 "~" H 1000 1700 50 0001 C CNN
F 4 "1" H 1000 1700 50 0001 C CNN "kicost.default:dnp"
F 5 "0.0" H 1000 1700 50 0001 C CNN "kicost.test:dnp"
F 6 "" H 1000 1700 50 0001 C CNN "kicost.production:nopop"
1 1000 1700
1 0 0 -1
$EndComp
$Comp
L Device:C C2
U 1 1 5F43CE1C
P 1450 1700
F 0 "C2" H 1565 1746 50 0000 L CNN
F 1 "1000 pF" H 1565 1655 50 0000 L CNN
F 2 "" H 1488 1550 50 0001 C CNN
F 3 "~" H 1450 1700 50 0001 C CNN
F 4 "production,test" H 1450 1700 50 0001 C CNN "version"
1 1450 1700
1 0 0 -1
$EndComp
$Comp
L Device:R R1
U 1 1 5F43D144
P 2100 1700
F 0 "R1" H 2170 1746 50 0000 L CNN
F 1 "1k" H 2170 1655 50 0000 L CNN
F 2 "" V 2030 1700 50 0001 C CNN
F 3 "~" H 2100 1700 50 0001 C CNN
F 4 "3k3" H 2100 1700 50 0001 C CNN "kicost.test:Value"
1 2100 1700
1 0 0 -1
$EndComp
$Comp
L Device:R R2
U 1 1 5F43D4BB
P 2500 1700
F 0 "R2" H 2570 1746 50 0000 L CNN
F 1 "1000" H 2570 1655 50 0000 L CNN
F 2 "" V 2430 1700 50 0001 C CNN
F 3 "~" H 2500 1700 50 0001 C CNN
F 4 "production default" H 2500 1700 50 0001 C CNN "Variant"
1 2500 1700
1 0 0 -1
$EndComp
$EndSCHEMATC

32
tests/test_plot/test_int_bom.py
View File

@ -34,6 +34,15 @@ Missing:
- number_boards
- XLSX/HTML colors (for real)
KiBoM Variants:
- kibom-variant_2.sch
- kibom-variant_5.sch
IBoM Variants:
- test_int_bom_variant_t2if + kibom-variant_3.sch + int_bom_var_t2i_csv
- test_int_bom_variant_t2is + kibom-variant_3.sch + int_bom_var_t2is_csv
- kibom-variant_4.sch
For debug information use:
pytest-3 --log-cli-level debug
@ -1239,6 +1248,7 @@ def test_int_bom_variant_t2b(test_dir):
def test_int_bom_variant_t2c(test_dir):
""" Test KiBoM variant and field rename filter, R1 must be changed to 3k3 """
prj = 'kibom-variant_2'
ctx = context.TestContextSCH(test_dir, 'test_int_bom_variant_t2c', prj, 'int_bom_var_t2c_csv', BOM_DIR)
ctx.run()
@ -1287,6 +1297,7 @@ def test_int_bom_variant_t2s(test_dir):
def test_int_bom_variant_t2if(test_dir):
""" IBoM variants test full """
prj = 'kibom-variant_3'
ctx = context.TestContextSCH(test_dir, 'test_int_bom_variant_t2if', prj, 'int_bom_var_t2i_csv', BOM_DIR)
ctx.run()
@ -1303,6 +1314,7 @@ def test_int_bom_variant_t2if(test_dir):
def test_int_bom_variant_t2is(test_dir):
""" IBoM variants test simple """
prj = 'kibom-variant_3'
ctx = context.TestContextSCH(test_dir, 'test_int_bom_variant_t2is', prj, 'int_bom_var_t2is_csv', BOM_DIR)
ctx.run(extra_debug=True)
@ -1312,6 +1324,26 @@ def test_int_bom_variant_t2is(test_dir):
ctx.clean_up(keep_project=True)
def test_int_bom_variant_t2kf(test_dir):
""" KiCost variants test full.
R1 must be changed to 3k3.
We also test the DNP mechanism. """
prj = 'kibom-variant_kicost'
ctx = context.TestContextSCH(test_dir, 'test_int_bom_variant_t2kf', prj, 'int_bom_var_t2k_csv', BOM_DIR)
ctx.run()
rows, header, info = ctx.load_csv(prj+'-bom.csv')
ref_column = header.index(REF_COLUMN_NAME)
check_kibom_test_netlist(rows, ref_column, 1, ['C1', 'C2'], ['R1', 'R2'])
rows, header, info = ctx.load_csv(prj+'-bom_(production).csv')
check_kibom_test_netlist(rows, ref_column, 2, ['C1'], ['R1', 'R2', 'C2'])
val_column = header.index(VALUE_COLUMN_NAME)
check_value(rows, ref_column, 'R1', val_column, '1k')
rows, header, info = ctx.load_csv(prj+'-bom_(test).csv')
check_kibom_test_netlist(rows, ref_column, 2, ['R2'], ['R1', 'C1', 'C2'])
check_value(rows, ref_column, 'R1', val_column, '3k3')
ctx.clean_up()
def test_int_bom_wrong_variant(test_dir):
ctx = context.TestContextSCH(test_dir, 'test_int_bom_wrong_variant', 'links', 'int_bom_wrong_variant', '')
ctx.run(EXIT_BAD_CONFIG)

43
tests/yaml_samples/int_bom_var_t2k_csv.kibot.yaml Normal file
View File

@ -0,0 +1,43 @@
# KiCost variants test
kibot:
version: 1
variants:
- name: 'production'
comment: 'Production variant'
type: kicost
file_id: '_(production)'
variant: production
- name: 'test'
comment: 'Test variant'
type: kicost
file_id: '_(test)'
variant: 't.*'
- name: 'default'
comment: 'Default variant'
type: kicost
variant: default
outputs:
- name: 'bom_internal'
comment: "Bill of Materials in CSV format"
type: bom
dir: BoM
options:
variant: default
- name: 'bom_internal_production'
comment: "Bill of Materials in CSV format for production"
type: bom
dir: BoM
options:
variant: production
- name: 'bom_internal_test'
comment: "Bill of Materials in CSV format for test"
type: bom
dir: BoM
options:
variant: test