[PcbDraw][Populate] Added filter expansion

- in `show_components` and `highlight`

CHANGELOG.md

@@ -21,6 +21,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
   - A `remap_components` option with better type checks
   - Better support for variants
   - Option to control the *SVG precision* (units scale)
+  - Filter expansion in `show_components` and `highlight`
 - SVG:
   - Option to control the *SVG precision* (units scale)
 - PCB_Print:

README.md

@@ -851,16 +851,18 @@ filters:
         - `regex`: [string=''] Regular expression to match.
         - *regexp*: Alias for regex.
         - `skip_if_no_field`: [boolean=false] Skip this test if the field doesn't exist.
+    - `exclude_bottom`: [boolean=false] Exclude components on the bottom side of the PCB.
     - `exclude_config`: [boolean=false] Exclude components containing a key value in the config field.
                         Separators are applied.
     - `exclude_empty_val`: [boolean=false] Exclude components with empty 'Value'.
     - `exclude_field`: [boolean=false] Exclude components if a field is named as any of the keys.
     - `exclude_refs`: [list(string)] List of references to be excluded.
                       Use R* for all references with R prefix.
-    - `exclude_smd`: [boolean=false] KiCad 5: exclude components marked as smd in the PCB.
+    - `exclude_smd`: [boolean=false] Exclude components marked as smd in the PCB.
-    - `exclude_tht`: [boolean=false] KiCad 5: exclude components marked as through-hole in the PCB.
+    - `exclude_tht`: [boolean=false] Exclude components marked as through-hole in the PCB.
+    - `exclude_top`: [boolean=false] Exclude components on the top side of the PCB.
     - `exclude_value`: [boolean=false] Exclude components if their 'Value' is any of the keys.
-    - `exclude_virtual`: [boolean=false] KiCad 5: exclude components marked as virtual in the PCB.
+    - `exclude_virtual`: [boolean=false] Exclude components marked as virtual in the PCB.
     - `include_only`: [list(dict)] A series of regular expressions used to include parts.
                       If there are any regex defined here, only components that match against ANY of them will be included.
                       Column/field names are case-insensitive.
@@ -2573,7 +2575,10 @@ Notes:
         - **`mirror`**: [boolean=false] Mirror the board.
         - **`output`**: [string='%f-%i%I%v.%x'] Name for the generated file. Affected by global options.
         - **`show_components`**: [list(string)|string=none] [none,all] List of components to draw, can be also a string for none or all.
-                                 The default is none. IMPORTANT! This option is relevant only when no filters or variants are applied.
+                                 The default is none.
+                                 There two ways of using this option, please consult the `add_to_variant` option.
+                                 You can use `_kf(FILTER)` as an element in the list to get all the components that pass the filter.
+                                 You can even use `_kf(FILTER1;FILTER2)` to concatenate filters.
         - **`style`**: [string|dict] PCB style (colors). An internal name, the name of a JSON file or the style options.
           * Valid keys:
             - **`board`**: [string='#208b47'] Color for the board without copper (covered by solder mask).
@@ -2589,13 +2594,14 @@ Notes:
         - `add_to_variant`: [boolean=true] The `show_components` list is added to the list of components indicated by the variant (fitted and not
                             excluded).
                             This is the old behavior, but isn't intuitive because the `show_components` meaning changes when a variant
-                            is used.
+                            is used. In this mode you should avoid using `show_components` and variants.
                             To get a more coherent behavior disable this option, and `none` will always be `none`.
                             Also `all` will be what the variant says.
         - `dnf_filter`: [string|list(string)='_none'] Name of the filter to mark components as not fitted.
                         A short-cut to use for simple cases where a variant is an overkill.
         - `dpi`: [number=300] [10,1200] Dots per inch (resolution) of the generated image.
-        - `highlight`: [list(string)=[]] List of components to highlight.
+        - `highlight`: [list(string)=[]] List of components to highlight. Filter expansion is also allowed here,
+                       see `show_components`.
         - `libs`: [list(string)=[]] List of libraries.
         - `margin`: [number|dict] Margin around the generated image [mm].
           * Valid keys:

docs/populate.md

@@ -74,6 +74,29 @@ For example:
 - `[[front | R1,R2 ]]` will render front side of the board and adds R1 and R2.
 - `[[back | ]]` will render the back side and no components will be added
 
+Note that KiBot also allows to include groups of components selected by a filter.
+If you use `[[front | R1,_kf(all_smd) ]]` and you have the following filter:
+
+```yaml
+  - name: all_smd
+    type: generic
+    exclude_smd: true
+    invert: true
+```
+The list will be expanded to R1 plus all the SMD components of the board.
+But suppose you want to select all the SMD components of the top side of the board,
+you could use `[[front | _kf(all_smd;all_front) ]]` adding the following filter:
+
+```yaml
+  - name: all_front
+    type: generic
+    exclude_bottom: true
+```
+
+Note that we use `;` as separator because `,` is the separator in the list of references.
+You can also use the `!` (not) operator, like this: `[[front | _kf(all_tht;!all_conn) ]]`
+This will select all THT components that aren't connectors, assuming you provide the
+correct filters. Here is an [example to try](../tests/data/with_filter_html.md).
 
 ## Handlebars template
 

docs/samples/generic_plot.kibot.yaml

@@ -1358,7 +1358,7 @@ outputs:
       # [boolean=true] The `show_components` list is added to the list of components indicated by the variant (fitted and not
       # excluded).
       # This is the old behavior, but isn't intuitive because the `show_components` meaning changes when a variant
-      # is used.
+      # is used. In this mode you should avoid using `show_components` and variants.
       # To get a more coherent behavior disable this option, and `none` will always be `none`.
       # Also `all` will be what the variant says
       add_to_variant: true
@@ -1371,7 +1371,8 @@ outputs:
       dpi: 300
       # [string='svg'] [svg,png,jpg,bmp] Output format. Only used if no `output` is specified
       format: 'svg'
-      # [list(string)=[]] List of components to highlight
+      # [list(string)=[]] List of components to highlight. Filter expansion is also allowed here,
+      # see `show_components`
       highlight: []
       # [list(string)=[]] List of libraries
       libs: []
@@ -1425,7 +1426,10 @@ outputs:
           val: ''
           # `value` is an alias for `val`
       # [list(string)|string=none] [none,all] List of components to draw, can be also a string for none or all.
-      # The default is none. IMPORTANT! This option is relevant only when no filters or variants are applied
+      # The default is none.
+      # There two ways of using this option, please consult the `add_to_variant` option.
+      # You can use `_kf(FILTER)` as an element in the list to get all the components that pass the filter.
+      # You can even use `_kf(FILTER1;FILTER2)` to concatenate filters
       show_components: none
       # [boolean=true] Show the solder paste layers
       show_solderpaste: true

kibot/out_pcbdraw.py

@@ -24,6 +24,8 @@ import subprocess
 from tempfile import NamedTemporaryFile
 # Here we import the whole module to make monkeypatch work
 from .error import KiPlotConfigurationError
+from .fil_base import BaseFilter
+from .kiplot import load_sch, get_board_comps_data
 from .misc import (PCBDRAW_ERR, PCB_MAT_COLORS, PCB_FINISH_COLORS, SOLDER_COLORS, SILK_COLORS, W_PCBDRAW)
 from .gs import GS
 from .layer import Layer
@@ -219,15 +221,19 @@ class PcbDrawOptions(VariantOptions):
             self.mirror = False
             """ *Mirror the board """
             self.highlight = Optionable
-            """ [list(string)=[]] List of components to highlight """
+            """ [list(string)=[]] List of components to highlight. Filter expansion is also allowed here,
+                see `show_components` """
             self.show_components = Optionable
             """ *[list(string)|string=none] [none,all] List of components to draw, can be also a string for none or all.
-                The default is none. IMPORTANT! This option is relevant only when no filters or variants are applied """
+                The default is none.
+                There two ways of using this option, please consult the `add_to_variant` option.
+                You can use `_kf(FILTER)` as an element in the list to get all the components that pass the filter.
+                You can even use `_kf(FILTER1;FILTER2)` to concatenate filters """
             self.add_to_variant = True
             """ The `show_components` list is added to the list of components indicated by the variant (fitted and not
                 excluded).
                 This is the old behavior, but isn't intuitive because the `show_components` meaning changes when a variant
-                is used.
+                is used. In this mode you should avoid using `show_components` and variants.
                 To get a more coherent behavior disable this option, and `none` will always be `none`.
                 Also `all` will be what the variant says """
             self.vcuts = False
@@ -269,6 +275,7 @@ class PcbDrawOptions(VariantOptions):
         super().__init__()
 
     def config(self, parent):
+        self._filters_to_expand = False
         # Pre-parse the bottom option
         if 'bottom' in self._tree:
             bot = self._tree['bottom']
@@ -285,6 +292,8 @@ class PcbDrawOptions(VariantOptions):
         # Highlight
         if isinstance(self.highlight, type):
             self.highlight = None
+        else:
+            self.highlight = self.solve_filters(self.highlight)
         # Margin
         if isinstance(self.margin, type):
             self.margin = (0, 0, 0, 0)
@@ -301,9 +310,11 @@ class PcbDrawOptions(VariantOptions):
         elif isinstance(self.show_components, str):
             if self.show_components == 'none':
                 self.show_components = None
-            else:
+            else:  # self.show_components == 'all'
                 # Empty list: means we don't filter
                 self.show_components = []
+        else:  # A list
+            self.show_components = self.solve_filters(self.show_components)
         # Resistors remap/flip
         if isinstance(self.resistor_remap, type):
             self.resistor_remap = []
@@ -333,6 +344,52 @@ class PcbDrawOptions(VariantOptions):
         self._expand_id = 'bottom' if self.bottom else 'top'
         self._expand_ext = self.format
 
+    def solve_filters(self, components):
+        """ Solves references to KiBot filters in the list of components to show.
+            They are not yet expanded, just solved to filter objects """
+        new_list = []
+        for c in components:
+            c_s = c.strip()
+            if c_s.startswith('_kf('):
+                # A reference to a KiBot filter
+                if c_s[-1] != ')':
+                    raise KiPlotConfigurationError('Missing `)` in KiBot filter reference: `{}`'.format(c))
+                filter_name = c_s[4:-1].strip().split(';')
+                logger.debug('Expanding KiBot filter in list of components: `{}`'.format(filter_name))
+                filter = BaseFilter.solve_filter(filter_name, 'show_components')
+                if not filter:
+                    raise KiPlotConfigurationError('Unknown filter in: `{}`'.format(c))
+                new_list.append(filter)
+                self._filters_to_expand = True
+            else:
+                new_list.append(c)
+        return new_list
+
+    def expand_filtered_components(self, components):
+        """ Expands references to filters in show_components """
+        if not components or not self._filters_to_expand:
+            return components
+        new_list = []
+        if self._comps:
+            all_comps = self._comps
+        else:
+            load_sch()
+            all_comps = GS.sch.get_components()
+            get_board_comps_data(all_comps)
+        # Scan the list to show
+        for c in components:
+            if isinstance(c, str):
+                # A reference, just add it
+                new_list.append(c)
+                continue
+            # A filter, add its results
+            ext_list = []
+            for ac in all_comps:
+                if c.filter(ac):
+                    ext_list.append(ac.ref)
+            new_list += ext_list
+        return new_list
+
     def get_targets(self, out_dir):
         return [self._parent.expand_filename(out_dir, self.output)]
 
@@ -362,28 +419,29 @@ class PcbDrawOptions(VariantOptions):
                                          no_warn_back=self.warnings == 'visible')
 
         filter_set = None
+        show_components = self.expand_filtered_components(self.show_components)
         if self._comps:
             # A variant is applied, filter the DNF components
             all_comps = set(self.get_fitted_refs())
             if self.add_to_variant:
-                # Old behavior
+                # Old behavior: components from the variant + show_components
-                all_comps.update(self.show_components)
+                all_comps.update(show_components)
                 filter_set = all_comps
             else:
                 # Something more coherent
-                if self.show_components:
+                if show_components:
                     # The user supplied a list of components
                     # Use only the valid ones, but only if fitted
-                    filter_set = set(self.show_components).intersection(all_comps)
+                    filter_set = set(show_components).intersection(all_comps)
                 else:
                     # Empty list means all, but here is all fitted
                     filter_set = all_comps
         else:
             # No variant applied
-            if self.show_components:
+            if show_components:
                 # The user supplied a list of components
                 # Note: if the list is empty this means we don't filter
-                filter_set = set(self.show_components)
+                filter_set = set(show_components)
         if filter_set is not None:
             logger.debug('List of filtered components: '+str(filter_set))
             plot_components.filter = lambda ref: ref in filter_set
@@ -391,7 +449,7 @@ class PcbDrawOptions(VariantOptions):
             logger.debug('Using all components')
 
         if self.highlight is not None:
-            highlight_set = set(self.highlight)
+            highlight_set = set(self.expand_filtered_components(self.highlight))
             plot_components.highlight = lambda ref: ref in highlight_set
         return plot_components
 

kibot/out_populate.py

@@ -87,12 +87,15 @@ class PopulateOptions(VariantOptions):
         logger.debug('Starting renderer with side: {}, components: {}, high: {}, image: {}'.
                      format(side, components, active_components, name))
         # Configure it according to our needs
+        options._filters_to_expand = False
         options.bottom = side.startswith("back")
         options.show_components = [c for c in components if c]
         if not options.show_components:
             options.show_components = None
+        else:
+            options.show_components = options.solve_filters(options.show_components)
         options.add_to_variant = False
-        options.highlight = [c for c in active_components if c]
+        options.highlight = options.solve_filters([c for c in active_components if c])
         options.output = name
         self._renderer.dir = self._parent.dir
         self._renderer._done = False
@@ -102,6 +105,7 @@ class PopulateOptions(VariantOptions):
     def save_options(self):
         """ Save the current renderer settings """
         options = self._renderer.options
+        self.old_filters_to_expand = options._filters_to_expand
         self.old_bottom = options.bottom
         self.old_show_components = options.show_components
         self.old_add_to_variant = options.add_to_variant
@@ -113,6 +117,7 @@ class PopulateOptions(VariantOptions):
     def restore_options(self):
         """ Restore the renderer settings """
         options = self._renderer.options
+        options._filters_to_expand = self.old_filters_to_expand
         options.bottom = self.old_bottom
         options.show_components = self.old_show_components
         options.add_to_variant = self.old_add_to_variant

tests/data/with_filter_html.md

@@ -0,0 +1,23 @@
+# Demo population with filters
+
+This is an example of populate output using KiBot filters.
+
+- [[front | ]] This is the front side of the board we are populating
+- [[back | ]] This is the back side of the board we are populating
+- [[front | _kf(all_smd;all_front) ]] First, populate all the SMD components on the front
+- [[back | _kf(all_smd;all_back)]] Now do the same for the back
+
+Now that you have all SMD components start soldering the THT components.
+But leave the connectors for the last step.
+
+- [[front | _kf(all_tht;!all_conn) ]] All THT, but connectors
+- [[front | _kf(all_conn) ]] Connectors added
+
+And here is the finished board
+
+- [[front | ]] Front
+- [[back | ]] Back
+
+## Conclusion
+
+You can add groups of components matching a filter.

tests/yaml_samples/pcbdraw_filters.kibot.yaml

@@ -0,0 +1,49 @@
+kiplot:
+  version: 1
+
+global:
+  solder_mask_color: blue
+  pcb_finish: ENIG
+
+filters:
+  - name: all_smd
+    type: generic
+    exclude_smd: true
+    invert: true
+
+  - name: all_tht
+    type: generic
+    exclude_tht: true
+    invert: true
+
+  - name: all_conn
+    type: generic
+    exclude_any:
+      - field: Value
+        regex: '.*CONN.*'
+      - field: Value
+        regex: 'SERVO.*'
+      - field: Reference
+        regex: 'JP.*'
+      - field: Reference
+        regex: 'U.*'
+    invert: true
+
+outputs:
+  - name: PcbDraw
+    comment: "PcbDraw test top"
+    type: pcbdraw
+    dir: PcbDrawFil
+    options: &pcb_draw_ops
+      format: png
+      show_components: ['_kf(all_tht;!all_conn)']
+      #show_components: ['_kf(all_conn)']
+      #show_components: ['_kf(all_smd)']
+
+  - name: PcbDraw2
+    comment: "PcbDraw test bottom"
+    type: pcbdraw
+    dir: PcbDrawFil
+    options:
+      <<: *pcb_draw_ops
+      bottom: True

tests/yaml_samples/populate_with_filter.kibot.yaml

@@ -0,0 +1,54 @@
+kiplot:
+  version: 1
+
+global:
+  solder_mask_color: blue
+  pcb_finish: ENIG
+
+filters:
+  - name: all_smd
+    type: generic
+    exclude_smd: true
+    invert: true
+
+  - name: all_front
+    type: generic
+    exclude_bottom: true
+
+  - name: all_back
+    type: generic
+    exclude_top: true
+
+  - name: all_tht
+    type: generic
+    exclude_tht: true
+    invert: true
+
+  - name: all_conn
+    type: generic
+    exclude_any:
+      - field: Value
+        regex: '.*CONN.*'
+      - field: Value
+        regex: 'SERVO.*'
+      - field: Reference
+        regex: 'JP.*'
+      - field: Reference
+        regex: 'U.*'
+    invert: true
+
+outputs:
+  - name: PcbDraw
+    comment: "How to draw a step"
+    type: pcbdraw
+    run_by_default: false
+    options:
+      format: png
+
+  - name: Populate
+    comment: "Populate example"
+    type: populate
+    dir: PopulateWithFilter
+    options:
+      renderer: PcbDraw
+      input: tests/data/with_filter_html.md