New output to join PDFs.

Closes #156
2022-03-03 16:13:00 -03:00 · 2022-03-03 16:13:00 -03:00 · 0c0c6ffd62
parent aeaabaf062
commit 0c0c6ffd62
9 changed files with 267 additions and 2 deletions
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@ -67,6 +67,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
  `solder_mask_color`, `silk_screen_color` and `pcb_finish`)
 - Report generation (for design house) (#93)
 - New output to print PCB layers in SVG format.
+- New output to join PDFs. (#156)

 ### Changed
 - Internal BoM: now components with different Tolerance, Voltage, Current
--- a/README.md
+++ b/README.md
@ -622,7 +622,9 @@ The available values for *type* are:
 - Documentation
    - `pdf_sch_print` schematic in PDF format
    - `svg_sch_print` schematic in SVG format
-    - `pdf_pcb_print`PDF file containing one or more layer and the page frame
+    - `pdf_pcb_print` PDF file containing one or more layer and the page frame
+    - `svg_pcb_print` SVG file containing one or more layer and the page frame
+    - `report` generates a report about the PDF. Can include images from the above outputs.
 - Bill of Materials
    - `bom` The internal BoM generator.
    - `kibom` BoM in HTML or CSV format generated by [KiBoM](https://github.com/INTI-CMNB/KiBoM)
@ -636,6 +638,7 @@ The available values for *type* are:
    - `compress` creates an archive containing generated data.
    - `download_datasheets` downloads the datasheets for all the components.
    - `pcbdraw` nice images of the PCB in customized colors.
+    - `pdfunite` joins various PDF files into one.
    - `qr_lib` generates symbol and footprints for QR codes.
    - `sch_variant` the schematic after applying all filters and variants, including crossed components.

@ -1624,6 +1627,33 @@ Next time you need this list just use an alias, like this:
    - `output_id`: [string=''] Text to use for the %I expansion content. To differentiate variations of this output.
    - `run_by_default`: [boolean=true] When enabled this output will be created when no specific outputs are requested.

+* PDF joiner
+  * Type: `pdfunite`
+  * Description: Generates a new PDF from other outputs.
+                 This is just a PDF joiner, using `pdfunite` from Poppler Utils.
+  * Valid keys:
+    - `comment`: [string=''] A comment for documentation purposes.
+    - `dir`: [string='./'] Output directory for the generated files. If it starts with `+` the rest is concatenated to the default dir.
+    - `disable_run_by_default`: [string|boolean] Use it to disable the `run_by_default` status of other output.
+                                Useful when this output extends another and you don't want to generate the original.
+                                Use the boolean true value to disable the output you are extending.
+    - `extends`: [string=''] Copy the `options` section from the indicated output.
+    - `name`: [string=''] Used to identify this particular output definition.
+    - `options`: [dict] Options for the `pdfunite` output.
+      * Valid keys:
+        - `output`: [string='%f-%i%I%v.%x'] Name for the generated PDF (%i=name of the output %x=pdf). Affected by global options.
+        - `outputs`: [list(dict)] Which files will be included.
+          * Valid keys:
+            - `filter`: [string='.*\.pdf'] A regular expression that source files must match.
+            - `from_cwd`: [boolean=false] Use the current working directory instead of the dir specified by `-d`.
+            - `from_output`: [string=''] Collect files from the selected output.
+                             When used the `source` option is ignored.
+            - `source`: [string='*.pdf'] File names to add, wildcards allowed. Use ** for recursive match.
+                        By default this pattern is applied to the output dir specified with `-d` command line option.
+                        See the `from_cwd` option.
+    - `output_id`: [string=''] Text to use for the %I expansion content. To differentiate variations of this output.
+    - `run_by_default`: [boolean=true] When enabled this output will be created when no specific outputs are requested.
+
 * Pick & place
  * Type: `position`
  * Description: Generates the file with position information for the PCB components, used by the pick and place machine.
--- a/docs/README.in
+++ b/docs/README.in
@ -435,7 +435,9 @@ The available values for *type* are:
 - Documentation
    - `pdf_sch_print` schematic in PDF format
    - `svg_sch_print` schematic in SVG format
-    - `pdf_pcb_print`PDF file containing one or more layer and the page frame
+    - `pdf_pcb_print` PDF file containing one or more layer and the page frame
+    - `svg_pcb_print` SVG file containing one or more layer and the page frame
+    - `report` generates a report about the PDF. Can include images from the above outputs.
 - Bill of Materials
    - `bom` The internal BoM generator.
    - `kibom` BoM in HTML or CSV format generated by [KiBoM](https://github.com/INTI-CMNB/KiBoM)
@ -449,6 +451,7 @@ The available values for *type* are:
    - `compress` creates an archive containing generated data.
    - `download_datasheets` downloads the datasheets for all the components.
    - `pcbdraw` nice images of the PCB in customized colors.
+    - `pdfunite` joins various PDF files into one.
    - `qr_lib` generates symbol and footprints for QR codes.
    - `sch_variant` the schematic after applying all filters and variants, including crossed components.

--- a/docs/samples/generic_plot.kibot.yaml
+++ b/docs/samples/generic_plot.kibot.yaml
@ -1036,6 +1036,28 @@ outputs:
      # [string=''] Board variant to apply.
      # Not fitted components are crossed
      variant: ''
+  # PDF joiner:
+  # This is just a PDF joiner, using `pdfunite` from Poppler Utils.
+  - name: 'pdfunite_example'
+    comment: 'Generates a new PDF from other outputs.'
+    type: 'pdfunite'
+    dir: 'Example/pdfunite_dir'
+    options:
+      # [string='%f-%i%I%v.%x'] Name for the generated PDF (%i=name of the output %x=pdf). Affected by global options
+      output: '%f-%i%I%v.%x'
+      # [list(dict)] Which files will be included
+      outputs:
+        # [string='.*\.pdf'] A regular expression that source files must match
+        - filter: '.*\.pdf'
+          # [boolean=false] Use the current working directory instead of the dir specified by `-d`
+          from_cwd: false
+          # [string=''] Collect files from the selected output.
+          # When used the `source` option is ignored
+          from_output: ''
+          # [string='*.pdf'] File names to add, wildcards allowed. Use ** for recursive match.
+          # By default this pattern is applied to the output dir specified with `-d` command line option.
+          # See the `from_cwd` option
+          source: '*.pdf'
  # Pick & place:
  # This output is what you get from the 'File/Fabrication output/Footprint position (.pos) file' menu in pcbnew.
  - name: 'position_example'
--- a/kibot/kiplot.py
+++ b/kibot/kiplot.py
@ -351,6 +351,8 @@ def get_output_dir(o_dir, obj, dry=False):


 def config_output(out, dry=False):
+    if out._configured:
+        return
    # Should we load the PCB?
    if not dry:
        if out.is_pcb():
@ -364,6 +366,8 @@ def config_output(out, dry=False):


 def run_output(out):
+    if out._done:
+        return
    GS.current_output = out.name
    try:
        out.run(get_output_dir(out.dir, out))
--- a/kibot/misc.py
+++ b/kibot/misc.py
@ -218,6 +218,7 @@ W_UNKFLD = '(W076) '
 W_ALRDOWN = '(W077) '
 W_KICOSTFLD = '(W078) '
 W_MIXVARIANT = '(W079) '
+W_NOTPDF = '(W080) '
 # Somehow arbitrary, the colors are real, but can be different
 PCB_MAT_COLORS = {'fr1': "937042", 'fr2': "949d70", 'fr3': "adacb4", 'fr4': "332B16", 'fr5': "6cc290"}
 PCB_FINISH_COLORS = {'hal': "8b898c", 'hasl': "8b898c", 'imag': "8b898c", 'enig': "cfb96e", 'enepig': "cfb96e",
--- a/kibot/out_pdfunite.py
+++ b/kibot/out_pdfunite.py
@ -0,0 +1,143 @@
+# -*- coding: utf-8 -*-
+# Copyright (c) 2022 Salvador E. Tropea
+# Copyright (c) 2022 Instituto Nacional de Tecnología Industrial
+# License: GPL-3.0
+# Project: KiBot (formerly KiPlot)
+import re
+import os
+import glob
+from sys import exit
+from subprocess import check_output, STDOUT, CalledProcessError
+from .gs import GS
+from .error import KiPlotConfigurationError
+from .kiplot import config_output, get_output_dir, run_output
+from .misc import MISSING_TOOL, WRONG_INSTALL, WRONG_ARGUMENTS, INTERNAL_ERROR, W_NOTPDF
+from .optionable import Optionable, BaseOptions
+from .registrable import RegOutput
+from .macros import macros, document, output_class  # noqa: F401
+from . import log
+
+logger = log.get_logger()
+
+
+class FilesList(Optionable):
+    def __init__(self):
+        super().__init__()
+        with document:
+            self.source = '*.pdf'
+            """ File names to add, wildcards allowed. Use ** for recursive match.
+                By default this pattern is applied to the output dir specified with `-d` command line option.
+                See the `from_cwd` option """
+            self.from_cwd = False
+            """ Use the current working directory instead of the dir specified by `-d` """
+            self.from_output = ''
+            """ Collect files from the selected output.
+                When used the `source` option is ignored """
+            self.filter = r'.*\.pdf'
+            """ A regular expression that source files must match """
+
+
+class PDFUniteOptions(BaseOptions):
+    def __init__(self):
+        with document:
+            self.output = GS.def_global_output
+            """ Name for the generated PDF (%i=name of the output %x=pdf) """
+            self.outputs = FilesList
+            """ [list(dict)] Which files will be included """
+        super().__init__()
+        self._expand_ext = 'pdf'
+
+    def config(self, parent):
+        super().config(parent)
+        if isinstance(self.outputs, type):
+            KiPlotConfigurationError('Nothing to join')
+        self._expand_id = parent.name
+
+    def get_files(self, output, no_out_run=False):
+        output_real = os.path.realpath(output)
+        files = []
+        out_dir_cwd = os.getcwd()
+        out_dir_default = self.expand_filename_pcb(GS.out_dir)
+        for f in self.outputs:
+            # Get the list of candidates
+            files_list = None
+            if f.from_output:
+                out = RegOutput.get_output(f.from_output)
+                if out is not None:
+                    config_output(out)
+                    files_list = out.get_targets(get_output_dir(out.dir, out, dry=True))
+                else:
+                    logger.error('Unknown output `{}` selected in {}'.format(f.from_output, self._parent))
+                    exit(WRONG_ARGUMENTS)
+                if not no_out_run:
+                    for file in files_list:
+                        if not os.path.isfile(file):
+                            # The target doesn't exist
+                            if not out._done:
+                                # The output wasn't created in this run, try running it
+                                run_output(out)
+                            if not os.path.isfile(file):
+                                # Still missing, something is wrong
+                                logger.error('Unable to generate `{}` from {}'.format(file, out))
+                                exit(INTERNAL_ERROR)
+            else:
+                out_dir = out_dir_cwd if f.from_cwd else out_dir_default
+                source = f.expand_filename_both(f.source, make_safe=False)
+                files_list = glob.iglob(os.path.join(out_dir, source), recursive=True)
+            # Filter and adapt them
+            for fname in filter(re.compile(f.filter).match, files_list):
+                fname_real = os.path.realpath(fname)
+                # Avoid including the output
+                if fname_real == output_real:
+                    continue
+                files.append(fname_real)
+        return files
+
+    def get_targets(self, out_dir):
+        return [self._parent.expand_filename(out_dir, self.output)]
+
+    def get_dependencies(self):
+        output = self.get_targets(self.expand_filename_pcb(GS.out_dir))[0]
+        files = self.get_files(output, no_out_run=True)
+        return files
+
+    def run(self, output):
+        # Output file name
+        logger.debug('Collecting files')
+        # Collect the files
+        files = self.get_files(output)
+        for fn in files:
+            with open(fn, 'rb') as f:
+                sig = f.read(4)
+                if sig != b'%PDF':
+                    logger.warning(W_NOTPDF+'Joining a non PDF file `{}`, will most probably fail'.format(fn))
+        logger.debug('Generating `{}` PDF'.format(output))
+        if os.path.isfile(output):
+            os.remove(output)
+        cmd = ['pdfunite']+files+[output]
+        logger.debug('Running: {}'.format(cmd))
+        try:
+            check_output(cmd, stderr=STDOUT)
+        except FileNotFoundError:
+            logger.error('Missing `pdfunite` command, install it (poppler-utils)')
+            exit(MISSING_TOOL)
+        except CalledProcessError as e:
+            logger.error('Failed to invoke pdfunite command, error {}'.format(e.returncode))
+            if e.output:
+                logger.error('Output from command: '+e.output.decode())
+            exit(WRONG_INSTALL)
+
+
+@output_class
+class PDFUnite(BaseOutput):  # noqa: F821
+    """ PDF joiner
+        Generates a new PDF from other outputs.
+        This is just a PDF joiner, using `pdfunite` from Poppler Utils. """
+    def __init__(self):
+        super().__init__()
+        with document:
+            self.options = PDFUniteOptions
+            """ [dict] Options for the `pdfunite` output """
+
+    def get_dependencies(self):
+        return self.options.get_dependencies()
--- a/tests/test_plot/test_misc.py
+++ b/tests/test_plot/test_misc.py
@ -1041,3 +1041,12 @@ def test_annotate_power_1(test_dir):
    ctx.compare_txt('deeper'+context.KICAD_SCH_EXT)
    ctx.compare_txt('sub-sheet'+context.KICAD_SCH_EXT)
    ctx.clean_up()
+
+
+def test_pdfunite_1(test_dir):
+    prj = 'light_control'
+    ctx = context.TestContext(test_dir, 'test_pdfunite_1', prj, 'pdfunite_1', POS_DIR)
+    ctx.run()
+    o = prj+'-PDF_Joined.pdf'
+    ctx.expect_out_file(o)
+    ctx.clean_up(keep_project=True)
--- a/tests/yaml_samples/pdfunite_1.kibot.yaml
+++ b/tests/yaml_samples/pdfunite_1.kibot.yaml
@ -0,0 +1,52 @@
+kiplot:
+  version: 1
+
+outputs:
+  - name: PDF_Joined
+    comment: "PDF files for top layer + mirrored bottom layer"
+    type: pdfunite
+    options:
+      outputs:
+        - from_output: PDF_top
+        - from_output: PDF_bottom
+
+  - name: PDF_top
+    comment: "PDF files for top layer"
+    type: pdf
+    dir: PDF
+    options:
+      exclude_edge_layer: false
+      exclude_pads_from_silkscreen: false
+      plot_sheet_reference: false
+      plot_footprint_refs: true
+      plot_footprint_values: true
+      force_plot_invisible_refs_vals: false
+      tent_vias: true
+
+      # PDF options
+      drill_marks: small
+      mirror_plot: false
+      negative_plot: false
+      line_width: 0.01
+    layers:
+      - layer: F.Cu
+        suffix: F_Cu
+      - layer: F.SilkS
+        suffix: F_Silks
+
+  - name: PDF_bottom
+    comment: "PDF files for bottom layer"
+    type: pdf
+    dir: PDF
+    options:
+      exclude_edge_layer: false
+      # PDF options
+      drill_marks: full
+      mirror_plot: true
+      negative_plot: true
+      line_width: 0.01
+    layers:
+      - layer: B.Cu
+        suffix: B_Cu
+      - layer: B.SilkS
+        suffix: B_Silks