With the Lumen 0.5.0 release finally coming up we've got one last chance to fix naming mistakes in Lumen. One major sticking point has always been the Target component. This name came about as a natural pair to the data Source, where data would flow from source to target. However, even then it never really made sense and since there's been quite a lot of refactoring that more cleanly separates views, filters and transforms (which used to be grouped together on a target).

This means that now a Target is defined as a component consisting of one or more view components that consume data from a pipeline and render the views into a layout. Therefore I suggest I finally pull the trigger and simply rename Target -> Layout. Alternatively I could also consider LayoutGroup as the name of the Python class but keep the YAML key as layouts.

I'd like to collect everyone's opinion so please chime in @Hoxbro, @maximlt, @jbednar, @droumis, @jlstevens and @eli-pinkus.

Adds a lumen builder CLI command that launches a Bokeh/Panel server running the builder.

• [x] Add tests
• [x] Decide on naming of builder, is it ui, gui, or builder?

• [x] import a component by its module path, e.g. you can do my_library.some_module.MyTransform

• Also made the install page more explicit about python installation after a good talk with @Hoxbro.

• Also included some minor fixes like syntax highlighting and a couple of links

It started with an example of how to use variables but failed with a verification error. This is fixed with 488ee9c5d935788602d1de50f4f8f93032040ee5.

Though, I still get this warning WARNING:param.Source: Resolving nested variable references currently not supported. which I think is why I get no automatic update when changing the variable input.

Maybe I'm doing something wrong with the specification.

variables:
  ticker:
    type: widget
    kind: TextInput
    default: https://raw.githubusercontent.com/matplotlib/sample_data/master/aapl.csv

sources:
  stock_data:
    type: file
    tables:
      ticker: $variables.ticker

targets:
- title: Table
  source: stock_data
  views:
    - type: table
      table: ticker

It can be hard to know if something is transform or transforms. I have tried to take account of this by using difflib to come up with suggestions if parameters are mistyped/wrong.

I have chosen to raise an error when an input parameter does not match the class parameters. My reasoning for this is that the error usually will arrive later, making it harder to debug.

I have used the following example to see if I can provoke an error by adding or removing an s.

from lumen.pipeline import Pipeline

data_url = 'https://raw.githubusercontent.com/rfordatascience/tidytuesday/master/data/2020/2020-07-28/penguins.csv'

pipeline = Pipeline.from_spec({
    'source': {
        'type': 'file',
        'table': {
            'penguins': data_url
        }
    },
    'filters': [
        {'type': 'widget', 'field': 'species'},
        {'type': 'widget', 'field': 'island'},
        {'type': 'widget', 'field': 'sex'},
        {'type': 'widget', 'field': 'year'}
    ],
    'transforms': [
        {'type': 'aggregate', 'method': 'mean', 'by': ['species', 'sex', 'year']}
    ]
})

The example will raise an error (try to find out what raises it).

I'm having trouble with an error message "ModuleNotFoundError: No module named 'panel.pane.perspective' " when attempting to launch lumen https://lumen.holoviz.org/index.html.

I have panel 0.11.0a8 installed through conda with pyviz/label/dev. Any insights please?

All components currently implement .from_spec methods which instantiate the component from the declarative specification. We want to be able to do the reverse and construct a specification from component instances by implementing .to_spec methods.

For all the basic component types this should be fairly straightforward, e.g. a Source, View or Filter simply has to serialize it's parameters and its type. It becomes a little more difficult if we are dealing with references and variables because View.pipeline should generally not inline and serialize the entire Pipeline specification.

def to_spec(self, allow_refs=True):
    """
    Converts the component to a declarative specification that can be serialized to YAML.
    Whether sub-component definitions are inlined depends on the type of component,
    e.g. Filter and Transform components will be inlined on a Pipeline but a Pipeline will
    not be inlined on a View.

    Arguments
    -----------
    allow_refs: boolean
      Whether to allow exporting references or to inline the materialized values.

    Returns
    --------
    Declarative specification containing the definition of this component.
    """

Goals

• We can serialize all component types individually but also a whole Dashboard or Pipeline definition.
• We can handle references and variables
• The exported specification faithfully roundtrips to an identical instance, i.e. we can go from instance -> specification -> instance and end up with an identical copy.

Includes various fixes to the DateFilter:

• On picker mode cast start/end/value to datetime.date
• Override the value Parameter by a CalendarDate or DateRange Parameters (let me know if there's a better way, that looks hacky!)
• Bidirectionally link the widget to the value Parameter, this allows URL query parameter changes to be synced back to the widget.

This is work in progress and requires other changes both in Param and Panel. I've also noticed accumulating calls in get_data on picker mode so there's something wrong there.

This issue summarizes some discussion I've had with @philippjfr and @jbednar about how best to generate widgets for custom view parameters. Here are the three options we talked about:

1. Use param precedence according to the usual panel semantics when converting parameters to widgets.
2. Declaring constant=True for the parameters that shouldn't be shown as widgets.
3. Explicitly list which parameters should become widgets (by name) in the yaml declaration.

Personally, I like option 3 the most as it is the most explicit and lets you easily set the widget order in the yaml. Then I like option 1 as it is consistent with how panel displays widgets for parameters and I like option 3 the least.

If different views are used (for example Perspective with Holoviews), the dashboard doesn’t build, as two separate extensions are needed.

It would be nice if the various extensions could be gracefully handled, if not an assertion error on crossing multiple extensions would be nice for users to understand the error instead of the dashboard not displaying anything.

Adds a manual_update option to the global config, each Target and Pipeline to be able to manually update the data using a button click. This is useful when applying a filter or transform is very slow.

• [x] Add tests
• [x] Decide on naming of manual_update parameter

Some methods in Lumen need different inputs depending on the class it is called from. An example of this could be from_spec, the example below. I would like to see these non-standard keywords being keyword-only arguments, so passing in the wrong arguments is impossible.

https://github.com/holoviz/lumen/blob/95809959b18eb31d069c084f4fabdefbc2cfb05e/lumen/views/base.py#L267-L269

https://github.com/holoviz/lumen/blob/95809959b18eb31d069c084f4fabdefbc2cfb05e/lumen/dashboard.py#L299

Currently, the penguins' dataset (https://raw.githubusercontent.com/rfordatascience/tidytuesday/master/data/2020/2020-07-28/penguins.csv) is used extensively in the new documentation.

It should be added to this repo or an S3 (which we control).

I just thought it made a lot of sense to check the websites asynchronously.

I have set a 10 seconds timeout, which could be removed again or given as a parameter.

Once a widget filter has been selected in a Lumen Pipeline within a notebook, there is no way to programmatically unselect or reset the filter.

ALL software version info

lumen: 0.5.0a64

Description of expected behavior and the observed behavior

A way to programmatically unselect options in a widget filter. @Hoxbro suggested: pipeline.reset_filters()

Complete, minimal, self-contained example code that reproduces the issue

import lumen
from lumen.views import Table
from lumen.pipeline import Pipeline
import panel as pn

pn.extension('tabulator', template='fast')

data_url = 'https://raw.githubusercontent.com/rfordatascience/tidytuesday/master/data/2020/2020-07-28/penguins.csv'

pipeline = Pipeline.from_spec({
    'source': {
        'type': 'file',
        'tables': {
            'penguins': data_url
        }
    },
    'filters': [
        {'type': 'widget', 'field': 'species'},
        {'type': 'widget', 'field': 'island'},
        {'type': 'widget', 'field': 'sex'},
        {'type': 'widget', 'field': 'year'}
    ],
    'transforms': [
        {'type': 'aggregate', 'method': 'mean', 'by': ['sex', 'year']}
    ]
})

pn.Row(pipeline.control_panel, Table(pipeline=pipeline, pagination='remote'))

Screenshots or screencasts of the bug in action

Changes addressed in this PR:

• split catalog to different files to test IntakeSource and IntakeSQLSource separately
• move tests for FileSource to a new file
• add tests to JSONSource
• add tests to JoinedSource
• move common testing logics to a utils file
• test Source.from_spec when loading from source in state.sources

Bug fixes:

• JSONSource._resolve_template_vars(): convert template to string to make sure re.findall(template) work properly. Currently, a local file path will be read as a pathlib.PosixPath object: https://github.com/holoviz/lumen/blob/master/lumen/sources/base.py#L528