Docs Connecting Your Data

Data Sources

A Data Source is where you tell Deepwell exactly what data to get from a tool you've connected, for example a list of tickets or your backup results. Reports read from Data Sources, so you set one up once and every Report can use it, and because it isn't tied to a single client, the same one supplies Reports for many of them.

A Data Source is where a Report's data starts, and from there you build a Visualization that turns the data into a chart. See Build a Visualization.

How to make a Data Source

The Data Sources page with the saved Data Sources, each with its Integration and endpoint.
Data Sources: each row shows its Integration, its endpoint, and where it is used. New Data Source is at the top right.

Describe the data you want, and Deepwell configures the request for you.

  1. Go to Data Sources, in the Admin menu, and select New Data Source. Only an MSP Admin can open Data Sources.
  2. Select the Connection to use. This is the connected tool that supplies the data.
  3. If you want to test with data for one client, select an Organization.
  4. If this Data Source reads a value from another saved Data Source, select Read from a previous Data Source. See How to chain a Data Source.
  5. Give a description of the endpoint, or copy the endpoint into the field. You can use API documentation, an example URL, or a short description.

Deepwell then configures the Data Source for you.

To start from a working example, select Examples. Deepwell shows endpoint shapes for the tool of your Connection under a heading that names the tool, for example ConnectWise examples. Select one to load it into the field.

Your input can be informal or exact. Here's an example in plain English:

Get service tickets, including summary, board, status, priority, company, owner, closed flag, date entered, date resolved, and actual hours.

Here's an example of endpoint documentation:

GET /v4_6_release/apis/3.0/service/tickets

headers:
    Accept: application/json

queryParameters:
    fields: id,summary,board/name,status/name,priority/name,company/identifier,owner/name,closedFlag,dateEntered,dateResolved,actualHours
    pageSize: 100

(This example is for ConnectWise. The endpoint of a different tool is different.)

Deepwell makes the configuration from either type of input, and you then review it.

What to include

Deepwell makes an HTTP GET request from your input, with a URL path, query parameters, and headers, and GET is the only method it supports. It builds the request, not the data, so each value comes from the real API when a Report runs.

As a minimum, Deepwell needs sufficient data to identify one endpoint: a path, documentation, or a URL, plus the fields that you want. It infers the other parts of the request.

More input makes the result faster and better. Include these items:

  • The exact fields that you want.
  • Each filter that the endpoint supports.
  • Whether the endpoint paginates, and a sensible page size.
  • Whether you want a Date Range. Write "include a start and end date" to connect the Date Range parameters.

Deepwell does these parts for you:

  • It keeps the path only, and removes the domain and the protocol.
  • It never includes authentication, because your Connection supplies that. Do not copy API keys, tokens, or authentication headers into a Data Source.
  • It inserts Parameters where the API needs them: {{start}} and {{end}} for the Date Range, and an organization parameter if you have an Organization selected in the builder.

After Deepwell makes the configuration, it shows a confidence of high or low and some suggestions. Low confidence means your input wasn't clear and Deepwell made assumptions, so clear, specific input gives you a better result.

How to review the configuration

Review these items and change them if necessary:

  • Name and Description: Deepwell suggests these from your input. Change them so that you can identify the Data Source later.
  • URL path: the endpoint that Deepwell calls.
  • Query parameters and Headers: the details of the request.

If you have an Organization selected when Deepwell makes the configuration, the configuration includes an organization parameter, which limits the data to one client. Select an Organization to test the Data Source with. When a Report runs, the parameter changes to the client of that Report.

Parameters

A Data Source can include Parameters. Deepwell puts a value in each Parameter at run time.

  • Date Range: Start (Days Ago) and End (Days Ago) set a moving period, for example the last 30 days.
  • Organization: an organization parameter changes to the correct client when a Report runs, so one Data Source is sufficient for each client.
  • Previous step: a Data Source can take a value from the step before it. See How to chain a Data Source.

How to chain a Data Source

A Visualization can use more than one Data Source. The Data Sources become the steps of a chain that run in order, and a step can put a value from the step before it into its own request. Some APIs need this: you get a company from one endpoint, then you need its id to get the tickets of that company from a second endpoint.

How to specify the chain

Make and save each Data Source that must run before the new Data Source. Then use the chain editor:

  1. In the Data Source AI Builder, select Read from a previous Data Source.
  2. Select the + control at the position where you want the saved Data Source.
  3. Select the saved Data Source.
  4. Repeat these steps for each earlier Data Source that the chain needs.
  5. If the order is incorrect, select the up or down control on a Data Source row.

The list ends with Current, which is the Data Source that you are making. The Data Source marked Previous step supplies the value to Current, while a Data Source marked Earlier step runs first but Current cannot read it directly.

The first Data Source in the list cannot read a previous step. If it does, Deepwell shows a warning that names it, for example Company Tickets needs an earlier Data Source. Select the + control before it and add the Data Source that runs earlier, until the first Data Source does not read a previous step.

The list is authoring context, because Deepwell uses the order for Test Request but does not save it with the Data Source, so add the same Data Sources to a Visualization in the same order.

To remove one Data Source, select its remove control. Select Clear all to remove the complete chain and its previous-step query parameter.

How to specify the value

The chain editor can add the value to a query parameter:

  1. Type the name that the API uses in Query parameter.
  2. Type the field path in Previous output path.
  3. Select Add query parameter.

Deepwell shows the complete value, for example {{previous.out:items[0].id}}, where the path names a field in the data that the previous step returned.

After Deepwell makes the configuration, you can also type a {{previous.out:<path>}} value in the URL path, in a query parameter, or in a header.

Example Meaning
{{previous.out:companyId}} The companyId field at the top level.
{{previous.out:company.id}} The id field inside company.
{{previous.out:items[0].id}} The id field of the first item.

A path holds field names and numeric indexes only, so Deepwell does not accept a function or a filter in it. If the value needs a filter, a condition, or a function, use a Previous Out Expression instead. See How to use a Previous Out Expression.

The value must be one text value, one number, or true or false. A path that gives an object, an array, or no value makes the step fail, and Deepwell then shows an error that names the parameter.

Remember these limits:

  • A step reads the immediately previous step only, and cannot reach two steps back.
  • The first step has no previous step, so a {{previous.out:...}} parameter in the first step always fails.
  • Deepwell does not cache a step that uses {{previous.out:...}}, so the step gets its data again on each run.

A chained Data Source cannot run on its own, but Deepwell can test it once the chain starts with a Data Source that does not read a previous step. After it makes the configuration, it runs each saved Data Source in order and then the unsaved Current Data Source, and the preview shows the data from Current. Select Test Request to run the complete chain again after a change.

How to use a Previous Out Expression

A simple path cannot tell items apart, for example when the previous step returns a list of tenants and you need the id of the one where tenant_kind is customer. For this, a Data Source can hold one Previous Out Expression: a JSONata expression that Deepwell runs against the output of the previous step.

  1. Save the Data Source. The Data Source AI Builder does not have this field.
  2. In the Data Sources list, select Edit on its row.
  3. In the Edit Data Source dialog, type the expression in Previous Out Expression, for example items.policy[tenant_kind = "customer"][0].tenant_id.
  4. Put {{previous.out:expression}} in the URL path, a query parameter, or a header, where the value must go.
  5. Save the Data Source.

The expression must give one text value, one number, or true or false, the same as a path. Deepwell does not review the expression when you save. An incorrect expression, or one that gives an object, an array, or no value, makes the step fail when it runs, with an error that names {{previous.out:expression}}.

A Data Source holds one Previous Out Expression, and one that uses {{previous.out:expression}} without an expression fails with the error previous out expression is not set. To remove the expression, clear the field and save.

Pagination

An endpoint can return its results on more than one page, so set pagination to on and Deepwell gets the complete set of results.

How to test a Data Source

For a Data Source that does not read a previous step, select Test Request to see real example data before you save. If the Data Source is for one Organization, also select an Organization to test with, and Deepwell shows the data that it received.

For a chained Data Source, first add each earlier Data Source in the chain editor, because Test Request stays unavailable until the first source does not read a previous step. Then select Test Request to run the complete chain. See How to chain a Data Source.

To build the request around a Date Range, set Include time window to on.

A Test window control then appears with the preview, with a Start offset for {{start}} and an End offset for {{end}}. If the configuration has neither Parameter, there is no Test window.

The offsets count back from now. Type 1D for one day, 7D for seven days, 1M for one month, or 0D for now, and the default is 1D. To run the test again with a different period, change the offset and select Test Request again.

The Test window applies to the test only, because a Report supplies its own Date Range when it runs.

A test isn't necessary before you save, but it is recommended, since an error is frequent with an unfamiliar endpoint. If the test returns one, review the URL path and the parameters against your tool's API documentation and correct them.

How to save a Data Source

Save the Data Source, and you can then build a Visualization from it. See Build a Visualization.

How to change a Data Source

Because many Reports can use one Data Source, a change has a wide effect: it applies to each Report that uses the Data Source, for each client.

The Data Sources list tells you what depends on each one before you change it. A Data Source in use reads Used by, with the Reports and the Visualization Blocks that use it, and marks how many of those Reports have a Schedule. One that nothing uses reads Not currently used.

When you save a change to a Data Source that is in use, Deepwell shows a Review Data Source Update step first. It lists the Visualization Blocks and Reports that use the Data Source and marks each Report that has an active Schedule. Review the list, then select Confirm update to save. A Data Source that nothing uses saves right away, with no review step.

The Edit Data Source dialog also holds the Previous Out Expression of a chained Data Source. See How to use a Previous Out Expression.

How to archive a Data Source

Deepwell doesn't delete a Data Source, so when you finish with one, archive it.

Go to Data Sources, in the Admin menu, and select the archive control on the Data Source. Deepwell shows this confirmation:

Archive this Data Source? You can restore it later. Reports and Visualization Blocks that use this Data Source will continue to work.

Select Archive Data Source. Deepwell confirms with Data Source archived.

An archived Data Source is not in your Data Sources list, and you can't select it for a new Visualization. The Reports and Visualization Blocks that already use it continue to run, and your existing Report Snapshots don't change.

You can archive only a Data Source that your MSP owns. A Data Source that Deepwell supplies to every MSP shows Read only and has no archive control.

How to restore an archived Data Source

Go to Data Sources, in the Admin menu, and select the Archived tab. Find the Data Source and select the Restore control. Deepwell confirms with Data Source restored and the Data Source goes back to the Active tab.

There's no confirmation step, because a restore removes nothing, and to reverse it you archive the Data Source again.

The Archived tab lists only the Data Sources that your MSP owns, so a Data Source that Deepwell supplies to every MSP can't be restored here.

Last updated