Simple Data Pipe: Analyzing Salesforce Data with Looker

The Simple Data Pipe (https://developer.ibm.com/clouddatase...) allows you to connect to data behind Web APIs, and land it all in one staging ground, in its native form, using IBM Cloudant. The world is then your oyster: we provide prebuilt connections to oodles of data sources, as well as integrations with both IBM dashDB, for data warehousing, and Bluemix’s Apache Spark™ service, for advanced analytics processing.

This video tutorial goes from the initial deployment, all the way through moving data from Cloudant to dashDB and visualizing it in the Looker business intelligence platform. See https://developer.ibm.com/clouddatase... for more examples, or https://developer.ibm.com/clouddatase... for pre-built connectors, along with a framework for building your own.

---

Closed Caption:

Today, I'm going to deploy the Simple Data Pipe example application from the IBM Cloud Data Services developer advocacy team.
Today, I'm going to deploy the Simple Data Pipe example application from the IBM Cloud Data Services developer advocacy team.
The app lets you connect to data that resides behind REST APIs, which is the standard for most modern web apps.
In this demo, we're going to connect to Salesforce.com data. Later in the tutorial,
I'll move data into IBM's dashDB cloud data warehouse
and hook up the Looker business intelligence platform to it
to start analyzing and exploring data.
Here are the services deployed with our app:
a Node.js runtime, a Cloudant database, and an optional service for single sign-on.
First, I need to configure my Salesforce instance for OAuth.
I populated some of the fields already,
but let me show you how to find the callback URL that you need.
Create your first pipe run in the Simple Data Pipe app.
Make sure you choose "SalesForce" and save it.
You can extend the Simple Data Pipe with more pre-built data source connectors,
or use our connector framework to build your own.
There's our callback URL.
Back in Salesforce, enable OAuth, and add all OAuth scopes.
Next, grab the OAuth consumer key and secret.
If you're using a Salesforce sandbox environment, check that option.
Now I'm connected and ready to pipe my data.
My Salesforce sandbox environment contains 35,000 records, spread across 42 tables.
I'm moving them all now.
I can view details on any pipe run to see the record counts for each table.
They actually link to their corresponding databases in Cloudant.
Let's hop over to Cloudant now.
I just need to grab the password for my Cloudant service in the Simple Data Pipe app.
Let's quickly look at one of our records in the leads database.
Here, Cloudant has captured the native JSON from the Salesforce API.
The good news is that I don't need to fully understand the schema of these JSON documents
in order to move them into a relational format for a data warehouse.
Cloudant will do that for me, thanks to its integration with dashDB.
Although I moved all my data in my Salesforce environment,
I'm only concerned with six tables for the BI dashboard I plan to build in Looker.
Here, Cloudant has automatically discovered the relational schema of my JSON documents --
-- but it also lets you customize the schema before import.
In dashDB, I'll head over to my connection settings and grab the parameters Looker needs.
Be sure to name your connection exactly -- dashdb -- all lowercase,
because the app references that name.
Now, I'm going to connect to a LookML project that Erin Franz from Looker was kind enough to design for us.
LookML is Looker's handy markup language. So let's look at that.
Fork the IBM CDS LookML project into your own GitHub repo, and give that URL to Looker.
I've already configured my deploy key, but note that Looker provides a link right to it.
Now, just sync with your GitHub project.
You can see the LookML for dashDB's leads table.
It defines the aggregate statistics and charts we're about to see.
And there we go: instant dashboard.
Executives love to see and explore their data.
Looker comes with a lot of features for drilling into data.
Think of it as a "pivot table on steroids."
For more on the Simple Data Pipe, see our complete tutorial and additional data source connectors at:
https://developer.ibm.com/clouddataservices/simple-data-pipe-salesforce-looker
There you have it. I'm Mike Broberg from IBM Cloud Data Services. Thanks.