The Syllabus The Notes

The Notes

Here’s where you’ll find resources, code, links, and notes for every class. Materials are added here generally at least one day before each class, but can change and get updated up until class time.

Class 1 • Basic internetting

Introductions

Engagement Projects in Journalism

What’s a Prototype (for us)?

Quick Examples

Logistics & plan

Exploring web pages

Assignment

This is the “Class 1” assignment. Do it now … it’s due on at high noon before Class 2.

Continue to play with this project in Glitch and make it your own. Change colors, play with formatting, play with the text. Get creative! Google around for html tips and tricks if you want. Post the live link to your project in Slack in #design-development so we can all try it.

Class 2 • Introduction to databases

Quick review of last week

Note on “assets”

A different structure

“Static”

What we did last week:

  1. Request: You -> https://curious-scout.glitch.me -> Glitch
  2. Response: Glitch -> index.html -> you

“Dynamic”

  1. Request: You -> https://newmark-bark.glitch.me -> Glitch
  2. Response: Glitch -> server.js -> index.html -> you

Instead of Glitch just serving up your fixed or static html page (as in Class 1) you’re using a “server” file that serves up the page.

Let’s take a look around …

See how the main page, index.html, is served.

We can also make a new “about” page by adding a new view page and a new route.

Here’s the view page, which we’ll call views/about.html:

<html>
  
  <head>
    <title>About Us</title>
  </head>
  
  <body>
      
    <h1>
      About Newmark Bark
    </h1>
    
    <p>
      Newmark Bark is a service of the Newmark Graduate School of Journalism at CUNY.
   </p>
   
  </body>
  
</html>

Here’s the route, which goes in the server.js file under the line that says put the routes under this line!

// this is the "home" page or the regular url

app.get('/about', function(request, response) {
  response.sendFile(__dirname + '/views/about.html');
});

(can also add the stylesheet to the about.html page by putting this code inside the <head> tags):

<!-- import the webpage's stylesheet -->
<link rel="stylesheet" href="/style.css">

Why all the hassle?

Here’s why … with a “dynamic” page, we can alter the page before sending it back to the user.

For example, try visiting: https://newmark-bark.glitch.me/random (note the “random” at the end!)

  1. Request: You -> https://newmark-bark.glitch.me/random -> Glitch
  2. Response: Glitch -> server.js -> get data -> inject data -> fact-page.html -> you

We can inject the page with data before serving it up.

But what if you have more than 6 rows of data?

Databases!

We’re going to play with a database that’s already up an running online.

A little bit of SQL

SQL stands for “Sequel Query Language” and is a very powerful, if arcane, way to ask questions of a database.

The format is generally:

SELECT

count() & WHERE

GROUP BY & ORDER BY

AS

Break!

Load the database into your Glitch project

  1. In your bark project, go to the Glitch terminal
    • Tools > Terminal
  2. Type: cd .data to change directory to the .data directory
  3. Copy-paste this and hit return: 
    wget "http://s3.amazonaws.com/media.johnkeefe.net/data/NYC_Dog_Licensing_Dataset_2018.csv"
sqlite3 my.db '.mode csv' '.import NYC_Dog_Licensing_Dataset_2018.csv nyc_dog_licenses'
  4. Type: sqlite3 my.db to start the database
  5. Type: SELECT count() FROM nyc_dog_licenses; to test it!
  6. Can even do: SELECT AnimalName, count(AnimalName) FROM nyc_dog_licenses GROUP BY AnimalName ORDER BY count(AnimalName) DESC LIMIT 10;
  7. Type: .exit to exit
  8. Close the Terminal with the “X” at the right

Dynamic with a database

  1. Request: You -> https://newmark-bark.glitch.me/first10 -> Glitch
  2. Response: Glitch -> server.js -> query a database -> get the answer -> turn into JSON -> you

Let’a make it!

First …

We’ll talk about adding this code in class:

Block one

// watch for "first10" in the URL
app.get('/first10', (request, response) => {
    
    // set the query
    let query = 'SELECT * FROM nyc_dog_licenses LIMIT 10'
  
    // run the query
    db.all(query, (err, rows) => {
    
        // let us know if something went wrong
        if (err) {
          console.log(err)
        }
    
        // otherwise, send a "response" with a JSON version of the rows we got
        response.send(JSON.stringify(rows))
  })
  
})

We’re seeing data on the web

Block two

// watch for "toppnames" in the URL
app.get('/topnames', (request,response) => {
    
    // set the query
    let query = 'SELECT AnimalName, count(AnimalName) as TotalDogs FROM nyc_dog_licenses GROUP BY AnimalName ORDER BY TotalDogs DESC LIMIT 10'
  
    // let us know if something went wrong
    db.all(query, (err, rows) => {

        // let us know if something went wrong
        if (err) {
          console.log(err)
        }

        // otherwise, send a "response" with a JSON version of the rows we got
        response.send(JSON.stringify(rows))

    })
})

Assignment

This is the “Class 2” assignment. Do it now … it’s due on at high noon before Class 3.

Add a new “route” to your web service called “topbreeds” that, when hit by a web browser, returns a list of the top FIVE dog breeds (by count) in the database.

Submit that “topbreeds” live link in the Google Classroom assignment form. That link will look like https://https://newmark-bark.glitch.me/topbreeds … but with your name instead of newmark.

Remember that your can find your base “live link” by clicking the “Share” button.

Class 3 • Making a dynamic webpage

Reset from last week

Data for the web

We’ve been making what’s known as an API. Officially, API stand for Application Program Interface. But I like to think of it as Another Person’s Information. It’s a way to get information from data on the web … even though you usually don’t actually see it. It’s often happening in the background.

Let’s make an a service specifically for providing data on how many dogs are named … whatever name we’re asked … and then make it easy to use.

Code walk-through

Design & Development - 03 Walkthrough from John Keefe on Vimeo.

The code walk-through is embeded above and also posted on Vimeo so you can go at your own pace — pause, speed up, go back. (Note that, oddly, the speed controls are only available on the embedded version above; they don’t show up on the Vimeo site.)

Quick SQL review

Remember, we used SQL to query the database for information.

The syntax, again, is: SELECT [columns or pseudo-columns] FROM [table] ... [more options]

Also remember we can put these queries into our “server.js” file on Glitch.

Making a lookup form on our site

We’re going to provide a service to our audience by making it super easy for them to find out how many dogs are named like theirs.

Block 3

app.get('/name/max', function(request,response) {
  db.all(`SELECT AnimalName, count(AnimalName) AS total FROM nyc_dog_licenses WHERE AnimalName LIKE "max"`, function(err, rows) {
    
    if (err) {
      console.log(err)
    }
    
    response.send(JSON.stringify(rows))
    
  })
})

Block 3A

app.get('/name/:input', function(request,response) {
  db.all(`SELECT AnimalName, count(AnimalName) AS total FROM nyc_dog_licenses WHERE AnimalName LIKE "${request.params.input}"`, function(err, rows) {
    
    if (err) {
      console.log(err)
    }
    
    response.send(JSON.stringify(rows))
    
  })
})

Let’s build a lookup form!

Block 4

<!DOCTYPE html>
<html>
<body>
    
    <form>
        What's your dog's name?<br>
        <input type="text" id="dog_box" onkeyup="hitAPI()"><br>
    </form>
    <div id="answer_line"></div>

    <script>
        function hitAPI() {
          
            var input_name = document.getElementById("dog_box").value
            var xhttp = new XMLHttpRequest()
            
            xhttp.onreadystatechange = function() {
                if (this.readyState == 4 && this.status == 200) {
                    data = JSON.parse(this.responseText)
                  document.getElementById("answer_line").innerHTML = `There are ${data[0].total} dogs named ${input_name} in NYC.`
                }
            }
                
            if (input_name != "") {
              xhttp.open("GET", `/name/${input_name}`, true)
              xhttp.send()
            } else {
              document.getElementById("answer_line").innerHTML = ""
            }
                    
        }
    </script>
</body>
</html>

Now we need a route to our form.

Block 5

In your server.js file, add this route. For organization’s sake, put it right after the first route … the one that sends people to index.html.

app.get('/form', function(request, response) {
  response.sendFile(__dirname + '/views/form.html');
});

Assignment

This is the “Class 03” assignment. Do it now … it’s due on at high noon before Class 04.

Complete the steps above so that your form works and answers the question for any name I enter. Paste the Show URL into the form below and submit it by the deadline. The URL should be like name-bark.glitch.me/form.

If you get stuck try backing up and following my steps again. If you still can’t get it to work, you have options:

  1. Go to the #design-development channel in the social-j Slack and post the “edit” link to your code. That link will begin like this: https://glitch.com/edit/#!/your-project .... Include a short description of what’s not working and what you’ve already tried.
  2. Join my “helpdesk hours” Wednesday at 12:30 pm. The link for that has been added to the syllabus and will be posted in the #design-development chat every Wednesday morning.

Class 4 • Introduction to conversational interfaces

Automatically processing what someone is saying – either in a chat, to a voice assistant, or in an email – is increasingly possible thanks to machine learning. We’ll play with one of these natural language processing tools (such as Dialogflow) to get a handle on how to make it work for you.

Utterances

These are things humans might say or type to a bot.

When building a bot, you need to have a sense of what kinds of things your users might ask — so it helps to declare up front what kinds of things it can answer!

Hi! I’m the Newmark Bark Bot and I can answer questions about dogs and dog ownership in New York City.

Utterances are used to train the bot, and in Dialogflow these called “Training Phrases.”

Intents

Intents are categories of questions we can answer, and usually track with some basic function the human is asking the bot to do. Many utterances can lead to the same intent:

What’s the intent here? We might say it’s air-travel-price.

But intents can be simpler than that, too.

… all mean “yes” to the bot. So we could group these into an intent called response-yes.

And these:

… all mean “no” to the bot. So we could group these into an intent called response-no.

And for our exclamations,

And words like these could mean the response-wow intent.

So you could make bot responses to those intents instead of those words, and let the Natural Language Processor decide which intent is, um, intended.

Entities

Consider our air-travel-price intent from above:

While the intent may be the same (“tell me the price of a plane trip”), the answers are definitely not going to be the same, right?

In order to formulate an answer, we’d need some information:

The portions of the intent you need to answer a specific question are called “entities” (and also “slots” and “elements” and probably many other things, sorry!)

Entities here would include “Miami,” “New York,” “Chicago,” and really any moderately sized city. They’d also include “Delta,” “American,” “JetBlue,” and really any airline. Also “September 18, 2020” and “8 pm” and any other date/time combination. These are all “Entities.”

Another example

Consider the following phrases:

You might call this a get-forecast intent. To answer a “forecast” intent, you need two additional pieces of information: Where and when. Entities!

So a get-forcast intent requires place and time.

We can set up our bot so that if the entity is missing, the bot asks for it.

Contexts

Context is information about where we are (or, more often, where we just were) in the conversation. Context becomes important when the same response has different meanings in different parts of the conversation.

For example, “yes” means something different if the question is “Do you like cats?” instead of “Do you like dogs?”

I like to think of “Contexts” as breadcrumbs we can leave along the way to keep track of the conversation.

Training

To train a bot — really a machine-learning model — we do the following:

  1. Set up an intent.

  2. Come up with as many training phrases as possible to trigger that intent. The more the better! Usually you’ll want at least 10.

  3. Write a response to that intent, which could be another question.

Interestingly, most natural language systems allow you to review its decisions and train it when it performed well and performed poorly.

Introduction to Dialogflow

There are lots of tools out there to use to make bots. We’ll play with Dialogflow.

Follow the instructions in this week’s coding walk-through video:

You can also see the video on the Vimeo site.

Setup

You’ll need to be logged into your J-School account or any Google/Gmail account.

OK, your “agent” is established.

Building a dog bot

Elaboration for all of these steps is in this week’s walk-through video.

Assignment

Check out Google Classroom for the “04 Assignment.” You’ll need the URL from the end of the video. Do it now … it’s due on at high noon on the day of our next class.

If you get stuck try backing up and following my steps again. If you still can’t get it to work, you have options:

  1. Go to the #design-development channel and provide a short description of what’s not working and what you’ve already tried.
  2. Join my “helpdesk hours” Wednesday at 12:30 pm. The link for that has been added to the syllabus and will be posted in the #design-development chat every Wednesday morning.

Class 5 • Talk to me: Making a voice service for Google Home

Let’s make it!

Video walk-through

Design & Development: Class 5 Walk-through from John Keefe on Vimeo.

Update our Glitch app to get ready for requests from Dialogflow

Chart showing flow of messages from human, to dialogflow and to Glitch ... and then back again.

Block 6

app.post('/dialogflow', function(request, response){
  
  var search_dog = request.body.queryResult.queryText
  
  db.all(`SELECT AnimalName, count(AnimalName) AS total FROM nyc_dog_licenses WHERE AnimalName LIKE "${search_dog}" GROUP BY AnimalName`, function(err, rows) {
    
    if (err) {
      console.log(err)
    }
    
    var total_dogs
    
    if (rows.length < 1) {
      total_dogs = "no"
    }  else {
      total_dogs = rows[0].total
    }
    
    var response_phrase = `There are ${total_dogs} dogs named ${search_dog} in New York City.`

    var data ={
      "fulfillmentText": response_phrase
    }

    response.send(JSON.stringify(data))
    
  })
  
})

Tell Dialogflow where to get the answers

Set up the conversation

Do it with your voice!

Note! Your J-School Google account isn’t authorized to do this part of the project. In order to try this yourself, you’ll need to share your bot with another Gmail account.

To do that …

OK, now we’re ready to continue.

Class 6 • Getting & giving insights with Google forms and sheets

We’ll explore how to collect and see the data in Google Sheets.

Pulling data FROM a Google Sheet

To continue in the walkthrough, you should have the following tabs open in your browser:

  1. These notes
  2. Your Glitch app
  3. A new, blank spreadsheet

Code walkthrough video!

Note: If you don’t get the “scary warning” I talk about during the authorization process, then just ignore that part of the video and click “Allow.”

Design & Development 2021 • 06 from John Keefe on Vimeo.

Turning your spreadsheet into an API

OK, now we’re going to add some magic …

Block 7

function doGet(e) {

  var sheetName = "Sheet1";
  var sheetId   = "YOUR_SHEET_ID";

  var book = SpreadsheetApp.openById(sheetId);
  var sheet = book.getSheetByName(sheetName);

  var json = convertSheet2JsonText(sheet);

  return ContentService
          .createTextOutput(JSON.stringify(json))
          .setMimeType(ContentService.MimeType.JSON);
}

function convertSheet2JsonText(sheet) {
  // first line(title)
  var colStartIndex = 1;
  var rowNum = 1;
  var firstRange = sheet.getRange(1, 1, 1, sheet.getLastColumn());
  var firstRowValues = firstRange.getValues();
  var titleColumns = firstRowValues[0];

  // after the second line(data)
  var lastRow = sheet.getLastRow();
  var rowValues = [];
  for(var rowIndex=2; rowIndex<=lastRow; rowIndex++) {
    var colStartIndex = 1;
    var rowNum = 1;
    var range = sheet.getRange(rowIndex, colStartIndex, rowNum, sheet.getLastColumn());
    var values = range.getValues();
    rowValues.push(values[0]);
  }

  // create json
  var jsonArray = [];
  for(var i=0; i<rowValues.length; i++) {
    var line = rowValues[i];
    var json = new Object();
    for(var j=0; j<titleColumns.length; j++) {
      json[titleColumns[j]] = line[j];
    }
    jsonArray.push(json);
  }
  return jsonArray;
}

Before we move on, we need to put your google sheet’s ID in the code!

OK, let’s get this running …

There’s your spreadsheet in JSON format!

Add the “request” library to our project

First we have to add some software to our code. This is a pre-made “library” that allows your Glitch project to go get data from another web service — also known as making a request. That’s why the library we are going to use is called “request.”

const request = require('request');

All set!

Build a “dog news” page

Block 8

View Block 8 code here.

Make a route to the news page

OK, now we need to make our route.

Block 9

app.get("/news", function(req, resp) {
  const url = "PUT_YOUR_GOOGLE_SCRIPT_URL_HERE";
  const options = { json: true };
  request(url, options, (error, res, body) => {
    if (error) {
      return console.log(error);
    }

    if (!error && res.statusCode == 200) {
      // do something with JSON, using the 'body' variable

      let data = { stories: body };

      // NOTE! If your page isn't updating, try increasing
      // this number by one. Just changing it makes the
      // news.html page refresh
      data.cachebust = 1;

      resp.render("news.html", data);
      console.log(data);
    }
  });
});

OK, in that code you just pasted, we need to change PUT_YOUR_GOOGLE_SCRIPT_URL_HERE to the JSON script for your spreadsheet. Go to that tab, copy the URL, and paste it over this PUT_YOUR_GOOGLE_SCRIPT_URL_HERE — making sure to keep the quote marks before and after.

Now try adding /news to your live URL!

Paste this full link into the Google Classroom assignment form to complete this week’s homework.

Assignment

Check out Google Classroom for the “06 Assignment.” You’ll need the URL from the end of the video. Do it now … it’s due on at high noon on the day of our next class.

If you get stuck try backing up and following my steps again. If you still can’t get it to work, you have options:

  1. Go to the #design-development channel and provide a short description of what’s not working and what you’ve already tried.
  2. Join my “helpdesk hours” Wednesday at 12:30 pm. The link for that has been added to the syllabus and will be posted in the #design-development chat every Wednesday morning.

Class 7 • Data security: For you, your sources, and your audience

Assignment

Check out Google Classroom for the “07 Assignment.” You need to do one of the security steps John described, submit it and also post it in the #design-development Slack channel.

Class 8 • Chatbots for collecting information in real time

Block 8A

const data = Date() + "," + search_dog + "\n"

fs.appendFile("searches.csv", data, err => {
  if (err) {
    console.log(err);
  }
});

Class 9 • Identifying and communicating your hopes and dreams

Class 10 • Make bots to do your bidding

We did a review of IFTTT, Zapier, Google Alerts, and Klaxon to help you monitor the internet and collect data.

Assignment

  1. Create a new applet/bot using IFTTT (at https://ifttt.com)
  2. Go to the applet’s “settings”
  3. Take a screenshot of the area where I can see both the trigger and the action.
  4. Submit that screenshot using the Google Classroom assignment page.

Class 11 • APIs, scraping and other data sources

You’ve made an API!

https://newmark-bark.glitch.me/name/juno

Note that this is the endpoint for our names API:

https://newmark-bark.glitch.me/name/

And this is our “first 10” endpoint:

https://newmark-bark.glitch.me/first10/

Using an API: Coinbase API + Google Spreadsheet

Let’s load this into a Google Spreadsheet, just because we can.

Steps:

Block 11-A

function onOpen() {

    // this code runs when the spreadsheet is opened
    var ui = SpreadsheetApp.getUi();
    ui.createMenu('API')
      .addItem('Update Bitcoin','callCoinbase')
      .addToUi();
      
}

function callCoinbase() {

    // Call coinbase for the latest data
    var response = UrlFetchApp.fetch("https://api.coinbase.com/v2/prices/BTC-USD/spot");

    var coinbase = JSON.parse(response.getContentText());
    var sheet = SpreadsheetApp.getActiveSheet();
    
    sheet.getRange(1,1).setValue([coinbase.data.amount]);
  
}

Let’s review the code.

Then …

More code snippets:

Add the currency value … by adding this line inside the callCoinbase function:

sheet.getRange(1,2).setValue([coinbase.data.currency]);

Add the timestamp!

sheet.getRange(1,3).setValue(Date());

Make it append to the end of the list … (calculate the “row” as the last row plus 1) … by replacing the current sheet.getRange... lines with these three:

var new_row = sheet.getLastRow() + 1;
sheet.getRange(new_row, 1).setValue([coinbase.data.amount]);
sheet.getRange(new_row, 2).setValue([coinbase.data.currency]);
sheet.getRange(new_row, 3).setValue(Date());

Extra info

A custom earthquake page using USGS data

Scraping-but-not-scraping

Actual Scraping:

Drawn from a Medium post by Julia Kho.

Hop over to this Python notebook on Google Colab.

More exploration

Assignment

Check out Google Classroom for the “11 Assignment.” Look for an API you want to use and fill out the form linked in that assignment.

Notes for subsequent classes will be posted prior to the class itself.