Skip to content

Commit

Permalink
Continue #25: Parse csv into array
Browse files Browse the repository at this point in the history
As of this commit, the imported file is sitting in a JS array in the
browser.  We still need to check the SSN against the API for duplicates,
and to POST each non-duplicate record via the API.

I've done some of the parsing inside the controller, which may be bad
form.  I just split the csv so that the headers and footers aren't sent
back to the browser.  Then, in the client-client (as we're calling it),
I use the MIT-licensed "PapaParse" library to transform the csv lines
into a JS array.

I also fixed a call in index.jade so that it refers to PapaParse instead
of to a different jquery library that I decided not to use.

I added papaparse.js so that we could use the library.  It has a license
header and I don't think we need any other files from that repo.  They
do offer a minified version, if we prefer.
  • Loading branch information
cecilia-donnelly committed Aug 4, 2015
1 parent 3484230 commit 5a8de1e
Show file tree
Hide file tree
Showing 4 changed files with 1,418 additions and 8 deletions.
10 changes: 7 additions & 3 deletions app/controllers/core.server.controller.js
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,6 @@
* Module dependencies.
*/
exports.index = function(req, res) {
var qs = require('querystring');
if (req.method == "POST"){
var body = '';
req.on('data', function (data) {
Expand All @@ -17,8 +16,13 @@ exports.index = function(req, res) {
// naming this "file_data" because I assume that we will
// only be POST-ing imported files. This may change in the
// future.
var file_data = qs.parse(body);
res.send(file_data);

// split out the contents from the headers
var csv_body = body.split("Content-Type: text/csv\r\n\r\n");
// and from the footers
var csv_only = csv_body[1].split("\r\n\r\n-----------------------------");
var csv_contents = csv_only[0];
res.send(csv_contents);
});
}
else{
Expand Down
2 changes: 1 addition & 1 deletion app/views/index.jade
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ html(lang="en")
script(type='text/javascript', src='js/lib/FileSaver.js/FileSaver.min.js')
script(type='text/javascript', src='js/lib/Blob.js/Blob.min.js')
script(type='text/javascript', src='js/lib/sampleData.js')
script(type='text/javascript', src='js/jquery.csv.js')
script(type='text/javascript', src='js/papaparse.js')
script(type='text/javascript', src='js/index.js')

body
Expand Down
11 changes: 7 additions & 4 deletions public/js/index.js
Original file line number Diff line number Diff line change
Expand Up @@ -522,6 +522,7 @@ $(function() {

function importAll() {
// get file
// need to check to make sure that it is a csv file, here
var formData = new FormData();
formData.append('file', $('input[type=file]')[0].files[0]);
$.ajax("/upload", {
Expand All @@ -530,12 +531,14 @@ $(function() {
processData: false,
contentType: false
}).done(function(response) {
console.log(response);
// read each line of file
var result = Papa.parse(response);
console.log("DEBUG: " + result);
// loop through array
// for each line, check whether ssn exists via API
// if it does, put that line in a list of duplicates
// if it doesn't, POST that line to the API
// display any duplicates to the user
// if it does, put that record in a list of possible duplicates
// if it doesn't, POST that record to the API
// display the list of possible duplicates to the user
});
}

Expand Down
Loading

2 comments on commit 5a8de1e

@kfogel
Copy link
Member

@kfogel kfogel commented on 5a8de1e Aug 5, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One comment:

When adding a third-party library (PapaParse), include full provenance information either in the commit message or in a README or some other easily-findable doc near where we put the library. See commit 7ce43ec for an example of the first way, which would be fine here.

I don't have an example handy of the second way, but it would be like if you had a third-party/ directory and you had a third-party/README.md file explaining where everything came from, then you could put the provenance information in there instead of into the commit message (because now it would be part of the commit's diff itself, so it's still "in the commit", even though not in the commit message).

@cecilia-donnelly
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kfogel Thank you! I wasn't sure how much information we needed about PapaParse. Thanks for pointing me to your commit.

Please sign in to comment.