ENH: add .formhandler()

CHANGELOG.md

+# Change log
+
+- `0.1.0`: Initial release with:
+    - [$.urlfilter](#urlfilter) changes URL query parameters when clicked. Used to filter data
+    - [g1.url.parse](#urlparse) parses a URL into a structured object
+    - [g1.url.join](#urljoin) joins two URLs
+    - [g1.url.update](#urlupdate) updates a URL's query parameters
+    - [$.dispatch](#dispatch) is like [trigger](https://api.jquery.com/trigger/) but sends a native event (triggers non-jQuery events too)
+- `0.2.0`: Added
+    - [$.template](#template) renders lodash templates
+    - [L.TopoJSON](#ltopojson) loads TopoJSON files just like GeoJSON
+- `0.2.1`:
+    - [$.template](#template) triggers a `template` event with the data and target nodes.
+      It also accepts a `src=` attribute that points to a template file.
+- `0.2.2`:
+    - [$.template](#template) can be applied to a container element like `body`. It supports
+      `data-selector=` which defaults to `script[type="text/html"]`

CONTRIBUTING.md

+# File structure
+
+- `dist/` has output files. It is re-created via  `npm run build`. It has:
+  - [g1.js](dist/g1.js) - non-minified source created via rollup on [index.js](index.js)
+  - [g1.min.js](dist/g1.min.js) - minified source created via rollup on [index.js](index.js)
+  - `<module>.min.js` for each module - minified source created via rollup on `index-<module>.js`
+- `./` has setup files.
+  - [index.js](index.js) is the rollup configuration to create the full `g1` package
+  - `index-<module>.js` is the rollup configuration to create each module
+  - Other support files
+- `src/` has source files. This includes:
+  - `<module>.js` - underlying source for each module, which may import other dependencies. TODO: rename jquery.* to this
+  - `<library>.js` - for internally used libraries
+- `test/` has test cases. It is run via `npm test`. It has:
+  - `test-<module>.html` for each browser module
+  - `test-<library>.js` for each library that can be tested directly on node.js
+  - `server.js` runs tests on [Puppeteer](https://github.com/GoogleChrome/puppeteer)
+  - `tape.js` is dynamically created using browserify to help with test cases. This is not committed
+  - Other test dependencies
+
+# Release
+
+To set up locally, clone this repo and run:
+
+    yarn install
+    npm run build     # Optional: build the dist/ directory
+    npm test          # Optional: run unit tests
+
+To publish a new version on npm:
+
+    # Run tests on dev branch
+    git checkout dev
+    npm test
+
+    # Update package.json version
+    # Update README.md change log
+    # Ensure that there are no build errors on the server
+    git commit -m"DOC: Release version x.x.x"
+    git push
+
+    # Merge into dev branch
+    git checkout master
+    git merge dev
+    git tag -a v0.x.x           # Add a one-line summary
+    git push --follow-tags
+    npm publish               # as sanand0
+
+    git checkout dev

README.md

@@ -12,17 +12,21 @@ To use all features, add this to your HTML:
 
     <script src="node_modules/g1/dist/g1.min.js"></script>
 
-Or import one of the individual libraries below. Each provides a set of utilities.
+Or import one of the individual libraries below. [g1.min.js](dist/g1.min.js) has all of these
+functions. [g.js](dist/g.js) is an un-minified version for debugging.
 
-- `urlfilter.min.js`: URL manipulation library:
+- [urlfilter.min.js](dist/urlfilter.min.js): URL filtering library
     - [$.urlfilter](#urlfilter) changes URL query parameters when clicked. Used to filter data.
     - [g1.url.parse](#urlparse) parses a URL into a structured object
     - [g1.url.join](#urljoin) joins two URLs
     - [g1.url.update](#urlupdate) updates a URL's query parameters
-- `jquery.min.js`: jQuery utilities:
+- [formhandler.min.js](dist/formhandler.min.js): Table renderer using [FormHandler](https://learn.gramener.com/guide/formhandler/)
+    - [$.formhandler](#formhandler) renders a HTML table from a [FormHandler URL](https://learn.gramener.com/guide/formhandler/)
+- [template.min.js](dist/template.min.js): template library
     - [$.template](#template) renders lodash templates. Requires [lodash](https://lodash.com/)
+- [event.min.js](dist/event.min.js): event library
     - [$.dispatch](#dispatch) is like [trigger](https://api.jquery.com/trigger/) but sends a native event (triggers non-jQuery events too)
-- `leaflet.min.js`: Leaflet utilities:
+- [leaflet.min.js](dist/leaflet.min.js): Leaflet utilities
     - [L.TopoJSON](#ltopojson) loads TopoJSON files just like GeoJSON. Requires [topojson](https://github.com/topojson/topojson)
 
 ## $.urlfilter
@@ -101,6 +105,95 @@ This activates all `.urlfilter` classes as below:
 ```
 
 
+## $.formhandler
+
+```html
+<div class="formhandler" src="formhandler-url"></div>
+<script>
+$('.target').formhandler()
+</script>
+```
+
+### $.formhandler attributes
+
+- `data-src`: [FormHandler](https://learn.gramener.com/guide/formhandler/) URL endpoint
+- `data-columns="col1, col2, ..."`: comma-separated column names to display (spaces stripped)
+- `data-page-size="100"`: number of rows per page
+- `data-size-values="10, 20, 50, 100"`: comma-separated number of page size values
+- `data-export-formats="csv, xlsx, html"`: comma-separated export formats
+
+### $.formhandler options
+
+Data attribute defaults can be set via the options. For example,
+`data-page-size` defaults to `pageSize`, and `data-columns` to `columns`.
+
+- `columns`: comma-separated column names to display, or a list of objects with these keys:
+    - `name`: column name
+    - `title`: display name of the column. Defaults to the same value as `name`
+    - `format`: function to format the value. Defaults sensibly based on type
+    - `sort`: `true` / `false` / options dict with:
+      - `show`: `true` (default) / `false`
+      - `operators`: `{'': 'Sort ascending', '-': 'Sort descending'}` (default)
+    - `filters`: `true` (default) / `false` / options dict with:
+      - `show`: `true` (default) / `false`
+      - `operators`: `{'', 'Equals...', '!', 'Does not equal...', ...}`.
+        The default list of operators is based on the auto-detected type of the column.
+    - `unique`: TODO: {dict of query parameter and display value} or [list of values] or function?
+    - `hideable`: `true` (default) / `false`
+- `table`: Shows the table control. Can be `true` (default) / `false`
+- `page`: Shows the page control. Can be `true` (default) / `false`.
+- `pageSize`: page size. Defaults to 100
+- `size`: Shows the page size control. Can be `true` (default) / `false`
+- `sizeValues`: Allowed page size values. Defaults to `[10, 20, 50, 100, 500, 1000]`
+- `export`: Shows the export control. Can be `true` (default) / `false`
+- `exportFormats`: {xlsx: 'Excel'}
+- `search`: Shows the search control. Can be `true` (default) / `false`
+- `filters`: Shows the applied filters control. Can be `true` (default) / `false`
+- `loading`: TODO: loading indicator
+- `namespace`: ''. URL prefix / namespace, if there are multiple tables on the same page
+- `urltarget`: '#'. Allows '#' / ''
+
+**Advanced**. Each component can have a target which specifies a selector. For
+e.g., to render the export button somewhere else, use
+`data-export-target=".navbar-export"`. This replaces the `.navbar-export`
+contents with the export button. (It searches within the table container for `.navbar-export` first, and if not found, searches everywhere.) Available targets are:
+- `tableTarget`
+- `pageTarget`
+- `sizeTarget`
+- `exportTarget`
+- `filtersTarget`
+- `searchTarget`
+
+**Advanced**: Each component's template string can be over-ridden. For example,
+`data-search-template="<input type='search'>"` will replace the search template
+with a simple input. Available template strings are:
+- `tableTemplate`
+- `pageTemplate`
+- `sizeTemplate`
+- `exportTemplate`
+- `filtersTemplate`
+- `searchTemplate`
+
+## $.formhandler examples
+
+Add a simple table using the FormHandler at `./data` that shows specific columns
+with a page size of 10 rows, and does not show the export filter.
+
+```html
+<div class="formhandler"
+     src="./data"
+     data-columns="id,country,state,sales"
+     data-page-size="10"
+     data-export="false"
+></div>
+<script>
+  $('.formhandler').formhandler()
+</script>
+```
+
+TODO: FormHandler on-done hook
+
+
 ## $.template
 
 ```html
@@ -379,57 +472,3 @@ g1.url.parse('/?a=1&b=2&c=3&d=4')       // Update this URL
           'a=del&b=toggle&c=add')       // Delete ?a, Toggle ?b, add ?c, update ?d (default)
 // Returns /?b=3&c=3&c=6&d=7
 ```
-
-
-# Change log
-
-- `0.1.0`: Initial release with:
-    - [$.urlfilter](#urlfilter) changes URL query parameters when clicked. Used to filter data
-    - [g1.url.parse](#urlparse) parses a URL into a structured object
-    - [g1.url.join](#urljoin) joins two URLs
-    - [g1.url.update](#urlupdate) updates a URL's query parameters
-    - [$.dispatch](#dispatch) is like [trigger](https://api.jquery.com/trigger/) but sends a native event (triggers non-jQuery events too)
-- `0.2.0`: Added
-    - [$.template](#template) renders lodash templates
-    - [L.TopoJSON](#ltopojson) loads TopoJSON files just like GeoJSON
-- `0.2.1`:
-    - [$.template](#template) triggers a `template` event with the data and target nodes.
-      It also accepts a `src=` attribute that points to a template file.
-- `0.2.2`:
-    - [$.template](#template) can be applied to a container element like `body`. It supports
-      `data-selector=` which defaults to `script[type="text/html"]`
-
-# Release
-
-Clone the repo and run:
-
-    yarn install
-
-To build locally, run:
-
-    npm run build
-
-To test locally, run:
-
-    npm test
-
-To publish a new version on npm:
-
-    # Run tests on dev branch
-    git checkout dev
-    npm test
-
-    # Update package.json version
-    # Update README.md change log
-    # Ensure that there are no build errors on the server
-    git commit -m"DOC: Release version x.x.x"
-    git push
-
-    # Merge into dev branch
-    git checkout master
-    git merge dev
-    git tag -a v0.x.x           # Add a one-line summary
-    git push --follow-tags
-    npm publish               # as sanand0
-
-    git checkout dev

gramex.yaml

+# Run Gramex in this directory to test formhandler
+
+url:
+  formhandler-data:
+    pattern: /formhandler-data
+    handler: FormHandler
+    kwargs:
+      url: test/formhandler.csv

index-event.js

 export { version } from './src/package.js'
-import { dispatch } from './src/jquery.dispatch.js'
+import { dispatch } from './src/event.js'
 
 if (typeof jQuery != 'undefined') {
   jQuery.extend(jQuery.fn, {

index-formhandler.js

+export { version } from './src/package.js'
+import { formhandler } from './src/formhandler.js'
+// FormHandler requires $().urlfilter
+export { url } from './index-urlfilter.js'
+
+if (typeof jQuery != 'undefined') {
+  jQuery.extend(jQuery.fn, {
+    formhandler: formhandler
+  })
+}

index-template.js

 export { version } from './src/package.js'
-import { template } from './src/jquery.template.js'
+import { template } from './src/template.js'
 
 if (typeof jQuery != 'undefined') {
   jQuery.extend(jQuery.fn, {

index-urlfilter.js

@@ -8,7 +8,7 @@ export var url = {
   update: update
 }
 
-import { urlfilter } from './src/jquery.urlfilter.js'
+import { urlfilter } from './src/urlfilter.js'
 
 if (typeof jQuery != 'undefined') {
   jQuery.extend(jQuery.fn, {

index.js

-export {version} from './src/package.js'
-export {url} from './index-urlfilter.js'
+export { version } from './src/package.js'
+export { url } from './index-urlfilter.js'
 import './index-template.js'
+import './index-formhandler.js'
 import './index-event.js'
 import './index-leaflet.js'

package.json

@@ -18,18 +18,22 @@
     "prepublishOnly": "npm test"
   },
   "devDependencies": {
+    "bootstrap": "4.0.0-beta.3",
     "browserify": "14",
     "component-emitter": "1",
     "eslint": "^4",
     "express": "4",
     "faucet": "^0.0.1",
+    "html-minifier": "3",
     "jquery": "3",
     "json2module": "0.0",
     "leaflet": "1",
+    "popper.js": "1",
     "puppeteer": "0.13",
     "rimraf": "2",
     "rollup": "0.52",
     "rollup-plugin-uglify": "2",
+    "rollup-pluginutils": "2",
     "tap-merge": "0.3",
     "tape": "4",
     "topojson": "3",

rollup-plugin-htmlparts.js

+/*
+Export all templates in a HTML file as variables. Their value is their inner HTML, minified.
+
+Usage in rollup.config.js:
+
+    {
+      input: "index",
+      plugins: [htmlparts('template.html')]
+    }
+
+Usage in index.js:
+
+    import { id1, id2, id3 } from './template.html'
+*/
+
+'use strict'
+
+const rollupPluginutils = require('rollup-pluginutils')
+const htmlMinifier = require('html-minifier')
+
+export default function htmlparts(includes) {
+  const filter = rollupPluginutils.createFilter([includes], [])
+  return {
+    name: 'htmlparts',
+    transform: function(code, id) {
+      if (filter(id)) {
+        const matches = code.match(/<!-- var .*? -->[\s\S]*?<!-- end -->/igm)
+        const result = matches.map(function(match) {
+          const lines = match.split(/\n/)
+          const name = lines[0].split(/ /)[2]
+          const html = htmlMinifier.minify(lines.slice(1, -1).join('\n'), {
+            collapseBooleanAttributes: true,
+            collapseInlineTagWhitespace: true,
+            collapseWhitespace: true,
+            decodeEntities: true,
+            removeComments: true,
+          })
+          return 'export var ' + name + ' = ' + JSON.stringify(html) + ';'
+        }).join('\n')
+        return { code: result, map: { mappings: '' } }
+      }
+    }
+  }
+}

rollup.config.js

 import uglify from 'rollup-plugin-uglify'
+import htmlparts from './rollup-plugin-htmlparts.js'
 
 export default [
   {
     input: "index",
+    plugins: [htmlparts('src/formhandler.template.html')],
     output: { file: "dist/g1.js", format: "umd", name: "g1" }
   },
   {
     input: "index",
-    plugins: [ uglify() ],
+    plugins: [htmlparts('src/formhandler.template.html'), uglify()],
     output: { file: "dist/g1.min.js", format: "umd", name: "g1" }
   },
   {
     input: "index-urlfilter",
-    plugins: [ uglify() ],
+    plugins: [uglify()],
     output: { file: "dist/urlfilter.min.js", format: "umd", name: "g1" }
   },
+  {
+    input: "index-formhandler",
+    plugins: [htmlparts('src/formhandler.template.html'), uglify()],
+    output: { file: "dist/formhandler.min.js", format: "umd", name: "g1" }
+  },
   {
     input: "index-template",
     plugins: [uglify()],
-    output: { file: "dist/template.min.js", format: "umd", name: "g1._modules" }
+    output: { file: "dist/template.min.js", format: "umd", name: "g1" }
   },
   {
     input: "index-event",
     plugins: [uglify()],
-    output: { file: "dist/event.min.js", format: "umd", name: "g1._modules" }
+    output: { file: "dist/event.min.js", format: "umd", name: "g1" }
   },
   {
     input: "index-leaflet",
-    plugins: [ uglify() ],
-    output: { file: "dist/leaflet.min.js", format: "umd", name: "g1._modules" }
+    plugins: [uglify()],
+    output: { file: "dist/leaflet.min.js", format: "umd", name: "g1" }
   }
 ]

src/formhandler.js

+// Import HTML sections as variables using rollup-plugin-htmlparts.js.
+// See rollup.config.js for usage.
+// Each <!-- var name --> section is imported as a minified variable string.
+import * as default_templates from './formhandler.template.html'
+import { parse } from './url.js'
+
+// Render components in this order. The empty component is the root component.
+var components = ['', 'table', 'page', 'size', 'export', 'search', 'filters']
+var default_options = {
+  table: true,
+  page: true,
+  pageSize: 100,
+  size: true,
+  sizeValues: [10, 20, 50, 100, 500, 1000],
+  export: true,
+  exportFormats: {
+    xlsx: 'Excel',
+    csv: 'CSV',
+    json: 'JSON',
+    html: 'HTML'
+  },
+  search: true,
+  filters: true
+}
+var meta_headers = ['filters', 'ignored', 'excluded', 'sort', 'offset', 'limit', 'count']
+
+export function formhandler(js_options) {
+  if (!js_options)
+    js_options = {}
+
+  this.each(function() {
+    var $this = $(this)
+    // Convert all .urlfilter classes into url filters that update location.hash
+    $this.urlfilter({
+      selector: '.urlfilter, .page-link',
+      target: '#',
+      remove: true                          // auto-remove empty values
+    })
+
+    // Pre-process options
+    var options = $.extend({}, default_options, js_options, $this.data())
+    options.src = $this.attr('src')
+    if (!options.columns)
+      options.columns = []
+    else if (typeof options.columns == 'string')
+      options.columns = _.map(options.columns.split(/\s*,\s*/), function(col) { return { name: col } })
+
+    // Compile all templates
+    var template = {}
+    _.each(components, function (name) {
+      var tmpl = options[name ? name + 'Template' : 'template'] || default_templates['template_' + name] || 'NA'
+      template[name] = _.template(tmpl)
+    })
+
+    function draw_table(data, args, meta) {
+      // Add metadata
+      meta.rows = data.length,
+      meta.columns = data.length ? _.map(data[0], function (val, col) { return { name: col } }) : []
+      // Render all components into respective targets
+      var template_data = {
+        data: data, meta: meta, args: args, options: options, idcount: 0, parse: parse
+      }
+      _.each(components, function(name) {
+        // Disable components if required. But root component '' is always displayed
+        if (name && !options[name])
+          return
+
+        var target
+        // The root '' component is rendered into $this.
+        if (!name)
+          target = $this
+        else {
+          // Rest are rendered into .<component-name> under $this
+          var selector = options[name + 'Target'] || '.' + name
+          target = $(selector, $this)
+          // But if they don't exist, treat the selector as a global selecctor
+          if (target.length == 0)
+            target = $(selector)
+        }
+        target.html(template[name](template_data))
+      })
+    }
+
+    function render() {
+      var url_args = parse(location.hash.replace(/^#/, '')).searchList
+      // TODO: Show a loading indicator
+      // Create arguments passed to the FormHandler. Override with the user URL args
+      var args = _.extend({
+        c: options.columns.map(function(d) { return d.name }),
+        _limit: options.pageSize,
+        _format: 'json',
+        _meta: 'y'
+      }, url_args)
+      $.ajax(options.src, {
+        dataType: 'json',
+        data: args,
+        traditional: true
+      }).done(function(data, status, xhr) {
+        var meta = {}
+        _.each(meta_headers, function(header) {
+          var val = xhr.getResponseHeader('Fh-Data-' + header)
+          if (val !== null)
+            meta[header] = JSON.parse(val)
+        })
+        // TODO: Stop loading indicator
+        draw_table(data, args, meta)
+      }).fail(function() {
+        // TODO
+      })
+    }
+    // Re-render every time the URL changes
+    $(window).on('hashchange', render)
+    // Initialize
+    render()
+  })
+}

src/formhandler.template.html

+<!--
+Each "var ..." template is embedded into formhandler.js as a minified HTML string variable.
+This uses rollup-plugin-htmlparts.js: our custom rollup plugin.
+
+Each template receives these variables:
+
+- data: JSON data from FormHandler
+- meta: Meta HTTP headers from FormHandler
+- options: Options passed to $().formhandler() (including defaults)
+- args: URL query parameters used to retrieve data
+-->
+
+<!-- This is the root template that renders all other components on this page -->
+<!-- var template_ -->
+  <div class="formhandler">
+    <div class="formhandler-table-header d-flex justify-content-between">
+      <div class="d-flex">
+        <div class="page mb-2 mr-2"></div>
+        <div class="size mb-2 mr-2"></div>
+      </div>
+      <div class="d-flex">
+        <div class="filters mb-2"></div>
+        <div class="export mb-2"></div>
+      </div>
+    </div>
+    <div class="table"></div>
+  </div>
+<!-- end -->
+
+<!-- var template_table -->
+<% var filtered_cols = args['_c'] && args['_c'].length != options.columns.length ? options.columns.filter(function(col) {return args['_c'].indexOf('-'+col.name) < 0}) : options.columns %>
+<% var cols = options.columns.length ? filtered_cols : meta.columns %>
+<table class="table table-sm table-striped">
+  <thead>
+    <% _.each(cols, function(colinfo) { %>
+      <% var sort_asc = args['_sort'] == colinfo.name,
+             sort_desc = args['_sort'] == '-' + colinfo.name %>
+      <th class="<%- sort_asc ? 'table-primary' : sort_desc ? 'table-danger' : '' %>">
+        <div class="dropdown">
+          <a href="#" class="dropdown-toggle text-nowrap" id="dropdownMenuButton-<%- idcount++ %>" data-toggle="dropdown" aria-haspopup="true" aria-expanded="false">
+            <%- colinfo.title || colinfo.name %>
+          </a>
+          <div class="dropdown-menu" data-col="<% colinfo.name %>" aria-labelledby="dropdownMenuButton-<%- idcount %>">
+            <a class="dropdown-item urlfilter <%- sort_asc ? 'active': '' %>" href="?_sort=<%- sort_asc ? '' : colinfo.name %>">Sort ascending</a>
+            <a class="dropdown-item urlfilter <%- sort_desc ? 'active': '' %>" href="?_sort=<%- sort_desc ? '' : '-' + colinfo.name %>">Sort descending</a>
+            <div class="dropdown-divider"></div>
+            <a class="dropdown-item urlfilter" href="?_c=-<%- colinfo.name %>" data-mode="add">Hide</a>
+          </div>
+        </div>
+      </th>
+    <% }) %>
+  </thead>
+  <tbody>
+    <% _.each(data, function(row) { %>
+      <tr>
+        <% _.each(cols, function(colinfo) { %>
+          <td>
+            <a class="urlfilter" href="?<%- colinfo.name %>=<%- row[colinfo.name] %>&amp;_offset=">
+              <%- row[colinfo.name] %>
+            </a>
+          </td>
+        <% }) %>
+      </tr>
+    <% }) %>
+  </tbody>
+</table>
+<!-- end -->
+
+<!-- var template_page -->
+<% var page = 1 + Math.floor(meta.offset / meta.limit),
+       last_page = 'count' in meta ? Math.floor((meta.count + meta.limit - 1) / meta.limit) : meta.rows < meta.limit ? page : null,
+       lo = Math.max(page - 2, 1),
+       hi = last_page !== null ? Math.min(last_page, page + 2) : page + 2