Commit 8258e5ab authored by S Anand's avatar S Anand
Browse files

ENH: create g1.datafilter. Fixes #3

parent 96be7cd6
Pipeline #46202 passed with stage
in 2 minutes and 5 seconds
......@@ -70,16 +70,16 @@ link *updates* the current page URL instead of replacing it.
For example:
```html
<a class="urlfilter" href="city=NY"> Change ?city= to NY</a>
<a class="urlfilter" href="city=NY" data-mode="add"> Add ?city= to NY</a>
<a class="urlfilter" href="city=NY" data-mode="del"> Remove NY from ?city=</a>
<a class="urlfilter" href="city=NY" data-mode="toggle"> Toggle NY in ?city=</a>
<a class="urlfilter" href="city=NY" data-target="pushState">Change ?city= to NY using pushState</a>
<a class="urlfilter" href="city=NY" data-target="#"> Change location.hash, i.e. #?city= to NY</a>
<a class="urlfilter" href="city=NY" data-target="iframe"> Change iframe URL ?city= NY</a>
<a class="urlfilter" href="?city=NY"> Change ?city= to NY</a>
<a class="urlfilter" href="?city=NY" data-mode="add"> Add ?city= to NY</a>
<a class="urlfilter" href="?city=NY" data-mode="del"> Remove NY from ?city=</a>
<a class="urlfilter" href="?city=NY" data-mode="toggle"> Toggle NY in ?city=</a>
<a class="urlfilter" href="?city=NY" data-target="pushState">Change ?city= to NY using pushState</a>
<a class="urlfilter" href="?city=NY" data-target="#"> Change location.hash, i.e. #?city= to NY</a>
<a class="urlfilter" href="?city=NY" data-target="iframe"> Change iframe URL ?city= NY</a>
<iframe src="?country=US"></iframe>
<a class="urlfilter" href="city=NY" data-target=".block"> Use AJAX to load ?city=NY into .block</a>
<a class="urlfilter" href="?city=NY" data-target=".block"> Use AJAX to load ?city=NY into .block</a>
<div class="block" src="?country=US"></div>
<script>
$('body').urlfilter() // Activate all the .urlfilter elements above
......@@ -160,8 +160,11 @@ Highlight containers use these attributes:
## datafilter
`g1.datafilter(data, filters)` returns the filtered data based on the filters. While urlfilter on [$.formhandler](#formhandler) applies filtering on data server side, `datafilter` applies urlfilter on frontend loaded data.
`g1.datafiilter(data, filters)` returns the filtered data based on the filters. For example:
`g1.datafilter(data, filters)` returns the filtered data based on the filters. While urlfilter on [$.formhandler](#formhandler) applies filtering on data server side, `datafilter` applies urlfilter on frontend loaded data.
For example:
```js
var data = [
......@@ -173,46 +176,83 @@ var data = [
{"ID": "5", "product": "Light", "sales": "100", "city": "London"}
]
g1.datafilter(data, [{col: 'sales', op: '>', val: 100},
{col: 'city', op: 'in', val: ['London', 'NY']},
{col: 'product', val: 'Fan'}])
g1.datafilter(data, {
'sales>': ['100'],
'city': ['London', 'NJ'],
'product': ['Fan']
})
// Returns [{"ID": "3", "product": "Fan", "sales": "120", "city": "NJ"}, {"ID": "4", "product": "Fan", "sales": "130", "city": "London"}]
```
## datafilter options
g1.datafilter with multiple datasets:
datafilter() contains three parameters:
```js
var data1 = [
{"ID": "1", "product": "Fan", "sales": "100", "city": "NY"},
{"ID": "2", "product": "Fan", "sales": "80", "city": "London"},
{"ID": "3", "product": "Fan", "sales": "120", "city": "NJ"},
{"ID": "4", "product": "Fan", "sales": "130", "city": "London"},
{"ID": "5", "product": "Light", "sales": "500", "city": "NY"},
{"ID": "5", "product": "Light", "sales": "100", "city": "London"}
]
- data: a list of objects
- filters: a list of objects, that will contains the below keys:
- col: column to be filtered.
- op: operator to be applied for filteration. default: `=`
- val: value of the selected column
- options: a dictionary that contains the below keys:
- limit: result is limited to. default: `1000`
- offset: filtering data should start from. default: `0`
- sort: a list of objects, that will contains the below keys:
- column: column to be sorted
- order: asc or desc. default: `asc`
- columns: a list of objects, that will contains the below keys:
- allow: a list of column names to be returned in the filtered data
- not: a list of column names to be skiped in the filtered data
var data2 = [
{"ID": "1", "city": "NY"},
{"ID": "2", "city": "London"},
{"ID": "3", "city": "NJ"},
{"ID": "4", "city": "London"},
{"ID": "5", "city": "NY"},
{"ID": "5", "city": "London"}
]
Rules:
g1.datafilter(data, {
'datsetname2:city': ['London', 'NJ'],
'sales>~': [100],
'datsetname1:product': ['Fan']
}, 'datsetname1'))
// ignores datsetname2:city: ['London', 'NJ']
// Returns [{"ID": "3", "product": "Fan", "sales": "120", "city": "NJ"}, {"ID": "4", "product": "Fan", "sales": "130", "city": "London"}, {"ID": "1", "product": "Fan", "sales": "100", "city": "NY"}]
var data2 = [
{"ID": "1", "city": "NY"},
{"ID": "2", "city": "London"},
{"ID": "3", "city": "NJ"},
{"ID": "4", "city": "London"},
{"ID": "5", "city": "NY"},
{"ID": "5", "city": "London"}
]
g1.datafilter(data2, {
'datsetname2:city': ['London', 'NJ'],
'sales>~': [100],
'datsetname1:product': ['Fan']
}, 'datsetname2'))
// ignores datsetname1:product: ['Fan']
- the key `op` may contains any one of the below values:
- `=`
- `!=`
- `>`
- `<`
- `>=`
- `<=`
- `~`
- `!~`
- `in`
// Return [
// {"ID": "2", "city": "London"},
// {"ID": "3", "city": "NJ"},
// {"ID": "4", "city": "London"},
// {"ID": "5", "city": "London"}
// ]
```
## datafilter options
datafilter() contains three parameters:
- `data`: a list of objects
- `filters`: [formhandler filters][formhandler-filters] extracted using
`g1.url.parse(url).searchList`. This converts `?city=London&sales>=1000` to
this filters object: `{'city': ['London'], 'sales>': ['1000']}`
- `namespace`: (optional) If `namespace` is not given, all filters are applied
on the dataset. If `namespace` is given, only filters that begin with
`<namespace>:` or that have no `:` are applied
[formhandler-filters]: https://learn.gramener.com/guide/formhandler/#formhandler-filters
## $.formhandler
......@@ -574,8 +614,8 @@ url.join(another_url, {query: false, hash: false})
For example:
```js
g1.url.parse('/').join('/?x=1#y=1', {hash: false}).toString() == '/?x=1';
g1.url.parse('/').join('/?x=1#y=1', {query: false}).toString() == '/#y=1';
g1.url.parse('/').join('/?x=1#y=1', {hash: false}).toString() == '/?x=1'
g1.url.parse('/').join('/?x=1#y=1', {query: false}).toString() == '/#y=1'
```
......
export { version } from './src/package.js'
export { datafilter } from './src/datafilter.js'
export { version } from './src/package.js'
export { types } from './src/types.js'
export { url } from './index-urlfilter.js'
export { datafilter } from './src/datafilter.js'
export { scale } from './src/scale.js'
export { datafilter } from './index-datafilter.js'
import './index-highlight.js'
import './index-template.js'
import './index-formhandler.js'
......
......@@ -12,6 +12,11 @@ export default [
plugins: [htmlparts('src/formhandler.template.html'), uglify()],
output: { file: "dist/g1.min.js", format: "umd", name: "g1" }
},
{
input: "index-datafilter",
plugins: [uglify()],
output: { file: "dist/datafilter.min.js", format: "umd", name: "g1" }
},
{
input: "index-urlfilter",
plugins: [uglify()],
......
var operators = {
'=': function(value, compare_with) { return value == compare_with },
'!=': function(value, compare_with) { return value != compare_with },
'>': function(value, compare_with) { return value > compare_with },
'<': function(value, compare_with) { return value < compare_with },
'>=': function(value, compare_with) { return value >= compare_with },
'<=': function(value, compare_with) { return value <= compare_with },
'in': function(value, compare_with) { return (compare_with.indexOf(value) != -1) },
'~': function(value, compare_with) { return (compare_with.indexOf(value) != -1) },
'!~': function(value, compare_with) { return (compare_with.indexOf(value) == -1) }
}
import { namespace } from './namespace_util.js'
var sorting = {
'string': function(value, compare_with) {
value = value.toUpperCase()
compare_with = compare_with.toUpperCase()
if (value < compare_with) {
return -1
function isEqual(value, compare_with, criteria_satisfied) {
// to handle: ( ...Shape!&... ) or ( ...&Shape&... )
if (!value) {
return criteria_satisfied ? (compare_with == null) : (compare_with != null)
}
if (value > compare_with) {
return 1
return value.indexOf(compare_with) != -1 ? !criteria_satisfied : criteria_satisfied
}
function greater_than(value, compare_with, include_equals) {
if ((isNaN(compare_with) && Date.parse(compare_with))) {
compare_with = Date.parse(compare_with)
value = Date.parse(value)
}
return 0
return include_equals ? (compare_with >= value) : (compare_with > value)
}
var operators = {
'=': function (value, compare_with) {
return isEqual(value, compare_with, false)
},
'!': function (value, compare_with) {
return isEqual(value, compare_with, true)
},
'number': function(value, compare_with) { return value - compare_with }
'>': function (value, compare_with) {
return greater_than(value, compare_with, false)
},
'<': function (value, compare_with) {
return greater_than(compare_with, value, false)
},
'>~': function (value, compare_with) {
return greater_than(value, compare_with, true)
},
'<~': function (value, compare_with) {
return greater_than(compare_with, value, true)
},
'~': function (value, compare_with) {
return isEqual(compare_with, value[0], false)
},
'!~': function (value, compare_with) {
return isEqual(compare_with, value[0], true)
}
}
var sorting = {
'string': function (value, compare_with, order) {
if (!order) order = 'asc'
// swap if 'desc'
if (order == 'desc')
value = [compare_with, compare_with = value][0]
export function datafilter(data, filters, options) {
filters = filters || []
return value.localeCompare(compare_with)
},
'number': function (value, compare_with, order) {
if (!order) order = 'asc'
// swap if 'desc'
if (order == 'desc')
value = [compare_with, compare_with = value][0]
options = options || {}
options.limit = options.limit || 1000
options.offset = options.offset || 0
return value - compare_with
}
}
options.sort = options.sort || []
function clone_pluck(source, include_keys, exclude_keys) {
if (include_keys.length == 0) include_keys = Object.keys(source)
var new_obj = {}
include_keys.forEach(function (key) {
if (exclude_keys.indexOf(key) < 0) new_obj[key] = source[key]
})
return new_obj
}
options.columns = options.columns || {}
options.columns.allow = options.columns.allow || []
options.columns.not = options.columns.not || []
var result_count = 0
var result = []
export function datafilter(data, filters, dataset_name) {
filters = filters || []
for(var index = options.offset; index < data.length; index++) {
var criteria_statisfied = true
var row = data[index]
var result = data
var operator, value
for(var i = 0; i < filters.length; i++) {
var col = filters[i].col || 'null'
var operator = filters[i].op || '='
var value = filters[i].val || null
// url namespace sanitize
filters = namespace(filters, dataset_name)
if(!(col in row))
continue
// apply WHERE clause
for (var key in filters) {
if (key[0] == '_') continue
var operator_index = (key.match(/(!|>|>~|<|<~|~|!~)$/)) ? key.match(/(!|>|>~|<|<~|~|!~)$/).index : key.length
operator = (key.slice(operator_index) != '') ? key.slice(operator_index) : '='
value = (filters[key][0] != "") ? filters[key] : null
if(!operators[operator](row[col], value)){
criteria_statisfied = false
break
}
}
var col = key.slice(0, operator_index)
if (criteria_statisfied) {
for(var column in row) {
if(options.columns.not.indexOf(column) != -1)
delete row[column]
else if(options.columns.allow.indexOf(column) == -1 && options.columns.allow.length != 0)
delete row[column]
result = result.filter(function (row) {
return (typeof row[col] != 'undefined') ? operators[operator](value, row[col]) : true
})
}
result.push(row)
result_count++
}
var offset = parseInt(filters['_offset']) || 0
var limit = parseInt(filters['_limit']) || 1000
if(result_count == options.limit) {
break
}
}
result = result.slice(offset, (offset + limit))
if(options.sort.length > 0) {
result.sort(function(a, b) {
var sort_status = false
options.sort.forEach(function(sort) {
var type = (isNaN(a[sort.column])) ? 'string' : 'number'
if(sort.order == 'asc') {
sort_status = sort_status || sorting[type](a[sort.column], b[sort.column])
} else if (sort.order == 'desc') {
sort_status = sort_status || sorting[type](b[sort.column], a[sort.column])
// apply SELECT clause
if (filters['_c']) {
var exclude_cols = [], include_cols = []
filters['_c'].forEach(function (column) {
column[0] == '-' ? exclude_cols.push(column.slice(1)) : include_cols.push(column)
})
result = result.map(function (row) {
return clone_pluck(row, include_cols, exclude_cols)
})
}
if (filters['_sort']) {
result.sort(function (a, b) {
var swap_rows = false
filters['_sort'].forEach(function (sort) {
var order = (sort[0] == '-') ? 'desc' : 'asc'
if (sort[0] == '-') sort = sort.substr(1)
if (typeof a[sort] == 'undefined') return
// if sort.column evaluates to false, it will proceed with evaluating || expression
var type = (isNaN(a[sort])) ? 'string' : 'number'
swap_rows = swap_rows || sorting[type](a[sort], b[sort], order)
})
return sort_status
return swap_rows
})
}
......
......@@ -3,6 +3,7 @@
// Each <!-- var name --> section is imported as a minified variable string.
import * as default_templates from './formhandler.template.html'
import { parse } from './url.js'
import { namespace } from './namespace_util.js'
// Render components in this order. The empty component is the root component.
var components = ['', 'table', 'page', 'size', 'count', 'export', 'filters', 'error']
......@@ -56,7 +57,7 @@ export function formhandler(js_options) {
if (!js_options)
js_options = {}
this.each(function() {
this.each(function () {
var $this = $(this)
// Convert all .urlfilter classes into url filters that update location.hash
$this.urlfilter({
......@@ -70,7 +71,7 @@ export function formhandler(js_options) {
if (!options.columns)
options.columns = []
else if (typeof options.columns == 'string')
options.columns = _.map(options.columns.split(/\s*,\s*/), function(col) { return { name: col } })
options.columns = _.map(options.columns.split(/\s*,\s*/), function (col) { return { name: col } })
// Compile all templates
var template = {}
......@@ -87,7 +88,7 @@ export function formhandler(js_options) {
// Assumption: Column name will not be '*'
if (options.columns.some(function (o) { return o['name'] === '*' }))
options.columns = _.map(meta.columns, function(col){
options.columns = _.map(meta.columns, function (col) {
var options_col = _.find(options.columns, function (o) { return o['name'] === col.name })
return options_col ? options_col : col
})
......@@ -99,7 +100,7 @@ export function formhandler(js_options) {
}
// Store template_data in $this
$this.data('formhandler', template_data)
_.each(components, function(name) {
_.each(components, function (name) {
// Disable components if required. But root component '' is always displayed
if (name && !options[name])
return
......@@ -122,9 +123,10 @@ export function formhandler(js_options) {
function render() {
var url_args = parse(location.hash.replace(/^#/, '')).searchList
url_args = namespace(url_args, options.name)
// Create arguments passed to the FormHandler. Override with the user URL args
var args = _.extend({
c: options.columns.map(function(d) { return d.name }),
c: options.columns.map(function (d) { return d.name }),
_limit: options.pageSize,
_format: 'json',
_meta: 'y'
......@@ -140,21 +142,21 @@ export function formhandler(js_options) {
// Hide loader
$('.loader', $this).addClass('d-none')
}
}).done(function(data, status, xhr) {
}).done(function (data, status, xhr) {
var meta = {}
_.each(meta_headers, function(header) {
_.each(meta_headers, function (header) {
var val = xhr.getResponseHeader('Fh-Data-' + header)
if (val !== null)
meta[header] = JSON.parse(val)
})
if (typeof options.transform == 'function') {
var result = options.transform({data: data, meta: meta, options: options, args: args}) || {}
var result = options.transform({ data: data, meta: meta, options: options, args: args }) || {}
data = 'data' in result ? result.data : data
meta = 'meta' in result ? result.meta : meta
}
draw_table(data, args, meta)
$this.trigger({ type: 'load', formdata: data, meta: meta, args: args, options: options })
}).fail(function(xhr, status, message) {
}).fail(function (xhr, status, message) {
$this.html(template['error']({
message: message
}))
......
export function namespace(search, name) {
// Return an object with all keys in search that begin with `<name>:` or
// do not have a `:` in them.
// If name is false-y, return search
if (!name)
return search
var result = {}
for (var key in search) {
var parts = key.split(':')
if (parts.length == 1)
result[parts[0]] = search[key]
else if (parts[0] === name)
result[parts[1]] = search[key]
}
return result
}
[{"Continent":"Europe"},{"Continent":"Asia"},{"Continent":"Asia"},{"Continent":"North America"},{"Continent":"Europe"},{"Continent":"Asia"},{"Continent":"Africa"},{"Continent":"South America"},{"Continent":"Europe"},{"Continent":"Oceania"},{"Continent":"Asia"},{"Continent":"Europe"},{"Continent":"North America"},{"Continent":"Asia"},{"Continent":"Europe"},{"Continent":"Africa"},{"Continent":"Europe"},{"Continent":"Asia"},{"Continent":"Africa"},{"Continent":"Africa"},{"Continent":"Asia"},{"Continent":"South America"},{"Continent":"South America"},{"Continent":"North America"},{"Continent":"Asia"},{"Continent":"Africa"},{"Continent":"Europe"},{"Continent":"North America"},{"Continent":"North America"},{"Continent":"Africa"},{"Continent":"Africa"},{"Continent":"Africa"},{"Continent":"Europe"},{"Continent":"Africa"},{"Continent":"South America"},{"Continent":"Africa"},{"Continent":"Asia"},{"Continent":"South America"},{"Continent":"North America"},{"Continent":"North America"},{"Continent":"Africa"},{"Continent":"Asia"},{"Continent":"Europe"},{"Continent":"Europe"},{"Continent":"Africa"},{"Continent":"Europe"},{"Continent":"North America"},{"Continent":"North America"},{"Continent":"Africa"},{"Continent":"South America"},{"Continent":"Europe"},{"Continent":"Africa"},{"Continent":"Africa"},{"Continent":"Africa"},{"Continent":"Europe"},{"Continent":"Africa"},{"Continent":"Europe"},{"Continent":"Oceania"},{"Continent":"Oceania"},{"Continent":"Europe"},{"Continent":"Africa"},{"Continent":"Europe"},{"Continent":"North America"},{"Continent":"Asia"},{"Continent":"Africa"},{"Continent":"Africa"},{"Continent":"Africa"},{"Continent":"Africa"},{"Continent":"Europe"},{"Continent":"North America"},{"Continent":"Africa"},{"Continent":"South America"},{"Continent":"North America"},{"Continent":"Europe"},{"Continent":"North America"},{"Continent":"Europe"},{"Continent":"Asia"},{"Continent":"Europe"},{"Continent":"Asia"},{"Continent":"Asia"},{"Continent":"Asia"},{"Continent":"Asia"},{"Continent":"Europe"},{"Continent":"Europe"},{"Continent":"North America"},{"Continent":"Asia"},{"Continent":"Asia"},{"Continent":"Africa"},{"Continent":"Asia"},{"Continent":"Asia"},{"Continent":"Oceania"},{"Continent":"Africa"},{"Continent":"North America"},{"Continent":"Asia"},{"Continent":"Asia"},{"Continent":"Europe"},{"Continent":"Asia"},{"Continent":"Asia"},{"Continent":"Asia"},{"Continent":"Asia"},{"Continent":"North America"},{"Continent":"Europe"},{"Continent":"Asia"},{"Continent":"Africa"},{"Continent":"Africa"},{"Continent":"Europe"},{"Continent":"Europe"},{"Continent":"Europe"},{"Continent":"Africa"},{"Continent":"Africa"},{"Continent":"Europe"},{"Continent":"Europe"},{"Continent":"Europe"},{"Continent":"Africa"},{"Continent":"Oceania"},{"Continent":"Europe"},{"Continent":"Africa"},{"Continent":"Asia"},{"Continent":"Asia"},{"Continent":"Africa"},{"Continent":"Europe"},{"Continent":"Africa"},{"Continent":"Asia"},{"Continent":"Africa"},{"Continent":"North America"},{"Continent":"Asia"},{"Continent":"Africa"},{"Continent":"Africa"},{"Continent":"Africa"},{"Continent":"Africa"},{"Continent":"North America"},{"Continent":"Europe"},{"Continent":"Europe"},{"Continent":"Asia"},{"Continent":"Oceania"},{"Continent":"Oceania"},{"Continent":"Asia"},{"Continent":"North America"},{"Continent":"South America"},{"Continent":"Oceania"},{"Continent":"Asia"},{"Continent":"Asia"},{"Continent":"Europe"},{"Continent":"Europe"},{"Continent":"Oceania"},{"Continent":"South America"},{"Continent":"Asia"},{"Continent":"Europe"},{"Continent":"Europe"},{"Continent":"Europe"},{"Continent":"Africa"},{"Continent":"Asia"},{"Continent":"Oceania"},{"Continent":"Africa"},{"Continent":"Africa"},{"Continent":"Europe"},{"Continent":"Asia"},{"Continent":"Europe"},{"Continent":"Europe"},{"Continent":"Africa"},{"Continent":"Europe"},{"Continent":"Africa"},{"Continent":"Africa"},{"Continent":"South America"},{"Continent":"Africa"},{"Continent":"North America"},{"Continent":"Asia"},{"Continent":"Africa"},{"Continent":"Africa"},{"Continent":"Africa"},{"Continent":"Asia"},{"Continent":"Asia"},{"Continent":"Asia"},{"Continent":"Asia"},{"Continent":"Africa"},{"Continent":"Oceania"},{"Continent":"Asia"},{"Continent":"North America"},{"Continent":"Oceania"},{"Continent":"Asia"},{"Continent":"Africa"},{"Continent":"Europe"},{"Continent":"Africa"},{"Continent":"North America"},{"Continent":"South America"},{"Continent":"Asia"},{"Continent":"Europe"},{"Continent":"North America"},{"Continent":"South America"},{"Continent":"Asia"},{"Continent":"Oceania"},{"Continent":"Oceania"},{"Continent":"Asia"},{"Continent":"Africa"},{"Continent":"Africa"},{"Continent":"Africa"}]
\ No newline at end of file
[
{
"देश": "भारत",
"city": "Hyderabad",
"product": "Biscuit",
"sales": 866.1,
"growth": -0.27
},
{
"देश": "भारत",
"city": "Hyderabad",
"product": "芯片",
"sales": 26.4,
"growth": -0.242
},
{
"देश": "भारत",
"city": "Hyderabad",
"product": "Crème",
"sales": 38.3,
"growth": -0.291
},
{
"देश": "भारत",
"city": "Hyderabad",
"product": "Eggs",
"sales": 513.7,
"growth": -0.113
},
{
"देश": "भारत",
"city": "Bangalore",
"product": "Biscuit",
"sales": 41.9,
"growth": -0.402
},
{
"देश": "भारत",
"city": "Bangalore",
"product": "芯片",
"sales": 52.2,
"growth": 0.064
},
{
"देश": "भारत",
"city": "Bangalore",
"product": "Crème",
"sales": 17.8,
"growth": -0.052
},
{
"देश": "भारत",
"city": "Bangalore",
"product": "Eggs",
"sales": 178.9,
"growth": -0.261
},
{
"देश": "भारत",
"city": "Coimbatore",
"product": "Biscuit",
"sales": 217.4,
"growth": 0.114
},
{
"देश": "भारत",
"city": "Coimbatore",
"product": "芯片",
"sales": "",
"growth": ""
},
{
"देश": "भारत",
"city": "Coimbatore",
"product": "Crème",
"sales": 94.4,
"growth": -0.288
},
{
"देश": "भारत",
"city": "Coimbatore",
"product": "Eggs",
"sales": 72.8,
"growth": -0.066
},
{
"देश": "Singapore",
"city": "Singapore",