Index
Index proccess for system tables
Now index service is uses one _changes stream for system tables. If rules for one or more tables has changed or added, first of all need fix rules in sys.config and create reindex task for reindexing all docs from 0 sequence. Rules from sys.config
will be applied only for new cluster.
Example of task file:
Path: table_files/system_index_task.txt
Body:
[[{table_rules, {user,[{"id",{"test_temp","id",[pk,{varchar,100}]}},
{"email",{"test_temp","email",[{varchar,100}]}},
{"phone",{"test_temp","phone",[{varchar,100}]}},
{"first_name",{"test_temp","first_name",[{varchar,100}]}},
{"last_name",{"test_temp","last_name",[{varchar,100}]}}]}},
{temp_table, "test_temp"},
{table, "users"}],
[{table_rules,{gtin,[{"id",{"test_temp_gtin","id",[pk,{varchar,100}]}},
{"owner",{"test_temp_gtin","owner",[{varchar,100}]}},
{"name",{"test_temp_gtin","name",[text]}},
{"brand_id",{"test_temp_gtin","brand_id",[{varchar,100}]}}]}},
{temp_table, "test_temp_gtin"},
{table, "gtin"}
]
].
In presented example new process will be indexing table test_temp
and test_temp_gtin
from 0 sequence. After ends, table test_temp_gtin
will be replaced to gtin
and test_temp
will be replaced to users
, task file system_index_task.txt
will be deleted and new rules from file will be applied.
Get the list of supported index types
gtin
- index type of products
slgtin
- index type of serialized products
gdti
- index type of gdti documents
GET /index-service/index-schema/types
Parameters
Name |
Type |
Description |
authenticateit_identity_ticket |
header |
Required: Session's ticket |
Example Response
["gtin", "slgtin", "gdti"]
Create participant index schema
POST /index-service/index-schema
Parameters
Name |
Type |
Description |
authenticateit_identity_ticket |
header |
Required: Session's ticket |
type |
string |
index type ("gtin" , "slgtin" or "gdti" ) |
fields |
object |
{"Made In":"Made In", "Other Title":"Name column in web"} |
link_fields |
list |
Link index rules |
source_types |
array::string |
sources - ["gs1","brand","expert","retailer","contributor"] |
language |
string |
language code in ISO e.g. en, ru |
country |
string |
country code e.g. 036 |
Example Request
{
"conditions": {
"owner": "urn:authenticateit:participant:735879621218609"
},
"fields": {
"field_0e76292794888d4f1fa75fb3aff4ca27c58f56a6": {
"image": "Image"
},
"field_3c6de1b7dd91465d437ef415f94f36afc1fbc8a8": {
"title": "Title"
},
"field_414f505575623b99b6fe418814b146c3f52f1bd4": {
"serial_number": "Serial Number"
},
"field_579233b2c479241523cba5e3af55d0f50f2d6414": {
"owner": "Owner"
},
"field_5bb1d40aad5728573e7b5d2db72c584057d74170": {
"company_prefix": "Company Prefix"
},
"field_87ea5dfc8b8e384d848979496e706390b497e547": {
"id": "ID"
},
"field_b584e85a40e6e2574fddf520398df3604a66fb9a": {
"doc_type": "Document Type"
}
},
"id": "index_schema_gdti_urn:authenticateit:participant:735879621218609",
"source_types": [
"brand",
"contributor",
"expert",
"gs1",
"retailer",
"skuvantage"
],
"link_fields": [
{
"text": "Link Text Str",
"image_field_name": "Web Column name img",
"url_field_name": "Web Column name url"
}
],
"type": "gdti"
}
Example Response
{
"conditions": {
"owner": "urn:authenticateit:participant:735879621218609"
},
"created_at": "2019-04-15T03:52:52Z",
"default_fields": {
"company_prefix": {
"company_prefix": "Company Prefix"
},
"doc_type": {
"doc_type": "Document Type"
},
"id": {
"id": "ID"
},
"image": {
"image": "Image"
},
"owner": {
"owner": "Owner"
},
"serial_number": {
"serial_number": "Serial Component"
},
"title": {
"title": "Title"
}
},
"fields": {
"field_0e76292794888d4f1fa75fb3aff4ca27c58f56a6": {
"image": "Image"
},
"field_3c6de1b7dd91465d437ef415f94f36afc1fbc8a8": {
"title": "Title"
},
"field_414f505575623b99b6fe418814b146c3f52f1bd4": {
"serial_number": "Serial Number"
},
"field_579233b2c479241523cba5e3af55d0f50f2d6414": {
"owner": "Owner"
},
"field_5bb1d40aad5728573e7b5d2db72c584057d74170": {
"company_prefix": "Company Prefix"
},
"field_87ea5dfc8b8e384d848979496e706390b497e547": {
"id": "ID"
},
"field_b584e85a40e6e2574fddf520398df3604a66fb9a": {
"doc_type": "Document Type"
}
},
"id": "index_schema_gdti_urn:authenticateit:participant:735879621218609",
"index_type": "gdti",
"link_fields": [
{
"image_field": "field_2ba36518745e34c4a06b945f1e050cea3af0b59c",
"image_field_name": "Web Column name img",
"text": "link text str",
"url_field": "field_41e726918505fd70292de8a204cf3093f8743350",
"url_field_name": "Web Column name url"
}
],
"participant_id": "urn:authenticateit:participant:735879621218609",
"source_types": [
"brand",
"contributor",
"expert",
"gs1",
"retailer",
"skuvantage"
],
"table": "index_schema_table_gdti_cc268bc2-63cb-480c-8193-3766b6d67e3b",
"type": "index_schema",
"updated_at": "2021-04-01T15:08:14Z",
"user_id": "urn:authenticateit:user:email:ethertest3@mailinator.com"
}
Get participant index schema
GET /index-service/index-schema
Parameters
Name |
Type |
Description |
authenticateit_identity_ticket |
header |
Required: Session's ticket |
Example Response
[
{
"conditions": {
"owner": "urn:authenticateit:participant:735879621218609"
},
"default_fields": {
"company_prefix": {
"company_prefix": "Company Prefix"
},
"doc_type": {
"doc_type": "Document Type"
},
"id": {
"id": "ID"
}
},
"fields": {
"field_87ea5dfc8b8e384d848979496e706390b497e547": {
"id": "ID"
},
"field_b584e85a40e6e2574fddf520398df3604a66fb9a": {
"doc_type": "Document Type"
}
},
"link_fields": [
{
"image_field": "field_2ba36518745e34c4a06b945f1e050cea3af0b59c",
"image_field_name": "Web Column name img",
"text": "link text str",
"url_field": "field_41e726918505fd70292de8a204cf3093f8743350",
"url_field_name": "Web Column name url"
}
],
"id": "index_schema_gdti_urn:authenticateit:participant:735879621218609",
"source_types": [
"brand",
"contributor",
"expert",
"gs1",
"retailer",
"skuvantage"
],
"table": "index_schema_table_gdti_cc268bc2-63cb-480c-8193-3766b6d67e3b",
"type": "gdti"
},
{
"conditions": {
"custody": "urn:authenticateit:participant:735879621218609",
"owner": "urn:authenticateit:participant:735879621218609"
},
"default_fields": {
"atk_sscc": {
"atk_sscc": "ATK SSCC"
},
"num_contributor_sources": {
"num_contributor_sources": "Contributor Sources"
},
"atk_gs1_sscc": {
"atk_gs1_sscc": "ATK GS1 SSCC"
},
"code": {
"code": "GS1 Databar"
},
"image": {
"image": "Image"
},
"task": {
"task": "Serialization Task"
},
"delivery_number": {
"delivery_number": "Delivery"
},
"user_reviews": {
"user_reviews": "Reviews"
},
"id": {
"id": "ID"
}
},
"fields": {},
"id": "index_schema_slgtin_urn:authenticateit:participant:735879621218609",
"source_types": [
"brand",
"contributor",
"expert",
"gs1",
"retailer"
],
"table": "index_schema_table_slgtin_07a62602-6cc9-430f-bd28-cf14d9747ac9",
"type": "slgtin"
},
{
"conditions": {
"owner": "urn:authenticateit:participant:735879621218609"
},
"default_fields": {
"brand_id": {
"brand_id": "Brand"
},
"gpc_brick": {
"gpc_brick": "GPC Brick"
},
"gpc_class": {
"gpc_class": "GPC Class"
},
"gpc_family": {
"gpc_family": "GPC Family"
},
"gpc_segment": {
"gpc_segment": "GPC Segment"
},
"id": {
"id": "ID"
}
},
"fields": {},
"id": "index_schema_gtin_urn:authenticateit:participant:735879621218609",
"source_types": [
"gs1",
"brand",
"expert",
"retailer",
"contributor"
],
"table": "index_schema_table_gtin_735879621218609",
"type": "gtin"
}
]
Create manual reindex task
POST /index-service/index-schema/reindex
Parameters
Name |
Type |
Description |
authenticateit_identity_ticket |
header |
Required: Session's ticket |
type |
string |
Required: Index type (gtin , slgtin or gdti ) |
Example Request
{
"type": "gdti"
}
Example Response
{
"id": "urn:authenticateit:participant:735879621218609:gdti:reindex"
}
Read manual reindex task status
GET /index-service/index-schema/reindex
Parameters
Name |
Type |
Description |
authenticateit_identity_ticket |
header |
Required: Session's ticket |
id |
string |
Required: Task Id |
Example Response
{
"key": "urn:authenticateit:participant:735879621218609:gdti:reindex",
"new_table": "gdti_ee259a99-f830-4972-8abf-59b12c11658f_reindex",
"original_table": "index_schema_table_gdti_cc268bc2-63cb-480c-8193-3766b6d67e3b",
"owner": "urn:authenticateit:participant:735879621218609",
"pid": "<0.28840.0>",
"progress": 7,
"status": "done"
}
Get index tables of ancestor brand participants
Endpoint for supply chain participants to find out SGTIN indexing
tables of supply chain ancestor brands. The response is a JSON object
with brand participant identifiers as keys. Each participant entry has
table
field (table name from participant's SGTIN/LGTIN indexing
schema).
GET /index-service/index-schema/brand-tables
Parameters
Name |
Type |
Description |
authenticateit_identity_ticket |
header |
Required: Session's ticket |
Example Response
{
"urn:authenticateit:participant:576318441844504": {
"table": "index_schema_table_slgtin_cf7372e8-40f9-4ab4-b2e0-75bd42b1a213"
}
}
Get table's profiles for RSQL
GET /index-service/rsql/profiles/:table
Parameters
Name |
Type |
Description |
authenticateit_identity_ticket |
header |
Required: Session's ticket |
table |
- |
Required: Table |
Example Response
[
{
"fields": [
"field_64fdc5235d65719db2313923ad30ce7fda154808",
"id"
],
"profile": "test_profile"
}
]
Request SQL
POST /index-service/rsql/:table
Parameters
Name |
Type |
Description |
authenticateit_identity_ticket |
header |
Required: Session's ticket |
query |
string |
rsql query |
skip |
- |
like offset in sql |
take |
- |
like limit in sql |
op |
- |
condition OR/AND |
include_scans |
- |
optional option (for data or count ) |
order |
- |
set order. Example: order=id-asc,name-desc |
type |
- |
set type of request: data , count or group |
profile |
- |
request profile (for data or count ) |
convert_date |
- |
For group type. Use for date conversion (as day, year ot month) |
group_by_fields |
- |
For group type. List of group types |
group_by_count_field |
- |
For group type. Count field name |
include_scans
- optional parameter for product table (boolean type: true or false). It is used to count user scans (the parameter user_scans
is added to the answer)
If the rsql filter value contains some special symbols (;
, "
), filter value should be quoted. For example:
id=like=|\"some value \\\" with one quote\"|
id=in=(\"Some value ; with rsql delimiter\",1,2,3);id!=123
Example rsql
id==123;name==wklnfer
id!=123;name!=wklnfer
| id=le=123;name=le=wklnfer | where id < 123 |
| id=lt=123;name=lt=wklnfer | where id ⇐ 123 |
| id=gt=123;name=gt=wklnfer | where id > 123 |
| id=ge=123;name=gt=wklnfer | where id >= 123 |
id=in=(d9683bd2-fd3e-4b0d-93bb-b58c8c2da284,1,2,3)
id=out=(d9683bd2-fd3e-4b0d-93bb-b58c8c2da284,1,2,3)
id=in=(d9683bd2-fd3e-4b0d-93bb-b58c8c2da284,1,2,3);name==test
id==123;name=like=|wklnfer| for like need set | instead of %
name=unlike=|wklnfer|
| name=null | name IS NULL |
| name=not_null | name IS NOT NULL |
| name=match_against=some value | Full-text search by field |
| name=match_against=\"\\\"some value\\\"\" | Full-text exact search by name field |
Example RSQL GROUP Type Request
{
"query": "owner==urn:authenticateit:participant:804355519238230;date_created=ge=2020-10-01T00:00:00Z",
"type": "group",
"group_by_fields": "issuer;owner;date_created",
"convert_date": [
{
"field": "date_created",
"format": "day"
}
],
"group_by_count_field": "id",
"order": "date_created_day-desc"
}
Example Response
{
"count": 1,
"data": [
{
"id": "04444455368975",
"image": null,
"name": "Test product 04444455368975",
"owner": "urn:authenticateit:participant:1489513906969",
"sources": null,
"user_scans": 12
}
]
}
Request SQL. partner_brand
products filter
POST /index-service/rsql/gtin
Parameters
Name |
Type |
Description |
authenticateit_identity_ticket |
header |
Required: Session's ticket |
partner_brand |
enum |
Required: partner_brand filter value: true ,false ,null |
query |
string |
rsql query |
skip |
- |
like offset in sql |
take |
- |
like limit in sql |
op |
- |
condition OR/AND |
order |
- |
set order. Example: order=id-asc,name-desc |
type |
- |
set type of request: data |
Method supports type data
only.
query
filter should be prepared only by predefined fields name
(gtin name) and/or id
(gtin value).
partner_brand
is required and could be set as true
, false
or null
Example RSQL GROUP Type Request
{
"query": "id=like=|Wow|;name=like=|Wow|",
"op": "or",
"type": "data",
"partner_brand": true,
"take": 5
}
Example Response
[
{
"brand_id": null,
"id": "00000093524919",
"name": "WOW Select Baby Shampoo & Conditioner 200ml",
"owner": "urn:authenticateit:participant:708023102856412"
},
{
"brand_id": null,
"id": "00000093524933",
"name": "WOW Select Head to Toe Baby Wash 200ml",
"owner": "urn:authenticateit:participant:708023102856412"
},
{
"brand_id": null,
"id": "00000093524957",
"name": "WOW Select Mild Baby Shampoo 200ml",
"owner": "urn:authenticateit:participant:708023102856412"
},
{
"brand_id": null,
"id": "00000093524964",
"name": "WOW Select Calming Baby Bath 200ml",
"owner": "urn:authenticateit:participant:708023102856412"
},
{
"brand_id": null,
"id": "00000093633154",
"name": "WOW SELECT TURKEY 85G",
"owner": "urn:authenticateit:participant:708023102856412"
}
]
Clear cache
Clear cache for RSQL count requests
POST /index-service/clear_cache
Parameters
Name |
Type |
Description |
authenticateit_identity_ticket |
header |
Required: Session's ticket |
date_from |
string |
“2017-03-15T15:10:23Z” |
date_to |
string |
“2017-03-15T15:10:33Z” |
Example Response
"Cache successfully deleted"
Request API
POST /index-service/rapi
Parameters
Name |
Type |
Description |
authenticateit_identity_ticket |
header |
Required: Session's ticket |
table |
string |
table to select e.g. "gtin", "participant" or "brand" |
page |
number |
page 0,1,2... |
limit |
number |
limit per page 10,20... |
company_prefix |
string |
"9300" |
Example Response
[
{
"id": "09300633912000",
"name": "Woolworths Select 5 Sectioning Clips Silver",
"owner": "urn:authenticateit:participant:1486385901460"
},
{
"id": "09300633911959",
"name": "Woolworths Select 4 Bun Nets Light Brown",
"owner": "urn:authenticateit:participant:1486385901460"
}
]
Extended RAPI
POST /index-service/rapi/:table
System can read all brands. Other participants able to read only self brands.
Parameters
Name |
Type |
Description |
authenticateit_identity_ticket |
header |
Required: Session's ticket |
table |
url |
Required: Only brand table is supported |
take |
number |
Like limit in sql |
skip |
number |
Like offset in sql |
Example Request
{
"take":3,
"skip":1
}
Example Response
{
"count": 281,
"data": [
{
"approved": null,
"budget": null,
"id": "1900",
"logo": null,
"name": null,
"owner": "urn:authenticateit:participant:155450266873886",
"owner_name": "1900",
"portfolio": null
},
{
"approved": 1,
"budget": 1,
"id": "2",
"logo": null,
"name": null,
"owner": "urn:authenticateit:participant:1",
"owner_name": "Authenticateit Pty Ltd",
"portfolio": null
},
{
"approved": null,
"budget": null,
"id": "21312312321",
"logo": null,
"name": null,
"owner": "urn:authenticateit:participant:488704009134617",
"owner_name": "TestCompany",
"portfolio": null
}
]
}
Current system tables
authentication_result |
id |
event_ts |
user_id |
code |
country |
language |
latitude |
longitude |
gtin |
id |
owner |
name |
brand_id |
index_schema |
id |
table |
participant_id |
index_type |
user_id |
created_at |
updated_at |
participant |
id |
name |
parent |
system |
partner |
product360 |
manufacturer |
outsource |
distributor |
retailer |
expert |
supplier |
company_prefix |
rewards_campaign |
id |
owner |
name |
products |
events |
start_date |
end_date |
users |
id |
email |
language |
country |
shutout |
id |
owner |
name |
products |
start_date |
end_date |
business_location |
id |
owner |
name |
longitude |
latitude |