v0.7.5

The v0.7.5 is a huge update over the last 3 weeks!!!

So many changes (even for my personal way of doing things) that I will mention the log messages. Note that in addition to the new chances themselves, I'm actually am (or at least wasn't) as proficient with Python language than my experience with PHP, JavaScript and IoC (e.g. Ansible, YAML). Anyway, part of the old code (like the ones on /bin folder at the top) are still there, so old comments should still work.

The release v0.7.5 is mostly to have a point in time way to reference changes. I'm not sure if we will reach an v1.0.0 or after v0.9 will go to v0.10.0, v0.11.0, v.0.12.0, etc. Also, as much as possible we're already trying to make the tools to extend functionality be self-explanatory (aka even your code editor explain how to extend, not just the documentation!) and also implement integration tests to allow create/update features with more confidence.

git log --oneline -153

430ffcb hxlm (#11): initial draft of hdpcli --objectivum-linguam NNN
40f9d79 hxlm (#11): re-enabled hdpcli --non-urn & hdpcli --verum-urn
6859bd8 hxlm (#11): re-enabled hdpcli --non-grupum & hdpcli --verum-grupum
7fa8f27 hxlm (#11): re-enabled hdpcli to use remote files as HDP, e.g. hdpcli http://example.org/path/my.hdp.yml
0efd3ac indigenous-language-brazil-sample.hxl.csv (test data) added; Source: all languages with Iso 639-3 Code from https://pt.wikipedia.org/wiki/L%C3%ADnguas_ind%C3%ADgenas_do_Brasil
abe2fd2 hxlm (#11): drafted cli commands to hdpcli; --non-adm0, --non-grupum, --non-nomen, --non-tag, --non-urn, --verum-adm0, --verum-grupum, --verum-tag, --verum-urn
1a41d14 i18n+l10n (#15): core_vocab, added attr.verum & attr.falsum; this is not as intuitive. In special attr.falsum.zho (because may have many, many alternatives) and attr.falsum.ara (because I'm not sure about the attr.falsum.ara.id) is welcome to get revision; the initial values at least are from Wikidata/Wikitionary.
78160bd i18n+l10n (#15): core_vocab, added attr.grupum; One way to group colletions (hsilos). While the tags allow use at any level, this attribute should be explicity named at top level.
f4f6a04 hxlm (#11): implicitly create an urn-like hash key for hsilos without explicitly use; something like 'urn:oo:hsilo:domain_base:container_base-container_item_index' were container_base ofte would be 'local' (localhost) and container_base often the filename itself
e228e24 hxlm (#11): force output Unicode YAML; if a single 'Olá Mundo\!' already was ugly, it would be much worse with anything beyond ASCII; so lets force it by default for everyone
27893a0 hxlm (#11): added  HDP._prepare_from_local_directory(); draft of HDP._update()
ed6dcc8 hxl-processing-specs (#14): added tests
2550222 hxl-processing-specs (#14): now is possible to specify inline data ('input_data', 'output_data', as part of hrecipe.exemplum; the underlining inplementation still not ready, but the idea is be able to specify self-contained example when creating recipes with YAML; the hrecipe.exemplum[N]objectivum.datum can be used for self-contained testing!
57d2cca i18n+l10n (#15), hxl-processing-specs (#14): now is possible to specify more than one source; also content without translation will be prefixed with a single _
4819547 i18n+l10n (#15): to core_vocab added objectivum; renamed attr.lingua -> attr.linguam
bfcd583 core_vocab Added attr.locum & attr.tempus
abce05b core_vocab Added attr.datum; renamed: desc -> descriptionem, lang -> lingua, name -> nomen, source -> fontem
7c367b3 core_vocab attr.exemplum added (see https://github.com/EticaAI/HXL-Data-Science-file-formats/issues/14#issuecomment-798939113)
beba373 hxlm.core.internal.formatter.beautify() created; already deal if output is piped to another command, like jq or hxlspec, without breaking
271385f Coloring all the things works. But how to know if is not stdout, but being piped? If piped, things broke hard with jq or hxlspec
07a30fa Testing this thing called pygments. Colors are nice. Let's try color all the things and see if don't break what already was working
47f8c87 v0.7.5 started
de9467a hxl-processing-specs (#14): HDP.export_json_processing_specs() works
57de6da hxlm(#11), hxl-processing-specs (#14): HDP._prepare_from_remote_iri() works
0dc88ee hxlm(#11), hxl-processing-specs (#14): HDP._prepare_from_local_file() now works
cc5462f hxlm(#11), hxl-processing-specs (#14): Added HDP.HDP_JSON_EXTENSIONS & HDP.HDP_YML_EXTENSIONS
10c7df8 hxlm(#11), hxl-processing-specs (#14): Added HDP._online_unrestricted_init, HDP._safer_zone_hosts, HDP._safer_zone_list, HDP.export_schema_json(), HDP.export_yml()
e516c56 hxlm(#11), hxl-processing-specs (#14): created hxlm.core.model.hdp HDP class
819196d hxl-processing-specs (#14): hdpcli --export-to-hxl-json-processing-spec added (draft)
314c0e3 hdp.json-schema.json: <3 it's working
e344f79 hdp.json-schema.json: ok, now we starting to have something; still broken, but at least I know where is broken
2559c6a schema/hdp.json-schema.json I think is better to restrict to json schema draft-07 (very recent one may not be supported
181274e hdp.json-schema.json: ok, re-read the current draft-bhutton-json-schema-00 spec; I think we actually can have subschemas
68e9fd3 schema/hdp.json-schema.json started
544e12e recipe-test-01.yml started
81db9aa HSToken draft (from internal.keystore)
a57f77d hdpcli, hxlm (#11): HKeystore created; proof of concept of how to parse encrypted URNs (urnresolver #13) may require some way generic adapter
6317513 hxlm (#11): _get_keypar() started
fe6cb62 hxlm (#11): HDataDispatch; flight dispatcher, but for data
da44daf hxlm, hdpcli (#11): Started draft of HWorkspace; lots of reading about which crypto high level library to use (and also about care about developers usability
b74dc68 hdpcli (#11): we will definely will need keyring, in special for who would not have hardware smartcards
d8992f6 hdpcli (#11): MVP of _get_salt() & _get_fernet(); created _entropy_pseudotest()
1840c6e hdpcli (#11): drafted _get_salt() & _get_fernet()
8816aad hdpcli (#11): _exec_hdp_init_data() works
62e4305 hdpcli (#11): created prompt_confirmation() and a bunch of if elses
3025360 hdpcli (#11): drafted cli --hdp-init, --hdp-init-home & --hdp-init-data
44e0191 hdpcli (#11): started, based on hxl2example
0edcecf hxlm (#11), urnresolver (#13): python cryptography lib is now an requeriment (need when dealing with encryted shared list of URN)
1dcb6d4 v0.7.4 started
a8fbfc2 urnresolver (#13): urn-data-specification exclusive folder started
881a96d hxlm (#11), urnresolver (#13): --no-urn-user-defaults & --no-urn-vendor-defaults CLI options added
36c80ef hxlm (#11), urnresolver (#13): Why use URN to identify resources is more than naming convention
3379f0a hxlm (#11), urnresolver (#13): documentation of urnresolver current state
2e6d30c hxlm (#11), urnresolver (#13): initialize with some defaults (if user did not customized yet), file urnresolver-default.urn.yml
e9a6d67 hxlm (#11), urnresolver (#13): 22 commits/hours later, now we start to have something testable
33eb104 hxlm (#11), urnresolver (#13): drafted concept of 'urnref' (something like when have several sources or URNs, allow urnresolver filter sources (at first just use file names)
041420e hxlm (#11), urnresolver (#13): added loader for YAML & JSON; for TravisCI, removed hardcoded path; renamed urn:data:xz:hxl:std:core:hashtag (std inspired on ISO RFC) to urn:data:xz:hxl:standard:core:hashtag (maybe core is not need?)
9aa8010 hxlm (#11), urnresolver (#13): now also with TSV files!
9d468ea hxlm (#11), urnresolver (#13): drafted HXLM_DATA_URN_EXTENSIONS_ENCRYPTED
926019b hxlm (#11), urnresolver (#13): working on get_urn_resolver_local(), file order is important
77895bc hxlm (#11), urnresolver (#13): created get_urn_resolver_local() and draft of get_urn_resolver_remote() & get_urn_resolver_remote_authenticated()
045fd2c urnresolver (#13): added .well-known proof of concept test files; renaming files to be just urn.csv,urn.json,urn.yml (this is more an suffix, since users could in fact search for entire paths)
aed3208 urnresolver (#13): Added URNResolver test files
ad3a6a0 hxlm (#11), urnresolver (#13): clean up
ba9f034 hxlm (#11), urnresolver (#13): fixed issue with non-ASCII internationalized organization names
a5e76a6 hxlm (#11), urnresolver (#13): for DataUrnHtype() testing using even domain names as quick namespace; full unicode support need more testing (fallbacking to GenericUrnHtype instead of DataUrnHtype)
b21beac hxlm (#11), urnresolver (#13): for DataUrnHtype() some baseline parser before need to resort to full grammar checking; this at least could help locallized namespaces get what matter for then, while keeping the start of the 'urn:data' predictable
99fceae hxlm (#11), urnresolver (#13): for DataUrnHtype(); so many options to buld full parser to avoid regex hell; I think maybe the early versions could be a bunch of effective if-elses
7102e28 hxlm (#11), urnresolver (#13): for DataUrnHtype(), reading about create a formal ABNF (like the ISO URN, RFC5141); but ANTLR seems more friendly
a8306ad hxlm (#11), urnresolver (#13): drafted HXLM_CONFIG_BASE, ..POLICY.. & ..VALT..
d7cc2b3 hxlm (#11), urnresolver (#13): started hxlm.core.schema.urn.util
0660705 hxlm (#11), urnresolver (#13): started DataUrnHtype() [urn:data:, urn:data-i, urn:data-p]
8468d30 hxlm (#11), urnresolver (#13): code clean up; draft of documentation
17900d4 hxlm (#11), urnresolver (#13): './.tox/py38/bin/urnresolver --help' at least dont return error;
7b6502f hxlm (#11), urnresolver (#13): moving urnresolver as entry_points instead of script (more strict control, force to be python script, not generic system scrypt)
56bfc7b hxlm (#11), urnresolver (#13): started urnresolver based on hxl2example
5f81dd8 hxlm (#11): improved IetfUrnHtype() works [for 'urn:ietf:rfc' prefixes']
b21d6c8 hxlm (#11): improved IetfUrnHtype(); added GenericUrnHtype().prepare() method
894a0a3 hxlm (#11): started IetfUrnHtype()
3270123 hxlm (#11): UrnHtype, testing how to work with dataclasses; created GenericUrnHtype to be an more generic method (so would work with URNs like the IETF RFC or the ISO ones)
6bd4ad5 hxlm (#11): URNs now are only on hxlm.core.htype.urn (removed from schema)
041c83e hxlm (#11): on hxlm.core.schema.urn.HURN() drafted as an Htype, URNHtype()
48b924d hxlm (#11): on HURN() added is_valid()
102de60 hxlm (#11): started hxlm.core.schema.urn.HURN class
17b6a79 hxlm (#11): added core_vocab hsilo (named internal code for hdataset/hfile; but not for hcompliance
a14f96e automated-testing (#12): documented that, with tox, is possible to use more than Travis CI, but Azule Pipelines, docker, etc etc etc
9366706 automated-testing (#12): Added .travis.yml; testing plugin tox-dev/tox-travis
b0f6a39 hxlm (#11): to hxlm.core.schema.ItemHVocab() added merge() and diff()
5046153 hxlm (#11): to hxlm.core.schema.ItemHVocab() added parse_yaml(), to_dict(), to_json(), to_yaml()
d5945e6 hxlm (#11): documented concepts behind schema.vocab
a2bba16 hxlm (#11): renamed hxlm.core.schema.{vocab.yml -> core_vocab.yml}; created VocabHSchema
b6f14ee hxlm (#11): improved organization of tests
c12f99f hxlm (#11): around 100 lint fixes
c3dfe19 hxlm (#11): improved tox; some general quick lint fixes
6cb2bbc hxlm (#11): added tox (automated testing) to help cope with essential complexity; some more advanced tests may come later
31b5044 hxlm (#11): previous fixes to avoid circular imports with new ConversorHSchema
4255f48 hxlm (#11): ConversorHSchema class started
fbadf87 hxlm (#11): Right-to-left (RTL) scripts will need extra attention; also vocab.yml change from CRLF to LF
d96483b hxlm (#11): vocab.yml, added attr.desc and quick overview of the reasoning of vocab.yml
f293164 hxlm (#11): vocab.yml, added attr.adm0, attr.lang, attr.meta
9d9a486 hxlm (#11): vocab.yml, added attr.tag & attr.source
bf97ac2 hxlm (#11): vocab.yml, added hcompliance; 'acceptable use policy (AUP)' actually seems the best term in English; maybe worthe change internal names
1051db7 hxlm (#11): vocab.yml, added attr.iri/attr.urn ('Internationalized Resource Identifier'/'Uniform_Resource_Name')
e1283cd hxlm (#11): vocab.yml, added hfile; hdataset now uses '-' to replace missing spaces betwen words
c5f55ce hxlm (#11): v0.7.3 started; created draft of vocab.yml
f34cf38 hxlm (#11): testing Declarative Programming more strict approach
8595c68 hxlm (#11): HFile implements is_available_sources() and reload_from_souces()
808e34f hxlm (#11): Draft of SStorage (not sure if abstract ramfs or let tempfs already on core); HFile now starts to check is_available_locally
818620e hxlm (#11): HRecipe (HXL-proxy JSON recipes) is getting better; almost there to make work with URL
554b37f hxlm (#11): HMeta export all subschemas (but still using YAML raw object; still need some extra abstraction
51f0c64 hxlm (#11): moved RecipeHType from HType to Model; improved HMeta way to parse raw schemas
4118708 hxlm (#11): improved way HMeta load hdatasets
cfe0824 hxlm (#11): started HMeta class, hxlm.core.model.meta
27092a9 hxlm (#11): draft of hrecipes/hrecipe on YAML file; needs more abstraction at Model level to load the YAML
8730fa7 hxlm (#11): RecipeHtype, HXLProxyRecipeHtype and ItemHXLProxyRecipeHtype created
287c996 hxlm (#11): hxml.core.data.baseline.hmeta.yml now as array (like Ansible playbooks, but allowing several hmetas on same file
ef006ab hxlm (#11): hxlm.core.schema.baseline.yml added HTYPE_PRIMITIVES
352289d hxlm (#11): hxlm.core.schema.baseline.yml created
270daa4 hxlm (#11): improved organization of hxlm.core.schema
afc3bbb hxlm (#11): started hxlm.core.schema
7cf25f8 hxlm (#11): testing YAML instead of JSON; RTLs like Arabic/Hebrew (think allow even localized also on key) may be harder to do quick tests since my prefered code editor have this https://github.com/Microsoft/vscode/issues/11770
ac74732 hxlm (#11): v0.7.2 started; let's try YAML instead of JSON for files used by HConteiner (temporary name)
74db2ba hxlm: hxlm.core.HFile class created; changed explanation of what HConteiner means (Maybe the name should go back to something with 'Meta', because we maybe could even reconstruct the files on disk just with metadata
03670a3 hxlm: started of hxlm.schema.json from Python classes (allow autocomplete on code editors\!); added hxlm.data.baseline: place (v_iso2, v_iso3, ...) and lang (v_iso639_3)
c8fcef5 hxlm: created hxlm.routing.request_priority_access()
cf47b27 hxlm: draft of ResourceRoutingHtype; created hxlm.routing.request_cache_resource()
0eb5bd9 hxlm: draft of PleaseRoutingHtype
02bd3d2 hxlm: draft of RoutingHtype; an specialized X-Forward-By, "X-HRouting-By"
9cff6d0 hxlm: hxlm.routing non-core subpackage started; print('We <3 MANRS')
7b93f12 hxlm: hxlm.core.util.debug() now output all levels submodules; improved hxml.core.compliance.verbose_event()
a991299 hxlm: added HConteiner().describe() and hxlm.core.compliance.verbose_event()
80d4019 hxlm: drafted LicenseHtype & CopyrightHtype
e28af3e hxlm: drafted HConteiner (explicitly different from plural of Datasets to avoid confusion; Also naming HDataset); not sure about names
167330e hxlm: hxlm.core.base.MDataset class created; but both Excel, CKan and formats like HDF5 work with MULDIPLE datasets. Still need to abstract
661fc7b tests/hxlated-xarray-pandas-dask-test.py: tests with dask (not dusk)
e06dd79 tests/hxlated-xarray-pandas-dust-test.py draft (quick test with other libraries)
cf08576 hxlm: Added HTYPE_ENCRYPTED="!"; HPTL_ [Data Processor (human, organization) Trust Level] and HITL_ [Data Infrastruture Trust Level]"
0f941ac hxlm: added file htype/actor.py; added check_compliance_on_after_decrypt_row(), check_compliance_on_before_decrypt_row()
5fc41dc hxlm: added check_compliance_on_after_decrypt_column(), check_compliance_on_before_decrypt_column() & check_compliance_on_termination()
c1dab73 hxlm: started hxlm.core.compliance module
9a8ccda hxlm: moved from setup.py to setup.cfg; created hxlm-crypto-eticaai as different project (eventually most plugins could be moved too);
51744fb hxlm: moved plugins to subfolder plugins (this clear a bit the namespace; so many plugins for a same country)
8a1a1c6 hxlm: added renamed variables related to Data Sensivity Level
c5ac16f hxlm: added cmp_sensitive_level()
7cc5fc4 hxlm: added Information sensitivity levels HSLVU, HSLV0, HSLV1, HSLV3, HSLV3, HSLV4
4f26b71 hxlm: added hxlm.taxonomy.get_lang(), hxlm.core.hxl_info()
cb2cba5 hxlm: added xa_rc and xl_por
e5be8c7 hxlm: added HXLmPoliticException & HXLmCrimeException
454f632 hxlm: added taxonomy plugin; renamed xa_brsus -> xa_sus
b4ce231 hxlm: added xa_gdpr, xa_humanitarian & xa_lgpd draft plugins
455c96e hxlm: documented prefixes
ef9a792 hxl2encryption & hxl2encryption v0.1-draft created, based on hxl2example
bf9bd92 hxlm: docstrings, pylint