-
-
Notifications
You must be signed in to change notification settings - Fork 23
/
CHANGES
1107 lines (994 loc) · 54.2 KB
/
CHANGES
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
3.8.2
Enhancements:
* The MergerConfig class now accepts overrides for config values as "keys" and
"rules" keyword arguments to the constructor.
Credit and my thanks go to https://github.com/leviem1!
BREAKING CHANGES:
* Support for Python 3.6 has been dropped. This is forced by incompatibilities
discovered with the latest version of pytest and because dependencies like
dateutil and ruamel-yaml-clib no longer support Python 3.6. Support for
Python 3.7 is tepid. While pytest is still working with Python 3.7, other
dependencies are no longer supporting Python 3.7; however, the extensive
tests for yamlpath show no issues with them, so far. For now, Python 3.12
support is pending, waiting for the dateutil library to resolve a
DeprecationWarning regarding its use of datetime.datetime.utcfromtimestamp().
3.8.1
Bug Fixes:
* Fixed issue #220 reported by https://github.com/abramov-oleg wherein novel
mergeat paths given to the yaml-merge tool with Array and Array-of-Hash data
in the RHS document would cause an interminable loop that would max out CPU
and eventually exhaust RAM.
3.8.0
Enhancements:
* The yaml-set and yaml-merge command-line tools now support a new option:
--json-indent/-J. This applies only to JSON output. By default, JSON
documents are written/printed as single-line documents. This new option
enables users to vertically expand JSON output with a specified indentation
size for nest levels.
* Additional *YAMLPathExceptions have been added to increase the specificity of
various error conditions. You do not need to change any of your code as a
result of this change; all new exceptions are subclasses of the original
YAMLPathException. The new exceptions include:
* UnmatchedYAMLPathException: No matching nodes
* DuplicateKeyYAMLPathException: Duplicate key creation attempted
* TypeMismatchYAMLPathException: Data Type mismatch
* BadAliasYAMLPathException: Anchor/Alias/YMK error
* NoDocumentYAMLPathException: Attempt to delete the entire document
* RecursionYAMLPathException: Recursion nesting error
* The Processor class now enables consumers to verify whether a YAMLPath would
resolve to at least one node in the present document. The result is a simple
Boolean (True when the path resolves; False, otherwise). The document is not
modified by this test. See: Processor.exists(yaml_path)
Bug Fixes:
* Pursuant to the addition of the intersection operator (&) added in 3.7.0, the
text of one of the YAMLPathExceptions has been updated:
Former:
Adjoining Collectors without an operator has no meaning; try + or -
between them, {yaml_path}
New:
Adjoining Collectors without an operator has no meaning; try +, -, or &
between them, {yaml_path}
3.7.0
Enhancements:
* Support for Python 3.11 has been added.
* A new Search Keyword has been added, [unique([NAME])]. This operates against
collections to return only values which have no duplicates within the
collection; i.e.: [1, 2, 2, 3] has unique values, [1, 3]. The NAME argument
is required when operating against Hashes (maps/dicts) and Arrays-of-Hashes
(sequences/lists of maps/dicts) to identify which field/property to evaluate
for uniqueness. This can be inverted to return only results which are
duplicated within the collection.
* A new Search Keyword has been added, [distinct([NAME])]. This operates
against collections to return exactly one of every value within the
collection, discarding duplicates; i.e.: [1, 2, 2, 3] has distinct values,
[1, 2, 3]. The NAME argument is required when operating against Hashes
(maps/dicts) and Arrays-of-Hashes (sequences/lists of maps/dicts) to identify
which field/property to evaluate for uniqueness. This cannot be inverted.
* Added a new Collector Math Operator: & (intersection)
As in set mathematics, this yields only elements of two collections which are
common to both collections. Unlike set mathematics, collections allow
duplicate elements. If you need to enforce distinctness of the intersected
results, use the [distinct([NAME])] Search Keyword against the collected
result, as in `((list1)&(list2))[distinct([NAME])]`.
Bug Fixes:
* A typographical error in yamlpath.enums has been corrected with backward-
compatible adapters in place to support both the correct and incorrect
spelling of PathSeparators (formerly PathSeperators). If the PathSeperators
version appears in your own code, please update to the new spelling. The
incorrectly spelled version of this enumeration is now deprecated and will be
removed in a future release. Thanks go entirely to
https://github.com/AndydeCleyre for working so hard to submit this fix!
* Processor.get_nodes would emit a nonobvious error message when mustexist is
left at its default value (False) and yaml_path contained ** or * segments.
Now, these segment types are excluded from generating "missing" nodes; only
nodes which exist can be matched by * and **. Credit and my thanks go to
https://github.com/gdubicki for discovering and reporting this issue.
3.6.9:
Enhancements:
* Partial support for updating bare Python dict and Python's native
collections.OrderedDict data structures was removed in version 3.6.8 because
compatible YAML/EYAML/JSON data never presented as these data types and if
anyone ever attempted to update a key by reference in a dict or
collections.OrderedDict, it would cause a Python stack dump due to neither
supporting the required insert method, which is provided only by ruamel.yaml.
This version not only restores this capability, but also solves the issue of
missing support for the insert logic, where applicable. It also adds support
for the ruamel.yaml.compat.ordereddict type. Thanks to
https://github.com/tsinggggg for requesting this feature be added!
3.6.8
Bug Fixes:
* Changes to format and value of child nodes under Anchored Hashes (maps/dicts)
caused unexpected expansion of the same key-value pair wherever a YAML Merge
Key Aliased the affected Anchored Hash. Thanks to
https://github.com/hemnstill for finding and reporting this issue!
Enhancements:
* Support for Python 3.10 has been added, thanks to https://github.com/mechie!
3.6.7
Bug Fixes:
* Release 3.6.6 had a broken package; it was unexpectedly missing the required
ruamel.yaml patch found in yamlpath.patches.timestamp. Thanks to
https://github.com/tsinggggg and https://github.com/kaniblu for reporting it!
3.6.6
Enhancements:
* Support ruamel.yaml up to version 0.17.21.
Bug Fixes:
* YAML timestamp values could not be created via yamlpath tools or its library,
per http://yaml.org/type/timestamp.html.
* CAUTION 1: ruamel.yaml seems to force all timestamps to UTC while it loads
YAML/JSON/Compatible data. So, when the timestamp value contains time-zone
data, it will be stripped off after it is applied to the timestamp value.
The yamlpath library reverses this when emitting the affected values but if
you attempt to load the timestamp values directly in the DOM, you'll end up
with the UTC-converted value, stripped of any time-zone specification. If
you need the original, unmodified data as a time-zone aware
datetime.datetime value, pass it through the helper method,
Nodes.get_timestamp_with_tzinfo(data: AnchoredTimeStamp).
* CAUTION 2: In order to support timestamp parsing, this project now depends
on the python-dateutil library. Be sure to install it!
Thanks again go to https://github.com/AndydeCleyre!
3.6.5
Bug Fixes:
* When using EYAML with block formatted values on Windows, the block formatting
was broken when written to YAML, corrupting the YAML file. Non-block
formatted values were unaffected and this issue only affected Windows EYAML
users.
* EYAML values were not being decrypted when the result appeared in a list.
* Boolean values were being unexpectedly output as base-10 representations of
the binary values 1 (True) and 0 (False) when emitting to JSON after setting
the Boolean. This change also brings this project into compilance with
https://peps.python.org/pep-0632/. Many thanks to
https://github.com/AndydeCleyre!
3.6.4
Enhancements:
* Support ruamel.yaml up to version 0.17.17.
Bug Fixes:
* Refactored single-star wildcard segment (*) handling to enable filtering
matches when subsequent segments exist; this fixes Issue #154.
3.6.3
Bug Fixes:
* The eyaml-rotate-keys command-line tool failed to preserve block-style EYAML
strings after key rotation.
3.6.2:
Bug Fixes:
* The eyaml-rotate-keys command-line tool would generate a stack-dump when the
key for an encrypted value contained dots. The underlying library for this
tool now safely generates the internal YAMLPaths it uses.
* The default encoding when opening files is now set to utf-8 to support
extended character sets on Windows.
3.6.1:
Enhancements:
* Enable verified support for ruamel.yaml up to version 0.17.10.
3.6.0:
Bug Fixes:
* Some peculiar data constructions could cause the ConsolePrinter to stack-dump
when writing debug messages.
* A NotImplementedError stack-dump was generated whenever a YAML Path was
evaluated when written improperly for Collector Math operations. This
specifically occurred when the RHS term was not demarcated with a parenthesis
pair. Now, a YAML Path parsing error is generated, indicating where the
missing character must appear.
* Use of an Array Element index in square-bracket notation ([N]) within a
sub-path -- such as in a Search Expression or Collector -- caused incorrect
YAML Path parsing. This usually manifested as a "not an integer index"
error.
* Byte strings were causing stack-dumps during JSON serialization; they are now
serialized as double-demarcated strings (a ' pair within a " pair) with a b
prefix, like: {"byte_value": "b'BYTEVAL'"}.
* Bare Anchor name references were treated as Hash key names when & was not the
very first non-separator character of a YAML Path or immediately following a
`[`. So, /my_hash/&my_anchor was not working as expected.
* The Merger (and thus, the yaml-merge command-line tool) would only return the
LHS document when neither LHS nor RHS documents were Hashes, no matter what
the merge options were set to. This did not affect content which was
children of Hashes.
Enhancements:
* YAML Path parsing errors are now a little more specific, indicating at which
character index the issue occurs. API users who have been scanning error
messages will need to update their code to accommodate the new text.
* Collector subtraction now handles Hash-of-Hashes and Array-of-Array results,
which were not possible before.
* Array-of-Hash nodes can now be searched for the presence of a given key in
its Hash elements using the . search operand, yielding matching elements (the
entire Hash elements having the given key). The difference can be
illustrated by contrasting these now-equivalent YAML Paths (where "books"
is an Array-of-Hashes; imagine only some Hash elements have an "isbn" key):
1. `/books/*[.=isbn]` or `books.*[.=isbn]`
2. `/books[.=isbn]/isbn` or `books[.=isbn].isbn`
3. `/books/*[has_child(isbn)]/isbn` or `books.*[has_child(isbn)].isbn`
4. `/books[has_child(isbn)]/isbn` or `books[has_child(isbn)].isbn`
All four of those queries yield exactly the same data. Note that example 2
behaves like examples 3 and 4. Examples 2-4 yield the entire matching Hash,
not just the "isbn" value. This enables access to other keys of the Hash
without necessitating use of a `[parent()]` search keyword, which would be
necessary for example 1 if you wanted to access any key other than "isbn"
from the matches.
* YAML Merge Keys can now be accessed directly by Anchor name, yielding the
entire original -- pre-merged -- reference Hash. This has _very limited_
utility. Using this in isolation will only reveal the default values for any
referenced keys, ignoring -- perhaps confusingly -- any local overrides. It
can however be helpful when reverse-engineering very complex merge
arrangements.
* The yaml-merge command-line tool (and the underlying Merger class) now offer
an option -- --preserve-lhs-comments (-l) -- that will attempt to preserve
LHS document comments. USE WITH CAUTION. At present, comment handling
during a merge is unwieldy, so some comments or new-line characters may
appear to become divorced from nodes they should obviously be attached to.
As such, the default behavior of the merge engine will continue to be removal
of all comments. At this time, RHS document comments will still be discarded
during merge operations. This will be revisited when ruamel.yaml refactors
how YAML comments are handled.
* The yaml-merge command-line tool now offers a new option, --multi-doc-mode
(-M), which accepts one of the following modes:
* CONDENSE_ALL: This is the default, which merges all multi-documents up
into single documents during the merge.
* MERGE_ACROSS: Condense no multi-documents; rather, only merge documents
"across" from right to left such that the first document in the RHS multi-
document merges only into the first document in the LHS multi-document, the
second across similarly, and so on.
* MATRIX_MERGE: Condense no multi-documents; rather, merge every RHS
document in a multi-document RHS into every LHS document in a multi-
document LHS.
* The [has_child(NAME)] Search Keyword now accepts an &NAME form of its first
(only) parameter. This switches the function to match against Anchor/Alias
names, including YAML Merge Keys.
* YAML Merge Keys can now be deleted by their Anchor/Alias name via the
yaml-set command-line tool and the underlying Processor class.
* YAML Merge Keys can now be created, offering run-time merging of
same-document Hash data. The yaml-set command-line tool offers a new option,
--mergekey, which applies to --change targets the new YAML Merge Key, as long
as each target is a Hash.
WARNING: As a consequence of adding this capability to the yaml-set command-
line tool, it is no longer possible to implicitly alias scalar nodes by
passing only the --change and --anchor parameters. The operation must now be
explicit by setting --aliasof or --mergekey along with --change and
optionally with --anchor.
* The yaml-diff tool now supports multi-document sources. Only one document of
any multi-document source on each side of the LHS-RHS comparison can be
selected for the operation (diffs are performed only between two documents).
Such selection is made via two new options, --left-document-index|-L and
--right-document-index|-R. An error is emitted whenever a multi-document
source is detected without an appropriate document index selection.
* YAML Unordered Sets -- https://yaml.org/type/set.html -- are now fully
supported in YAML Paths, this project's API, and the reference command-line
tools. Because an Unordered Set is effectively a Hash (map/dict) where the
entries are key-value pairs all having null (None) values, their entries are
accessible only by their exact key. While they look in YAML data like Arrays
(sequences/lists) with a leading `?` rather than a `-`, they are not; their
entries cannot be accessed by a numerical index because they are defined in
the YAML specification as deliberately unordered.
API Changes:
* The common.nodes utility class now has a generally-useful static method which
accepts any String data and safely converts it to its native Python data-type
equivalent with special handling for case-insensitive Booleans via
ast.literal_eval: typed_value.
* The common.searches utility class now requires both terms to be of the same
data-type for comparisons. When they types materially differ -- int and
float are treated as similar enough -- a String comparision is performed.
This is how it has always been excepting that types were lazily coalesced in
older versions; they are now converted before the comparison is considered.
* The NodeCoords wrapper now supports more utility properties and methods:
* .unwrapped_node is the same output as calling
NodeCoords.unwrap_node_coords(data) except it can be called directly upon
instances of NodeCoords rather than as a static method call. The static
method is still available.
* .deepest_node_coord returns whichever NodeCoord instance is most deeply
wrapped by an instance of NodeCoords. Such wrapping comes from nesting
Collectors. This method simplfies getting to the original data element(s).
* .wraps_a(Type) indicates whether the deepest wrapped data element is of a
given data-type.
3.5.0:
Bug Fixes:
* Search expressions against Boolean values, [key=True] and [key=False], were
impossible. Now, they are working and are not case-sensitive, so [key=True],
[key=true], [key=TRUE], and such all work as expected.
* When null values were present, Deep Traversal (**) segments would always
return every node with a null value even when they would not match filter
conditions after the ** segment. When mustexist=False, this would also cause
a YAMLPathException.
* Descendent searches were considering only the first child of the search
ancestor. Now, ANY matching descendent node will correctly yield the
ancestor.
* Some Python-generated complex data types were escaping JSONification,
leading to unexpected stack-dumps when writing out JSON data for data types
like date and datetime.
Enhancements:
* An entirely new segment type has been added to YAML Path and is now supported
by the library and reference implementation command-line tools: Keyword
Searches. Similar to programming language keywords, these reserved Keywords
work much like functions, accepting parameters and performing algorythmic
operations or returning data not otherwise accessible to other YAML Path
segment types. These new capabilities -- explored on the project Wiki --
include:
* [has_child(NAME)]
* [name()]
* [max(NAME)]
* [min(NAME)]
* [parent([STEPS])]
* When stringified, YAML Paths with a solitary * wildcard segment were printed
using their internal RegEx variant, [.=~/.*/]. They are now printed as they
are entered, using a solitary *. As a consequence, any deliberate RegEx of
[.=~/.*/] is also printed as its equivalent solitary *.
* The yaml-paths command now allows printing YAML Paths without protective
escape symbols via a new --noescape option. While this makes the output more
human-friendly, the unescaped paths will not be suitable for use as YAML Path
input to other YAML Path processors where special symbols require escaping.
* [API] The NodeCoords class now tracks ancestry and the last YAML Path segment
responsible for triggering its generation. The ancestry stack --
List[AncestryEntry] -- was necessary to support the [parent()] Search
Keyword. The responsible YAML Path segment tracking was necessary to enable
Hash/map/dict key renaming via the [name()] Search Keyword. These optional
attributes may be set when the NodeCoords is generated.
* [API] YAMLPath instances now have a pop() method. This mutates the YAMLPath
by popping off its last segment, returning that segment.
3.4.1:
Bug Fixes:
* yaml-set (and the underlying Processor class) were unable to change nodes
having a null (None) value to anything else. This changes how null/None
values are handled by the Processor during node retrieval; they are no longer
discarded, so you will receive None as the data of any retrieved NodeCoords
for appropriate null/None leaf nodes.
Enhancements:
* Python 3.9 is now supported (because common testing tools finally work with
Python 3.9).
* The node deletion capability of the yaml-set command is now part of the
library. See Processor::delete_nodes(...) and
Processor::delete_gathered_nodes(...) for details.
* The node aliasing capability of the yaml-set command is now part of the
library. See Processor::alias_nodes(...) and
Processor::alias_gathered_nodes(...) for details.
* The node tagging capability of the yaml-set command is now part of the
library. See Processor::tag_nodes(...) and
Processor::tag_gathered_nodes(...) for details.
* The library now supports loading YAML from String rather than only from file.
Simply pass a new `literal=True` keyword parameter to
Parsers::get_yaml_data(...) or Parsers::get_yaml_multidoc_data(...) to
indicate that `source` is literal serialized (String) YAML data rather than a
file-spec. This mode is implied when reading from STDIN (source is "-").
* The emitter_write_folded_fix.py patch file for ruamel.yaml has been removed
in favor of an author-supplied solution to the problem --
https://sourceforge.net/p/ruamel-yaml/tickets/383/ -- for which the patch was
originally written.
Known Issues:
* ruamel.yaml version 0.17.x is a major refactoring effort by the project's
owner. As such, only select versions will be marked as compatible with
yamlpath. Such marking occurs in this project's dependencies list via the
setup.py file. This is necessary because I use yamlpath in production
environments where stability is paramount; I need the freedom to update
yamlpath at-will without incurring any unexpected failures due to
incompatible ruamel.yaml changes. I will try to test some -- but not all --
ruamel.yaml releases from time to time and update yamlpath dependency
compatibilities accordingly.
* ruamel.yaml version 0.17.4 somewhat resolves a previously reported issue --
https://sourceforge.net/p/ruamel-yaml/tickets/351/ -- wherein certain
arrangements of comments or new-lines within YAML files near aliased hash
keys would cause a total loss of data when the stream was written to file.
Now, the data is no longer entirely lost. However, the preceding comment or
new-line is deleted when the stream is written to file. This is deemed to be
an acceptable compromise, for now, because the alternative is to either lose
the entire document or lose all attempted changes to the affected document.
Until the issue is properly fixed, an XFAIL test will continue to be in the
yamlpath unit test suite.
3.4.0:
Bug Fixes:
* For the yaml-diff command-line tool, custom identity keys for specific
records in AoHs of the RHS document were not being considered for comparison.
Any identity key set for the whole AoH was being applied to every record in
the set.
Enhancements:
* The yaml-get command-line tool -- and the underlying Processor::get_nodes
method -- now retrives nodes with `null` values. Non-JSON null results from
yaml-get are printed using the ASCII NULL control-character (Hexadecimal 00)
wherever encountered. While most terminals and shells won't visibly print
this character, it is there and can be picked up by downstream parsers of
STDOUT. When the output is rendered as JSON, the unquoted "null" identifier
is used.
* The yaml-set command-line tool now enables assigning null values using a new
--null|-N input option.
* The yaml-set command-line tool now supports assigning YAML Aliases to target
--change|-g nodes, referencing any other node via --aliasof|-A whether it is
already Anchored, or not. Should the Anchor not already exist, a unique name
will be automatically assigned. This automatic name can controlled via
--anchor|-H. Setting --change|-g and --aliasof|-A to the same node along
with a new --anchor|-H explicitly renames an existing Anchor and its Aliases.
The same is implicitly possible by specifying --change|-g and --anchor|-H
without --aliasof|-A. Using these against non-YAML files merely duplicates
the target value to the indicated --change|-g nodes.
* With a new --tag|-T auxilliary option, the yaml-set command-line tool now
supports assigning custom YAML tags (data-type specifiers) to nodes created
or updated during the operation.
* The yaml-merge and yaml-diff command-line tools now also handle YAML tags.
* The single-star wildcard segment translation was changed from [.!=""] to
[.=~/.*/] which enables it to match any key or value (including empty-string
and null).
API Changes:
* The yamlpath.func library is too big and some very useful general code has
become attached to special-purpose classes. As such, it's time for
refactoring before this becomes untennable. New, better-organized static
classes were created under a new yamlpath.common namespace. Since this
would be destructive to other Python code which depends on the old
organization, this is a heads-up. Your existing code will continue to work
without any changes, but only until 4.x (which is not yet planned). Until
then, you must update your own code to use the new static classes for the
following functions and methods so as to avoid fatal errors down the road:
- yamlpath.func.append_list_element
-> yamlpath.common.Nodes.append_list_element
- yamlpath.func.build_next_node
-> yamlpath.common.Nodes.build_next_node
- yamlpath.func.clone_node
-> yamlpath.common.Nodes.clone_node
- yamlpath.func.create_searchterms_from_pathattributes
-> yamlpath.common.Searches.create_searchterms_from_pathattributes
- yamlpath.func.ensure_escaped
-> yamlpath.YAMLPath.ensure_escaped
- yamlpath.func.escape_path_section
-> yamlpath.YAMLPath.escape_path_section
- yamlpath.func.get_node_anchor
-> yamlpath.common.Anchors.get_node_anchor
- yamlpath.func.get_yaml_data
-> yamlpath.common.Parsers.get_yaml_data
- yamlpath.func.get_yaml_editor
-> yamlpath.common.Parsers.get_yaml_editor
- yamlpath.func.get_yaml_multidoc_data
-> yamlpath.common.Parsers.get_yaml_multidoc_data
- yamlpath.func.make_float_node
-> yamlpath.common.Nodes.make_float_node
- yamlpath.func.make_new_node
-> yamlpath.common.Nodes.make_new_node
- yamlpath.func.search_anchor
-> yamlpath.common.Searches.search_anchor
- yamlpath.func.search_matches
-> yamlpath.common.Searches.search_matches
- yamlpath.func.stringify_dates
-> yamlpath.common.Parsers.stringify_dates
NOTE that this method is deprecated and will be eliminated in favor of
using its more comprehensive replacement,
yamlpath.common.Parsers.jsonify_yaml_data
- yamlpath.func.unwrap_node_coords
-> yamlpath.wrappers.nodecoords.NodeCoords.unwrap_node_coords
- yamlpath.func.wrap_type
-> yamlpath.common.Nodes.wrap_type
- yamlpath.merger.Merger.combine_merge_anchors
-> yamlpath.common.Anchors.combine_merge_anchors
- yamlpath.merger.Merger.delete_all_comments
-> yamlpath.common.Anchors.delete_all_comments
- yamlpath.merger.Merger.rename_anchor
-> yamlpath.common.Anchors.rename_anchor
- yamlpath.merger.Merger.replace_anchor
-> yamlpath.common.Anchors.replace_anchor
- yamlpath.merger.Merger.scan_for_anchors
-> yamlpath.common.Anchors.scan_for_anchors
- yamlpath.merger.Merger.set_flow_style
-> yamlpath.common.Anchors.set_flow_style
Until you update your code, a deprecation warning will be printed to STDERR
every time the yamlpath.func file is imported and the first time one of the
relocated Merger static methods is called. To be rid of the message, update
your code to use the new sources of the deprecated functions/methods and
remove every import of and from yamlpath.func.
3.3.0:
Bug Fixes:
* It was impossible to install yamlpath 3.x without first installing
ruamel.yaml via pip for Python 3.x. Not only has this been fixed but
explicit tests have been created to ensure this never happens again.
Enhancements:
* A new command-line tool, yaml-diff, now compares exactly two
YAML/JSON/Compatible documents, producing a GNU diff-like report of any
differences in the data they present to parsers. Along with diff's "a"
(added), "c" (changed), and "d" (deleted) report entries, affected YAML Paths
are printed in lieu of line numbers. Further, a report entry of "s" (same)
is available and can be enabled via command-line options. This tool also
features optional special handling of Arrays and Arrays-of-Hashes, which can
be configured as CLI options or via an INI file for distinct settings per
YAML Path. See --help or the Wiki for more detail.
API Changes:
* NodeCoords now employ a new `path` attribute. This is an optional parameter
which is assigned during construction to later report the translated origin
YAML Path; this is where the node was found or created within the DOM. Note
that Collector segments work against virtual DOMs, so the YAML Path of an
outer Collector will be virtual, relative to its parent at construction; when
nested, this will be a bare list index. Any NodeCoords in the virtual
container which point to real nodes in the DOM will have their own concrete
YAML Paths.
* YAMLPath instances now support nonmutating addition of individual segments
via the + operator. Whereas the append() method mutates the YAMLPath being
acted upon, + creates a new YAMLPath that is the original plus the new
segment. In both cases, the orignal YAMLPath's separator is retained during
both operations. As with .append(), new segments added via + must also be
properly escaped -- typically via path.escape_path_section -- before being
added.
3.2.0:
Enhancements:
* Expanded YAML Path Search Expressions such that the OPERAND of a Search
Expression may be a sub-YAML Path. This enables searching descendent nodes
-- without moving the document pointer -- to yield ancestors with matching
descendants. This has more utility when searching against Arrays-of-Hashes.
Bug Fixes:
* Date values in YAML could not be written to JSON streams; the JSON renderer
would generate an incompatibility error. Now, dates are written as Strings
to JSON. This affected: yaml-get, yaml-set (in stream mode), yaml-merge, and
yaml-paths.
3.1.0:
Enhancements:
* yaml-set can now delete nodes when using --delete rather than other input
arguments.
* A new command-line tool has been created, yaml-validate, which validates
YAML/JSON/compatible single- and multi-documents (files or STDIN).
Bug Fixes:
* The yaml-merge command-line tool wasn't allowing Scalar values supplied via
STDIN -- with no structure, just bare Scalar values -- to be appended to
exising Array data structures; rather, it was wholly overwriting the
destination, deleting all pre-exisiting elements.
* The yaml-merge command-line tool wasn't accepting empty-strings as STDIN
documents; it was reporting a document-read error, instead. This turns out
to be useful when you want to use yaml-merge instead of yaml-set to
deliberately write an empty-string value at some --mergeat location within
the LHS document.
* The yaml-merge command would not accept any variation of "false" as a Scalar
value input; it was instead reporting a document-read error. This turns out
to be useful when using yaml-merge as if it were yaml-set to write a false
Boolean value to the LHS document at some --mergeat location.
API Changes:
* The func.get_yaml_data and func.get_yaml_multidoc_data functions now
return/yield tuples. The first field is the desired yaml_data (can be None
for empty documents) and the second field is a Boolean which indicates True
when the document loaded with no errors or False when an error occurred.
This is necessary in order to accept a bare "false" Scalar value as a
STDIN-supplied document.
3.0.0:
Enhancements:
* Added a new YAML Path Segment Type: *
This is identical to a Search segment where the search term is `[.!=""]`.
This translates to "match every Hash key for which its name is not empty and
every Array element which is not empty". This operator also vertically
expands results from Collectors, effectively breaking them out from an Array-
per-line to one-Scalar-per-line. If you place this inside a Collector, the
results will still be collected into an Array.
* The * character now also serves as a wildcard character for key-names, Hash
values, and Array value comparisons, converting the segment to a Search. For
example, a YAML Path like `abc.d*` becomes `abc[.^d]`, `abc.*f` becomes
`abc[.$f]`, and `abc.*e*` becomes `abc[.=~/^.*e.*$/]`, and so on.
* Added a new YAML Path Segment Type: **
This new type is a "Traversal" segment which causes YAML Path operations to
deeply traverse the document from that point. When there are no further
segments in the YAML Path, every leaf node (Scalar value) is matched. When
the YAML Path has at least one further segment, it (and all further segments)
must match subsequent nodes (anywhere deeper than that point in the document)
or none are matched. Results can be collected.
* The yaml-merge and yaml-get command-line tools now treat the - pseudo-file as
implicit when NOT specified AND the session is non-TTY. This can be blocked
with --nostdin|-S. This enables, for example, piping into these commands
without being forced to specify the - pseudo-file as an argument to them.
* The yaml-merge command now enables users to force the merged document to be
written out as YAML or JSON via a new --document-format (-D) command-line
argument. When unset, the format will be based on the file-name extension of
the --output file when provided, or (last-resort) that of the first document
(AUTO).
* The yaml-merge command now accepts multi-document YAML files, created when
the YAML standard-specified End-of-Document, Start-of-Document marker pair
(...<EOL> followed by ---<EOL>) is present, like:
```yaml
---
document: 1
...
---
document: 2
```
* The yaml-merge command now accepts multi-document JSON files, created when
there are multiple root-level entities, like:
```json
{"document": 1}
{"document": 2}
```
* Because any document to yaml-merge can be a multi-document, it no longer
requires at least 2 YAML_FILEs be supplied on the command-line. If users
pass only a single file or stream that is not a multi-document file, its
content will merely be written out without any merging into it. This can be
useful for trivially converting any file from YAML to JSON or JSON to YAML,
like `yaml-merge --document-format=json file.yaml` or
`yaml-merge --document-format=yaml file.json`.
* The `yaml-set` command-line tool can now write changes to empty or minimally-
viable files, enabling users to build up new data files from scratch. The
file must already exist, even if completely empty. A non-empty, minimally-
viable file depends on document type. For example:
A minimally-viable YAML file:
```yaml
---
# The triple-dash is required.
```
Two versions of a minimally-viable JSON file:
```json
{}
```
or:
```json
[]
```
However, minimally-viable structure is necessary only for files with unusual
file-name extensions. When the file-name extension is one of yaml, yml, or
json (case-insensitive), the file can be completely empty and still result in
a YAML or JSON data structure.
* The `yaml-set` command-line tool now accepts empty-String values.
* The `yaml-merge` command-line tool now permits overwriting one of its input
files as long as `--overwrite` is used instead of `--output`; these are
mutually-exclusive options. To help users protect against accidental change,
a new `--backup` flag will cause the to-be-overwritten file to be renamed
with a ".bak" file-name extension. A pre-existing backup file with the same
name will be unceremoniously replaced.
* The `yaml-set` command-line tool now accepts an arbitrary set of characters
from which to derive `--random` values via a new `--random-from` argument.
This is especially useful when you need to limit or expand the characters
used and when you wish to favor some characters more than others (simply
repeat the favored characters more than other characters in the argument
value but do so under caution because doing so reduces randomness).
* The `yaml-set` command-line tool now accepts documents from STDIN, causing it
to write the resulting changes to STDOUT. This enables `yaml-set` to operate
as a stream editor, like `yaml-get` and `yaml-merge`.
* The `yaml-paths` command-line tool now accepts documents from STDIN. It also
now accepts multi-document YAML and JSON as file or STDIN input. Because it
is impossible to determine whether a file or stream contains multi-document
data without reading through the entire file more than once, output now always
displays the file-name (or STDIN) and -- new -- the document-index in which
matches were found. As before, users can turn off file-name display by
setting --nofile|-F. In previous, single-document versions, the file-name
display was automatically muted when there was only one YAML_FILE to process.
Bug Fixes:
* Collectors were breaking search nodes with Regular Expressions, making it
impossible for collected searches to return expected matches.
* Fixed the Known Issue which was logged at version 2.4.0; setting values which
override aliased key-value pairs now correctly adds the new key-value pair to
the DOM.
* When the left-most document was JSON, yaml-merge and yaml-set would both
improperly write out a YAML document start mark (---) and then a hybrid
JSON/YAML result rather than valid JSON.
* The yaml-merge command would "explode" LHS Anchored Hashes wherever they were
aliased when the RHS document modified the same Hash into which the alias was
used.
* Setting a Python-style Boolean value via `yaml-set` (True or False) without
setting --format=boolean would cause an error because ruamel.yaml was
expecting an integer, instead. It is no longer necessary to set --format in
this case. However, --format=boolean can still be useful to convert more
"Boolean like" values into true|false, like on, off, yes, no, true, false,
True, False, 1, 0.
Non-Breaking API Changes:
* The various protected _get_* methods of Processor were changed to reduce the
number of positional parameters while also allowing for new special-use
parameters for future changes. The formerly optional positional `parent` and
`parentref` parameters are now optional keyword arguments by the same names.
Because these were all protected methods and the affected parameters were
optional anyway, this is not deemed a breaking change; no one should have
been directly calling them.
* The get_yaml_editor function now supports several keyword arguments which
provide for some customization of the returned ruamel.yaml.YAML instance.
See its documentation for details. This is a non-breaking change as the
defaults for each new keyword argument set the behavior identical to what it
was before this change.
* The get_yaml_data function now returns False rather than None when there is
an issue attempting to load data. This is because an empty-but-viable
document correctly returns None but there is no valid YAML or JSON document
which can be comprised only of a Scalar Boolean. This is a non-breaking
change because None and False are equivalent for code like:
```python
data = get_yaml_data(get_yaml_editor(), ConsoleLogger(), "file.yaml")
if not data:
print("No data")
```
However, you can now differentiate between "No data" and "Invalid document"
like so:
```python
data = get_yaml_data(get_yaml_editor(), ConsoleLogger(), "file.yaml")
if data is None:
print("No data")
elif not data and isinstance(data, bool):
print("Invalid document")
else:
print("Got a non-empty document")
```
* The ConsolePrinter's debug method now prints vastly more detail and allows
for customization of the output. Read its documentation for details.
From this release forward, the version reported by all command-line tools is
synchronized with the version of the overall yamlpath installation.
2.4.3:
Bug Fixes:
* Array-of-Hashes were not being detected for the purpose of applying merge
rules defined via the INI-style configuration file.
* Array-of-Hashes identity key inference was looking first to the LHS document.
This was backwards for an RHS-to-LHS merge and has been corrected.
The yaml-merge command now reports version 0.0.4 to reflect these changes.
2.4.2:
Enhancements:
* In the INI file's [rules] section, different merge rules can now be applied
to specific parts -- no matter how deeply nested -- of the same Hash
structure.
Bug Fixes:
* The 3rd-party Python INI file parser had no way of differentiating between
the key and value of a YAML Path entry containing an = sign in its key, like
"/path[.=name]/key = left". This update reconstitutes such lines and
correctly parses an affected YAML Path from the merge rule.
The yaml-merge command now reports version 0.0.3 to reflect these changes.
2.4.1:
Bug Fixes:
* The yaml-merge tool (and underlying Merger class) incorrectly assigned "None"
Anchors to all floating-point numbers. This prevented all merging when both
the LHS and RHS documents contained at least one floating-point number, each.
The yaml-merge command now reports version 0.0.2 to reflect this change.
2.4.0:
Enhancements:
* Added new reference command-line tool: yaml-merge. This is a very complex
tool, so a comprehensive treatise will be added to the project Wiki to
explore its capabilities. Along with those of its component classes, its
unit tests also provide many examples of the same.
* YAMLPath instances now support arbitrary changes to separator.
* YAMLPath instances now support equality testing (against the stored path),
immune to differences in separator.
* The get_yaml_data function now supports "-" as a source file. This is
interpreted as a read from STDIN.
* Due to the change to the get_yaml_data function, the yaml-get reference
command-line tool now supports retrieving nodes from YAML/Compatible data
passed to it via STDIN when its YAML_FILE argument is set to -. The new
yaml-merge reference command-line tool also reads from STDIN when one of its
YAML_FILE input arguments is -. No other reference command-line tools
support this change at this time.
Known Issues:
1. Neither yaml-set nor yaml-merge will add override keys to a Hash which uses
the YAML merge operator (<<:) and which does not already have a matching
override key. This issue has existed for a very long time but was only
discovered during preparation for this release. This will be logged and
tracked as a Known Issue for this release -- to be fixed at another time --
because no one (not even myself) has yet encountered/reported this issue, it
is non-trivial to fix, and it is an edge-case. Here is an example of this
issue:
For ex.yaml:
---
anchored_hash: &its_anchor
ah_key: Base value
merging_hash:
<<: *its_anchor
mh_key: Implementation value
... both of these commands:
`yaml-set --change=/merging_hash/ah_key --value='Override value' ex.yaml`
`echo 'Override value' | yaml-merge -m /merging_hash/ah_key ex.yaml -`
... will fail to affect the expected change. The expectation would be:
---
anchored_hash: &its_anchor
ah_key: Base value
merging_hash:
<<: *its_anchor
mh_key: Implementation value
ah_key: Override value
... but the actual result is (without any indication of an error):
---
anchored_hash: &its_anchor
ah_key: Base value
merging_hash:
<<: *its_anchor
mh_key: Implementation value
2.3.7:
Bug Fixes:
* Setting negative floats could cause the leading "-" symbol to be replaced
with an unexpcted "0" when specifying a float format, or crash when using
the default format.
2.3.6:
Bug Fixes:
* When using yaml-set with --format=folded and --eyamlcrypt, the encrypted
value was being mistakenly appended with a spurious newline character at its
end. Note that this affected only Puppet's Hiera lookup and not EYAML
itself; on the command-line, the eyaml command would not show the extra
newline character. The yaml-get command was also not affected, making it
very difficult to set up a unit-test for this case.
2.3.5:
Bug Fixes:
* Certain YAML constructs trigger AssertionErrors in ruamel.yaml during YAML
data writes. This was causing yaml-set to generate empty files. Until
https://sourceforge.net/p/ruamel-yaml/tickets/351/ is fixed, this patch
will revert the file contents to mitigate data loss under these conditions.
A specific test has been created to detect when the upstream issue is fixed.
2.3.4:
Bug Fixes:
* Minor security patch: Python already makes non-shell subprocess calls safe
(mitigating shell command injection). This patch makes that already-present
protection explicit rather than implicit.
2.3.3:
Bug Fixes:
* Subtraction Collector math had no effect when the RHS was a list of scalar
values (because LHS was a list of NodeCoords, so comparison was always
false). Also reduced O(3N) to O(2N) during Collector subtraction.
Enhancements:
* The console logger's debug method now includes the type of each element in a
list while it is being dumped.
2.3.2:
Bug Fixes:
* Subtraction Collector math crashed when the RHS result was non-iterable.
2.3.1:
Bug Fixes:
* Under certain conditions, some YAML changes were affecting unexpected nodes
which had identical original values.
* YAMLValueFormats.DEFAULT was identifying all CLI-supplied values as String
rather than their most-likely data-type.
API COMPATIBILITY WARNING:
Previous versions of the library returned only selected data nodes. Now,
instead of data nodes, NodeCoords objects are returned. This is necessary in
order to resolve Issue #44, which was caused by Python over-aggressively
optimizing memory, making non-String data nodes references to any other node
with the same value. This made changing exactly one of those data nodes
impossible because all nodes with the same original value would change with it.
Now, the NodeCoords class carries DOM tracking data along with each data
node making precise node changes possible. However, your queries will now
return this more complex additional data. In order to evaluate just the data,
please refer to this before-and-after example:
BEFORE:
for node in processor.get_values(yaml_path):
do_something_with(node)
AFTER:
for node_coordinate in processor.get_values(yaml_path):
do_something_with(node_coordinate.node)
If you need to recursively remove DOM tracking data from the results, a new
utility function is available: func.unwrap_node_coords(data). Note however
that you need that tracking data in order to change data within the DOM.
This does not affect the output of the sample command-line utilities.
2.3.0:
Bug Fixes:
* The get_yaml_data helper function now contains ruamel.yaml errors/warnings
without disrupting calling context handlers.
Enhancements:
* yaml-paths version 0.2.0 now has more detailed output control, trading
--pathonly for --values, --nofile, --noexpression, and --noyamlpath.
2.2.0:
Bug Fixes:
* YAML construction errors are now caught and more cleanly reported by all
command-line tools.
Enhancements:
* yaml-paths version 0.1.0 now has more specific flags, enabling:
* more precise handling of anchors in YAML data, and
* expanding parent node results to instead return all their child leaf nodes.
2.1.1:
Bug Fixes:
* yaml-paths was recursing into nodes for which the name had already matched,
causing unnecessary search results. Version 0.0.2 fixes this; when a node is
matched by name, any children are ignored because they will have already been
yielded as the parent node's value.
2.1.0:
Enhancements:
* Added a new yaml-paths command-line tool. In short, it enables searching
YAML/Compatible files, returning YAML Paths for any matches. As an Alpha-
grade tool, it is being released at version 0.0.1. Feedback welcome!
* All command-line tools which accept --pathsep now accept symbolic separators
rather than only names; so, --pathsep=/ is idental to --pathsep=fslash, etc.
Minor changes were also made to all command-line tools to consolidate some
repeat code. Each has a version bump to reflect this minor refactoring
effort.
2.0.2:
Bug Fixes:
* eyaml-rotate-keys was broken by the refactoring for 2.0.0. eyaml-rotate-keys
v1.0.2 restores functionality.
Enhancements:
* Command-line tools are now managed via pip as entry_points/console_scripts
rather than external binaries. This enables superior cross-platform
compatibility as well as unit testing. As such, all of the CLI tools have
been updated pursuant to (generally trivial, excepting eyaml-rotate-keys)
issues discovered during their newfound CI tests.
2.0.1:
Bug Fixes:
* yaml-set v1.0.4 lost track of EYAML block formatting between read and write,
causing replacement values to use unexpected formatting. This is fixed in
yaml-set v.1.0.5.
2.0.0:
Enhancements:
* Added Collectors to YAML Path expressions. These take the form of "(YAML
Path)" -- parenthesis () are used to demarcate each Collector -- resulting in
a list of zero or more matches for the sub-query. Operators between
Collectors include + and -, like "(...)+(...)", "(...)-(...)", and nested
Collectors are possible, like "(...)-((...)+(...))". Collectors may appear
anywhere within the outer YAML Path, effectively setting the point within the
data at which each Collector is rooted.
* A major code refactoring was undertaken to break YAMLPath out as its own class
and improve code quality (per mypy and pylint).
Bug Fixes:
* yaml-set v1.0.4 now implies --mustexist when --saveto is set. Otherwise,
--saveto was trying to save nothing when --change (without --mustexist)
pointed nowhere.
1.2.5:
Bug Fixes:
* yaml-set v1.0.3 no longer requires --privatekey unless decryption is requested
via enabling --check. As a side-effect, the script will also no longer ignore
requests to set the same value as was already set.
1.2.4:
Bug Fixes:
* yaml-set v1.0.2 now preserves newlines for pre-folded EYAML values when saving
the old encrypted value to another node.
* ruamel.yaml v0.15.96 is now the minimum acceptable version in order to adopt
better round-trip editing.