-
Notifications
You must be signed in to change notification settings - Fork 10
/
CHANGES
1435 lines (1010 loc) · 62 KB
/
CHANGES
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
========================
beancount: CHANGES
========================
Note: This file contains only changes in the 'default' branch.
2015-11-22
- Removed beancount.core.data.AttrDict and replace it by a regular dict. This
removes some annoyances when pickling and copying metadata. Metadata is now
represented by a plain old dict. (Fixes issue #79.)
2015-11-09
- Implemented a simple pickle-date on beancount.loader.load_file(), which will
reload the parsed & processed contents of a top-level file if a cache file
is present and more recent than the file itself, or update the contents of
the cache after loading if not. (This works a bit like .pyc byte-compilation
in Python or .elc in Emacs LISP.) This makes the loading of large files
multiple times ~15x faster, which is a big deal since Beancount has become a
little bit slow (I need to spend time optimizing at some point).
This feature is a bit experimental, so for now it only gets turned on if the
BEANCOUNT_LOAD_CACHE environment variable is set.
2015-10-04
- Implemented --output=<filename> option for bean-query tool.
2015-10-02
- Implemented an experimental new Query directive whose purpose is to define
pre-canned SQL queries in the Beancount file itself and to then build tools
to automate the generation of reports of those queries.
For the moment being, I'm just playing with this to convince myself that
this can be powerful and generic enough to provide more generally. I've
found myself running specific queries on speific files, e.g., during trips
and I'd like to automate those to some extend, perhaps replace the script
code that lives in beancount.plugins.split_expenses with this.
Anyhow, this is not supported but if you understand this and really want to
turn it on, set this option in your input file:
option "experimental_query_directive" "TRUE"
The directives aren't used anywhere yet, and you'll have to read source code
to figure out what they look like.
- Implemented a "reload" command in the shell. Typing "reload" will refresh
the data from the latest file contents.
2015-09-29
- Tightened the screws on the Amount() constructor a bit and made changes ini
order to allow it to accept a sentinel value for MISSING values from the
parser. Changes to beancount.core.amount.Amount are:
1. A string for the number is not accepted anymore, that is, Amount(str,
str) is not supported anymore. Amount() now unconditionally requires a
Decimal instance, so you must use Amount(Decimal, str). In any case, this
constrcutros was likely abused only by me in the tests. I'd rather have
a tighter, less permissive constructor.
2. The convenience constructor beancount.core.amount.from_string() is now
officially aliased as beancount.core.amount.A, so you can "from
beancount.core.amount import A". I now commit to using this constructor
and converted all the tests to use it. This is great for writing tests.
Amount really is a fundamental piece that deserves a very short alias
like this.
In summary, how to create Amount instances:
* If you use explicit values, use Amount(Decimal, str), e.g.,
Amount(D('100.00'), 'USD'), Amount(number, currency)
* If you need a convenience constructor, use A(str), e.g.,
A('100.00 USD'), A('32,000 KRW')
2015-09-29
- Added a new column 'description' to the SQL client in order to produce more
compact reports.
Oftentimes the payee is left to an empty value and journals
have a lot of wasted whitespace. Concatenating the payee and the narration
in a single field goes a long way towards producing reports that aren't
nearly as wide. I could have added a CONCAT() function but this is so common
and convenient that a dedicated 'description' alias is justified. You can
use it in posting context ("SELECT description ..." or "SELECT ... WHERE
description ~ '...'") or transaction context ("SELECT ... FROM description ~
'...'")..
2015-09-27
- Implemented a plugin that can book price conversions using a particular
booking method, in this case, FIFO. This idea stemmed from a suggestion I
made on the mailing-list a few weeks ago, which was to process a particular
account's price conversions to insert cost basis specifications and in doing
so to book reducing postings to FIFO order automatically. This could all be
done via a plugin, so I did it. (See tag {c395de691449} on the mailing-list
for more context.)
You can find it under "beancount.plugins.book_conversions". You use it like
this: first you load the plugin:
plugin "beancount.plugins.book_conversions" "Assets:Bitcoin,Income:Bitcoin"
The configuration requires you provide two accounts:
1. An assets account to process, whereby all price conversions (in any currency)
occurring in that account will have their cost basis automatically inserted,
and,
2. An Income account that is inserted on reducing transactions to absorb the
profit or loss of the sale.
Then you book your transactions as simple price conversions, without worrying
about cost basis, for example, this input:
2015-09-04 *
Assets:Bank -750.00 USD
Assets:Bitcoin 3.000000 BTC @ 250.00 USD
2015-09-05 *
Assets:Bank -780.00 USD
Assets:Bitcoin 3.000000 BTC @ 260.00 USD
2015-09-20 *
Assets:Bitcoin -2.000000 BTC @ 300.00 USD
Expenses:Something
2015-09-21 *
Assets:Bitcoin -2.000000 BTC @ 310.00 USD
Expenses:Something
2015-09-22 *
Assets:Bitcoin -2.000000 BTC @ 330.00 USD
Expenses:Something
Would get translated by the plugin into the equivalent of this:
2015-09-04 *
Assets:Bitcoin 3.000000 BTC {250.00 USD} @ 250.00 USD
trades: "trade-ff4c3e592147,trade-6b9c8689ed50"
Assets:Bank -750.00 USD
2015-09-05 *
Assets:Bitcoin 3.000000 BTC {260.00 USD} @ 260.00 USD
trades: "trade-91e2245de593,trade-d622a89c614d"
Assets:Bank -780.00 USD
2015-09-20 *
Assets:Bitcoin -2.000000 BTC {250.00 USD} @ 300.00 USD
trades: "trade-ff4c3e592147"
Income:Bitcoin -100.00000000 USD
Expenses:Something 600.00000000 USD
2015-09-21 *
Assets:Bitcoin -1.000000 BTC {250.00 USD} @ 310.00 USD
trades: "trade-6b9c8689ed50"
Assets:Bitcoin -1.000000 BTC {260.00 USD} @ 310.00 USD
trades: "trade-91e2245de593"
Income:Bitcoin -110.00000000 USD
Expenses:Something 620.00000000 USD
2015-09-22 *
Assets:Bitcoin -2.000000 BTC {260.00 USD} @ 330.00 USD
trades: "trade-d622a89c614d"
Income:Bitcoin -140.00000000 USD
Expenses:Something 660.00000000 USD
Note that the price annotations are kept as in the input file. The plugin only
inserts the cost basis and splits postings where required.
Secondly, a unique trade identifier is allocated for each matching pair of
postings that were booked and is added to the metadata of those postings.
Its value is a comma-separated list. This metadata is useful because it
allows you to load the entries and from it, to build a list of trades to
produce a suitable report for reporting.
So the plugin also contains code to do this. You can invoke it as a main
program, like this:
python3 -m beancount.plugins.book_conversions <filename.beancount>
On the file above, it should generate the following output, which I believe is
the report you're really after for reporting purposes:
Units Currency Cost Currency Buy Date Buy Price Sell Date Sell Price P/L
-------- -------- ------------- ---------- --------- ---------- ---------- ------
2.000000 BTC USD 2015-09-04 250.00 2015-09-20 300.00 100.00
1.000000 BTC USD 2015-09-04 250.00 2015-09-21 310.00 60.00
1.000000 BTC USD 2015-09-05 260.00 2015-09-21 310.00 50.00
2.000000 BTC USD 2015-09-05 260.00 2015-09-22 330.00 140.00
-------- -------- ------------- ---------- --------- ---------- ---------- ------
Use the '--format=csv' option if you want to produce output suitable for
including in a spreadsheet. If you need to customize the output further, see the
source code (file:book_conversions.py). It is not possible to generate this list
of trades using the SQL client yet, to change the output columns you would have
to create a standalone script; you can copy just the main() function in the
plugin if you need to do that and import the rest. (Note: The function that
extracts the list of matches provides both the posting and its parent
Transaction object, so you can output data from the transaction if you want,
e.g. the narration of the selling transaction.)
2015-09-14
- Merged more changes to beancount.el (Emacs support) by Stefan Monnier:
* Use syntax-tables for comments and strings.
* Make ido use optional in Emacs config.
* Quote filenames on command lines
Thank you Stefan!
2015-09-12
- Added to beancount.plugins.split_expenses a new script that is able to
generate the typical kinds of reports you would need to provide to
participants in a shared project or travel on their specific expenses and
contributions, as well as a breakdown of their expenses per category
(expenses account) and final balances for each participants. Invoke it like
this:
python3 -m beancount.plugins.split_expenses <filename>
The script lives in the same file as the split_expenses plugin, because it
is tightly related to it. It presumes the input file will process through
this plugin. Try to run it on the example files under
beancount/examples/sharing.
- Added a new file "examples/sharing/duxbury2015.beancount" which contains a
new example of sharing expenses on a trip between three people.
- Starter cleanup of beancount.el contributed by Stefan Monnier. More changes
to come, this is basically a no-op patch. It just moves code around and
makes other such cosmetic changes to silence some compiler warnings, prepare
for subsequent changes, better follow coding conventions etc. Detail:
* beancount.el: Add some standard file headers, license, and
sectioning comments. Sprinkle a few FIXMEs while I'm here.
(beancount-font-lock-defaults): Rename from beancount-font-lock-keywords,
since this corresponds to font-lock-keywords and not to
font-lock-defaults.
(beancount-directive-names, beancount-font-lock-keywords,
beancount-accounts): Move variables before their first use (to silence
the compiler).
(beancount-mode-map): Use a proper defvar instead of define-minor-mode's
half-broken :keymap.
(beancount-mode): Merge the make-local-variables into their corresponding
`set'.
(beancount-accounts): Move make-variable-buffer-local to top-level since
it's a global operation.
(beancount): Declare this group, already used in define-minor-mode.
(beancount-hash-keys): Mark unused arg `v' with an underscore.
(beancount-transaction-set-flag): Don't use replace-string (designed for
interactive use) from Elisp.
(beancount-for-line-in-region): Use copy-marker and eobp.
(beancount-align-numbers): Break lines to avoid hiding arguments.
(compilation-read-command): Declare variable to silence byte-compiler.
2015-09-06
- Filter out platform-specific comments from the code generated by GNU Bison,
in order to avoid unnecessary diffs when I switch between Linux and Mac OS X.
2015-09-05
- Changed the semantics of the account_rounding account, so that it matches
that of the other account options, that is, they all specify a sub-account
of the Equity account. So if you had
option "account_rounding" "Equity:RoundingError"
in your input file, change this to:
option "account_rounding" "RoundingError"
To ease the transition, the parser now checks if the first component is one
of the five categories and if so, issues a warning and removes the prefix.
(One side-effect of this change is that the rounding error account may only
be an Equity account, but I'm hoping this won't be a problem for anyone.)
2015-08-31
- The column "weight" was broken; fixed by yagebu@.
2015-08-30
- Fixed a very minor bug in split_expenses plugin whereby the generated
postings did not contain the __automatic__ metadata field, and in some
particular situations, their automatically calculated values would end up
being used for inferring the tolerances.
2015-08-15
- Changed the semantics of the parsing stage, in a fairly profound way.
This should have no visible changes to users, but people writing scripts
should revise their code if they were using
beancount.parser.parser.parse_*() functions.
Just to be clear: beancount.loader.load_*() has not changed. If you just use
the loader, there are no changes. Changes are only at the parser level.
Here's what's going on and why: The parser used to carry out interpolation
of missing numbers on postings for each transaction locally, while parsing
the transactions. This was done by calling by calling
beancount.core.interpolate.balance_incomplete_postings(), here:
https://bitbucket.org/blais/beancount/src/ee2073aae080aaa8e260abe8a501abf872948f0e/src/python/beancount/parser/grammar.py?at=default#grammar.py-803
Loading a list of entries was carried out in two steps:
,-----------------------load----------------------.
(recursively)
,---------------------. ,---------. ,------------.
(input)--->| parse + interpolate |-->| plugins |-->| validation |--> entries
`---------------------' `---------' `------------'
First, the parser would run on the input and process all the input files
recursively (processing includes). "Interpolation", the process of filling
in missing numbers, was carried out at that stage, during parsing, and only
locally, that is, for each transaction in isolation. "Booking" of lots, that
is, selecting which of an account's inventory lots to match and reduce, was
explicit. This booking could not take advantage of the accumulated
inventories in order to vary its behavior. You had to specify the entire lot
information unambiguously.
After this, in a second stage, the plugins were run on the entries and a
final validation step was run at the end.
To implement the booking proposal
(http://furius.ca/beancount/doc/proposal-booking), we want for the user to
be able to provide a partial specification of lots to be matched against an
account's accumulated inventory at the date the transaction is to be
applied. The idea is that if there is no ambiguity, the user should be able
to specify very little information about a lot (for example if there is a
single lot in the account when processing the transaction an empty spec of
"{}" should be sufficient). Moreover, where the specification is ambiguous,
we also want to support automatic selection of lots according a method
specified by the user, e.g., FIFO booking, LIFO booking, etc.
For this to work, we need to have parsed all the inputs to some sort of
incomplete specification, a representation of inputs that hasn't yet been
resolved to specific lots, in order to carry out booking. The parser has
been modified to output such incomplete postings:
,-------------------------load-------------------------.
(recursively)
,-------. ,-------------. ,---------. ,------------.
(input)--->| parse |-->| booking |-->| plugins |-->| validation |--> entries
`-------' | + | `---------' `------------'
| interpolate |
`-------------'
incomplete
entries
Because "interpolation" runs on the result of specific lot values, "booking"
must run before it, and so they are inter-related. Thus, booking and
inteprolation has been moved to a dedicated step that runs on the list of
incomplete entries and resolves them to the regular entries to be processed
further by plugins and validation.
This also has a nice side-effect: the booking step is where all the
complexity is, and it is now isolated and I will be able to test and
experiment on it in isolation. This is where all the fun will be.
A description of the incomplete specifications output by the parser can be
found here in the parser.py file, this is the description of the
intermediate state of postings whose lots haven't yet been matched and
resolved to specific inventory lots:
https://bitbucket.org/blais/beancount/src/18282452e265959b69d3d10c6d9cf32e5815c522/src/python/beancount/parser/parser.py?at=booking
Essentially, a posting's 'lot' attribute contains a "LotSpec" tuple instead
of a "Lot" tuple, and several numbers may be left unfilled (for interpolate
values). I don't imagine anyone will ever have to manipulate such
intermediate entries, only the booking code.
The previous booking algorithm has been moved to the booking stage and the
semantics should be identical to what they used to be. This is still the
default algorithm--it just runs in its own dedicated stage, still operating
locally on each transation. You should observe no difference in behaviour.
I've merged these changes now in order to minimize the differences between
the booking and default branches and because I was able to do it without
changing any of the semantics (despite the large number of lines and tests
modified). This was a necessary refactoring. Because that code is now
isolated to its own stage I should be able to begin implementing the more
complex state-dependent booking algorithms in the 'booking' branch. (I'm
excited about this.) I could even implement different booking heuristics and
switch between them.
Because the parser used to spit out regular, complete entries,
parser.parsedoc() was used in much of the tests instead of loader.loaddoc(),
so that many of those tests would not have to concern themselves with making
sure the input passed the validation stage run by the loader. For example,
creating Open directives just to create some Transaction test object wasn't
necessary in the tests. All of those tests had to be revised and I made them
all depend on beancount.loader.load_*() instead of the now weakened
parser.parse_*() functions which output incomplete, unbooked entries. This
cleans up the dependencies a bit as well. If you wrote your own unit tests
and were using parser.parsedoc(), you should convert them to use
loader.loaddoc(). In order for this change not to go unnoticed (and for
naming consistency with parse_string() and parse_file()) I'll probably
rename parser.parsedoc() to parser.parse_doc() and ditto with the loader.
(Note: In some cases I've had to specifically setup the input of some tests
in "raw" plugin processing mode to avoid triggering some unwanted and
unrelated errors. I'm tempted to remove even more from the default plugins.
This may happen in a future CL.)
- Renamed beancount.parser.parser.parsedoc() to parse_doc() and
beancount.loader.loaddoc() to load_doc(), for consistency with the other
parse_*() and load_*() functions. Kept a stub that will issue a warning if
you use it.
2015-07-23
- In the conversion to the new booking syntax, I had inadventently removed
support for the total cost "{{ ... }}" syntax. I brought this back next to
the new booking syntax, and uncommented tests that had been made to skip.
2015-07-21
- Merged ongoing work from the 'booking' branch that will eventually change
the semantics of inventory booking; in the interest of a smooth transition,
and for me to be able to use either branch interchangeably, I introduced a
few changes that should have no user-visible effect on 'default':
* The parser.parse_string() and parser.parse_file() routines don't
interpolate anymore. For this reason, the tests all had to be adjusted not
to include interpolation. The interpolation of incomplete postings
(grammar.interpolation()) has moved to a separate phase and is now run by
loader._load() *AFTER* parsing. This is key to implementing fuzzy matching
semantics for matching reducing lots: we need to have all the incomplete
transactions parsed and sorted in order to select matching lots in date
order.
This only affects you if you wrote scripts against the parser
interface directly (this is highly unlikely).
* For writing unit tests, parser.parsedoc() and loader.loaddoc() are now
decorator factories. This allowed me to add options to parser.parsedoc()
to perform interpolation, and when not specified, to check that entries
with interpolation are not present in the tests. Also, I coudl merge the
functionality of parser.parsedoc_noerrors() and loader.loaddoc_noerrors()
in their respective equivalents. The docstring tests now validate that
there are no errors in the docstrings by default. This makes the tests
tighter (a few bugs in the tests themselves were found and fixed).
* The new syntax for cost specification that will be in effect for the
inventory booking proposal is now supported. The syntax is
backward-compatible with the previous one. Previously, the following
syntaxes were supported for specifying the cost and optionally a lot
acquisition date:
Assets:Investments 1 HOOL {123.00 USD}
Assets:Investments 1 HOOL {123.00 USD / 2013-07-20}
Instead of these options, the new syntax supports a comma-separated list
of cost-specifiers which can be one of
<cost> -> e.g. 123.00 USD
<lot-date> -> e.g. 2015-07-20
<label> -> e.g. "first-lot"
<merge-cost> -> e.g. *
In order to keep current input working, either a comma (,) or a slash (/)
is supported to separate the components. This is valid:
Assets:Investments 1 HOOL {2013-07-20 / 123.00 USD / "first-lot"}
Those can be provided in any order. For example, these are all also valid
syntaxes for cost:
Assets:Investments 1 HOOL {}
Assets:Investments 1 HOOL {"first-lot"}
Assets:Investments 1 HOOL {2013-07-20, 123.00 USD}
Assets:Investments 1 HOOL {2013-07-20, "first-lot"}
Assets:Investments 1 HOOL {*}
Assets:Investments 1 HOOL {*, 123.00 USD}
Moreover, the cost amount now supports a compound amount that is expressed
not in terms of each unit of the currency, but in terms of the total
amount over all units, for example, this is how you could fold in the cost
of a commission:
Assets:Investments 1 HOOL {123.00 # 9.95 USD}
The syntax for a compound amount follows this pattern:
[<per-unit-cost>] # [<total-cost>] <currency>
The numbers are both optional. If no '#' separator is present, the total
cost component is assumed to be zero. This will eventually subsume the
{{...}} total cost syntax, by specifying only the total cost portion of
the compound amount:
Assets:Investments 100 HOOL {# 12300.00 USD}
Of course, this combines with the other spec formats, so this is valid:
Assets:Investments 1 HOOL {123.00 # 9.95 USD, 2015-07-22}
Finally, while the new syntax is supported in the parser, the old
semantics for inventory lot specification is still in order. If you
provide an unsupported combination of lot specifiers (e.g., you use a
label, a compound amount, or a merge-cost marker), an error will be issued
accordingly.
So just use the cost and date as previously; I will bring in new semantics
incrementally, semantics that will take advantage of this new syntax. I
will try to do so in a way that minimizes changes.
- I removed forwarded symbols from beancount.parser. Generally, in order to be
able to mock functions, you should always import packages, not symbols.
2015-07-20
- Numbers rendered from bean-query are now rendered using the display context
inferred from the input file. This means numbers are rounded nicely, using
the most common precision seen in the input file.
- I fixed a bug on writing UTF-8 output to the console on Mac OS X and how it
interacts with the 'less' pager.
2015-07-12
- Implemented support for arithmetic operations: +, -, *, / and parenthesis
groupings are now supported, anywhere that a number can be seen in the input
file, including postings, costs & prices, and balance numbers.
2015-07-12
- Changed how web & report rendering code renders numbers so that it uses the
DisplayContext inferred from the numbers in the input file. Numbers in the
reports were previously using the default precision and that led to all
sorts of inconsistencies. The numbers are now rendered using the most common
precision which has been seen in the input files. (This work had been
planned for a while, as part of converting everything to use the
DisplayContext. Next up will be converting the shell's rendering routines.)
2015-07-11
- WARNING: API CHANGES AHEAD. If you wrote plugins, you will need to remove
the 'entry' parameter to constructors of Posting().
I removed the Posting.entry back-reference of Postings to their owning
Transaction objects. This makes programmatically creating our data
structures much more straight-forward and avoids us having to run pirouettes
around which object to create first. It also removes all manners of "fixing
up" transactions and removes one ultimatley unnecessary degree of freedom
from plugins outputs: we don't have to check that the plugins generate
objects with correct back-references. It also makes the documentation
simpler: I won't have to explain backreferences and how to deal with them.
Finally, it avoids the circular reference which could potentially make
Python's GC do some work for no other reason.
This happened because it occurred to me that the only really important place
where back-references were used were in the beancount.core.realization code,
and that in the RealAccount objects I could easily replace the Posting
instances in lists of "postings or entries" into a new tuple object
"TxnPosting" that contains a reference to the parent transaction and a
reference to the posting. This voids the need for a back-reference, and
makes the API really straightforward.
- WARNING: USER-LEVEL CHANGES FOLLOW. I repatriated the following optional
plugins to the beancount.plugins package, so that all the plugins are in the
same place:
beancount.ops.auto_accounts -> beancount.plugins.auto_accounts
beancount.ops.implicit_prices -> beancount.plugins.implicit_prices
This is so that the "beancount.ops" package contains only plugins run by
default (non-optional), and so that the "beancount.plugins" only contains
optional plugins the user must provide.
In doing so, I also made the "implicit_prices" plugin NOT INCLUDED BY
DEFAULT. This plugin creates automatically Price directives for all Postings
which have a @ <price> annotation, thus feeding those prices into the price
database. If you want to keep this behaviour -- and you probably should --
you need to add the following in your input file:
plugin "beancount.plugins.implicit_prices"
(I'm considering moving move of the default plugins to be optional, in
particular, beancount.ops.documents, not sure yet.)
2015-07-05
- Implemented support for parsing strings from files in various encodings.
Only string objects encoded in non-ASCII are supported, e.g., payee,
narrations, description, etc. This means the account names are still
restricted to ASCII, in particular. This works with UTF8 encoding (tested),
latin1 encoding (tested), and probably more. The support is at the lexer
level. You may now supply an 'encoding' parameter to parser.parse_file(),
parser.parse_string(), loader.load_file(), loader.load_string() functions.
Passing in an invalid encoding should not mail it fail (though the contents
of the strings will be empty). Unit tests for all those cases have been
added.
- Completely reviewed all lexer and grammar error handling code. The parser
code used to bail out with abort() on an unhandled exception and dump core
after issuing its error message (this is ugly but was never a big issue
since this occurred pretty rarely). All errors are now handled gracefully
(invalid lexer token, exception raised in lexer builder, invalid gramamr
rule, exception raised in grammar builder Python code). A comprehensive
suite of unit tests has been built to catch all these. You should never
experience a core dump anymore. A number of other unexpected cases has also
been dealt with.
- The lexer used to have a fixed-size buffer for parsing its strings. The
buffer is now dynamically allocated. Lexing should also be slightly faster.
- Mmeory management for the parser has been reviewed. Memory clean up for
temporaries generated during parsing has been reinstated (it was disabled
somehow) and cleanup rules are invoked even in the case of errors. This may
make the parser somewhat slower.
2015-07-02
- Built support for setuptools and rolled back changes. Setuptools is broken
beyond what I can bear. Forget this. You'll have to install dependencies by
hand. It's a one-liner, like this:
pip3 install python-dateutil bottle ply lxml
2015-07-01
- Wrote a post-mortem/assessment doc on LedgerHub, detailed future plans for
LedgerHub and plans to eventually merge the codebase into Beancount again.
See: http://furius.ca/ledgerhub/doc/postmortem
2015-06-27
- Complete the design doc and made it public.
2015-06-26
- Created a generic script that can be used to automate the upload of
arbitrary CSV files to a Google Spreadsheet. This can be used along with the
holdings reports in order to create a spreadsheet tracking your assets.
Unfortunately, this script has to be written in Python 2.x, because of
the unavailability of the gdata-python-client library which is not available
at this time for Python 3.x. The script is self-contained. If you want
to use this, you must install the following libraries in your Python 2.x
install:
* gdata-python-client
* oauth2client
* httplib2
2015-06-25
- Posting objects now have (filename, lineno) in their metadata. This could
potentially be helpful for people who want to insert text programmatically
in their input file without having to write their own parser.
See: https://groups.google.com/d/msg/beancount/zNCFk-YGdr8/8P2_vGGH9xoJ
2015-06-15
- All the code to bake a static HTML archive of a bean-web interface has been
rewritten.
* There used to be a number of bugs caused by the wget tool. These bugs have
been fixed.
* The new code does not depend on wget -- Beancount does not require wget
anymore. Explicit code to do web scraping has been implemented. On the
other hand, a new dependency is introduced: lxml. This dependency was
already required for development and install in the same way as all the
other dependencies: using pip3. Should be a breeze to install on all
platforms.
* A verification layer has been tacked on after the web scraping step to
check that there are no dangling links.
* Intentionally skipped links are now converted to plain text, such that
there are no dangling links for pages we aren't saving (these just used
to be ignored).
I repeat: Beancount does not depend on wget anymore.
Please install lxml with: "pip3 install lxml"
- The 'exportpf' report which is used to produce OFX that can be imported in
Google Finance now issues a warning if you did not provide either 'ticker'
or 'export' metadata for holdings that get exported. The reason for the new
adminition is that it trips up the OFX import and when Google Finance sees
a ticker it does not understand, the import fails. At least with the warning
we're telling the user they should be providing tickers or export directives
for all their commodities if they export the portfolio.
2015-06-10
- Removed dependency on curses. If you don't have curses, everything should
work fine now, errors are being caught. Curses was only used to provide good
default values for rendering tables in terminals on e.g. bean-report
journal and to decide whether to use a pager in multi-page bean-query.
The default values are fixed if curses isn't available, e.g. 80 chars wide,
but as previously, you can override them manually. I also removed the
dependency from checkdeps and from the install document.
2015-06-09
- Rewrote the Installation Instructions document completely to make it
simpler. Some users had found it daunting.
http://furius.ca/beancount/doc/install
- Tested Windows installation using Cygwin and added it to the installation
instructions.
http://furius.ca/beancount/doc/install
2015-06-07
- Moved the following representation related symbols to their own module:
beancount.core.amount.D -> beancount.core.number.D
beancount.core.amount.Decimal -> beancount.core.number.Decimal
beancount.core.amount.ZERO -> beancount.core.number.ZERO
beancount.core.amount.ONE -> beancount.core.number.ONE
beancount.core.amount.HALF -> beancount.core.number.HALF
beancount.core.amount.round_to -> beancount.core.number.round_to
I also moved the tests. All the code specifically related to numbers is
now in beancount.core.number. I also removed the relative imports, as
Ledgerhub is importing from the Beancount source code now.
This is not a user-visible change. However, if you wrote scripts against
the codebase, you will need to rename your imports. In the meantime I added
stubs which issue a warning if you import from the obsoleted names.
- Moved documentation scripts from experiments to the main Beancount codebase.
There is a new package for it: beancount.docs. These codes are used to maintain
the Beancount documentation and there is common code for authentication in there,
code that I plan to use for automating the exporting ot portfolios.
This introduces some optional dependencies:
* google-api-python-client
* oauth2client
* httplib2
These are optional, and at the moment none of the actual Beancount tools
make any use of them. I moved this into the main Beancount codes because
the common authentication code will be used to export portfolios directly
to a Google spreadsheet in Google Drive, which will be super convenient
to build custom portfolio tracking views.
2015-06-06
- Added an 'inferred_tolerance_multiplier' option to allow the user to adjust
the magnitude of the inferred tolerance.
When the tolerance values aren't specified explicitly via the
'default_tolerance' option, the tolerance is inferred from the numbers in
the input file. For example, if a transaction has posting with a value
like '32.424 CAD', the tolerance for CAD will be inferred to be 0.001
times some multiplier. This is the muliplier value.
We normally assume that the institution we're reproducing this posting
from applies rounding, and so the default value for the multiplier is
0.5, that is, half of the smallest digit encountered.
You can now customize this multiplier by changing this option, typically
expanding it to account for amounts slightly beyond the usual tolerance,
for example, if you deal with institutions with bad of unexpected rounding
behaviour.
- Promoted the 'experiment_infer_tolerance_from_cost' option to
'infer_tolerance_from_cost': not an experiment anymore. See thread on
mailing-list w/ mharris. This feature works well and actually solves real
problems for users.
A corresponding section has been added in the documentation at
http://furius.ca/beancount/doc/tolerances
2015-06-04
- When using the 'experiment_infer_tolerance_from_cost' option, the inferred
tolerance is now expanded to include the SUM of all maximum rounding errors
from postings held at cost, not just the maximum. Users with multiple
rounding events in a single transactions could occur rounding that many
times. We assume no more rounding events than the number of postings held at
cost.
- I wrote a new document to explain how to deal with restricted stock units
vesting events:
http://furius.ca/beancount/doc/vesting
2015-06-03
- bean-doctor context now outputs weight and residual numbers at their full
precision instead of their rounded one.
2015-06-02
- Created a new document to keep track of user contributions:
http://furius.ca/beancount/doc/contrib
2015-05-30
- Converted the Syntax Cheat Sheet to a Google Doc and removed it here.
- Added a check to bake the source code's checksums into the C extension
module and upon loading beancount.parser.parser check that the checksums
match. This automatically issues a warning on all programs that use the
parser if the compiled extension module is out-of-date and the user needs to
rebuild. This should explain the occasional unexpected situation where the
extension module doesn't match the Python code, for those running from
source or during development and in switching branches.
- Added an experimental boolean option: 'experiment_infer_tolerance_from_cost'
that expands the inferred tolerance to include values inferred from the cost
and price amounts on those cost and price currencies.
Here is the doc for it:
Enable an EXPERIMENTAL feature that expands the maximum tolerance inferred
on transactions to include values on cost currencies inferred by postings
held at-cost or converted at price. Those postings can imply a tolerance
value by multiplying the smallest digit of the unit by the cost or price value
and taking half of that value.
For example, if a posting has an amount of "2.345 RGAGX {45.00 USD}"
attached to it, it implies a tolerance of 0.001 x 45.00 / 2 = 0.045 USD
and this is added to the mix to enlarge the tolerance allowed for units of
USD on that transaction. All the normally inferred tolerances (see
http://furius.ca/beancount/doc/tolerances) are still taken into account.
Enabling this flag only makes the tolerances potentially wider.
WARNING: This feature may go away in the future. It is an exploration to see
if it is truly useful.
- Added a dependency check for 'curses' library and updated documentation.
This is for OpenSuse which packages curses separately.
2015-05-22
- Fixed a simple bug (#47) where the default value used for legacy tolerances
wasn't being initialized properly and added a test for it.
2015-05-20
- Fixed a potential important bug (#48) just introduced a few days ago whereby
enabling the 'account_rounding' option would inadvertently disable balance
checks. Thanks to Matthew Harris for finding and reporting it.
- Added a 'dedent' option to both beancount.parser.parser.parse_string() and
beancount.loader.load_string(). This is a convenience for writing quick
tests.
2015-05-17
- MAJOR CHANGES: Precision and tolerances in Beancount have been entirely
revamped. The logic used to compute tolerances on transaction balances and
balance assertions is new. This previously discussed proposal:
http://furius.ca/beancount/doc/proposal-rounding
has been implemented. The new behaviour is documented here:
http://furius.ca/beancount/doc/tolerances
Here is a high-level summary of changes (see doc above for details):
* The tolerance used for transaction balances is inferred automatically from
the amounts on the transactions themselves. When not possible, a default
tolerance is looked up from values you can provide (as well as a global
default).
* There is a new 'default_tolerances' option to provide tolerances to be
used used when it cannot be inferred automatically. You use it like this:
option "default_tolerance" "*:0.005"
option "default_tolerance" "INR:0.5"
...
* The 'tolerance' option has been deprecated. It now has no effect.
* Balance assertions and Pad directives use a similar inference mechanism,
where the tolerance is based on the precision of the expected number in
the input.
* Rounding errors may be automatically posted to a special account (new
postings are inserted during parsing where necessary). This is disabled by
default; you enable this by setting the new 'account_rounding' option to
the account you want the rounding errors to be posted to:
option "account_rounding" "Equity:Rounding-Errors"
* These changes may cause some of your transactions to report as not
balancing, because the tolerances and now inferred automatically and the
inferred tolerances may be smaller than the fixed value we used
previously. YOU NEED TO MAKE SUITABLE ADJUSTMENTS TO YOUR INPUT FILE.
However, because this might cause some minor disruption, I have provided
an option to allow users to revert Beancount to its previous tolerance
behavior for a few months:
option "use_legacy_fixed_tolerances" "TRUE"
* There is a new EXPERIMENTAL feature which makes Beancount accept an
explicit tolerance value on balance assertions, like this:
YYYY-MM-DD balance <account> <number> ~ <tolerance> <currency>
For example,
2015-05-17 balance Assets:Vanguard 23.2134 ~ 0.002 RGAGX
By default, this feature is not enabled; this is being tested out and it
might go away eventually. To enable it, add the following experiment flag:
option "experiment_explicit_tolerances" "TRUE"
- The 'plugin' option has now been deprecated and will issue a warning if you
use it. You must now use the 'plugin' directive instead.
- The parser now supports boolean types, with this syntax: TRUE, FALSE. This
can be useful to set metadata fields.
- Postings that were inserted automatically have some metadata set on them to
distinguish this fact ('__automatic__').
2015-05-12
- Allow currency names up to 24 characters (not 12). This is often required
for people using financial options.
2015-05-11
- Added parsing of boolean types: TRUE, FALSE will produce values of
Python "True" and "False". This is useful to used as metadata values.
- Removed extraneous reference release in key_value_value in parser.
2015-05-03
- Added Equity to the list of accounts that count as proceeds for
beancount.plugins.sellgains plugin. This is so that old transactions
may use an unknown equity account and still pass the sellgains check.
2015-04-19
- Fixed a bug with the pad directive that did not look for account name
boundaries when looking at a parent account.
- In output of bean-doctor context, now print the residual amount.