-
Notifications
You must be signed in to change notification settings - Fork 5
/
THOUGHTS.txt
6526 lines (4570 loc) · 216 KB
/
THOUGHTS.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
Thu Sep 22 00:12:42 BST 2022
Making quite reasonable progress, though only running under emulation.
Since almost everything so far has been a recap of nixwrt, that's to
be expected.
The example config starts some services at boot, or at least attempts
to. Next we shoud
- add some network config to run-qemu
- implement udhcp and odhcp properly to write outputs
and create resolv.conf and all that
- write some kind of test so we can refactor the crap
- not let the tests write random junk everywhere
Thu Sep 22 12:46:36 BST 2022
We can store outputs in the s6 scan directory, it seems:
> There is, however, a guarantee that s6-supervise will never touch subdirectories named data or env. So if you need to store user information in the service directory with the guarantee that it will never be mistaken for a configuration file, no matter the version of s6, you should store that information in the data or env subdirectories of the service directory.
https://skarnet.org/software/s6/servicedir.html
> process 'store/pj0b27l5728cypa5mmagz0q8ibzpik0h-execline-mips-unknown-linux-musl-2.9.0.1-bin/bin/execlineb' started with executable stack
https://skarnet.org/lists/skaware/1550.html
Thu Sep 22 16:14:49 BST 2022
what network peers do we want to model for testing?
- wan: pppoe
- wan: ip over ethernet, w/ dhcp service provided
- wan: l2tp over (ip over ethernet, w/ dhcp service provided)
- lan: something with a dhcp client
https://accel-ppp.readthedocs.io/en/latest/ could use this for testing
pppoe and l2tp?
Thu Sep 22 22:57:47 BST 2022
To build a nixos vm with accel-ppp installed (not yet configured)
nix-build '<nixpkgs/nixos>' -A vm -I nixos-config=./tests/ppp-server-configuration.nix -o ppp-server
QEMU_OPTS="-display none -serial mon:stdio -nographic" ./ppp-server/bin/run-nixos-vm
To test it's configured I thought I'd run it against an OpenWrt qemu
install, so, fun with qemu networking ensues. This config in ../openwrt-qemu
is using two multicast socket networks -
nix-shell -p qemu --run "./run.sh ./openwrt-22.03.0-x86-64-generic-kernel.bin openwrt-22.03.0-x86-64-generic-ext4-rootfs.img "
so hopefully we can spin up other VMs connected either to its lan or
its wan: *however* we do first need to configure its wan to use pppoe
uci set network.wan=interface
uci set network.wan.device='eth1'
uci set network.wan.proto='pppoe'
uci set network.wan.username='db123@a.1'
uci set network.wan.password='NotReallyTheSecret'
(it's ext4 so this will probably stick)
Fri Sep 23 10:27:22 BST 2022
* mcast=230.0.0.1:1234 : access (interconnect between router and isp)
* mcast=230.0.0.1:1235 : lan
* mcast=230.0.0.1:1236 : world (the internet)
Sun Sep 25 20:56:28 BST 2022
TODO - bugs, missing bits, other infelicities as they occur to me:
DONE 1) shutdown doesn't work as its using the busybox one not s6.
2) perhaps we shouldn't have process-based services like dhcp, ppp
implement "address provider interface" - instead have a separate
service for interface address that depends on the service and uses its
output
* ppp is not like dhcp because dhcp finds addresses for an existing
interface but ppp makes a new one
3) when I killed ppp it restarted, but I don't think it reran
defaultroute which is supposed to depend on it. (Might be important
e.g. if we'd been assigned a different IP address). Investigate
semantics of s6-rc service dependencies
DONE 4) make the pppoe test run unattended
5) write a test for udhcp
6) squashfs size is ~ 14MB for a configuration with not much in it,
look for obvious wastes of space
7) some of the pppoe config should be moved into a ppp service
8) some of configuration.nix (e.g. defining routes) should be moved into
tools
DONE 9) split tools up instead of having it all one file
10) is it OK to depend on squashfs pseudofiles if we might want to
switch to ubifs? will there always be a squashfs underneath? might
we want to change the pseudofiles in an overlay?
11) haven't done (overlayfs) overlays at all
12) overlay.nix needs splitting up
13) upgrade ppp to something with an ipv6-up-script option
14) add ipv6 support generally
15) "ip address add" seems to magically recognise v4 vs v6 but
is that specified or fluke?
16) tighten up the module specs. (DONE) services.foo should be a s6-rc
service, (DONE) kernel config should be checked in some way
DONE 17) rename nixwrt references in kernel builder
18) maybe stop suffixing all the service names with .service
19) syslogd - use busybox or s6?
chat -s -S ogin:--ogin: root / "ip address show dev ppp0 | grep ppp0" 192.168.100.1 "/nix/store/*-s6-linux-init-*/bin/s6-linux-init-hpr -p"
Working towards a general goal of having a derivation we can
usefully run `nix path-info` on - or some other tool that will
tell us what's making the images big. The squashfs doesn't
have this information.
Towards that end (really? can't remember how ...) what would be a
way for packages to declare "I want to add files to /etc"? Is that
even a good idea?
Thinking we should turn s6-init-files back into a real derivation.
Tue Sep 27 00:31:45 BST 2022
> Thinking we should turn s6-init-files back into a real derivation.
This turns out to be Not That Simple, because it contains weird shit
(sticky bits and fifos).
Tue Sep 27 09:50:44 BST 2022
* allow modules to register activation scripts that are run on the
root filesystem once all packages are installed
- do they run on build or on host? if we're upgrading in place
how do we ship filesystem changes to the host?
or:
* allow modules to declare environment.*, use pseudofile on build and
create real files on host. will need to keep the implementation on
host faily simple because restricted environment
Tue Sep 27 16:14:18 BST 2022
TODO list is getting both longer and shorter, though longer on
average.
2) perhaps we shouldn't use process-based services like [ou]dhcp as
queryable endpoint for interface addresses (e.g. when adding routes).
Instead have a separate service for interface address that depends on
the *dhcp and uses its output
3) when I killed ppp it restarted, but I don't think it reran
defaultroute which is supposed to depend on it. (Might be important
e.g. if we'd been assigned a different IP address). Investigate
semantics of s6-rc service dependencies
4) figure out a nice way to fit ppp into this model as it actually
creates the interface instead of using an existing unconfigured one
5) write a test for udhcp
7) some of the pppoe config should be moved into a ppp service
11) haven't done (overlayfs) overlays at all
13) upgrade ppp to something with an ipv6-up-script option, move ppp and pppoe derivations into their own files
14) add ipv6 support generally
15) "ip address add" seems to magically recognise v4 vs v6 but
is that specified or fluke?
19) ship logs somehow to log collection system
21) dhcp, dns, hostap service for lan
22) support real hardware
Tue Sep 27 22:00:36 BST 2022
Found the cause of huge image size: rp-pppoe ships with scripts that
reference build-time packages, so we have x86-64 glibc in there
We don't need syslog just to accommodate ppp, there's an underdocumented
option for it to log to a file descriptor
Wed Sep 28 16:04:02 BST 2022
Based on https://unix.stackexchange.com/a/431953 if we can forge
ethernet packets we might be able to write tests for e.g. "is the vm
running a dhcp server"
Wed Sep 28 21:29:05 BST 2022
We can use Python "scapy" to generate dhcp request packets, and Python
'socket' model to send them encapsulated in UDP. Win
It's extremely janky python
Thu Sep 29 15:24:37 BST 2022
Two points to ponder
1) where service config depends on outputs of other services, we
do that rather ugly "$(cat ${output ....})" construct. Can we improve on
that? Maybe we could have some kind of tooling to read them as environment
variables ...
2) we have given no consideration yet to secrets. we want the secrets to
be not in the store; we want some way of refreshing them when they change
Sat Oct 1 14:24:21 BST 2022
The MAC80211_HWSIM kernel config creates virtual wlan[01] devices
which hostapd will work with, and a hwsim0 which we can use to monitor
(though not inject) trafic. Could we use this for wifi tests? How do
we make the guest hwsim0 visible to the host?
Sat Oct 1 18:41:31 BST 2022
virtual serial ports: I struggled with qemu for ages to get this to work.
You also need the unhelpfully named CONFIG_VIRTIO_CONSOLE option in
kconfig
QEMU_OPTIONS="-nodefaults -chardev socket,path=/tmp/wlan,server=on,wait=off,id=wlan -device virtio-serial-pci -device virtserialport,name=wlan,chardev=wlan"
Sun Oct 2 09:34:48 BST 2022
We could implement the secrets store as a service, then the secrets
are outputs.
Things we can do in qemu
1) make interface address service that depends on dhcp, instead of
being set by it directly
2) check out restart behaviour of dependent services when depended-on
service dies
3) pppd _creates_ an interface, work out how to fit it into this model
5) add bridge support for lan
8) upgrade ppp to something with an ipv6-up-script option, move ppp and pppoe derivations into their own files
9) get ipv6 address from pppoe
10) get ipv6 delegation from pppoe and add prefix to lan
11) support dhcp6 in dnsmasq, and advertise prefix on lan
12) firewalling and nat
- default deny or zero trust?
14) write secrets holder as a service with outputs
20) should we check that references to outputs actually correspond with
those provided by a service
Things we probably do on hardware
6) writable filesystem (ubifs?)
7) overlay with squashfs/ubifs - useful? think about workflows for
how this thing is installed
16) gl-ar750
17) mediatek device - gl-mt300 or whatever I have lying around
18) some kind of arm (banana pi router?)
19) should we give routeros a hardware ethernet and maybe an l2tp upstream,
then we could dogfood the hardware devices. we could run an l2tp service
at mythic-beasts, got a /48 there
https://skarnet.org/software/s6/s6-fghack.html looks like a handy thing
we hope we'll never have to use
Sun Oct 2 22:22:17 BST 2022
> make interface address service that depends on dhcp, instead of being set by it directly
We can do this for dhcp, but we can't do it for ppp. Running the ppp service
creates a ppp[012n] interface and assigns it an ipv4 address and there's not
a whole lot we can easily do to unbundle that.
So
- the ppp service needs to behave as if it were a "link" service
- either it *also* needs to behave as an address service, or we could
have an address service that subscribes to it and does nothing other than
translate output formats
Note regarding that second bullet: at the moment the static address
service has no outputs anyway!
Tue Oct 4 22:43:02 BST 2022
While trying to make the TFTP workflow not awful I seem to have written
a TFTP server.
Thu Oct 6 19:26:40 BST 2022
We have a booting kernel on gl-ar750, but we aren't at a point that it can
find a root filesystem
I'd *like* to be able to use the same delivery mechanism (kernel uimage
concatenated monolithic
Sat Oct 8 11:12:09 BST 2022
We have it booting on hardware, mounting root fs, running getty :-)
For NixWRT TFTP boots we used a single image with both kernel and squashfs, and
relied on CONFIG_MTD_SPLIT_FIRMWARE to identify where the boundary was and create
/dev/mdtn devices at the right offsets so that the kernel could find the
squashfs
For Liminix we're not going to do that.
* CONFIG_MTD_SPLIT_FIRMWARE is only available in OpenWrt patches
* it's an uncomfortable level of automagic just to save us doing two TFTPs
instea of one
* the generated image is anyway not the one we'd write to flash (has unneeded
PHRAM support)
* it means we need to memmap out enough ram for the whole image inc kernel when really
all we need to reserve is the rootfs bit
Sat Oct 8 11:23:08 BST 2022
"halt" and "reboot" don't work on gl-ar750
Sat Oct 8 13:10:00 BST 2022
Where do we go with this ar750?
- wired networking
- wifi
Sun Oct 9 09:57:35 BST 2022
We want to be able to package kernel modules as regular derivations, so that
they get added to the filesystem
This means they need access to kernel.modulesupport
This means kernel.modulesupport needs to be in pkgs too?
This is fine, probably, but we'd like to avoid closing over vmlinux because
there's no need for it to be in the filesystem
Mon Oct 10 22:57:23 BST 2022
The problem is that kernel kconfig options are manipulated in the
liminix modules, which means that data must be (transitively) available
to modules, so they can't be regular packages as they're tied so tightly
to the exact config. Unless we define a second overlay that references
the configuration object, but my head hurts when I start to think about that
so maybe not.
Tue Oct 11 00:00:13 BST 2022
Building ag71xx (ethernet driver) as a module doesn't work because
it references a symbol ath79_pll_base in the kernel that hasn't been
marked with EXPORT_SYMBOL.
We could forge an object file that "declares" it with a gross and disgusting hack like this
$ echo > empty # not actually "empty", objcopy complains about that
$ grep ath79_pll_base /nix/store/jcc114cd13xa8aa4mil35rlnmxnlmv09-vmlinux-mips-unknown-linux-musl-modulesupport/System.map
ffffffff807b2094 B ath79_pll_base
$ mips-unknown-linux-musl-objcopy -I binary -O elf32-big --add-section .bss=empty --add-symbol ath79_pll_base=.bss:0x807b2094 empty f.o
I don't claim this is a good idea, just an idea. Thought was that we would not
have to declare its type this way. Also it might not work with kaslr
https://stackoverflow.com/a/68903503
Backstory: why are we trying to build this as a module? because the
openwrt fork of it seems to be a bit more advanced than the mainline,
and I *suspect* that the mainline version doesn't work with our
openwrt-based device tree which ahs the mdio as a nested node inside
the ag71xx node - in mainline the driver seems to have all the mdio
stuff inline. So, could we build the openwrt driver without patching
the crap out of our kernel
Sun Oct 16 15:25:33 BST 2022
Executive decision: let's use the openwrt kernel (at least for
gl-ar750). Mainline kernel doesn’t have devicetree support for this
device or the SoC it’s based on, and the OpenWrt dts for it doesn’t
have the same "compatible"s, which makes me think that an indefinite
amount of patching will be necessary to make dts/modules for one of
them work with a kernel for the other
As a result: now we have eth0 appearing, but not eth1? Guessing we
need to add some kconfig for the switch
Mon Oct 17 21:23:37 BST 2022
we are spending ridiculous amounts of cpu/io time copying kernel source
trees from place to place, because we have kernel tree preparation
and actual building as two separate derivations.
I think the answer is to have a generic kernel build derivation
in the overlay, and then have the device overlays override it with
an additional phase to do openwrt patching or whatever else they
need to do.
Tue Oct 18 23:02:43 BST 2022
* previous TODO list is Aug 02, need to review
* dts is hardcoded to gl-ar750, that needs cleaning up
* figure out persistent addresses for ethernet
* fix halt/reboot
* "link" services have a "device" attribute, would much rather
have everything referenced using outputs than having two
different mechanisms for reading similar things
* Kconfig.local do we still need it?
* check all config instead of differentiating config/checkedConfig
Sun Feb 5 18:14:02 GMT 2023
We have resumed.
commit eb4efab6a215bf03cf5aab10d4ac909e83e9c148
Author: Daniel Barlow <dan@telent.net>
Date: Sat Jan 28 23:18:28 2023 +0000
* find out what works
* add that stuff to hydra
* fix the rest
* add that stuff to hydra
* convert to flake
* check if routeros can be run interactively
* some per-device docs in a form that can be transcluded for website
ci builds
* each of the tests has hardcoded device/config/etc
* build an "empty" configuration for each target device
* build an unstable configuration for qemu
Wed Feb 8 16:52:22 GMT 2023
We have hydra builds for all the previously-working devices, though we
don't yet know if any of those builds actually boots or does anything
useful.
[DONE] Would be nice to clean up the run-qemu and connect-qemu scripts
and put them in the buildEnv
Some thought needed about how to hook up the gl-ar750 to the internets,
ideally in a way that mirrors typical real uses. AAISP have an L2TP
service, but I would prefer to use pppoe on the device, so how to
translate one to t'other on an intermediary/gateway machine?
https://www.rfc-archive.org/getrfc.php?rfc=3817#gsc.tab=0 exists
as an RFC but I can't find anything that actually implements it
Actual Documentation (e.g. user and developer manuals) should live in
the liminix repo so it corresponds with the code, and can be rsynced
from there to the web site, maybe with a deploy hook or something.
Haven't decided what a good doc format is yet
If we create a flake for Hydra to run on, that _more or less_ means we
don't have any manual hydra jobset configuration to document.
There are still some tests that need adding to CI
[DONE] Should the per-device config be a module not an overlay? Given that
half of what's in it is kernel config (a module could set this)
and the rest is source tarball download specs (needs nixpkgs,
a module has this and could set it too) I wonder why it isn't already
[ALREADY DOES] Can we make Hydra report output sizes so we can plot closure size
trends and see if it all goes awful?
Thu Feb 9 08:14:39 GMT 2023
For better developer experience, I am thinking that either (1)
swap tasks 2 and 3 (writable filesystem before module system)
or (2) add NBD support so I can iterate on a real device without
full rebuilds every time
Fri Feb 10 06:18:25 PM GMT 2023
did the overlay->module thing
[DONE] Need to fix all the configuration around PHRAM, I can't see how it
would ever work
Sat Feb 11 14:37:45 GMT 2023
Consolidated TODO
* figure out persistent addresses for ethernet (?)
[SEEMS DONE] * fix halt/reboot
[DONE, NO] * Kconfig.local do we still need it?
[DONE] * check all config instead of differentiating config/checkedConfig
Things we can do in qemu
* "link" services have a "device" attribute, would much rather
have everything referenced using outputs than having two
different mechanisms for reading similar things
1) make interface address service that depends on dhcp, instead of
being set by it directly
2) check out restart behaviour of dependent services when depended-on
service dies
3) pppd _creates_ an interface, work out how to fit it into this model
5) add bridge support for lan
8) upgrade ppp to something with an ipv6-up-script option, move ppp and pppoe derivations into their own files
9) get ipv6 address from pppoe
10) get ipv6 delegation from pppoe and add prefix to lan
11) support dhcp6 in dnsmasq, and advertise prefix on lan
12) firewalling and nat
- default deny or zero trust?
14) write secrets holder as a service with outputs
20) should we check that references to outputs actually correspond with
those provided by a service
* Actual Documentation (e.g. user and developer manuals)
* make a flake
* There are still some tests that need adding to CI
Things we probably do on hardware
[DONE] * dts is hardcoded to gl-ar750, that needs cleaning up
6) writable filesystem (ubifs?)
7) overlay with squashfs/ubifs - useful? think about workflows for
how this thing is installed
16) gl-ar750
[DONE] * decide how to hook up the gl-ar750 to the internets
17) mediatek device - gl-mt300 or whatever I have lying around
18) some kind of arm (banana pi router?)
[DONE DIFERENTLY] 19) should we give routeros a hardware ethernet and maybe an l2tp upstream,
then we could dogfood the hardware devices. we could run an l2tp service
at mythic-beasts, got a /48 there
Sat Feb 11 15:57:31 GMT 2023
The reason we would like to run PPPoE instead of L2TP on the "rotuer" device is
- closer to real world scenario
- means no need to run dhcp client on the wan interface before we
even get to start the l2tpd
Sun Feb 12 14:57:28 GMT 2023
https://github.com/katalix/go-l2tp#kpppoed
Mon Feb 13 04:44:09 PM GMT 2023
if the gl-ar750 is connected to an ethernet card that linux is ignoring,
we're going to have to set up _some_ qemu thing just to run tftp from.
Tue Feb 14 17:59:34 GMT 2023
We should do a derivation that creates an ISO image and a qemu shell
script based on a configuration.nix, and put it in buildEnv. We'll
call it "borderNetVm" :
> A broadband remote access server (BRAS, B-RAS or BBRAS) routes
traffic to and from broadband remote access devices such as digital
subscriber line access multiplexers (DSLAM) on an Internet service
provider's (ISP) network.[1][2] BRAS can also be referred to as a
broadband network gateway or border network gateway (BNG).[3]
(for consistency we should rename the "access" qemu socket network to
match whatever we call this)
rm border.qcow2 ; nix-shell --argstr liminix `pwd` --argstr nixpkgs `pwd`/../nixpkgs --argstr unstable `pwd`/../unstable-nixpkgs/ ci.nix -A buildEnv --run "run-border-vm"
Wed Feb 15 22:56:59 GMT 2023
configuration for border vm needs to come from somewhere so it's good
for more people than just me
- pci device for setting up the ethernet
- lns address
- uid so it can do 9p shares? do we need to map things here?
also need to document the host-side bits so that people can set up
their spare ethernet as vfio
next step for hacking is to figure out what I was doing with pppoe
Wed Feb 15 22:59:56 GMT 2023
docs ...
* introduction
* user guide
** how to build it
** how to flash it on your device
** what to put in configuration.nix
** modules
* developer guide
** building/running with qemu
*** emulated upstream
** building/running on hardware
*** run in place with TFTP
*** emulated upstream
** CI
** Roadmap
** Contributing
nix-shell -p sphinx --run "make -C doc html"
https://francis.begyn.be/blog/nixos-home-router contains information about avahi reflector
Fri Feb 17 00:09:34 GMT 2023
29 11.282085831 81.187.76.242 → 8.8.8.8 ICMP 106 Echo (ping) request id=0x0187, seq=2/512, 4
30 11.286314642 90.155.53.19 → 81.187.76.242 ICMP 78 Destination unreachable (Communication admin)
We're getting packets over the pppoe-l2tp relay thing. Just have to
work out now why we're not routing
Fri Feb 17 16:54:41 GMT 2023
Haha. We weren't routing because we'd used the wrong CHAP password
Fri Feb 17 16:58:27 GMT 2023
This TODO is for nlnet task 1 and for bits of subsequent tasks that
are annoying enough that I might poke at them anyway:
1) gl-ar750, why do we get "ag71xx 19000000.eth: invalid MAC address, using random address"
2) gl-ar750, wifi
3) document services so I can remember how they work. Refer back to Oct 18 for notes that no longer make sense
4) check out restart behaviour of dependent services when depended-on service dies
5) pppd _creates_ an interface, work out how to fit it into this model
6) add bridge support for lan
7) upgrade ppp to something with an ipv6-up-script option, move ppp and pppoe derivations into their own files
8) get ipv6 address from pppoe
9) get ipv6 delegation from pppoe and add prefix to lan
10) support dhcp6 in dnsmasq, and advertise prefix on lan
11) firewalling and nat - default deny or zero trust?
13) should we check that references to outputs actually correspond with
14) make a flake?
15) see if there are other tests that need adding to CI
15a) is bordervm derivation tested?
18) gl-mt300a
19) gl-mt300n-v2
20) publish the manual using CI
12) write secrets holder as a service with outputs
16) writable filesystem (ubifs?)
17) overlay with squashfs/ubifs - useful? think about workflows for how this thing is installed
I could plug tninkpad into the gl-ar750 LAN port to dogfood the wired
networking
Sat Feb 18 14:26:45 GMT 2023
Apparently we're not currently doing anything special with busybox,
just using the default nixos build with the default applets.
We'd like to be able to say in modules which applets they need,
so that we build all necessary applets but don't waste any space.
But we don't want to build a busybox for each module because that
would be a big waste of space.
One option:
- add busybox configuration to `config` so that modules can maul it
- add a busybox module that builds it with union of all config and
adds link in /bin
- make everything else look in /bin instead of referencing pkgs.busybox
It would be good if services could assert somehow that their required
config is present
Sat Feb 18 23:45:13 GMT 2023
# lsmod
cd /lib/modules/mac80211
insmod ./compat/compat.ko
insmod ./net/wireless/cfg80211.ko
insmod ./net/mac80211/mac80211.ko
insmod ./drivers/net/wireless/ath/ath.ko
insmod ./drivers/net/wireless/ath/ath9k/ath9k_hw.ko
insmod ./drivers/net/wireless/ath/ath9k/ath9k_common.ko
insmod ./drivers/net/wireless/ath/ath9k/ath9k.ko
insmod ./drivers/net/wireless/ath/ath10k/ath10k_core.ko
insmod ./drivers/net/wireless/ath/ath10k/ath10k_pci.ko
[21.344930] ath9k 18100000.wmac: failed to load calibration data from mtd device
[21.352728] ath: phy0: parsing configuration from OF node
[21.362576] ath: phy0: serialize_regmode is 0
[21.367092] ath: phy0: UNDEFINED -> AWAKE
[21.372051] ath: phy0: Trying EEPROM access at Address 0x03ff
[21.377999] ath: phy0: Trying EEPROM access at Address 0x0fff
[21.383940] ath: phy0: Trying EEPROM access at Address 0x01ff
[21.389879] ath: phy0: Trying OTP access at Address 0x03ff
[21.400396] Data bus error, epc == 8027964c, ra == 83125880
[21.406156] Oops[#1]:
Sun Feb 19 18:15:27 GMT 2023
We have ath9k listening for packets. To make this ready to use:
- need to load the modules
- enable bridging lan with wlan
- packet forwarding
- firewall
Mon Feb 20 20:41:17 GMT 2023
need to fix all the other broken ci jobs :-(
The wlan test is failing because we moved mac80211 to a module and
there's nothing running to insmod it
Wed Feb 22 18:17:17 GMT 2023
bridge is e2b3738d0f8c3f2fd76ebcef65612de502a7b121 but it's the wrong
way around: the master interface needs to be up whether or not all
of its children are, so members depend on master not vice versa
Next steps:
- re-implement bridge, enable bridging lan with wlan
- packet forwarding
- firewall
- ath10k
- ipv6
Fri Feb 24 23:37:56 GMT 2023
bridging wlan was made complex because can't add a device to a bridge
until it's operational, and wlan0 is not operational until hostapd
has churned awhile. Therefore, "waitup" listens for netlink messages
and notifies s6 readiness stuff
we have a firewall nft script but we're not running it on boot
we have forwarding but no dns, maybe because we haven't told
dnsmasq about any upstream servers
Sun Feb 26 21:08:47 GMT 2023
to add firmware we need to put files in /lib/firmware, which means
a module
i guess we should do that in the device module
we can create the firmware files as packages
for the cal data we would like to get it from the device MTD "art"
partition at
boot time.
f
====from openwrt
case "$FIRMWARE" in
"ath10k/cal-pci-0000:00:00.0.bin")
case $board in
allnet,all-wap02860ac|\
araknis,an-500-ap-i-ac|\
araknis,an-700-ap-i-ac|\
engenius,eap1200h|\
engenius,enstationac-v1|\
glinet,gl-x750|\
watchguard,ap300)
caldata_extract "art" 0x5000 0x844
ath10k_patch_mac $(macaddr_add $(mtd_get_mac_binary art 0x0) 2)
caldata_extract part offset count
caldata_dd $mtd /lib/firmware/$FIRMWARE $count $offset || \
caldata_die "failed to extract calibration data from $mtd"
dd if=$source of=$target iflag=skip_bytes,fullblock bs=$count skip=$offset count=1 2>/dev/null
=======
part=$(basename $(dirname $(grep -l art /sys/class/mtd/*/name)))
dd if=/dev/$part \
of=/run/cal-pci-0000:00:00.0.bin iflag=skip_bytes,fullblock \
bs=0x844 skip=0x5000 count=1
Mon Feb 27 22:46:37 GMT 2023
Found and fixed a bunchg of things that were stopping ath10k from
working. The remaining problem is (I think) that insmod is not
synchronous, so "ip link set up dev wlan1" doesn't work immediately
after the module is inserted. Maybe we need another netlink thing
to wait until the interface is present.
Wed Mar 1 18:26:44 GMT 2023
ath10k works, but the wlan module loading stuff is quite kludgey
I wonder if wlan0, wlan1, eth0, eth1 etc should be defined per-device
- how does the aplication config know which devices exist? If we
decide to switch to some form of persistent device naming, the names
will differ from one device to the next. Perhaps the device should
also provide standard names where possible?
services.network.links = {
lan = interface { ... };
wan = interface { ... };
wlan_24 = interface { ... };
wlan_5 = interface { ... };
}
Thu Mar 2 22:45:11 GMT 2023
We have a flashable image!
Now we can use the gl-ar750 for internet access in the shed, we can
apppropriate the other device that's in there and try Liminix on it
Fri Mar 3 23:08:58 GMT 2023
If we're going to unplug serial console from the gl-ar750 maybe we
should install an ssh server first.
0) set a root password
1) allow setting a root password from configuration.nix
(means defining config.users properly)
2) allow authorizedKeys per user
3) dropbear service
4) see if the wired lan works! :-)
Sat Mar 4 12:31:07 GMT 2023
To improve logging, each service should have its own s6-log service
which prefixes the service name onto the log line and then sends to
stdout
https://skarnet.org/software/s6/servicedir.html
https://skarnet.org/software/s6/s6-log.html
As far as I can tell, the `log` directory inside the service
directory should itself be a service directory for the s6-log
process that does this
.... hahaha no that doesn't work
s6-rc, for some reason, ignores the `log` directory and requires
that loggers be done with consumer-for and producer-for instead
Sat Mar 4 23:27:00 GMT 2023
notes for this week's news update
* ath10k kernel support and and firmware
- 5GHz wifi works
- need to retrieve the firmware from a special - partition on the
device itself, so we do that using a service that - the wlan
interface depends on
* replace waitup with more generally useful ifwait
to make the ath10k load at boot, we need to insert the module and then
wait for it to do something or other in the background before we can
configure the interface. so we need something like waitup but
for presence not operational state
it turns out that a program that just waits for a particular interface
state and then exits is quite simple to add into run scripts and
we don't need all that notification-fd stuff anyway
* move FW_LOADER* config to modules/base
* rejig config a bit.
- device hardware characteristics are now under
the `hardware` key and include the available network interfaces.
- options for users and groups are now defined a bit more
specifically than "attrset", making it possible to e.g. set a
root password
- dts is moved from `boot` to `hardware`
* now producing flashable images, so you can generate a liminix config
and write it to the device instead of having to boot using TFTP and
a serial console every time
* ssh support
* prefix logs with the service name
Sun Mar 5 22:51:21 GMT 2023
Added swconfig: it was a straight copy from nixwrt and hasn't changed
upstream since. But don't need it, because the lan port works fine
without it (I assume both lan ports and the cpu are all connected
untagged)
Mon Mar 6 09:42:33 GMT 2023
Today I plugged in the mt300a.
echo 17 >/sys/class/gpio/export
echo out >/sys/class/gpio/gpio17/direction
why are our images getting big
- lua links ncurses
- hostapd links openssl and sqlite
- nftables needs
- iptables?
- jansson? what is that?
- libedit/readline
- ifwait needs bash
File: result/squashfs
Size: 10371072 Blocks: 20256 IO Block: 4096 regular file
with smaller nftables: 9617408 Blocks: 18784
hostapd wqithout sqlite 9003008 Blocks: 17584
without bash: 8622080 Blocks: 16840 IO Block: 4096 regular file
without lua readline: bigger?! 8769536 Blocks: 17128 IO Block: 4096 regular file
Mon Mar 6 20:57:49 GMT 2023
[ 0.539992] mtk_soc_eth 10100000.ethernet: mdio-bus disabled
[ 10.493918] platform regulatory.0: Direct firmware load for regulatory.db fail
ed with error -2
[ 10.502828] cfg80211: failed to load regulatory.db
Check in morning, but whichever port the ethernet cable is plugged into,
is considered by the kernel as port 0 - which I think we should treat as
WAN
VLAN 1:
vid: 1
ports: 1 2 3 4 5 6t
VLAN 2:
vid: 2
ports: 0 6t
ip link add link eth0 name lan type vlan id 1
ip link add link eth0 name wan type vlan id 2
figure out how to add these to gl-mt300a device config
then extedner.nix can add a bridge
Tue Mar 7 20:13:15 GMT 2023
We need NTP or some other way to get accurate time
[done] Need to add regulatory.db somewhere standard, maybe modules/wlan?
Tue Mar 7 21:43:56 GMT 2023
When we get to phase 2, need to review how network interfaces and
their addresses interplay. It should be possible to have a network