Wiring an SPI SD Card Adapter to an ESP32 NodeMCU

[sebi5361]

I would like to log data on an SD card.
Inspired from what I read about the Olimex ESP32-EVB board diagram, I connected a SPI SD Card Adapter Board to my ESP32 NodeMCU as follows:
3.3V - 3.3V
CS - GND
MOSI - GPIO15
CLK - GPIO14
MISO - GPIO2
GND - GND
Unfortunately this doesn't work as the card doesn't seem to communicate with the MCU.
Any help would be appreciated.

[fredlcore]

Have you tried using one of the SD card examples available on the net? If I'm not mistaken, t here are also several SD card libraries around, so the one we use in BSB-LAN may not be the one for your adapter. And of course you have to activate #define ESP32_USE_SDin BSB_LAN_config.h - have you done that?

[sebi5361]

Yes indeed #define ESP32_USE_SD is activated in 'BSB_LAN_config.h'.

To run your software without any modification, I tried to make my hardware similar to the one of the Olimex ESP32-EVB board that your software supports out of the box.
I chose a SPI SD Card adapter with very rudimental electronics only (mainly pull-ups):

... that I connected according to the wiring scheme of the Olimex ESP32-EVB board, to my Joy-It ESP32 NodeMCU:

Thus:

SPI SD Card Adapter	Joy-It ESP32 NodeMCU
3.3V	3.3V
CS	3.3V*
MOSI	GPIO15
CLK	GPIO14
MISO	GPIO2
GND	GND

(*): or GND as CS is supposed to be active low??? I tried both without success.

Could you point me into your code to the place where you specify the pin numbers of the ESP32 SPI bus connector used for communicating with the SD Card?

[fredlcore]

This is all done in the respective SD card library, nothing is specifically defined in BSB-LAN. That's why I recommended you to first try to get the SD adapter working with basic examples outside BSB-LAN.

[sebi5361]

EDIT: Wrong wiring, correct wiring below!

After more investigation it works with the wiring above.
My mistake was that I was using a 2.2k resistor (as on the Olimex diagram) to bring CS high and I had to remove it.
For convenience, I decided to use the UEXT port of the BSB-LAN adapter to connect the SD-Card holder to the Joy-it MCU (see picture below).

Now it just works fine:

Milliseconds;Date;Parameter;Description;Value;Unit
661569;08.02.2024 21:19:09;8410;Return temperature heat pump;38.0;°C
667120;08.02.2024 21:19:14;8410;Return temperature heat pump;38.0;°C
672655;08.02.2024 21:19:20;8410;Return temperature heat pump;38.0;°C
678024;08.02.2024 21:19:25;8410;Return temperature heat pump;38.0;°C
683617;08.02.2024 21:19:31;8410;Return temperature heat pump;38.0;°C
689204;08.02.2024 21:19:36;8410;Return temperature heat pump;38.0;°C
694685;08.02.2024 21:19:42;8410;Return temperature heat pump;38.0;°C
700318;08.02.2024 21:19:48;8410;Return temperature heat pump;38.0;°C
705676;08.02.2024 21:19:53;8410;Return temperature heat pump;38.0;°C
711062;08.02.2024 21:19:58;8410;Return temperature heat pump;38.0;°C
716502;08.02.2024 21:20:04;8410;Return temperature heat pump;38.0;°C
721909;08.02.2024 21:20:09;8410;Return temperature heat pump;38.0;°C
727325;08.02.2024 21:20:15;8410;Return temperature heat pump;38.0;°C
732677;08.02.2024 21:20:20;8410;Return temperature heat pump;38.0;°C

Thanks!

[fredlcore]

Thanks providing this solution, so it means you just connect the adapter 1:1 without any other parts to the UEXT pinholes? That would really be easy. Or do you have some extra wires on the back of the adapter?

[fredlcore]

And CS now also goes to 3.3V?

[sebi5361]

Thanks providing this solution, so it means you just connect the adapter 1:1 without any other parts to the UEXT pinholes? That would really be easy. Or do you have some extra wires on the back of the adapter?
And CS now also goes to 3.3V?

Yes ~1:1, easy, no extra wire but a drop of solder between 3.3V and CS to connect them directly.

[fredlcore]

Thanks, but on my adapter some of the UEXT pins are not connected to anything, so how did you make the connection from the SD adapter via the UEXT pins to GPIOs 2, 14 and 15 respectively?

[sebi5361]

Ohh you make me doubt now.

Indeed it was my assumption that all the pins of UEXT were connected to the Joy-it connector rails on the BSB-LAN adapter.
I have version 4.3 of the adapter. Do you know if it is the case? I will check with a multimeter.

But I did check the wiring of the SD adapter on a breadboard (without using UEXT, nor the BSB-LAN adapter) removing the card once populated with the log files and checking their existence directly from my PC. So at least, I am sure of that part.

Do you know if the software switches to the local EPROM if the SD-Card is not available? ... as currently I am logging data:

[fredlcore]

Only UEXT pins 1, 2, 4 and 10 are connected to VCC, GND, RX and TX, all other pins are not connected. It wouldn't make much sense to connect them to some random pins on the Joy-It rails because these may or may not have the same function. Even within the Olimex range, the UEXT pins refer to different GPIOs on the ESP32. For things like you are doing, I have connected the outer double pin row next to the UEXT connector on a 1:1 basis. So if you solder a pin header on that one, you have access to those pins. If these are not the "right" ones, then most software/library lets you configure other GPIOs so that you should be able to make it match accordingly.

And no, BSB-LAN does not fall back to writing on the EEPROM if you have no SD card connected. This would lead to messed up data log files if you insert and remove the SD card because part of it would be written to the card and other parts to the EEPROM then.
These are actually two different libraries that are included based on the compiler directive that you (de)activate in the _config.h file, so a fallback would even be impossible in the current code. The configuration page in the webinterface lists the compiler directives used, so you can see if #define ESP32_USE_SD is actually active or not - in the latter case, you have logged to EEPROM all the time.

[sebi5361]

Thank you Fred!
A wrong firmware ('#define ESP32_USE_SD' was commented; I used a different ESP32 to experiment on my breadboard) didn't let me notice my wiring was wrong.
With the correct firmware and the correct wiring (using the outer double pin rows) logging data on a SD-Card using this adapter works just fine (see pictures below):

SPI SD Card Adapter	Joy-It ESP32 NodeMCU
3.3V	3.3V
CS	3.3V
MOSI	GPIO15
CLK	GPIO14
MISO	GPIO2
GND	GND

[fredlcore]

Great, thanks! How did you connect the blue cable (I assume GPIO2?) to the BSB-LAN adapter?

[sebi5361]

I am not sure to understand perfectly your question: I connected the blue wire the same way I soldered the other wires but on the top rail.
Blue wire = GPIO14 = CLK

[fredlcore]

Sorry for this not being so straightforward as it maybe could be, but the "upper" double row is not connected to the Joy-It header row. That's because this double row is used also for the optional use of the adapter on a Raspberry Pi. Therefore these pins cannot be shortened. Of course you could do so yourself with just a drop of solder, but if you just connected the blue wire to the adjacent pin of D14, then this means that your setup doesn't need it ;). You can try and see for yourself that on that row, the pins are not connected, so the pin would just float.
BTW: I used small silkscreen traces to show which pins are connected and which ones are not.

[sebi5361]

As unbelievable as it sounds recording data on the SD-Card was working fine without the clock signal!!!

I added a droplet of solder to shorten the blue wire pad to the D14 pad as you suggested:

... and it still works ;)
The silkscreen is difficult to see once the rails are soldered, but I should have taken the time to check for continuity between the pads.

[sebi5361]

Yes it is.
When you asked me to investigate about this CS pin being high, I bought another SD card adapter and used a different ESP32 board I had laying around to perform tests on a breadboard while leaving my original setup in place connected to my heater.

Yesterday I performed OTA update (v3.3.2-> v3.3.3) of my original setup. It works fine with the SD card adapter connected.

[fredlcore]

OTA is not affected by the upload problem. I assume that if you do an update via USB, you will encounter the problems you experienced in the beginning and will have to detach the SD card adapter from the ESP32.
But more importantly, could you please do a log download from your setup (e.g. via wget or curl) and measure the speed? If it is more or less the same speed, then I'd just promote the SPI solution which seems to work on all ESP32s and all card reader combinations. Thank you!

[fredlcore]

I got my adapter working with the SPI library, using the following pins:

SPI SD Card Adapter	Joy-It ESP32 NodeMCU
VCC	VIN*
CS	GPIO5
MOSI	GPIO23
CLK	GPIO18
MISO	GPIO19
GND	GND

*Note: My adapter did not work when connecting it to 3V3, but it worked when using VIN. Other adapters, however, may work with 3V3 as well. Try VIN first, and if everything works fine, you can try 3V3 as it is easier to connect to, as you will have all required pins on one side.
Transfer speed with SPI is around 250kB/s. Can you give us the speed with SD_MMC, @sebi5361?

[sebi5361]

Using wget, the speed I get is 444KB/s (11.17M in 26s).

[fredlcore]

Great, thanks, that's significant. I've therefore added the definement FORCE_SD_MMC_ON_NODEMCU that will allow users with a NodeMCU to access their SD card via SD_MMC instead of SPI. If at some point you have the time to do a firmware upload via USB with your SD card reader still connected that way, it would be great if you could confirm whether or not the upload is only possible after disconnecting the SD card reader. Thank you!

[sebi5361]

I confirm it is possible to upload the firmware by USB with the SD card adapter connected and the SD card inserted (see below photos) on my WROOM (Joy-It?) board (see pictures below).
On my other board (WROVER), I noticed that it is possible to upload a new firmware by USB as soon as the MISO pin from the adapter (board pin 2) gets disconnected.
By the way, I get an error compiling firmware 3.3.4 with #define FORCE_SD_MMC_ON_NODEMCU:

C:\TINKERING\BSB-LAN\BSB-LAN-master_v3.3.4\BSB_LAN\BSB_LAN.ino:4976:12: error: 'SD' was not declared in this scope
     return SD.usedBytes();
            ^~

[fredlcore]

That is really strange. I have an exactly same-looking WROOM-board (although not from Joy-It), and I have the same problems that you describe with the WROVER (GPIO pin 2 has to be detached to perform an upload).

If you had no problems with both boards, I would assume that your card reader already provides the required pull-ups that mine is missing. But I don't understand why one board has no problems with the upload with GPIO2 connected and the other one has that problem. Especially since the Espressif lists this problem for both the WROVER as well as the DevKit boards:
https://docs.espressif.com/projects/esp-idf/en/stable/esp32/api-reference/peripherals/sd_pullup_requirements.html#sdio-dev-kits
This is my card reader:

If there was a way to determine which boards are "good" and/or which card readers are "good", then that would be a great help, because doubling the SD card speed is certainly an advantage. But unless we can figure that out, I won't be able to recommend that solution because it will only lead to bug reports here with people claiming that either the card and/or the upload is no longer working :(...

[fredlcore]

The IC on the top is a 74HC125, i.e. a quad-tri-state buffer, the four resistors are 3.3k...

[fredlcore]

...and as for the compiler error, that should be fixed now...

[sebi5361]

Recommending this hack or not is up to you, but disconnecting an external circuitry to upload a firmware is a well known manipulation in the Arduino community, and can be done easily if the board is plugged in your BSB-LAN board, not soldered.
Additionally the error message when trying to upload a new firmware by USB is quite explicit:
WARNING: Failed to communicate with the flash chip, read/write operations will fail.
Try checking the chip connections or removing any other hardware connected to IOs.

Sadly, I am still getting the same compilation error ('SD' was not declared in this scope) with your latest firmware version.
To reproduce: download the latest code, rename files, remove comment in front of #define FORCE_SD_MMC_ON_NODEMCU, compile.

[fredlcore]

Well, (un)fortunately, most users of BSB-LAN are not part of the Arduino community. For the majority of new users, this is rather the first time they do anything with a microcontroller, so the error messages above could easily be understood in a way that BSB-LAN is the problem here (because it is connected to the IOs), and then users report "bugs" that are not related to BSB-LAN.
If we can identify why certain combinations of ESP32s and card readers do (not) work, then I can make a clear recommendation in the manual. But as long as this remains a mystery (that I would really like to solve), I'll stick to recommending the SPI connection. Even if it's just 50% of the speed, it should work across all boards and card readers.

And sorry for the bug, there was one last instance I didn't adjust. Here it compiles fine now.

[sebi5361]

Yes the firmware works fine now with the latest commit!
As unplugging the SD card adapter before uploading the firmware works for all MCU / Adapter combinaisons I would make sure to mention it once for all.
Anyhow I understand your motivation to find the reasons behind those different cases.
To help you with that, I used my multimeter to reverse engineer my SD card adapter and measured a 10K pullup resistor on the MISO pin.

[fredlcore]

Thanks! I'll try to get a multimeter tomorrow and check if the resistance on mine is 3k3. Just to be sure, the 10k go from MISO to VCC, right? It's not 10k from the SD card MISO pin to the corresponding pin on the ESP32?
Looking more closely at my adapter, it seems to me that all data-related lines go via the 74HC125. This could be an indication that my reader is actually a SPI(-only) reader because I assume that this tri-state buffer is used to take the card reader of the SPI bus in case other devices are being accessed. Maybe the buffer is activated in conjunction with the SPI bus' CS signal, so I might have to try and pull the CS signal low to get the buffer working when used in SDMMC mode... Also, maybe my 3k3 resistors are used for level shifting, as my board only works with VIN, not with 3V3, so the 3k3 resistors might be running in series with the data line, and not as pullups. Too bad I can't find any kind of datasheets for our two card readers. That would make things a lot easier ;)...

[fredlcore]

Ok, found one diagram here which shows that my 3k3 resistors are in serial with the data line and probably used as part of level shifting:

[fredlcore]

Since I can't find a diagram for your adapter, could you just measure the connections of the four resistors and the capacitor to see where they go to / come from? I assume they are more or less 1:1 connections from the connector to the SD card with four pullups and some kind of stabilizing capacitor...

[fredlcore]

I was wondering if it's possible to make use of the CD pin of the SD card, but I'm not good enough in electronics to just test this by only using my imagination: If the SD card is ejected, CD is outputting low (i.e. tied to GND?). GPIO2, on the other hand, needs to be tied low when a FW update is made. Could the low CD output be used to also pull GPIO2 low as well? Because Espressif themselve suggest a similar solution:

One way to do this is to connect GPIO0 and GPIO2 using a jumper, and then the auto-reset circuit on most development boards will pull GPIO2 low along with GPIO0, when entering download mode.

What I don't know is what kind of transistor or something like that would be necessary so that the low signal from CD is only connecting GPIO2 to GND if CD is low. I guess that would require a PNP transistor with CD connected to the base, GPIO2 to the collector and GND to the emitter? Or vice versa? So if CD is high, nothing would happen, and if CD goes low, GPIO2 is pulled to GND and the upload will work - at least in my theory ;)...

[sebi5361]

Here is the reverse engineering of my card:

• There are 6 lines GND, 3v3, MOSI, MISO, CLK and CS. I call line a connection between a board pin and the corresponding connector pin.
• 2 capacitors are mounted between the 3v3 and the GND lines (to smoothen the current).
• 3x 10K pullup resistors are mounted on the MISO, MOSI and CLK lines (from that signal line to 3v3).
• CS line is direct (no additional component)
• a 10K resistor is mounted between the 3v3 line and {both-connector-pins-1-and-8}
• connector CD1 and CD2 pins (Card Detect, open without a card) are left unconnected

[fredlcore]

Wow, great, thanks a lot! Pins 1 and 8 would be the additional two lines needed for 4-bit SDMMC communication and they are tied high if 1-bit mode is in use.
Just for reference, this is the pinout for SDMMC and SPI modes:

SDMMC	SPI
DAT2	--
DAT3/CD	CS
CMD	MOSI
3V3	3V3
CLK	CLK
GND	GND
DAT0	MISO
DAT1	--

[sebi5361]

Regarding the pullups resistors on my board, yes they are NOT mounted in series on the line but rather in parallel with 3v3.

About your board, I think the 3K3 resistors are only there to limit the current going to the 74HC125. This component acts like a repeater (1->1, 0->0) but at a different voltage. What I mean is that your board could be either 3v3 or 5v logic, whereas the SD card needs to be 3v3.

I am still confused about this CS line (as I thought it was a SD card output) but I think your adapter is wired very similarly to mine, and thus could work the same way (in MMC mode).

CD1 and CD2 are the two poles of a switch that is open without a card and close with a SD card in. Thus you could connect those two poles between the adapter board MISO pin and the MCU board GPIO2 pin to allow disconnecting the adapter board MISO line pullup resistor when removing the SD card. It would still be necessary for those two pins to be accessible and to mention the users to remove the SD card when uploading a new firmware (which is not far from mentioning them to disconnect the SD card adapter).

[fredlcore]

Yes, I read elsewhere that most boards can be used interchangeably. Probably mine would work if I add the pullups (or maybe even the internal ones of the ESP32 could work via pinMode(X,INPUT_PULLUP);. If these are also needed for lines 1 and 8 (DAT2 and DAT3), that would be difficult because my board doesn't have those, and since the lines are not (easily) accessible, I'd have to fiddle directly with the tiny connectors on the card connector. But maybe/hopefully that's optional.

CS is the same as CD, but the former is for SPI mode, the latter for SDMMC mode, and each have a different direction, depending on how the card is initialized and in which state of initialization it is:
http://elm-chan.org/docs/mmc/mmc_e.html#spiinit
So after your card has been initialized in SDMMC mode, pin 2 is CD (output), and not CS (input). Apparently, according to Sandisk specs (https://electronics.stackexchange.com/questions/207335/what-is-the-way-to-detect-sd-card-insertion-removal#comment1252393_207335) SD cards have an internal pullup on that pin which is why you have measured the 3.3V on that pin. That's why I was wondering if that signal could be used in conjunction with a transistor to connect the adapter board's MISO pin to GPIO2 via the transistor. So in that case, could a NPN transistor be used to connect the two pins if CD ouputs a 3.3V signal? Upon card removal, the transistor would no longer connect the two pins and an update could be done. If that makes sense, would you be able to test something like that, as I don't have the necessary components available here?

I would certainly prefer this solution because quite a few people are using enclosures for their setups. So if you could just remove the SD card when you want to do an update without opening up the enclosure, that would be much easier than unplugging the wires and then make sure you reconnect them properly. Of course, if this is not possible to do, then trying to route the pin via CD1 and CD2 is of course a more difficult operation.

[sebi5361]

I am afraid I don't have the skills to do that unfortunatly.
But I don't see any reason why it shouldn't work eventually.
Here are some thoughts though:
GPIO2 = MOSI AND CS right?
Thus MOSI and CS are interchangeable as Base and Collector of the NPN transistor. What option is preferable?
Let's imagine the option where CS is on the base. As i_Base->Emitter always flow on a transistor, it needs to be limited not to trigger a high signal on GPIO2 just by itself. How to compute the required value of the resistor placed in series for such a purpose? Are additional resistors required at the collector and emitter? What about the other option?
Those questions should be addressed. Those are the skills I lack...

[fredlcore]

Thanks a lot for these thoughts - you already have more knowledge on this matter than me because I didn't know that there is always a flow from Base to emmitter. That of course would rule out having CD as the base. Having MISO (which is the DAT0 line in SDMMC, not MOSI which would be the CMD line) as the base would then make more sense because if the base is active through current from MISO, there would only be current flowing when there is also data being transmitted. It would be an easy test because MISO and CD are so close together that one could solder the two B/C legs of the transistor right to the two pins and then attach the jumper wire not to the MISO pin but to the emitter of the transistor...
I could try and get the materials here and test it by myself, but I'd currently still lack a scope with which I could test in case things are not going as they should :(...

[sebi5361]

Unfortunately the problem remains the same with MISO for the base as one would have to make sure its pullup current gets limited enough not to trigger a high signal on GPIO2 by itself.
My (poor) knowledge comes from this video.

[fredlcore]

Aaah, I forgot about the pullup which I don't have on my adapter. Never mind, then it won't work that way, and that's ok. Thanks for all your testing and the idea in the first place!

[fredlcore]

@sebi5361: Sorry to bother you again, but I got a SD card reader today which kind of looks like yours, except that it's for full-size SD-cards:

I have no problems getting it to work via SPI and the SD.h library when I power it via VIN (5V). However if I want to run it via SD_MMC.h and power it via 3V3, the SD_MMC.begin() function hangs and the watchdog kicks in after a while. If I power it via VIN, at least it doesn't hang, but it gives me the "Failed to initialize the card (0x107). Make sure SD card lines have pull-up resistors in place" error message I had with my previous card reader. But here, all lines seem to have the same 10k pullups as they do on your card. I really wonder why the reader doesn't work with 3V3, although it has a dedicated pin for 5V and 3.3V supply, and then giving me the pullup error when it's powered via 5V.
Schematics should be like this one, as the module looks identical:
https://forum.arduino.cc/t/sd-card-reader-connection-to-arduino-uno/1025932/16
Did you run into any similar behaviour when you did your initial tests?

[sebi5361]

You are not bothering me.
Yes according to the schematics that board is comparable to mine: it is already populated with 10K pullups resistors. And powering it with 5V or 3.3V should be similar, as 3.3V is generated from 5V thanks to the LDO.
I don't recall any particular manipulation to make my adapter work (except I had bad connections with the Dupond wires and had to try a few times giggling the circuitry around).
Remember that once the connection is lost it won't recover: you have to try again from a fresh start.
Make sure your SD card is formatted in FAT32. Try with another card.
I would try with and without the card to see if I get different error messages to understand if my card is at least detected.
Do you have a multimeter now? It could help to check the different voltages at rest (MISO, MOSI, SCK, and CS aka CD should be 3.3V).
Any progress?

[fredlcore]

Thanks, I got a multimeter today, but just an analogue one, so measurements are not really exact. I also changed the Dupont jumper wires to make sure they are not worn out. I measure a bit above 3V on the adapter on +3.3V and MOSI. MISO and CLK are around 1.6V, both with inserted SD card and without.
All this is measured after booting BSB-LAN while it waits for the SD card initialization.
If keep the power supply pin in +3.3V, but switch it to VIN on the ESP32, it completes the SD card initialization, but with the error message that the pull-up resistors might be missing. This could probably be that the higher-than-expected voltage on the 3.3V pin leads to undefined behaviour, but this is just a guess.
So, yes, if you could measure the voltages, especially on MISO and CLK, that would be helpful.

[sebi5361]

While powered but disconnected from the MCU I get MOSI = MISO = CLK = 3.3V (thanks to the pullup resistors).
While connected to the MCU and running, I get MOSI = MISO = 3.3V and CLK = 0V (The multimeter can't measure signal spikes there are during communications that can be observed using a scope)
It seems there are 10K pulldown resistors on your MCU board, which together with the SD card adapter pullup resistors, make voltage divider bridges of ratio 1/2 (1.6V ~ 3.3V/2)

[fredlcore]

Yes, I was thinking the same, but couldn't explain how this could be because Pullups need to be activated through code, and I don't even know if there are Pulldowns on the ESP32 board.
I guess, I'll try another board, I found one that looks exactly like yours, and if that also isn't working, then there are either different types of Joy-It boards around or... I really don't know ;)...

[fredlcore]

I have finally find the time to do some measurements with a digital multimeter and also ordered an SD card reader such as the one that you are having. Again, it did not work. I haven't tested voltages yet, but what I did find out now was that on the ESP32, there is a pull-up resistance between D2 and 3V3 of around 11.8k and a pull-up between D15 and 3V3 of around 70k. Now, with the micro SD card reader that looks exactly like yours and which has pull-ups of 10k between MOSI and MISO, of course these pull-up resistances are affected just by connecting the adapter. The resistance between D2 (MISO) and 3V3 drops to a bit over 5V and the pull-up resistance between D15 (MOSI) and 3V3 drops to around 22k. When I power on the ESP32, it seems that the resistances drop to 0, but maybe I have a measuring problem here.
What changes now is that I do no longer get error no. 0x107, but 0x108 which seems to stand for "invalid response". But it remains that I cannot get any of the three adapters to work.
Before I finally quit on this, could you check the resistances on your device between D2/D15 and 3V3 with and without the SD card reader attached to it? Thanks!

[sebi5361]

Hi Fred. Today is a day-off in France and I finally found the time to do the measures you asked.
It was done on my secondary esp32 as the main one is not accessible easily. Results are:

Resistance	ESP32 board only	ESP32 + card reader
3V3-D2	O.L.	10K
3V3-D15	10K	5K

Hope this helps...