Yes, that's right. I now remember that I have seen that option in Electrum before but totally forgot about it. I guess it's fair to assume that the exported file can only be imported back to Electrum and not any other wallets, right? which is another reason why Craig Raw created BIP0329 in an attempt to standardize it.
Exporting labels from Electrum to Sparrow is only possible in .CSV format. Furthermore, when testing, if you import labels from Electrum to Sparrow Wallet through the standard method (as suggested by Abdussamad), even if you have labels on addresses and transactions, Sparrow won't show any errors, however, no labels will actually be imported.
However, if you export labels directly from the transaction list in .CSV format (history tab > tools in the top right corner > Export) and then import them into Sparrow, you'll see the transaction labels in your history, but not the address labels:
