I think the secret is that when serializing to JSON, Qt writes out a UTF-8 binary encoded version of the text into a QByteArray. You can recover the string using QString::fromUtf8. Also, even though toJson writes out a QByteArray it is not a true binary representation; as I just mentioned, it is the bytes of the UTF-8 encoded string representation of the JSON. For binary serialization you must use either Qt's own QJsonDocument::toBinaryData or find a BSON library.
Text serialization
Use QJsonDocument::toJson to serialize as text via a QString, and QJsonDocument::fromJson to deserialize from text.
Tip: Consider passing the QJsonDocument::Compact format option to get compact JSON output during serialization.
To QString
QString json = QString::fromUtf8(doc.toJson(QJsonDocument::Compact));
From QString
QJsonDocument doc = QJsonDocument::fromJson(json);
Binary serialization
Use QJsonDocument::toBinaryData to serialize as binary via a QByteArray, and QJsonDocument::fromBinaryData to deserialize from binary.
Caveat: The binary format used by the above functions is Qt's own binary format (qbjs) so may not be suitable for interoperation with other binary formats such as BSON. For that, you might want to look into choosing a BSON implementation.
To QByteArray
QByteArray bytes = doc.toBinaryData();
From QByteArray
QJsonDocument doc = QJsonDocument::fromBinaryData(bytes);