Load Struct from binary file into SDRAM (C)

#1

This has been partially covered in other threads (including this one https://sebiik.github.io/community.axoloti.com.backup/t/two-questions-arrays-and-where-to-put-them-solved/4252/28 ), I think, but the specifics are confusing me.

I have some structs saved to a binary file. The struct is defined as:

struct lpcTables {
    uint16_t    subtype;
    uint8_t     num_k;
    uint8_t     energy_bits;
    uint8_t     pitch_bits;
    uint8_t     kbits[10];
    uint8_t     energytable[64];
    uint8_t     pitchtable[64];
    int16_t     ktable[10][64];
    uint8_t     interp_coeff[8];
};

lpcTables *lpf_coeffs_data;

and there are 8 of them rolled into the binary file.

I want to load this data into SDRAM, but I'm not sure how exactly to do this.
I currently have this in my Init Code:

// FatFS file_erroror message container
FRESULT file_error;
FIL bin;

LogTextMessage("Attempting to load binary data");

// Attempt to open binary file, and keep return status in file_error var
file_error = f_open(&bin, "attr_table", FA_READ | FA_OPEN_EXISTING);

// Exit if binary file not found
if(file_error != FR_OK) {
	LogTextMessage("Unable to open binary data file");
	return;
}

/////////////////////////////
// Allocate array in SDRAM //
/////////////////////////////

LogTextMessage("Allocating space in SDRAM for LPC Coefficents tables");
static lpcTables lpc_coeffs_data_sdram[attr_poly][11872] __attribute__ ((section (".sdram")));

// Set pointer to SDRAM array
LogTextMessage("Setting pointer to LPC Coefficents tables");
lpf_coeffs_data = &lpc_coeffs_data_sdram[parent -> polyIndex][0];

I get this error message in the console:

region `sdram' overflowed by 8612096 bytes
collect2: error: ld returned 1 exit status
make: *** [/Users/adrin009/Documents/axoloti/build/xpatch.bin] Error 1
shell task failed, exit value: 2
Compiling patch failed ( /Users/adrin009/Library/Mobile

I managed to load a binary before, but that was just a straight array of bytes. This time, I want to load structured data, and I can't work out how to do it.

Sorry if this is really basic stuff, but it's confusing the hell out of me..

a|x

#2

Also, I'd like to be able to allocate exactly the right amount of space in SDRAM for the file, but I can't work out how to do that, either, because it seems the number of bytes required needs to set in advance (ie as an object attribute).

a|x

#3

the binary file has no structure at all, it is just number after number. there is also no concept of int16 or uint8_t or similar. this means:

you have to know where your stuff is in the binary. you can load e.g. the first 8 bytes and save them into 8 uint8_t variables or four uint16_t.

to get the right amount of sdram allocated you first get the file size with f_size, after that you can allocate the right amount of memory with sdram_malloc(filesize).

you don't need to use the table stuff and "attribute...".

for practical reasons it would be easier if all your variables had the same size, then you could just load them all into one big array and read from there. otherwise you have to load the first x bytes into uint16_t then into uint8_t etc... i would not want to do that :slight_smile:

or make different bin files for the different variable types.

look at the .h file i sent you, it should be pretty clear howto open the file on sdcard and allocate the memory from there.

#4

I realise the binary data is completely unstructured.

You can do this, though.

https://overiq.com/c-programming/101/pointer-to-a-structure-in-c/

It’s the old school way :wink:

I’ll have another look at you file, and see if I can do it your way.

a|x

a|x

#5

but the struct is not contained in the binary file. (just the numbers within) so i also don't think you can just read out the data and have your struct magically reappear. maybe i don't get what you say though.

#6

i would look at the binfile with a hex editor and see how the data is stored in there. my guess is like i said:

if you save one uint8_t and one uint16_t (127 and 4096 for example) you will see:

7F 10 00

regardless of any structs. without knowing the sizes this could well be:

32528(7F10) and 0(00) or 127 16 and 0...

but i suspect this is not your point...

#7

It’s certainly possible to load data into a struct from a binary file.

I must admit, though, pointers and structs are a bit confusing to me.

a|x

#8

you can load a struct if it’s flat ( so that’s includes arrays) ,
one thing I’d need to check though is byte alignments. It may be your struct needs padding to deal with this ... but can’t remember off hand.

If your using pointers then you need to a ‘resolving’ step which can be non trivial, depending on requirements.

@lokki I’d assume @toneburst code is casting the data to the struct ie it assumes this binary data Ian in the format of the struct ... as such like a static cast, the compiler doesn’t care ... unless you start using it as a pointer :wink:
( and also if byte alignment is required, hence point above)

#9

This kind of works, at least for extracting data from the first 'record' in the binary file.

// FatFS file_erroror message container
FRESULT file_error;
FIL bin;
UINT bytes_read;

LogTextMessage("Attempting to load binary data");

// Attempt to open binary file, and keep return status in file_error var
file_error = f_open(&bin, "attr_table", FA_READ | FA_OPEN_EXISTING);

// Exit if binary file not found
if(file_error != FR_OK) {
	LogTextMessage("Unable to open binary data file");
	return;
}

const uint32_t binsize = f_size(&bin);

LogTextMessage("binsize: %d", binsize);

/////////////////////////////
// Allocate array in SDRAM //
/////////////////////////////

LogTextMessage("Allocating space in SDRAM for LPC Coefficents tables");

lpf_coeffs_data = (lpcTables *)sdram_malloc(binsize);

// Load lpcdata into sdram
int remaining_bytes = binsize;
int offset = 0;

while (remaining_bytes > 0) {
	if (remaining_bytes > sizeof(fbuff)) {
		file_error = f_read(&bin, fbuff, sizeof(fbuff), &bytes_read);
		if (bytes_read == 0)
			break;
		memcpy((char*)(&lpf_coeffs_data[0]) + offset,(char *)fbuff,bytes_read);
		remaining_bytes -= bytes_read;
		offset += bytes_read;
	} else {
		file_error = f_read(&bin, fbuff, remaining_bytes, &bytes_read);
		memcpy((char*)(&lpf_coeffs_data[0]) + offset,(char *)fbuff,bytes_read);
		remaining_bytes = 0;
	}
}

if (file_error != FR_OK) {
	LogTextMessage("Failed reading lpc data file, aborting...\n");
	return; 
}

file_error = f_close(&bin);
if (file_error != FR_OK) {
	LogTextMessage("Failed closing lpc data file, aborting...\n");
	return; 
}

/*struct lpcTables {
    uint16_t    subtype;
    uint8_t     num_k;
    uint8_t     energy_bits;
    uint8_t     pitch_bits;
    uint8_t     kbits[10];
    uint8_t     energytable[64];
    uint8_t     pitchtable[64];
    int16_t     ktable[10][64];
    uint8_t     interp_coeff[8];
};*/

LogTextMessage("subtype: %d", lpf_coeffs_data->subtype);
LogTextMessage("num_k: %d", lpf_coeffs_data->num_k);
LogTextMessage("energy_bits: %d", lpf_coeffs_data->energy_bits);
LogTextMessage("pitch_bits: %d", lpf_coeffs_data->pitch_bits);
LogTextMessage("kbits[9]: %d", lpf_coeffs_data->kbits[9]);
LogTextMessage("energytable[2]: %d", lpf_coeffs_data->energytable[2]);
LogTextMessage("pitchtable[5]: %d", lpf_coeffs_data->pitchtable[5]);
LogTextMessage("ktable[5][3]: %d", lpf_coeffs_data->ktable[5][3]);
LogTextMessage("interp_coeff[4]: %d", lpf_coeffs_data->interp_coeff[1]);

This is what I see in the console:

Attempting to load binary data
binsize: 11872
Allocating space in SDRAM for LPC Coefficents tables
subtype: 1
num_k: 10
energy_bits: 4
pitch_bits: 5
kbits[9]: 3
energytable[2]: 1
pitchtable[5]: 49
ktable[5][3]: -163
interp_coeff[4]: 42

Everything checks out fine (even with the 2-dimensional array at 'coeffs_data-> ktable), except that it goes a bit screwy when it hits the array at 'coeffs_data->interp_coeff'

The array should have the values {0,3,3,3,2,2,1,1}, so value at index 4 should be 2, NOT 42. Is this where byte-alignment rears it's ugly head, @thetechnobear?

a|x

#10

If I increment the pointer

lpc_coeffs_data++;

I get garbage values.

subtype: 57602
num_k: 197
energy_bits: 2
pitch_bits: 95
kbits[9]: 220
energytable[2]: 43
pitchtable[5]: 0
ktable[5][3]: 0
interp_coeff[4]: 0

a|x

#11

would it not be lpc_coeffs_data[1] to access the second entry? EDIT: that is of course bullshit since it is a struct. sorry

#12

you are reading index 1 there, at least according to the source you posted...

#13

Yeah, I think I know what's going wrong. Not all the arrays are the right length, so I'm adding padding to the binary data, now.

a|x

#14

And now it works!

I've padded out all the arrays in my C binary-export program so that they all match the struct definition

struct lpcTables {
    uint16_t    subtype;
    uint8_t     num_k;
    uint8_t     energy_bits;
    uint8_t     pitch_bits;
    uint8_t     kbits[10];
    uint8_t     energytable[16];
    uint8_t     pitchtable[64];
    int16_t     ktable[10][32];
    uint8_t     interp_coeff[8];
};

Now, I can increment the pointer

lpc_coeffs_data++;

lpc_coeffs_data+=2;

etc., to get the correct values for the second/third etc. set of tables.

a|x

#15

yes!! very nice... :wink:

#16

:slight_smile:

Multiple instances of the object will load the data into SDRAM multiple times, but, since the file is less that 6kb, I'm not sure that's too much of an issue.

a|x

#17

i guess that is ok. :slight_smile: