Read raw data in Matlab

Discussion in 'Respiration Sensor' started by Mo Frz, Sep 14, 2017.

  1. Mo Frz

    Mo Frz New Member

    Hi,

    I am trying to read the raw data recorded by X2M200. The data is saved in amplitude and phase format. Do you have any Matlab script to read the data? Unfortunately, I can not find any good documentation on how the data is formatted!

    Here is the code I used but did not work:

    fileID = fopen('filename', 'rb');
    while ~feof(fileID)
    contentID = fread(fileID, 1, 'uint32');

    framecounter = fread(fileID, 1, 'uint32');
    numofbins = fread(fileID, 1, 'uint32');
    timestamp = fread(fileID, 1, 'int64');
    binlength = fread(fileID, 1, 'float');
    samplingfreq = fread(fileID, 1, 'single');
    carrierfreq = fread(fileID, 1, 'single');
    rangeoffset = fread(fileID, 1, 'single');
    power = fread(fileID, numofbins, 'single');
    phase = fread(fileID, numofbins, 'single');
    end
    fclose(fileID);

    Can somebody help me on what is wrong and how the data is formatted in .dat files?

    Thanks!
     
  2. R.Adam

    R.Adam New Member

  3. Mo Frz

    Mo Frz New Member

    Hi R. Adam,

    Thanks for your reply. The matlabscript really helped.

    However, I still get error when reading my own recorded data. I discovered that some of the values that I read do not make sense. For example, sampling frequency Fs in one my my recordings was read as 0.1084, and the carrier Frequency Fc was read as 0.1891. The rest of the recording had similar issues. The number of bins is always 1.0253e+09 in my recordings which is also weird.

    Any idea what is wrong here?

    Thanks a lot!
     
  4. Mo Frz

    Mo Frz New Member

    I have attached an example of the recorded files here.
     
  5. R.Adam

    R.Adam New Member

    Hi!

    I am not sure but these values really don't make sense.
    I watched your .dat file, and it's totally wrong. There are suggested values for recording:
    For example a suggest you to use sleeping mode and:
    - set your range: eg. 0.5-2m
    - sensitivity: 9
    -led control: full or what ever you want
    -UI settings: for me everything on the max
    -Set directory to save data
    next step:
    Maybe thats your fault:
    -include baseband A/P output
    (you can choose either A/P or I/Q),but only one! I recomend Amplitude/Phase because its easy to understand.
    -include sleep massage

    Than start the record.

    If you open the .dat file it should look something like this (the starting header):

    0D 00 00 00 51 05 00 00 34 00 00 00 88 65 DC 96 5A 01 00 00 2B 54 1F 3D 5F A7 0F 51 EE 44 DC 4F 9A 99 99 3E ....Here comes the datas(amp and phase) and after that the next frame start with 0D 00 00 00 ......

    Where:

    ContentId: 0D 00 00 00 = 13
    FrameCounter: 51 05 00 00 = 1361
    NumOfBins: 34 00 00 00 = 52 (if the mode is resp: it's 26)
    Timestamp: 88 65 DC 96 5A 01 00 00 = 1.488589710728000e+12
    BinLength: 2B 54 1F 3D = 0,0388986282050610
    SamplingFrequency: 5F A7 0F 51 = 3.856177152000000e+10
    CarrierFrequency: EE 44 DC 4F = 7.391009792000000e+09
    RangeOffset: 9A 99 99 3E = 0.3

    (The XeThru serial protocol assumes litle-endian byte order.)

    Hope you can make a correct record after this.
    Bye: Adam
     
  6. Mo Frz

    Mo Frz New Member

    Hi R. Adam,

    Thanks a lot for your reply! I really appreciate that!

    I have used different settings and almost have followed what you have described and I had the same issue.

    There are however a few things that I am not sure about:

    1- I can not set the range to 0.5-2 meters. The default is 0.5-1.2 and if I increase the upper limit to 2 meter the lower limit can not be anything lower than 1.3 meters.

    2- The other thing is that I can get the respiration pattern correctly in the .csv files. Though the raw data looks weird!

    3- On my sensor board it says "X2M03 290003-008", do you think this could be an older version of the respiration sensor that may not be well supported by XethruExplorer 2.5.0?

    Thanks a lot!
     
  7. R.Adam

    R.Adam New Member

    Hi Mo!

    Just from curiosity I upgraded my Xethru explorer from 2.1.1 to 2.5, and I just got a very similar file you also get.
    So I was looking for same periodicity and find the file is periodic in 440byte, this is the size of the frame which is made by 2.5 explorer. When I used the explorer 2.1.1 (almost a year ago) it was 452byte and the header looked like as I already write down. Looking to this frame (the 440byte long), the header is something like this:

    3F 04 00 00 34 00 00 00 2B 54 1F 3D 5F A7 0F 51 EE 44 DC 4F 9A 99 99 3E

    3F 04 00 00 => This have to be the framecounter
    34 00 00 00 => NumberOfBins (52 because sleeping profile)
    2B 54 1F 3D => BinLength (0,0388986282050610 [m])
    5F A7 0F 51 => SamplingFrequency: 3.856177152000000e+10 [Hz]
    EE 44 DC 4F => CarrierFrequency: 7.391009792000000e+09 [Hz]
    9A 99 99 3E => RangeOffset : 0.3[m]

    And than follow the 4byte*52 Amplitude data + 4byte*52 Phase data
    And you can check that the next frame is begining with 40 04 00 00, so the counter is increased by 1.

    So as you can see the ContentId(32bit) and the Timestamp(64bit) is not included in the new Xethru explorer.
    You should modify the matlabscrit likewise.


    As for 1. I really hope that's only because you choose respiration mode instead of sleeping. You say in the 2. point that you can't get respiration, so I think that way your range is only 0.5-1.2.
    I really recomend you to use sleeping mode instead of respiration, the BinLength is the same and you get twice the range.

    Converting hexa values you can use this site: https://www.h-schmidt.net/FloatConverter/IEEE754.html
    Just don't forget the little-endian byte order.

    bye : Adam

    PS:
    Don't care about the .csv files, all the important data in the .dat file.

    .
    .
    .
    I saw others have similar problems, and because it didn't seemed too much time to make the modifications, you can use this reader until an offical is out, but I recomend you to get an older version of the explorer instead, because the Timestamp can be useful!
     

    Attached Files:

    Last edited: Sep 16, 2017
    Mo Frz likes this.
  8. Mo Frz

    Mo Frz New Member

    Thanks for the info. Now it makes more sense what I am recording! Do you have by any chance access to the older version of the XethruExplorer or do you know where I can download it?

    Thanks again
     
  9. R.Adam

    R.Adam New Member

    Hi,

    I don't think I still have the installer to the old version, but the xethru staff should have a copy or at least I think that's the biggest possibility that they store it.
    I checked and unfortunately on the site you can see only the version history now.
    I hope you can get somehow a version 2.21 or later somehow, but if not you only loose the timestamp data, and time beetwen frames can be calculated by Fs (Frame/sec) which is around 20.

    bye : Adam

    PS: Meanwhile I found out that you can download all the previous versions here:

    https://www.xethru.com/community/resources/xethru-explorer-for-windows.5/history
     
    Last edited: Sep 25, 2017 at 6:18 PM
  10. bulut

    bulut New Member

    Hello Adam,

    I have a problem with the .dat file. You write in your mail that you open the data file and then look something like this.

    "0D 00 00 00 51 05 00 00 34 00 00 00 88 65 DC 96 5A 01 00 00 2B 54 1F 3D 5F A7 0F 51 EE 44 DC 4F 9A 99 99 3E ....Here comes the datas(amp and phase) and after that the next frame start with 0D 00 00 00 ...... "

    I have opened it with the program HxD. Is this the right way to open it. Or is this not able with HxD ?
    How you open it ?

    Best Reagrds
    Bulut
     
  11. R.Adam

    R.Adam New Member

    Hi Bulut!

    I think you can open it with any hex-editor/reader. I opened it with HxD too, so I can see the structure of the frame, but that's all you can do with a hex-editor.

    If would like to use the datas stored in the .dat file you use Matlab (or python or C++).

    If you made the .dat file with an older version of the XeThruExplorer, than your .dat should start with a ContentId like 0D 00 00 00. If that's the case you can use this link to get the right file reader for matlab:

    https://www.xethru.com/community/resources/x2m200-baseband-amplitude-example.63/

    But if you use a newer version of the explorer, than you get a different frame structure which is lack of the ContentId and the Timestamps. If you would like to read it in matlab you may use the modified matlab reader which you can find under reply 7 with explanation on the change too.

    To help you see the change here is a picture of the old frame and the new both in SLEEPING mode:
    https://i.imgur.com/E3VFmE8.jpg

    bye: Adam
     
    Last edited: Sep 21, 2017 at 10:35 PM
  12. Hi Adam,
    Please, could give details on the meaning of the timestamp below (previous version of xethru explorer) ? It is 64bit epoch (milliseconds since 1970.01.01) according to xethru file format. For me this is a huge number !
    As you have explained in your previous post, timestamp is very important for data processing but it is not generated in the lastest version of xethru explorer. Please could you give any example on the use of this timestamp ? Do you have any idea why the latest version of xethru explorer doesn't include this 64bit data in the header of each frame ?

    Timestamp: 88 65 DC 96 5A 01 00 00 = 1.488589710728000e+12

    Best regards,
    Chouaibou.
     
  13. R.Adam

    R.Adam New Member

    Hi Chouaibou!

    Yes it's a 64bit epoch (milliseconds since 1970.01.01). If you do the math, you can see it's correct:

    1.488589710728000e+12 ms / 1000 / 60 / 60 / 24 / 365.25 = 47,17056.... year

    So 47years 62days 7hours 7minutes ..... passed when this record has been made since 1970.01.01.
    => Date of the record: 2017.03.04. if I calculated right.

    So it's obviously a great data to sort your records, according to which happend when. Of course dat. file names are created with a date, but if you for some reason rename same record, that's the only info which can tell you the recording date.

    Another use was when I did fft on a recorded data. I needed the pulse repetition frequency which is 20 according the data sheet, but if you don't know it you can count it like this:

    Timeinterval=TimeVec(numFrames) - TimeVec(1)
    Timeinterval= Timeinterval/1000;
    Fs=numFrames/Timeinterval;

    You get for example Fs = 20.000889125989154


    Or if you monitoring a place and searching for a particular change in amplitude or phase you can search the frame and with the help of the time stamp you can tell the very precise time of the change.
    Unlike the previouse situation you can't use Fs=20, because you dont have a reference time like 1970.01.01.

    I don't have any idea why the latest version of xethru explorer doesn't include this 64bit data, because I am just a member like you :).

    bye: Adam
     
    Last edited: Sep 25, 2017 at 10:07 AM
  14. Hi Adam,
    Great ! Many thanks for your cristal clear reply.

    Best regards,
    Chouaibou.
     
  15. Olav Liseth

    Olav Liseth Administrator Staff Member

    Hi,

    We've updated the file format documentation: https://www.xethru.com/community/resources/xethru-file-formats.55/

    ModuleConnector is the simplest way of using your recordings and you will find examples on how to read your files with the different support programming languages, e.g.: matlab\examples\IQbaseband_playback_data.m

    Olav
     
  16. Hi Olav,
    Thank you for the link and your reply and the new file format. I always use xethru module connector as it is a great tool to get access to baseband output of all modules. I really appreciate all examples in modules connectors.

    Best regards,
    Chouaibou.