Discussion:
[Freetel-codec2] Please prominently suggest a file suffix for codec2 encoded files
Frieder Ferlemann
2017-05-22 18:16:17 UTC
Permalink
Hi,

and thanks for Codec 2!

Maybe I'm just overlooking something but I couldn't find a recommended file suffix for codec 2.

The examples in https://sourceforge.net/p/freetel/code/HEAD/tree/codec2-dev/README
use the suffix ".bit" appended after "_c2" after the stripped file name.

So encoded files in the README look like "hts1a_c2.bit".
This is awkward because
a) ".bit" is too generic (as such used for many formats) and
b) "_c2" will be a challenge for file browsers.

Please suggest a 3 letter and a 4 letter suffix for codec2 encoded files!

Thanks again,
Frieder
Bruce Perens
2017-05-22 18:28:33 UTC
Permalink
See https://www.xiph.org/ogg/doc/rfc3533.txt . Make it work with the
Ogg Encapsulation Format.

Thanks

Bruce

On Mon, May 22, 2017 at 11:16 AM, Frieder Ferlemann
Post by Frieder Ferlemann
Hi,
and thanks for Codec 2!
Maybe I'm just overlooking something but I couldn't find a recommended file suffix for codec 2.
The examples in https://sourceforge.net/p/freetel/code/HEAD/tree/codec2-dev/README
use the suffix ".bit" appended after "_c2" after the stripped file name.
So encoded files in the README look like "hts1a_c2.bit".
This is awkward because
a) ".bit" is too generic (as such used for many formats) and
b) "_c2" will be a challenge for file browsers.
Please suggest a 3 letter and a 4 letter suffix for codec2 encoded files!
Thanks again,
Frieder
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Freetel-codec2 mailing list
https://lists.sourceforge.net/lists/listinfo/freetel-codec2
Kevin Otte
2017-05-22 20:35:44 UTC
Permalink
Or Matroska (.mka):
https://matroska.org/technical/specs/index.html
Post by Bruce Perens
See https://www.xiph.org/ogg/doc/rfc3533.txt . Make it work with the
Ogg Encapsulation Format.
Albert Cahalan
2017-05-22 21:14:28 UTC
Permalink
...
Post by Bruce Perens
See https://www.xiph.org/ogg/doc/rfc3533.txt . Make it work with the
Ogg Encapsulation Format.
Either way, it still needs a file suffix.

The idea that "ogg" would work for arbitrary audio/video data has been
a disaster. Apps claim the file extension, but can't decode all possible
ways of encoding the data. You may have 5 apps that handle a format,
covering all possible encodings in total, but no app handles everything.

This is why TIFF never caught on, but JFIF and PNG and GIF were fine.
You can toss JPEG2000 or JBIG2 or Wang vector markup in a TIFF,
and even 2 different JPEG storage styles. Good luck with reliable reading.
For the most part, users gave up. They reject it. Web browsers don't try.

In actual usage, the "ogg" suffix is Vorbis audio. That is all.

You can't even safely use the container format unless you can mark the
header at a fixed offset with a clear indicator that will ensure that file
magic never just claims "ogg", because that would imply Vorbis. You'll
get files saved as "ogg" and served up with a MIME type that doesn't
make it clear that codec2 is in use. Files won't play right.

This should work properly:

Matroska: filename.mk2 (Mac type MAT2)
Ogg: filename.og2 (Mac type OGG2)
Headerless: filename.codec2 (Mac type DEC2)
MIME type: prefix the file extension with "audio/", w/o "x-"

That said, Matroska and Ogg are only suitable if file magic is
trivial. There must
not be any chance of misdirecting the data to a Vorbis-only player or similar.
Tomas Härdin
2017-05-24 09:35:47 UTC
Permalink
Hi

Going to supply some feedback as an FFmpeg dev, because this is not the
first time the subject of containers have come up on this list
Post by Albert Cahalan
...
Post by Bruce Perens
See https://www.xiph.org/ogg/doc/rfc3533.txt . Make it work with the
Ogg Encapsulation Format.
Either way, it still needs a file suffix.
The idea that "ogg" would work for arbitrary audio/video data has been
a disaster. Apps claim the file extension, but can't decode all possible
ways of encoding the data. You may have 5 apps that handle a format,
covering all possible encodings in total, but no app handles everything.
Works fine for mkv and mka. The only reason why you'd need a
codec-specific file extension is if you have a bunch of different
single-codec programs for dealing with them. This hasn't been the case
since the early 2000's. Programs like VLC and mpv exist now.
Differentiating on essence kind (audio vs video+audio) makes some sense
however, since a good audio player may suffer from having to also be a
good video player (and vice versa). Exception might be if you want to
associate with say the FreeDV program
Post by Albert Cahalan
You can't even safely use the container format unless you can mark the
header at a fixed offset with a clear indicator that will ensure that file
magic never just claims "ogg", because that would imply Vorbis. You'll
get files saved as "ogg" and served up with a MIME type that doesn't
make it clear that codec2 is in use. Files won't play right.
Yes, the essence codec used is typically marked in some kind of header
or footer. codec2 mode would likely be coded in extradata (WAVEFORMATEX
in RIFF containers, 'wave' in MOV etc.) with one or two bytes taking the
value of CODEC2_MODE_* in codec2.h

MIME is also poorly implemented in many Web servers (apache2 for
example), this is hardly new. Format probing (file, ffprobe, mediainfo)
should always be used since the truth is in the data. Similarly Web
browsers should rely on MIME or possibly format probing if the server is
crap
Post by Albert Cahalan
Headerless: filename.codec2 (Mac type DEC2)
How do you differentiate modes without a header? This should not be
encouraged
Post by Albert Cahalan
MIME type: prefix the file extension with "audio/", w/o "x-"
Can MIME specify modes I wonder.. RFC 4281 seems to suggest syntax like:

"audio/ogg; codecs=codec2.700C" or perhaps
"audio/ogg; codecs=codec2.8" for an Ogg containing codec2 in mode 700C.
"audio/wav; codecs=49362.8" ditto for .wav (assuming codec2 gets
assigned ID=0xC0D2, see RFC 2361)
"audio/codec2; mode=8" raw codec2 mode 700C data, assuming we can use a
key like "mode"
Post by Albert Cahalan
That said, Matroska and Ogg are only suitable if file magic is
trivial. There must
not be any chance of misdirecting the data to a Vorbis-only player or similar.
Vorbis-only player? What is this, 1997? :)

That's all I have time to write on this right now. This does not address
issues with picking a container, which I've touched on before on this
list. TL;DR it must handle seeking in a decent fashion, for which Ogg
sucks. MOV and WAV are OK. MKV seems fine too. I'd go with WAV since
it's tiny, stupid and simple

/Tomas SA2TMS
Kevin Otte
2017-07-23 23:56:33 UTC
Permalink
I was going back over this thread and realizing that, despite resembling
a Not Invented Here solution, a full on container format might be
overkill. Seeking, as Tomas suggested, would be nice, but for the most
part we're just trying to play short streams.

So, to the original poster's request, I humbly suggest ".c2" as the
extension. I don't think there are any issues with a two character
extension as opposed to a three or four.

That said, the notion of managing the different bitrates does present a
challenge. I therefore suggest any such encoded file have some "magic"
bytes at the beginning, even if this is something as simple as 0x43 0x32
("C2"). I then propose the next byte be the mode ID as defined in
codec2.h. This would allow c2enc/c2dec to be retrofitted or separate
tools to be constructed to Do The Right Thing™ based on these.

If we can agree on these bytes, I'm happy to take a stab at coding this
up. It doesn't look like it should be too difficult (famous last words!)

73 de Kevin N8VNR
Post by Kevin Otte
https://matroska.org/technical/specs/index.html
Post by Bruce Perens
See https://www.xiph.org/ogg/doc/rfc3533.txt . Make it work with the
Ogg Encapsulation Format.
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Freetel-codec2 mailing list
https://lists.sourceforge.net/lists/listinfo/freetel-codec2
David Rowe
2017-07-24 07:45:58 UTC
Permalink
Hi Kevin,

That sounds fine to me. A lot of my work (e.g. feeding Codec 2
compressed audio into a modulator input) requires a raw, headerless bit
stream.

So it would be great if any changes to c2dec/c2enc support (say via a
command line option) a headerless version.

Thanks,

David
Post by Kevin Otte
I was going back over this thread and realizing that, despite resembling
a Not Invented Here solution, a full on container format might be
overkill. Seeking, as Tomas suggested, would be nice, but for the most
part we're just trying to play short streams.
So, to the original poster's request, I humbly suggest ".c2" as the
extension. I don't think there are any issues with a two character
extension as opposed to a three or four.
That said, the notion of managing the different bitrates does present a
challenge. I therefore suggest any such encoded file have some "magic"
bytes at the beginning, even if this is something as simple as 0x43 0x32
("C2"). I then propose the next byte be the mode ID as defined in
codec2.h. This would allow c2enc/c2dec to be retrofitted or separate
tools to be constructed to Do The Right Thing™ based on these.
If we can agree on these bytes, I'm happy to take a stab at coding this
up. It doesn't look like it should be too difficult (famous last words!)
73 de Kevin N8VNR
Post by Kevin Otte
https://matroska.org/technical/specs/index.html
Post by Bruce Perens
See https://www.xiph.org/ogg/doc/rfc3533.txt . Make it work with the
Ogg Encapsulation Format.
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Freetel-codec2 mailing list
https://lists.sourceforge.net/lists/listinfo/freetel-codec2
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Freetel-codec2 mailing list
https://lists.sourceforge.net/lists/listinfo/freetel-codec2
Kevin Otte
2017-07-24 21:18:48 UTC
Permalink
I like your magic_id better :)

The reason I looked at the mode byte is because that's what c2enc/c2dec
(by way of a #define) use to set up the codec2 instance.

codec2.h:
#define CODEC2_MODE_1300 4

c2enc.c:
mode = CODEC2_MODE_1300;
...
codec2 = codec2_create(mode);

Samples per frame, bits per frame, etc are all defined in the codec2
internal structure:

nsam = codec2_samples_per_frame(codec2);
nbit = codec2_bits_per_frame(codec2);

So it really doesn't make sense to represent these values in the header
when the code can figure it out from a single byte.

samplerate is something of a sticky wicket. For now the only outlier is
WB at 16Khz. Ideally I'd like to see the whole pipeline work at 48KHz
and let the FFT do the voodoo, but that's a whole other thread. I think
I'll leave that alone for now.

It might make sense to keep a flags byte around for --natural and
--softdec. I haven't used these so I don't know much about them.

At present we don't do any FEC in the raw streams, only on the air (eg:
with freedv_tx/rx not c2enc/dec).

The version stuff might be useful for detecting future backward
incompatible changes, but my gut feeling is YAGNI. I'd be interested to
hear David's thoughts on that.

So, that leaves us with a simpler header:
struct Codec2Header{
magic_identification :u24b, // offset 0 bytes
// const 0xC0DEC2 ([0xC0,0xDE,0xC2])
major_version :u8, // offset 3 bytes //
minor_version :u8, // offset 4 bytes //
mode_id :u8, // offset 5 bytes //
flags :u8, // offset 6 bytes // default 0
};

Two bytes less if we ignore the version stuff.

My plan is to add some code to c2enc/dec to see if the codec2 filename
argument ends in ".c2".

In the case of the encoder, if it sees that file extension, it will
write the file header and then proceed to write as normal. Otherwise
behave as it does today.

In the decoder, if that extension, try to parse the header and set the
mode. If the magic is wrong, seek the file back to zero and honor the
command line mode argument / flags. I don't want to change the argument
order as to not break old scripts. This means when using a valid .c2
file the mode argument could be invalid, eg: ./c2dec foobar test.c2 -

73 de Kevin N8VNR
Hi folks!
Post by Kevin Otte
That said, the notion of managing the different bitrates does present a
challenge. I therefore suggest any such encoded file have some "magic"
bytes at the beginning, even if this is something as simple as 0x43 0x32
("C2"). I then propose the next byte be the mode ID as defined in
codec2.h. This would allow c2enc/c2dec to be retrofitted or separate
tools to be constructed to Do The Right Thing™ based on these.
May I suggest a bitstream header format that encodes all the profile
parameters, so that new profiles can be added without changing the
header format or adding new "magic" numbers for profiles.
73 de txp0wer
Post by Kevin Otte
If we can agree on these bytes, I'm happy to take a stab at coding this
up. It doesn't look like it should be too difficult (famous last words!)
73 de Kevin N8VNR
Post by Kevin Otte
https://matroska.org/technical/specs/index.html
Post by Bruce Perens
See https://www.xiph.org/ogg/doc/rfc3533.txt . Make it work with the
Ogg Encapsulation Format.
<code language="pseudocode" style="Rust">
// u8 == unsigned 8-bit integer
// u16b == unsigned 16-bit integer, big endian
struct Codec2Header{
magic_identification :u24b, // offset 0 bytes
// const 0xC0DEC2 ([0xC0,0xDE,0xC2])
major_version :u8, // offset 3 bytes //
minor_version :u8, // offset 4 bytes //
forward_ec_level :u8, // offset 6 bytes //
flags :u8, // offset 5 bytes // default 0
subframes_per_frame :u8, // offset 7 bytes // 1 to 4
samples_per_subframe :u16b, // offset 8 bytes
samplerate_hz :u16b // offset 10 // default 8000
};
struct Codec2SubframeParameters{ // offset 12
// repeated [subframes_per_frame] times
lsp_bits_per_frame :u8, // offset 12+4*n
pitch_bits_per_frame :u8, // offset 12+4*n+(0..1)
energy_bits_per_frame :u8, // offset 4*n+(0..1)
voicing_bits_per_frame :u8 // offset 4*n+(0..1)
}
match forward_ec_level{
0=>none,
1=>"1200",
2=>"1850",
3=>"2000",
_=>undefined
}
0x01=NATURAL_BYTE_ORDER // --natural
0x02=ALTERNATE_MODE_1 // "B"
0x04=ALTERNATE_MODE_2 // "C", "D"
0x08=ALTERNATE_MODE_4 // "E" .. "H"
0x10=unused
0x20=unused
0x40=unused
0x80=ALTERNATE_CODEBOOK_1
</code>
example (hex)
# "3200" default profile
# begin Codec2Header
00: C0 DE C2
03: 00 04 # version 0.4
05: 00 # no forward error correction
06: 00 # flags: gray code, no alternate profile, no alternate codebook
07: 02 # 2 subframes per frame
08: 00 a0 # 160 samples per subframe => 320 samples per frame
0a: 1f 40 # 8000 Hz
# end Codec2Header, begin Codec2SubframeParameters
# # begin subframe 0 # if I understand codec2_encode_3200() right
0c: 20 # 32 bits LSP
0d: 00 # 0 bits pitch
0e: 00 # 0 bits energy
0f: 00 # 0 bits voicing
# # end subframe 0, begin subframe 1
10: 12 # 18 bits LSP
11: 07 # 0 bits pitch
12: 05 # 0 bits energy
13: 02 # 0 bits voicing
# # end subframe 1
# end Codec2SubframeParameters, begin data bitstream
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Freetel-codec2 mailing list
https://lists.sourceforge.net/lists/listinfo/freetel-codec2
Loading...