環境
当方の環境は以下である。
zig version: 0.16.0
ffmpeg version: 6.1.1-3ubuntu5
Windows 11 + x86_64 + wsl2 + ubuntu24.04 + zsh
FFmpegにおけるC APIの習得方法
FFmpeg公式HPのExamplesを参考にC APIの叩き方と呼び出し順を勉強する。RTSPで受信しH.264をデコードしフレームを取得するまではdemux_decode.cを参考とする。フレームYCbCr(YUV)からRGBに変換する前はscale_video.cを参考とする。尚、各関数の説明はリンク先の公式ドキュメントより引用している。
demux_decode.cから分かる叩き方と呼び出し順
- avformat_open_input
ストリームを開いてヘッダを読む。
※avformat_close_inputで閉じること。
/* Open an input stream and read the header.
The codecs are not opened. The stream must be closed with avformat_close_input(). */
int avformat_open_input(AVFormatContext **ps, const char *url, const AVInputFormat *fmt, AVDictionary **options);- avformat_find_stream_info
パケットを読んでストリーム情報を取得する。
/* Read packets of a media file to get stream information.
This is useful for file formats with no headers such as MPEG. This function also computes the real framerate in case of MPEG-2 repeat frame mode. The logical file position is not changed by this function; examined packets may be buffered for later processing. */
int avformat_find_stream_info(AVFormatContext *ic, AVDictionary **options); - av_find_best_stream
AVMediaTypeに一致するストリームを探し出す。
int av_find_best_stream(AVFormatContext *ic, enum AVMediaType type, int wanted_stream_nb, int related_stream, const AVCodec **decoder_ret, int flags); - avcodec_find_decoder
コーデックIDに一致するデコーダを探す。
/* Find a registered decoder with a matching codec ID. */
const AVCodec *avcodec_find_decoder(enum AVCodecID id);- avcodec_alloc_context3
AVCodecContextを確保し初期化する。
※avcodec_free_context()で解放すること。
/* Allocate an AVCodecContext and set its fields to default values.
The resulting struct should be freed with avcodec_free_context(). */
AVCodecContext *avcodec_alloc_context3(const AVCodec *codec);- parameters_to_context
AVCodecContextにAVCodecParametersを反映する。
/* Fill the codec context based on the values from the supplied codec parameters.
Any allocated fields in codec that have a corresponding field in par are freed and replaced with duplicates of the corresponding field in par. Fields in codec that do not have a counterpart in par are not touched. */
int avcodec_parameters_to_context(AVCodecContext *codec, const struct AVCodecParameters *par);- avcodec_open2
AVCodecを使用するためにAVCodecContextを初期化する。AVDictionaryにオプションを設定できる。
/* Initialize the AVCodecContext to use the given AVCodec.
Prior to using this function the context has to be allocated with avcodec_alloc_context3().
The functions avcodec_find_decoder_by_name(), avcodec_find_encoder_by_name(), avcodec_find_decoder() and avcodec_find_encoder() provide an easy way for retrieving a codec.
Depending on the codec, you might need to set options in the codec context also for decoding (e.g. width, height, or the pixel or audio sample format in the case the information is not available in the bitstream, as when decoding raw audio or video).
Options in the codec context can be set either by setting them in the options AVDictionary, or by setting the values in the context itself, directly or by using the av_opt_set() API before calling this function.
Example:
av_dict_set(&opts, "b", "2.5M", 0);
codec = avcodec_find_decoder(AV_CODEC_ID_H264);
if (!codec)
exit(1);
context = avcodec_alloc_context3(codec);
if (avcodec_open2(context, codec, opts) < 0)
exit(1);
In the case AVCodecParameters are available (e.g. when demuxing a stream using libavformat, and accessing the AVStream contained in the demuxer), the codec parameters can be copied to the codec context using avcodec_parameters_to_context(), as in the following example:
AVStream *stream = ...;
context = avcodec_alloc_context3(codec);
if (avcodec_parameters_to_context(context, stream->codecpar) < 0)
exit(1);
if (avcodec_open2(context, codec, NULL) < 0)
exit(1);
*/
int avcodec_open2(AVCodecContext *avctx, const AVCodec *codec, AVDictionary **options);
int av_dict_set(AVDictionary **pm, const char *key, const char *value, int flags);- av_image_alloc
画像バッファを確保し、各面のポインタと対応する面の行サイズを返す。面数はAVPixelFormatによって異なり、RGB24は1面、YUV420Pは3面である。
※av_freep(&pointers[0])で解放すること。
/* Allocate an image with size w and h and pixel format pix_fmt, and fill pointers and linesizes accordingly.
The allocated image buffer has to be freed by using av_freep(&pointers[0]). */
int av_image_alloc(uint8_t *pointers[4], int linesizes[4], int w, int h, enum AVPixelFormat pix_fmt, int align); - av_frame_alloc
フレームを確保する。
※av_frame_free()で解放すること。
※av_frame_unref()で参照を解放すること。
/* Allocate an AVFrame and set its fields to default values.
The resulting struct must be freed using av_frame_free(). */
AVFrame *av_frame_alloc(void);- av_packet_alloc
パケットを確保する。
※av_packet_free()で解放すること。
※av_packet_unref()で参照を解放すること。
/* Allocate an AVPacket and set its fields to default values.
The resulting struct must be freed using av_packet_free(). */
AVPacket *av_packet_alloc(void);- av_read_frame
ストリームの次の枠を返す。AVPacketにパケットが返る。
ループで回して使う。
/* Return the next frame of a stream.
This function returns what is stored in the file, and does not validate that what is there are valid frames for the decoder. It will split what is stored in the file into frames and return one for each call. It will not omit invalid data between valid frames so as to give the decoder the maximum information possible for decoding.
On success, the returned packet is reference-counted (pkt->buf is set) and valid indefinitely. The packet must be freed with av_packet_unref() when it is no longer needed. For video, the packet contains exactly one frame. For audio, it contains an integer number of frames if each frame has a known fixed size (e.g. PCM or ADPCM data). If the audio frames have a variable size (e.g. MPEG audio), then it contains one frame.
pkt->pts, pkt->dts and pkt->duration are always set to correct values in AVStream.time_base units (and guessed if the format cannot provide them). pkt->pts can be AV_NOPTS_VALUE if the video format has B-frames, so it is better to rely on pkt->dts if you do not decompress the payload.
Returns
0 if OK, < 0 on error or end of file. On error, pkt will be blank (as if it came from av_packet_alloc()).*/
int av_read_frame(AVFormatContext *s, AVPacket *pkt); - avcodec_send_packet
パケットをデコーダに入力する。
/* Supply raw packet data as input to a decoder.
Internally, this call will copy relevant AVCodecContext fields, which can influence decoding per-packet, and apply them when the packet is actually decoded. (For example AVCodecContext.skip_frame, which might direct the decoder to drop the frame contained by the packet sent with this function.) */
int avcodec_send_packet(AVCodecContext *avctx, const AVPacket *avpkt);- avcodec_receive_frame
デコーダから出力されたフレームを返す。
/* Alias for avcodec_receive_frame_flags(avctx, frame, 0). */
int avcodec_receive_frame(AVCodecContext *avctx, AVFrame *frame);scale_video.cから分かる叩き方と呼び出し順
- sws_getContext
SwsContextを確保して返す。
※sws_freeContext()で解放すること。
/* Allocate and return an SwsContext.
You need it to perform scaling/conversion operations using sws_scale(). */
SwsContext *sws_getContext(int srcW, int srcH, enum AVPixelFormat srcFormat, int dstW, int dstH, enum AVPixelFormat dstFormat, int flags, SwsFilter *srcFilter, SwsFilter *dstFilter, const double *param); - sws_scale
画像バッファsrcSlice[]をスケール変換しdst[]に格納する。
/* Scale the image slice in srcSlice and put the resulting scaled slice in the image in dst.
A slice is a sequence of consecutive rows in an image. Requires a context that has previously been initialized with sws_init_context().
Slices have to be provided in sequential order, either in top-bottom or bottom-top order. If slices are provided in non-sequential order the behavior of the function is undefined. */
int sws_scale(SwsContext *sws, const uint8_t *const srcSlice[], const int srcStride[], int srcSliceY, int srcSliceH, uint8_t *const dst[], const int dstStride[]);上述の関数に必要なヘッダ
パッケージ管理ツールでインストールする。
zsh
% sudo apt install ffmpeg libavformat-dev libavcodec-dev libavutil-dev libswscale-dev必要なヘッダは以下である。
- libavformat/avformat.h
- avformat_open_input()
- avformat_find_stream_info()
- av_find_best_stream()
- av_read_frame()
- libavcodec/avcodec.h
- avcodec_find_decoder()
- avcodec_alloc_context3()
- avcodec_parameters_to_context()
- avcodec_open2()
- avcodec_send_packet()
- avcodec_receive_frame()
- libavutil/dict.h
- av_dict_set()
- libavutil/frame.h
- av_frame_alloc()
- packet.h
- av_packet_alloc()
- libavutil/imgutils.h
- av_image_alloc()
- libswscale/swscale.h
- sws_getContext()
- sws_scale()
- libavutil/error.h
- エラー処理
zigでH.264対応IPカメラのRTSPストリーム配信をキャプチャする
zigで書いてみた。とりあえずRGBに変換してppmで保存する処理を61フレーム分実行する。RTSPストリーム配信は前回記事：H.264対応IPカメラのRTSPストリーム配信をiPyCamでシミュレーションしたい～バーチャルIPカメラ～に記載した内容でシミュレーションした。
zig: main.zig
const std = @import("std");
const builtin = @import("builtin");
const print = std.debug.print;
const debug = std.log.debug;
pub const std_Options: std.Options = .{ .log_level = if (builtin.mode == .Debug) .debug };
const c = @cImport({
@cInclude("libavformat/avformat.h");
@cInclude("libavcodec/avcodec.h");
@cInclude("libavutil/imgutils.h");
@cInclude("libswscale/swscale.h");
@cInclude("libavutil/error.h");
});
const NBuf = 256;
var buf: [NBuf]u8 = undefined;
//関数内でバッファを作ると関数終了時に消えるため、引数で受け取る必要がある
fn wrap_av_strerror(errbuf: []u8, errnum: c_int) []const u8 { //[]:スライス、[_]配列長さ推論
_ = c.av_strerror(errnum, errbuf.ptr, errbuf.len); //スライスのポインタは.ptr
return std.mem.sliceTo(errbuf, 0); //NULL終端まで
}
pub fn main(init: std.process.Init) !void {
const io = init.io;
const cwd: std.Io.Dir = std.Io.Dir.cwd();
const output_dir_name = "ppm_zig";
cwd.createDir(io, output_dir_name, .default_dir) catch |e| switch (e) {
error.PathAlreadyExists => {
print("\x1b[31m[return error]\x1b[0mPathAlreadyExist: {s}\n", .{output_dir_name});
},
else => return e,
};
debug("Successfully created {s}", .{output_dir_name});
var output_dir: std.Io.Dir = try cwd.openDir(io, output_dir_name, .{});
defer output_dir.close(io);
var ciRet: c_int = undefined; //C API返り値用変数
const src_url = "rtsp://172.26.230.129:8554/video_main";
//コマンドオプションを設定
var options: ?*c.AVDictionary = null; //ちなみに、.?でOpitionalを外すことができる
defer c.av_dict_free(&options);
_ = c.av_dict_set(&options, "rtsp_transport", "tcp", 0);
_ = c.av_dict_set(&options, "fflags", "nobuffer", 0);
_ = c.av_dict_set(&options, "flags", "low_delay", 0);
//ストリームを開いてヘッダを読む
var fmt_ctx: [*c]c.AVFormatContext = null; //[*c]はcポインタ、nullも入る
ciRet = c.avformat_open_input(&fmt_ctx, src_url, null, &options);
defer c.avformat_close_input(&fmt_ctx);
if (ciRet < 0) {
print("\x1b[31m[return error]\x1b[0mc.avformat_open_input(): {s}\n", .{wrap_av_strerror(&buf, ciRet)});
return;
}
//パケットを読んでストリーム情報を取得する
ciRet = c.avformat_find_stream_info(fmt_ctx, null);
if (ciRet < 0) {
print("\x1b[31m[return error]\x1b[0mavformat_find_stream_info(): {s}\n", .{wrap_av_strerror(&buf, ciRet)});
return;
}
//AVMediaTypeに一致するストリームを探し出す
ciRet = c.av_find_best_stream(fmt_ctx, c.AVMEDIA_TYPE_VIDEO, -1, -1, null, 0);
if (ciRet < 0) {
print("\x1b[31m[return error]\x1b[0mav_find_best_stream(): {s}\n", .{wrap_av_strerror(&buf, ciRet)});
return;
}
//コーデックIDに一致するデコーダを探す
const video_stream_index: usize = @intCast(ciRet);
const video_stream: [*c]c.AVStream = fmt_ctx.*.streams[video_stream_index]; //.*でポインタを外す
const dec: [*c]const c.AVCodec = c.avcodec_find_decoder(video_stream.*.codecpar.*.codec_id);
if (dec == null) {
print("failed to find AVMEDIA_TYPE_VIDEO codec\n", .{});
return;
}
//AVCodecContextを確保し初期化する
var video_dec_ctx: [*c]c.AVCodecContext = c.avcodec_alloc_context3(dec);
defer c.avcodec_free_context(&video_dec_ctx);
if (video_dec_ctx == null) {
print("failed to allocate the AVMEDIA_TYPE_VIDEO codec context\n", .{});
return;
}
//AVCodecContextにAVCodecParametersを反映する
ciRet = c.avcodec_parameters_to_context(video_dec_ctx, video_stream.*.codecpar);
if (ciRet < 0) {
print("\x1b[31m[return error]\x1b[0mavcodec_parameters_to_context(): {s}\n", .{wrap_av_strerror(&buf, ciRet)});
return;
}
//AVCodecを使用するためにAVCodecContextを初期化する
ciRet = c.avcodec_open2(video_dec_ctx, dec, null);
if (ciRet < 0) {
print("\x1b[31m[return error]\x1b[0mavcodec_open2(): {s}\n", .{wrap_av_strerror(&buf, ciRet)});
return;
}
//画像バッファを確保し、各面のポインタと対応する面の行サイズを返す
//面数はAVPixelFormatによって異なり、RGB24は1面、YUV420Pは3面である
var video_dst_data: [4][*c]u8 = undefined;
var video_dst_linesize: [4]c_int = undefined;
const width = video_dec_ctx.*.width;
const height = video_dec_ctx.*.height;
const src_pix_fmt = video_dec_ctx.*.pix_fmt; //YUV(origin)
//const pix_fmt = c.AV_PIX_FMT_RGB24; //RGB
const dst_pix_fmt = c.AV_PIX_FMT_BGR24; //BGR to OpenCV
debug("width: {}, height: {}, src_pix_fmt: {}, dst_pix_fmt: {}", .{ width, height, src_pix_fmt, dst_pix_fmt });
ciRet = c.av_image_alloc(video_dst_data[0..].ptr, video_dst_linesize[0..].ptr, width, height, dst_pix_fmt, 1);
defer c.av_freep(@ptrCast(&video_dst_data[0]));
if (ciRet < 0) {
print("\x1b[31m[return error]\x1b[0mav_image_alloc(): {s}\n", .{wrap_av_strerror(&buf, ciRet)});
return;
}
//フレームを確保する
var frame: [*c]c.AVFrame = null;
frame = c.av_frame_alloc();
defer c.av_frame_free(&frame);
if (frame == null) {
print("Could not allocate frame\n", .{});
return;
}
//パケットを確保する
var pkt: [*c]c.AVPacket = null;
pkt = c.av_packet_alloc();
defer c.av_packet_free(&pkt);
if (pkt == null) {
print("Could not allocate frame\n", .{});
return;
}
//SwsContextを確保して返す
const sws_ctx: ?*c.SwsContext = c.sws_getContext(width, height, src_pix_fmt, //src
width, height, dst_pix_fmt, //dst
c.SWS_FAST_BILINEAR, null, null, null //etc.
);
defer c.sws_freeContext(sws_ctx);
//ストリームの次の枠を返す、AVPacketにパケットが返る
var count: usize = 0;
while (true) {
ciRet = c.av_read_frame(fmt_ctx, pkt);
debug("\x1b[36m[in loop]\x1b[0mav read frame; pkt: {any}", .{pkt});
if (ciRet < 0) {
print("\x1b[31m[return error]\x1b[0mav_read_frame(): {s}\n", .{wrap_av_strerror(&buf, ciRet)});
return;
}
if (pkt.*.stream_index != video_stream_index) {
print("\x1b[36m[continue]\x1b[0mpkt.*.stream_index != video_stream_index", .{});
_ = c.av_packet_unref(pkt);
continue;
}
//パケットをデコーダに入力する
ciRet = c.avcodec_send_packet(video_dec_ctx, pkt);
debug("\x1b[36m[done]\x1b[0mavcodec_send_packet", .{});
if (ciRet < 0) {
print("\x1b[31m[return error]\x1b[0mavcodec_send_packet(): {s}\n", .{wrap_av_strerror(&buf, ciRet)});
return;
}
//デコーダから出力されたフレームを返す
while (true) {
ciRet = c.avcodec_receive_frame(video_dec_ctx, frame);
debug("\x1b[36m[in loop]\x1b[0mavcodec_receive_frame; frame: {any}", .{frame});
// those two return values are special and mean there is no output
// frame available, but there were no errors during decoding
if (ciRet == c.AVERROR_EOF or ciRet == c.AVERROR(c.EAGAIN)) {
debug("\x1b[36m[break]\x1b[0mavcodec_receive_frame() ciRet:{} -> 0", .{ciRet});
ciRet = 0;
break;
}
if (ciRet < 0) {
print("\x1b[31m[return error]\x1b[0mavcodec_receive_frame(): {s}\n", .{wrap_av_strerror(&buf, ciRet)});
return;
}
if (video_dec_ctx.*.codec.*.type != c.AVMEDIA_TYPE_VIDEO) {
print("\x1b[33m[status changed]\x1b[0mvideo_dec_ctx.*.codec.*.type != c.AVMEDIA_TYPE_VIDEO", .{});
return;
}
if (frame.*.width != width or frame.*.height != height or frame.*.format != src_pix_fmt) {
debug("frame.*.width:{}, width:{}, frame.*.height:{}, height:{}, frame.*.format:{}, src_pix_fmt:{},", .{ frame.*.width, width, frame.*.height, height, frame.*.format, src_pix_fmt });
const errmsg =
"\x1b[33m[status changed]\x1b[0m" ++
"Width, height and pixel format have to be " ++
"constant in a rawvideo file, but the width, height or " ++
"pixel format of the input video changed";
print("{s}\n", .{errmsg});
return;
}
//画像バッファsrcSlice[]をスケール変換しdst[]に格納する
ciRet = c.sws_scale(sws_ctx, frame.*.data[0..].ptr, frame.*.linesize[0..].ptr, 0, height, //src
video_dst_data[0..].ptr, video_dst_linesize[0..].ptr //dst
);
if (ciRet < 0) {
print("\x1b[31m[return error]\x1b[0msws_scale(): {s}\n", .{wrap_av_strerror(&buf, ciRet)});
return;
}
//とりあえずppmにして保存
count += 1;
var strbuf: [NBuf]u8 = undefined;
const file_name = try std.fmt.bufPrint(&strbuf, "sws_frame_{d:0>4}.ppm", .{count});
const file: std.Io.File = try output_dir.createFile(io, file_name, .{});
//defer file.close(io);
debug("Successfully created {s}", .{file_name});
var file_writer = file.writer(io, &.{});
const writer = &file_writer.interface;
_ = try writer.print(
\\P6
\\{} {}
\\255
\\
, .{ width, height });
debug("ppm header: successfully wrote header", .{});
//linesizeはalignmentに合わせてpaddingされるためwidthに一致するとは限らない
const stride: usize = @intCast(video_dst_linesize[0]);
const x: usize = @intCast(width * 3);
debug("stride: {}, x: {}", .{ stride, x });
for (0..@intCast(height)) |y| {
try writer.writeAll(video_dst_data[0][y * stride .. y * stride + x]);
}
debug("ppm data: successfully wrote all data", .{});
file.close(io);
debug("count = {}", .{count});
if (count == 61) { //61フレームで強制終了
ciRet = -1;
print("\x1b[32m[success]\x1b[0m 61 frames ppm created\n", .{});
return;
}
_ = c.av_frame_unref(frame);
}
_ = c.av_packet_unref(pkt);
}
//動画ファイル入力などの場合はデコーダに残っているフレームを吐き出させる処理が必要
//ciRet = c.avcodec_send_packet(video_dec_ctx, null);
//・・・
}zig: build.zig
const std = @import("std");
pub fn build(b: *std.Build) void {
const target = b.standardTargetOptions(.{});
const optimize = b.standardOptimizeOption(.{});
const exe = b.addExecutable(.{
.name = "rtsp_ffmpeg",
.root_module = b.createModule(.{
.root_source_file = b.path("main.zig"),
.target = target,
.optimize = optimize,
.link_libc = true,
}),
.use_llvm = false,
});
//ライブラリリンク
exe.root_module.linkSystemLibrary("avformat", .{});
exe.root_module.linkSystemLibrary("avcodec", .{});
exe.root_module.linkSystemLibrary("avutil", .{});
exe.root_module.linkSystemLibrary("swscale", .{});
b.installArtifact(exe);
}実行方法
まずはデバッグ表示させてみる。
zsh
% zig build
% ./zig-out/bin/rtsp_ffmpeg
debug: Successfully created ppm_zig
debug: width: 1280, height: 720, src_pix_fmt: 0, dst_pix_fmt: 3
debug: [in loop]av read frame; pkt: cimport.struct_AVPacket@11892d00
debug: [done]avcodec_send_packet
debug: [in loop]avcodec_receive_frame; frame: cimport.struct_AVFrame@118a5e80
debug: [break]avcodec_receive_frame() ciRet:-11 -> 0
debug: [in loop]av read frame; pkt: cimport.struct_AVPacket@11892d00
debug: [done]avcodec_send_packet
debug: [in loop]avcodec_receive_frame; frame: cimport.struct_AVFrame@118a5e80
debug: Successfully created sws_frame_0001.ppm
debug: ppm header: successfully wrote header
debug: stride: 3840, x: 3840
debug: ppm data: successfully wrote all data
debug: count = 1
debug: [in loop]avcodec_receive_frame; frame: cimport.struct_AVFrame@118a5e80
debug: [break]avcodec_receive_frame() ciRet:-11 -> 0
debug: [in loop]av read frame; pkt: cimport.struct_AVPacket@11892d00
debug: [done]avcodec_send_packet
debug: [in loop]avcodec_receive_frame; frame: cimport.struct_AVFrame@118a5e80
debug: Successfully created sws_frame_0002.ppm
debug: ppm header: successfully wrote header
debug: stride: 3840, x: 3840
debug: ppm data: successfully wrote all data
debug: count = 2
・・・省略・・・
debug: [in loop]avcodec_receive_frame; frame: cimport.struct_AVFrame@118a5e80
debug: [break]avcodec_receive_frame() ciRet:-11 -> 0
debug: [in loop]av read frame; pkt: cimport.struct_AVPacket@11892d00
debug: [done]avcodec_send_packet
debug: [in loop]avcodec_receive_frame; frame: cimport.struct_AVFrame@118a5e80
debug: Successfully created sws_frame_0061.ppm
debug: ppm header: successfully wrote header
debug: stride: 3840, x: 3840
debug: ppm data: successfully wrote all data
debug: count = 61
[success] 61 frames ppm created
% ffprobe -hide_banner ppm_zig/sws_frame_0001.ppm #作成したppmを調査
Input #0, ppm_pipe, from 'ppm_zig/sws_frame_0001.ppm':
Duration: N/A, bitrate: N/A
Stream #0:0: Video: ppm, rgb24, 1280x720, 25 fps, 25 tbr, 25 tbn
% #ffplay ppm_zig/sws_frame_0001.ppm #GUI環境なら画像表示可能続いて、ReleaseFastでビルドする。上述の処理で既にppm出力先フォルダを作成してしまったので、既に存在する旨のエラーが出るが、そのエラーの場合のみそのまま次の処理に進むようにしてある。すなわち、ppmは上書きする。
zsh
% zig build -Doptimize=ReleaseFast
% ./zig-out/bin/rtsp_ffmpeg
[return error]PathAlreadyExist: ppm_zig
[success] 61 frames ppm created
%zigでC APIを叩くノウハウが少しは身についた気がするにゅんちゅ！ffmpegにも少しは詳しくなった気がするにゅんちゅ！
