Decoding Live Feed Devices¶
DeFFcode's FFdecoder API provide effortless support for any Live Feed Devices using two parameters:
source
parameter which accepts device name or its path, andsource_demuxer
parameter to specify demuxer for the given input device.
We'll discuss the Live Feed Devices support using both these parameters briefly in the following recipes:
DeFFcode APIs requires FFmpeg executable
DeFFcode APIs MUST requires valid FFmpeg executable for all of its core functionality, and any failure in detection will raise RuntimeError
immediately. Follow dedicated FFmpeg Installation doc ➶ for its installation.
Additional Python Dependencies for following recipes
Following recipes requires additional python dependencies which can be installed easily as below:
-
OpenCV: OpenCV is required for previewing video frames. You can easily install it directly via
pip
:OpenCV installation from source
You can also follow online tutorials for building & installing OpenCV on Windows, Linux, MacOS and Raspberry Pi machines manually from its source.
Make sure not to install both pip and source version together. Otherwise installation will fail to work!
Other OpenCV binaries
OpenCV maintainers also provide additional binaries via pip that contains both main modules and contrib/extra modules
opencv-contrib-python
, and for server (headless) environments likeopencv-python-headless
andopencv-contrib-python-headless
. You can also install any one of them in similar manner. More information can be found here.
Always use FFdecoder API's terminate()
method at the end to avoid undesired behavior.
Never name your python script deffcode.py
When trying out these recipes, never name your python script deffcode.py
otherwise it will result in ModuleNotFound
error.
Capturing and Previewing frames from a Webcam using Custom Demuxer¶
Example Assumptions
FFmpeg provide set of specific Demuxers on different platforms to read the multimedia streams from a particular type of Video Capture source/device. Please note that following recipe explicitly assumes:
- You're running Linux Machine with USB webcam connected to it at node/path
/dev/video0
. - You already have appropriate Linux video drivers and related softwares installed on your machine.
- You machine uses FFmpeg binaries built with
--enable-libv4l2
flag to supportvideo4linux2, v4l2
demuxer. BTW, you can list all supported demuxers using theffmpeg --list-demuxers
terminal command.
These assumptions MAY/MAY NOT suit your current setup. Kindly use suitable parameters based your system platform and hardware settings only.
In this example we will decode BGR24 video frames from a USB webcam device connected at path /dev/video0
on a Linux Machine with video4linux2
(or simply v4l2
) demuxer, and preview them using OpenCV Library's cv2.imshow()
method.
Identifying and Specifying Video Capture Device Name/Path/Index and suitable Demuxer on different OS platforms
Windows OS users can use the dshow (DirectShow) to list video input device which is the preferred option for Windows users. You can refer following steps to identify and specify your input video device's name:
-
Identify Video Devices: You can locate your video device's name (already connected to your system) using
dshow
as follows:c:\> ffmpeg.exe -list_devices true -f dshow -i dummy ffmpeg version N-45279-g6b86dd5... --enable-runtime-cpudetect libavutil 51. 74.100 / 51. 74.100 libavcodec 54. 65.100 / 54. 65.100 libavformat 54. 31.100 / 54. 31.100 libavdevice 54. 3.100 / 54. 3.100 libavfilter 3. 19.102 / 3. 19.102 libswscale 2. 1.101 / 2. 1.101 libswresample 0. 16.100 / 0. 16.100 [dshow @ 03ACF580] DirectShow video devices [dshow @ 03ACF580] "Integrated Camera" [dshow @ 03ACF580] "USB2.0 Camera" [dshow @ 03ACF580] DirectShow audio devices [dshow @ 03ACF580] "Microphone (Realtek High Definition Audio)" [dshow @ 03ACF580] "Microphone (USB2.0 Camera)" dummy: Immediate exit requested
-
Specify Video Device's name: Then, you can specify and initialize your located Video device's name in FFdecoder API as follows:
-
[OPTIONAL] Specify Video Device's index along with name: If there are multiple Video devices with similar name, then you can use
-video_device_number
parameter to specify the arbitrary index of the particular device. For instance, to open second video device with name"Camera"
you can do as follows:# define video_device_number as 1 (numbering start from 0) ffparams = {"-ffprefixes":["-video_device_number", "1"]} # initialize and formulate the decoder with "Camera" source for BGR24 output decoder = FFdecoder("Camera", source_demuxer="dshow", frame_format="bgr24", verbose=True, **ffparams).formulate()
Linux OS users can use the video4linux2
(or its alias v4l2
) to list to all capture video devices such as from an USB webcam. You can refer following steps to identify and specify your capture video device's path:
-
Identify Video Devices: Linux systems tend to automatically create file device node/path when the device (e.g. an USB webcam) is plugged into the system, and has a name of the kind
'/dev/videoN'
, whereN
is a index associated to the device. To get the list of all available file device node/path on your Linux machine, you can use thev4l-ctl
command.You can use
sudo apt install v4l-utils
APT command to installv4l-ctl
tool on Debian-based Linux distros. -
Specify Video Device's path: Then, you can specify and initialize your located Video device's path in FFdecoder API as follows:
-
[OPTIONAL] Specify Video Device's additional specifications: You can also specify additional specifications (such as pixel format(s), video format(s), framerate, and frame dimensions) supported by your Video Device as follows:
You can use
ffmpeg -f v4l2 -list_formats all -i /dev/video0
terminal command to list available specifications.# define video device specifications ffparams = {"-ffprefixes":["-framerate", "25", "-video_size", "640x480"]} # initialize and formulate the decoder with "/dev/video0" source for BGR24 output decoder = FFdecoder("/dev/video0", source_demuxer="v4l2", frame_format="bgr24", verbose=True, **ffparams).formulate()
MacOS users can use the AVFoundation to list input devices and is the currently recommended framework by Apple for streamgrabbing on Mac OSX-10.7 (Lion) and later as well as on iOS. You can refer following steps to identify and specify your capture video device's name or index on MacOS/OSX machines:
QTKit is also available for streamgrabbing on Mac OS X 10.4 (Tiger) and later, but has been marked deprecated since OS X 10.7 (Lion) and may not be available on future releases.
-
Identify Video Devices: Then, You can locate your Video device's name and index using
avfoundation
as follows:$ ffmpeg -f avfoundation -list_devices true -i "" ffmpeg version N-45279-g6b86dd5... --enable-runtime-cpudetect libavutil 51. 74.100 / 51. 74.100 libavcodec 54. 65.100 / 54. 65.100 libavformat 54. 31.100 / 54. 31.100 libavdevice 54. 3.100 / 54. 3.100 libavfilter 3. 19.102 / 3. 19.102 libswscale 2. 1.101 / 2. 1.101 libswresample 0. 16.100 / 0. 16.100 [AVFoundation input device @ 0x7f8e2540ef20] AVFoundation video devices: [AVFoundation input device @ 0x7f8e2540ef20] [0] FaceTime HD camera (built-in) [AVFoundation input device @ 0x7f8e2540ef20] [1] Capture screen 0 [AVFoundation input device @ 0x7f8e2540ef20] AVFoundation audio devices: [AVFoundation input device @ 0x7f8e2540ef20] [0] Blackmagic Audio [AVFoundation input device @ 0x7f8e2540ef20] [1] Built-in Microphone
-
Specify Video Device's name or index: Then, you can specify and initialize your located Video device in FFdecoder API using its either the name or the index shown in the device listing:
When specifying device's name, abbreviations using just the beginning of the device name are possible. Thus, to capture from a device named "Integrated iSight-camera" just "Integrated" is sufficient:
-
[OPTIONAL] Specify Default Video device: You can also use the default device which is usually the first device in the listing by using "default" as source:
If these steps doesn't work for you then reach us out on Gitter ➶ Community channel
# import the necessary packages
from deffcode import FFdecoder
import cv2
# initialize and formulate the decoder with "/dev/video0" source for BGR24 output
decoder = FFdecoder("/dev/video0", source_demuxer="v4l2", frame_format="bgr24", verbose=True).formulate()
# grab the BGR24 frames from decoder
for frame in decoder.generateFrame():
# check if frame is None
if frame is None:
break
# {do something with the frame here}
# Show output window
cv2.imshow("Output", frame)
# check for 'q' key if pressed
key = cv2.waitKey(1) & 0xFF
if key == ord("q"):
break
# close output window
cv2.destroyAllWindows()
# terminate the decoder
decoder.terminate()
Capturing and Previewing frames from your Desktop¶
Example Assumptions
Similar to Webcam capturing, FFmpeg provide set of specific Demuxers on different platforms for capturing your desktop (Screen recording). Please note that following recipe explicitly assumes:
- You're running Linux Machine with
libxcb
module installed properly on your machine. - You machine uses FFmpeg binaries built with
--enable-libxcb
flag to supportx11grab
demuxer. BTW, you can list all supported demuxers using theffmpeg --list-demuxers
terminal command.
These assumptions MAY/MAY NOT suit your current setup. Kindly use suitable parameters based your system platform and hardware settings only.
In this example we will decode live BGR video frames from your complete screen as well as a region in FFdecoder API, and preview them using OpenCV Library's cv2.imshow()
method.
Specifying suitable Parameter(s) and Demuxer for Capturing your Desktop on different OS platforms
Windows OS users can use the gdigrab to grab video from the Windows screen. You can refer following steps to specify source for capturing different regions of your display:
For Windows OS users dshow
is also available for grabbing frames from your desktop. But it is highly unreliable and don't works most of the times.
-
Capturing entire desktop: For capturing all your displays as one big contiguous display, you can specify source, suitable parameters and demuxers in FFdecoder API as follows:
-
Capturing a region: If you want to limit capturing to a region, and show the area being grabbed, you can specify source and suitable parameters in FFdecoder API as follows:
x_offset
andy_offset
specify the offsets of the grabbed area with respect to the top-left border of the desktop screen. They default to0
.# define suitable parameters ffparams = { "-framerate": "30", # input framerate "-ffprefixes": [ "-offset_x", "10", "-offset_y", "20", # grab at position 10,20 "-video_size", "640x480", # frame size "-show_region", "1", # show only region ], } # initialize and formulate the decoder with "desktop" source for BGR24 output decoder = FFdecoder("desktop", source_demuxer="gdigrab", frame_format="bgr24", verbose=True, **ffparams).formulate()
Linux OS users can use the x11grab to capture an X11 display. You can refer following steps to specify source for capturing different regions of your display:
For X11 display, the source input has the syntax: "display_number.screen_number[+x_offset,y_offset]"
.
-
Capturing entire desktop: For capturing all your displays as one big contiguous display, you can specify source, suitable parameters and demuxers in FFdecoder API as follows:
-
Capturing a region: If you want to limit capturing to a region, and show the area being grabbed, you can specify source and suitable parameters in FFdecoder API as follows:
x_offset
andy_offset
specify the offsets of the grabbed area with respect to the top-left border of the X11 screen. They default to0
.# define suitable parameters ffparams = { "-framerate": "30", # input framerate "-ffprefixes": [ "-video_size", "1024x768", # frame size ], } # initialize and formulate the decoder with ":0.0" desktop source(starting with the upper-left corner at x=10, y=20) # for BGR24 output decoder = FFdecoder(":0.0+10,20", source_demuxer="x11grab", frame_format="bgr24", verbose=True, **ffparams).formulate()
MacOS users can use the AVFoundation to list input devices and is the currently recommended framework by Apple for streamgrabbing on Mac OSX-10.7 (Lion) and later as well as on iOS. You can refer following steps to identify and specify your capture video device's name or index on MacOS/OSX machines:
QTKit is also available for streamgrabbing on Mac OS X 10.4 (Tiger) and later, but has been marked deprecated since OS X 10.7 (Lion) and may not be available on future releases.
-
Identify Video Devices: You can enumerate all the available input devices including screens ready to be captured using
avfoundation
as follows:$ ffmpeg -f avfoundation -list_devices true -i "" ffmpeg version N-45279-g6b86dd5... --enable-runtime-cpudetect libavutil 51. 74.100 / 51. 74.100 libavcodec 54. 65.100 / 54. 65.100 libavformat 54. 31.100 / 54. 31.100 libavdevice 54. 3.100 / 54. 3.100 libavfilter 3. 19.102 / 3. 19.102 libswscale 2. 1.101 / 2. 1.101 libswresample 0. 16.100 / 0. 16.100 [AVFoundation input device @ 0x7f8e2540ef20] AVFoundation video devices: [AVFoundation input device @ 0x7f8e2540ef20] [0] FaceTime HD camera (built-in) [AVFoundation input device @ 0x7f8e2540ef20] [1] Capture screen 0 [AVFoundation input device @ 0x7f8e2540ef20] AVFoundation audio devices: [AVFoundation input device @ 0x7f8e2540ef20] [0] Blackmagic Audio [AVFoundation input device @ 0x7f8e2540ef20] [1] Built-in Microphone
-
Capturing entire desktop: Then, you can specify and initialize your located screens in FFdecoder API using its index shown:
-
[OPTIONAL] Capturing mouse: You can also specify additional specifications to capture the mouse pointer and screen mouse clicks as follows:
If these steps doesn't work for you then reach us out on Gitter ➶ Community channel
For capturing all your displays as one big contiguous display in FFdecoder API:
# import the necessary packages
from deffcode import FFdecoder
import cv2
# define framerate
ffparams = {"-framerate": "30"}
# initialize and formulate the decoder with ":0.0" desktop source for BGR24 output
decoder = FFdecoder(":0.0", source_demuxer="x11grab", frame_format="bgr24", verbose=True, **ffparams).formulate()
# grab the BGR24 frames from decoder
for frame in decoder.generateFrame():
# check if frame is None
if frame is None:
break
# {do something with the frame here}
# Show output window
cv2.imshow("Output", frame)
# check for 'q' key if pressed
key = cv2.waitKey(1) & 0xFF
if key == ord("q"):
break
# close output window
cv2.destroyAllWindows()
# terminate the decoder
decoder.terminate()
For limit capturing to a region, and show the area being grabbed:
x_offset
and y_offset
specify the offsets of the grabbed area with respect to the top-left border of the X11 screen. They default to 0
.
# import the necessary packages
from deffcode import FFdecoder
import cv2
# define suitable parameters
ffparams = {
"-framerate": "30", # input framerate
"-ffprefixes": [
"-video_size", "1024x768", # frame size
],
}
# initialize and formulate the decoder with ":0.0" desktop source(starting with the upper-left corner at x=10, y=20)
# for BGR24 output
decoder = FFdecoder(":0.0+10,20", source_demuxer="x11grab", frame_format="bgr24", verbose=True, **ffparams).formulate()
# grab the BGR24 frames from decoder
for frame in decoder.generateFrame():
# check if frame is None
if frame is None:
break
# {do something with the frame here}
# Show output window
cv2.imshow("Output", frame)
# check for 'q' key if pressed
key = cv2.waitKey(1) & 0xFF
if key == ord("q"):
break
# close output window
cv2.destroyAllWindows()
# terminate the decoder
decoder.terminate()