Build ffmpeg with 10-bit x265 support for encoding on Windows

So I was trying to create some 10-bit test HEVC content with ffmpeg and ran into the following error:

ffmpeg -i "input.mkv" -c:v libx265 -preset slow -crf 18 -pix_fmt yuv420p10le -c:a copy -y output_10bit.mkv

...

Incompatible pixel format 'yuv420p10le' for codec 'libx265', auto-selecting format 'yuv420p'

I discovered that by default - most ffmpeg builds include only 8-bit support for HEVC encoding with libx265.

Certainly, you can get builds that already enable 10 or even 12-bit support - but I’m usually a glutton for punishment so I thought I would see how hard it was to just build it manually.

Here are the exact steps I did to build both ffmpeg and libx265 with Visual Studio 2017 on Windows. Some of the steps will also detail some of the errors I ran into along the way - in case someone else runs into trouble, hopefully then it will show how to overcome any future hurdles.

Note: This guide will only build the 64-bit versions, but you should be able to adjust for 32-bit.

Build Tools

• git - to get the ffmpeg source code
• TortoiseHg - to get the x265 source code
• Visual Studio 2017 Community Edition (you can probably get most versions 2015 and newer to work)
  • When installing VS 2017 - make sure you select the options for Desktop development with C++ so the C++ compiler and tools are installed.
• CMAKE 3.xx
• YASM
  • Download YASM 64-bit and copy to C:\tools\yasm
  • Rename yasm-1.3.0-win64.exe (or whatever version) to yasm.exe
  • Add c:\tools\yasm to your PATH environment variable
• NASM (x265 v2.6 and older will use YASM, new versions will use NASM)
  • Download NASM 2.13 or newer to C:\tools\nasm
  • Add c:\tools\nasm to your PATH environment variable
• Msys2
  • Download Msys2 installer for 64-bit and install to C:\tools\msys64

  • With a text editor, open the file C:\tools\msys64\msys2_shell.cmd and un-comment the line

    rem set MSYS2_PATH_TYPE=inherit
    

    This is done so msys2 can use your existing PATH environment variable. (thanks to Scott Davies for pointing this out)

  • Now run c:\Tools\msys64\msys2_shell.cmd. When the msys2 shell opens - use pacman to install the following:

    pacman -S gcc
    pacman -S diffutils
    pacman -S make
    pacman -S pkg-config
    pacman -S autoconf
    

  • Close the msys2 shell for now - we’ll relaunch later from a Visual Studio command prompt

Build libx265 for 10-bit support

1. Download the x265 source code

   Create the directory c:\third_party and cd from a command prompt to it. Run the command:

   hg clone https://bitbucket.org/multicoreware/x265
   

2. Navigate to third_party\x265\build. Find the directory that matches the VC compiler you want to use. It is likely there will not be one matching the newest version of Visual Studio, so use the latest one available (In my case it was vc12-x86_64)

   Using a text editor, open make-solutions.bat. Change the generator command to match the version of VS you want to use. I’m using 2017, so my command is:

   cmake -G "Visual Studio 15 Win64" ..\..\source && cmake-gui ..\..\source
   

   Update 12/9/2017 - libx265 should now have a corresponding make-solutions.bat file for Visual Studio 2017. This step is no longer required

3. Run the make-solutions.bat - during the processing, the CMake GUI will open with properties highlighted red.
   • Check the HIGH_BIT_DEPTH for 10-bit
   • Set the CMAKE_INSTALL_PREFIX to a directory the x265 lib, headers, and importantly the pkg-config .pc file will be installed to

   • Verify the NASM_EXECUTABLE property is pointing the location where you installed NASM

     
     
     

4. Press Configure
   • Another property called MAIN12 will probably get highlighted in red. This is for 12-bit support (if you want that check it).
   • Press Configure again
   • Press Generate

5. Open the x265.sln in Visual Studio - change the build type to Release and the Arch to x64

6. Change the C++ runtime to use static CRT (/MT) rather than the default (/MD) to match ffmpeg

   This should be done for the cli, common, and encoder projects.

7. Build the ALL_BUILD project.

8. Build the INSTALL project - this is important for building ffmpeg next

9. Navigate to the location used in the CMAKE_INSTALL_PREFIX property above. Verify you have bin, include and lib folders.

Build ffmpeg with custom x265 build from above

1. Download the ffmpeg source code

   Create the directory c:\third_party\ffmpeg and cd from a command prompt to it. Run the command:

   git clone https://git.ffmpeg.org/ffmpeg.git ffmpeg
   

   This will result in the ffmpeg source tree located at c:\third_party\ffmpeg\ffmpeg which will be handy for an out-of-tree build

2. Create the directory c:\third_party\ffmpeg\ffbuild

3. Open a VS command prompt - I used the x64 Native Tools Command Prompt for VS 2017

4. Change the directory - cd to c:\tools\msys64

5. Run the msys2_shell.cmd for msys2

6. Verify our x265 build with pkg-config

   pkg-config --exists --print-errors x265
   

   This should fail with an error similar to the following, since we haven’t told pkg-config where our x265 is.

   Package x265 was not found in the pkg-config search path.
   Perhaps you should add the directory containing `x265.pc'
   to the PKG_CONFIG_PATH environment variable
   No package 'x265' found
   

   Add the x265.pc install location to the PKG_CONFIG_PATH environment variable

   PKG_CONFIG_PATH=$PKG_CONFIG_PATH:/c/x265/lib/pkgconfig
   

   Run the verify command again - you should get no error messages

   pkg-config --exists --print-errors x265
   

7. Configure ffmpeg for a build.

   cd /c/third_party/ffmpeg/ffbuild/
   

   There are many different ways to build ffmpeg - I’m doing the bare minimum here to get a ffmpeg build that can encode 10-bit x265:

   ../ffmpeg/configure --toolchain=msvc prefix="win-build" \
   --pkg-config-flags="--static" \
   --disable-shared \
   --enable-libx265 \
   --enable-gpl \
   --enable-asm \
   --enable-x86asm
   

   If you’re following this far, the above command should fail :(

   ERROR: x265 not found using pkg-config
   

   We already verified that pkg_config could locate our x265 above, but the ffmpeg configure script is complaining …

   Open the c:\third_party\ffmpeg\ffbuild\ffbuild\ffbuild\config.log and scroll to the bottom. You should see an error something like:

   LINK : fatal error LNK1181: cannot open input file 'x265.lib'
   ERROR: x265 not found using pkg-config
   

   Navigate to C:\x265\lib. The x265 build from above created a static lib named x265-static.lib … rename it to x265.lib

   I’m pointing this failure out because often when building ffmpeg you will need to refer to the config.log when there is a problem

   Run the ffmpeg configure command again - this time it should be successful.

8. Build and install ffmpeg

   make
   make install
   

9. Your fresh ffmpeg build should be at c:\third_party\ffmpeg\ffbuild\win-build

Transcode some content to 10-bit HEVC

1. Create a test clip with something like:

   ffmpeg -i "input.mkv" -c:v libx265 -preset slow -crf 18 -pix_fmt yuv420p10le -c:a copy -y output_10bit.mkv
   

2. Verify the video stream in the output file is 10-bit (we’re looking for confirmation the yuv420p10le pixel format was selected)

   ffmpeg -i "output_10bit.mkv"
         
   ... 
         
   Stream #0:0(eng): Video: hevc (Main 10), yuv420p10le(tv, progressive), 1280x720 [SAR 1:1 DAR 16:9], 59.94 fps, 59.94 tbr, 1k tbn, 59.94 tbc