Download HLS Streaming-Videos With FFMPEG. No-Encoding!

Posted at

Those notes were designed for developers,
they are complex and advance, require many installations,
but you can't actually f*** anything-up in the process,
so if you're up for it, go ahead.

We'll be getting a playlist URL from a browser,
copying cookie-authentication and downloading and merging streaming video-segments into a playable video file, and fast too.

Here are the steps to follow...



  1. You'll need:

    FFMPEG
    https://ffmpeg.zeranoe.com/builds/.
    Choose the static build (prefer x64 bit if your machine supports it).


    Burp.
    You can get a (free) community license in here: portswigger.net/burp/releases/professional-community-2020-2-1?requestededition=community.
    It's a way we'll view the network-traffic,
    and it will help us to copy the needed-data in-order to download the video.
    I've used Fiddler. It sucked! (and sometime ago stopped working..),
    Chrome had issues with the specific HTML video-player I've needed to scrape (was blocked..),
    and Firefox worked by the nightly version I've used had the major issues with the network-inspection tab.


    Java.
    Burp needs Java.
    You can download the exe (if your operation-system is Windows),
    which probably includes a Java-runtime.
    Or you can download the Jar from the Burp-website (better) and use your own java.
    You can download JDK (which includes java-runtime) for Windows from here: https://github.com/eladkarako/JDK-archive.
    You'll probably be fine with just keeping the JRE folder, no need to install or configure anything,
    just be specific to where you run the java.exe from.


    A modern browser. You can try https://www.google.com/chrome/canary or Firefox nightly https://ftp.mozilla.org/pub/firefox/nightly/latest-mozilla-central-l10n/.


    Some kind of notepad (Notepad++ or Notepad2)


    About 10-minutes to set everything up for the first time.
    It can be done in 5-minutes or less afterwards.


  2. Firefox sucks. But I prefer it for this task.

    I'll explain.

    Burp (the network-traffic-inspection-tool) creates a proxy on your machine,
    it uses the 127.0.0.1 address and (by default) the 8080 port.

    To use Burp you'll need to set the proxy for the browser to that address and port.

    While Chrome is great, it (sadly) uses the system proxy,
    which means a system wide switch, and possibly a lot of traffic directed.

    Firefox has "its own" proxy settings.

    Secondly, Certificates!
    to inspect an SSL connection and traffic,
    you'll need to make the browser trust the certificate that comes with the website,
    Burp replaces the certificate (and decodes the secure connection so you'll be able to inspect it..).

    Again, Chrome uses the system-wide certificate installing (which means installing a certificate on your operation system, which is not always idle..),
    while Firefox - Has its own certificate trusting system.

    Nice.




    Open Firefox and goto options, preferences, connection settings, manual proxy configurations, put in: 127.0.0.1 and 8080, click OK.

    You won't be able to browse any-website until you'll provide an actual proxy running in that port,
    so in the next step we'll be opening Burp for the first time.

    Note:
    If you're trying to automate the scraping process you might want to look on
    pref("network.proxy.type", 1); if you're using your own user.js in the Firefox profile,
    (or user_pref( for the prefs.js generated by the profile (although you should not edit it, probably) or lockPref( if you're using mozilla.cfg in the Firefox-execute folder).
    also:
    pref("network.proxy.backup.ftp", "127.0.0.1");
    pref("network.proxy.backup.ftp_port", 8080);
    pref("network.proxy.backup.ssl", "127.0.0.1");
    pref("network.proxy.backup.ssl_port", 8080);
    pref("network.proxy.ftp", "127.0.0.1");
    pref("network.proxy.ftp_port", 8080);
    pref("network.proxy.http", "127.0.0.1");
    pref("network.proxy.http_port", 8080);
    pref("network.proxy.ssl", "127.0.0.1");
    pref("network.proxy.ssl_port", 8080);


  3. You'll need to open up Burp.
    if you've downloaded the JAR-file just run Java -jar "burp_v2.1.jar".

    Change the path and the file name to match where you've placed the JAR-file and your JAR-filename.
    You'll start with a dialog-box, go ahead and click the "Next"-button (keep the option "Use Burp defaults") and click the "Start Burp"-button.

    From the "Dashboard" tab, switch to "Proxy" and click the "Intercept is on"-button,
    it will turn it off, we don't need it.

    Switch to "Target" tab.
    (when you'll start browsing you'll be getting some resource-tree filled up there).


  4. This is needed to be done just one time:

    In your browser of choice (Firefox!), browse http://burp,
    click "CA Certificate" on the top right and download the file to your desktop.

    Go to "options", "privacy&security", "view certificates", select "authorities", click "import".
    Browse and select the certificate-file from your desktop, check-ON all the checkboxes, trusting the certificate (SSL data from HTTPS-secured websites can be viewed now).


  5. Browse somepage with a streaming player,
    for example: https://13tv.co.il/item/news/haolam-haboker/season-01/clips/s7m7l-2011988/?webview=true,
    click the play button, switch to full screen, hover your mouse and click the gear-button, select HD-quality (you need to press play and switch to full-screen in order to switch quality in that specific player.. it's buggy!).

    Wait couple of seconds and we're done with the browser,
    all the data we need is in the Burp-program,
    pause the video, close the tab, go ahead to "Options", "Connection Settings", select "Use system proxy settings" and click OK. You can close Firefox.


  6. View your Burp program,
    on the "Target" tab, select the "Site Map" tab (if not already selected).

    See reshet-vod-il.ctedgecdn.net? click once, selecting it,
    now press and hold your RIGHT keyboard arrow,
    it will open-up the sub-tree(s) until it will get to the files.

    We are looking for a playlist file (ends with m3u8) that has the highest-resolution,
    in this case: chunklist_b1400000.m3u8 (if you're not seeing it, you'll need to go back to the browser and select the HD - high-resolution in the gear-button).

    Right click the chunklist_b1400000.m3u8 line,
    click "Copy as curl command".

    Open up notepad (Notepad++ or Notepad2 will do fine, Windows' Notepad not so much..),
    paste the content of the clipboard on to the notepad.


  7. You can close down Burp, we've got our data.


  8. we'll need two things, the cookie content which usually looks like this: Cookie: Ccookie=0x9c2c387ccf2578a3bc517b0de44e86e76851e9402fb913960396c704d7f4a423&?ctoken=&str=1587779070&exp=1587780270 (we don't need the wrapping ' characters!) and the URL itself which usually looks like this: https://reshet-vod-il.ctedgecdn.net/reshet-vod/_definst_/amlst:mediaroot/nana10/media/iiscdn/2020/04/05/news-haolam-haboker-season-01-clips-375-part-06_,550,850,1400,.mp4/chunklist_b1400000.m3u8

    Open up a console window,
    if you're using Windows, use the Win+R combination,
    write down cmd and press Enter,
    You'll be getting into the folder of your Windows-user-profile.
    Type: cd Desktop and press Enter.

    Finally - we're going to download the video-content.

    Here is the FFMPEG-command:
    ffmpeg.exe -headers "Cookie: Ccookie=0x9c2c387ccf2578a3bc517b0de44e86e76851e9402fb913960396c704d7f4a423&?ctoken=&str=1587779070&exp=1587780270" -i "https://reshet-vod-il.ctedgecdn.net/reshet-vod/_definst_/amlst:mediaroot/nana10/media/iiscdn/2020/04/05/news-haolam-haboker-season-01-clips-375-part-06_,550,850,1400,.mp4/chunklist_b1400000.m3u8" -c copy "myvideo.mkv"

    Go back to your notepad and copy the actual cookie-value, and URL for your video.
    You may choose any filename for your video, but make sure to use the .mkv-extension,
    press Enter to execute the command and wait until FFMPEG will finish downloading and mergin the video file. It will be done as fast as FFMPEG can download it, the "joining" part is done automatically and it will not encode the video (see -codec copy), so it take very little CPU power and no-additional processing-time.


  9. Tips:
    Say you'll be uploading the video to YouTube,
    have you noticed its resolution?

    You can open your desktop as a folder by running the command C:\Users\YOUR-USER-NAME\Desktop\,
    and if you have the details pane opened it will show you some of the video-properties.
    You can also open the video in VLC or Media-Player-Classic (https://codecguide.com/download_kl.htm) and look for the properties.

    This praticular video is "near HD" resolution,
    its width is 960 and its height is 540, once you'll upload it to YouTube,
    the nearest resolution in the YouTube player is 720p,
    which is a shame, since the current resolution is much better.

    There is "a cheat" in which we'll resize the video, not much,
    until it will get to width of 1920 and height of 1080,
    this will allow the users to view the video at 1080p,
    although the official-YouTube app on smartphones won't usually allow the 1080p (try https://github.com/TeamNewPipe/NewPipe) but YouTube on smart-TVs will!

    You don't have to do this step if you're fine with the video file "as is",
    and we're have to re-encode it, which means that ("officially") some quality will be lost,
    although you won't see it.

    We're going to use h.265 encoding which will result a half of file-size that would be in the old h.264 encoding (although it can take a lot of CPU and about 15-30 minutes of encoding time) - but at least your upload time will be very short.

    Also we're going to smooth the picture a bit with the built-in hqdn3d video-filter,
    it usually helps the video-compression process and removes noise and pixelations.

    ..also we're going to make sure the input video has (in this case it is already is) a proper 16:9 ration of width to height, but we'll pad it (with black background) in case needed before the scale-up so the original proportions will be kept.
    Next we'll use the scaling filter with -2 this makes sure the frame size is divisable by 2 as required by the MP4-frame size, with the proportion we gave before this isn't needed but this line is beautifuly-generic and I don't want to change it..

    Here is the command:
    ffmpeg -hide_banner -y -i "myvideo.mkv" -c:v libx265 -crf 26 -preset ultrafast -c:a copy -vf "fifo,pad=width=ih*16/9:height=ih:x=(ow-iw)/2:y=(oh-ih)/2:color=#00000000,scale=-2:1080,hqdn3d" "myvideo_hd.mkv"

    Keep in mind to specify the path to FFMPEG and your input and output file.

    Note that CRF 28 (for h.265) is like CRF 23 (for h.264) quality wise, but half the size.
    I've went to 26 to preserve more quality (will make the file-size a bit larger).

    If you need a smaller file-size, you can change the CRF switch to 30 (lower qulity, smaller file-size), and (if you don't care about waiting an hour+) switch to PRESET of veryslow (don't change quality, just lowers the file-size).

    If you have a computer with a strong graphic card, you might want to try using some GPU-processing,
    which help you to lower the CPU while processing.
    It is done by adding -hwaccel dxva2 before the -i "myvideo.mkv". It works well on most Windows, but if you have a Lenovo x220 (like me) or similar (where the GPU is part of the CPU, named something like "Intel 3000") don't bother, it won't get you any better result. If you DO have a proper-GPU you can try it (browse https://trac.ffmpeg.org/wiki/HWAccelIntro for information regarding Intel, nVidia and AMD GPUs)


    Here are some links about the video-filters:
    http://ffmpeg.org/ffmpeg-filters.html#fifo_002c-afifo
    http://ffmpeg.org/ffmpeg-filters.html#hqdn3d-1
    http://ffmpeg.org/ffmpeg-filters.html#scale-1

    And about h.265 encoding:
    https://trac.ffmpeg.org/wiki/Encode/H.265