Simple Introduction#
Expand
Overall Plan#
Expand
The plan is based on existing tutorials available on the internet, using virtualization to build a personal streaming media service for practice, aiming to achieve automated watching while learning some knowledge related to virtualization and servers. The overall framework of the streaming media service is as follows:
- proxmox supports virtualization operation
- docker or k8s serves as the foundational platform for service operation
- qbittorrent serves as the downloader
- jackett acts as the subscription indexer
- radarr and sonarr play the role of movie and series scrapers
- overseerr provides scraping integration services
- plex and emby are responsible for media center services
- bazarr matches subtitle information
The above is based on an X99 platform 2U server, of course, it doesn't have to be a rack-mounted server; other DIY devices can also implement this solution as long as there are enough cores and sufficient memory to handle virtualization. The projects used can all be found in the world's largest open-source community.
Flowchart#
Q&A:#
Expand
Q: Why not use NAS products like unraid, black 群,openmediavault, turnas, etc.?
A: It must be admitted that NAS products have better ecological support, a complete operating experience, and relatively low learning costs, especially black 群 unraid. However, using NAS products reduces some playability 😑. Choosing virtualization, although more cumbersome, allows for various problems encountered and the process of solving them, which I believe is the greatest joy of tinkering, and it can also yield a lot of knowledge in server operation and maintenance 🤩.
Q: What are the pros and cons of this solution in terms of viewing experience?
A: The advantage is that it is relatively simple to configure, and the interface is beautiful and easy to operate. Once perfected, you only need to click request on overseerr for the shows you want to watch, and it will be pushed to qb for download. No other operations are needed throughout the process, but the disadvantages are also obvious. Sonarr does not support Chinese, leading to anime, domestic, and Japanese and Korean dramas being renamed in English, and overly long file names are not very aesthetically pleasing when displayed on emby. Newer resources can be effectively supported, but if there are no PT site resources, it is difficult to find older film and television resources. (This problem exists in any solution 😟)
Q: What about other extensions?
A: For example, downloaders like utorrent can be extended, and media centers can add kodi, jillyfin, etc. You can also achieve mobile subscriptions through webhook message pushing. For anime series, you can extend subscription methods through AutoBangumi, etc. In short, there is still a lot of play space; this solution is just a foundation or a start, and there is much exploration space to come 😉.
Q: Other automation solutions?
A: The core experience of automated viewing lies in scraping video, allowing the media center to recognize film information. The focus of scraping is on renaming video files or their parent directories. From these perspectives, tools like nastools, tMM, etc., can serve as alternatives. For subtitles, ChineseSubFinder can be a substitute. In terms of virtualization, proxmox can also use exsi as a replacement.
Virtualization Underlying System#
Some notes about proxmox
Expand
- Image file download address: proxmox VE
- USB boot tool: Rufus or Ventoy, the latter is more recommended
- Proxmox does not have a default operation page; you can control it via the web interface using the connection provided after system installation, defaulting to https://ip:8006/
- Change source
Enterprise source
## Remove the built-in source rm /etc/apt/sources.list.d/pve-enterprise.list ## Add the official non-subscription source (choose one between this and the domestic non-subscription source below) echo 'deb http://download.proxmox.com/debian/pve bullseye pve-no-subscription' >> /etc/apt/sources.list.d/pve-no-subscription.list ## This is the domestic non-subscription source echo 'deb http://mirrors.ustc.edu.cn/proxmox/debian/pve bullseye pve-no-subscription' >> /etc/apt/sources.list.d/pve-no-subscription.list
Software source
## Edit source file: /etc/apt/sources.list cp /etc/apt/sources.list /etc/apt/sources.list.bak && vim /etc/apt/sources.list ## Delete all content in the file or comment it out with #, and add the following content deb https://mirrors.aliyun.com/debian bullseye main contrib deb https://mirrors.aliyun.com/debian bullseye-updates main contrib deb https://mirrors.aliyun.com/debian-security bullseye-security main contrib
- Multiple disks can be configured as raid5 to provide sufficient storage space for virtual machines while ensuring relatively reliable data storage.
Virtual Machine#
About managing Linux and necessary configurations in the plan
Expand
At least 3 virtual machines need to be created:
157.20.20.1 --> jumpserver --> Bastion host for remote operation and maintenance
157.20.20.2 --> nfs --> NFS file sharing system
157.20.20.3 --> docker --> Dedicated container for deploying streaming media
If choosing the k8s method, at least 2 more virtual machines need to be virtualized:
157.20.20.3 --> k8s-master --> k8s master node
157.20.20.4 --> k8s-node01 --> k8s worker node
157.20.20.4 --> k8s-node02 --> k8s worker node
Among them, the jumpserver needs at least 4C8G of resources, while other virtual machines should allocate at least 2 cores, with memory allocated as needed, without special requirements. If it is a k8s environment, the master node needs at least 2C6G, and the node nodes need at least 2C4G.
The reason for creating multiple virtual machines: to divide space by function, avoiding all components becoming unavailable due to improper operations leading to server file damage. Once this happens, repair and maintenance will become exceptionally troublesome.
Creating Virtual Machines#
ISO images need to be uploaded to pve's local storage for installing the basic operating system.
Various basic image download addresses: CentOS7, Rocky Linux, Debian
Click "Create Virtual Machine" in the upper right corner and select the appropriate image for installation.
Installing Jumpserver on the Virtual Machine#
Simple Introduction
Jumpserver is an open-source bastion host product under Feizhi Cloud, which is much more convenient than various ssh and scp tools. Official website
Deployment
curl -sSL https://resource.fit2cloud.com/jumpserver/jumpserver/releases/latest/download/quick_start.sh | bash
████████████████████████████████████████ 100%
[Success]: download install script to /opt/jumpserver-installer-v3.2.0
[Info]: Start executing the installation script.
[Info]: In an automated script deployment, note the message prompts on the screen.
████████████████████████████████████████ 100%
[Success]: The Installation is Complete.
For more commands, you can enter jmsctl --help to view help information.
After installation, access it in the browser at http://157.20.20.1 and log in with the default account admin/admin.
Creating the First Asset
Add assets in Asset Management --> Asset List.
Create authorization rules for user asset authorization in Permission Management --> Asset Authorization.
After authorization is complete, you can enter the console in the upper right corner to log in to the virtual machine and perform other operations. The flexibility and convenience are much higher than ssh tools. At this point, the jumpserver deployment is complete.
Creating NFS Shared File Server#
Create a second Linux virtual machine and allocate sufficient disk space. Plan data space and media space in advance to facilitate management and avoid confusion.
Create media and storage folders in the system, respectively serving as storage space and media space, and mount the disks properly.
## Confirm the space to be mounted
$ lsblk
## Create media and storage folders
$ cd /mnt && mkdir media && mkdir storage
## Format and mount the disks to the folders for the operating system to access
## File system formats can be xfs or ext4, etc.
$ mkfs.xfs /dev/sdb
$ mkfs.xfs /dev/sdc
$ mount /dev/sdb /mnt/storage && mount /dev/sdc /mnt/storage
## Modify /etc/fstab to ensure automatic mounting at boot
$ vim /etc/fstab
/dev/sdb /mnt/storage xfs defaults 0 2
/dev/sdc /mnt/media xfs defaults 0 2
Deploy the NFS server, taking Rocky Linux as an example.
- Install NFS server components:
$ yum install nfs-utils
- Set permissions for the directories to be shared:
$ chmod 777 /mnt/storage
$ chmod 777 /mnt/media
- Configure NFS access permissions in the file /etc/exports. For example:
$ vim /etc/exports
/mnt/storage 157.20.20.0/24(rw,sync,no_subtree_check)
/mnt/media 157.20.20.0/24(rw,sync,no_subtree_check)
## This will provide read-write access to clients in the 157.20.20.0/24 subnet
- Start the NFS service:
$ systemctl start nfs-server
$ systemctl enable nfs-server # Set to start automatically at boot
- Open the NFS connection ports in the firewall:
$ firewall-cmd --permanent --add-service=nfs
$ firewall-cmd --permanent --add-port=2049/tcp
$ firewall-cmd --reload
- Test the NFS configuration:
$ showmount -e localhost
/mnt/storage 157.20.20.0/24
/mnt/media 157.20.20.0/24
Tip
If you modify the contents in exports, you need to re-export the exports as follows:$ exportfs -r $ systemctl restart nfs-server
### Docker Server
Create a third Linux virtual machine and deploy the NFS server, taking Rocky Linux as an example.
```bash
$ yum install -y yum-utils device-mapper-persistent-data lvm2
$ yum-config-manager --add-repo https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
$ yum makecache fast && yum -y install docker-ce
$ systemctl start docker && systemctl enable docker
Upload the docker-compose file downloaded from GitHub through the bastion host's file management. Download link
The uploaded file is located in /tmp
by default.
$ cp /tmp/docker-compose-linux-x86_64 /usr/local/sbin/docker-compose
$ chmod +x /usr/local/sbin/docker-compose
After completion, you can check the versions of docker and docker-compose.
$ docker version
$ docker-compose version
Quick Deployment of k8s Environment#
Open-source project: KuboardSpray-github
This project can be directly deployed on the jumpserver virtual machine. Note that port 80 is already occupied, so you need to change the port when mapping.
docker run -d \
--restart=unless-stopped \
--name=kuboard-spray \
-p 8081:80/tcp \
-e TZ=Asia/Shanghai \
-v /var/run/docker.sock:/var/run/docker.sock \
-v ~/kuboard-spray-data:/data \
eipwork/kuboard-spray:latest-amd64
# If you can't grab this image, you can try this backup address:
# swr.cn-east-2.myhuaweicloud.com/kuboard/kuboard-spray:latest-amd64
Access it via http://157.20.20.1:8081, with the default account admin/Kuboard123.
Using kuboard-spray, you can manage nodes from 157.20.20.3 to 157.20.20.6 (specifically see which virtual machines to use as k8s environment nodes).
It is worth noting that a single server can only simulate a k8s cluster for testing and learning purposes. A real k8s cluster should have multiple master nodes and worker nodes distributed across different physical servers to ensure that a single machine's downtime does not affect the cluster's operation.
For detailed installation and deployment references, see the kuboard-spray official documentation.
Optimization reference:
To be updated
Downloader#
About deploying and using qbittorrent via docker or k8s
Expand
Deployment#
Docker Method#
Mount the shared folder to the local machine.
## Confirm if it can be mounted
$ showmount -e 157.20.20.2
## Mount
$ cd /mnt && mkdir media && mount -t nfs 157.20.20.2:/mnt/media /mnt/media
$ mkdir storage && mount -t nfs 157.20.20.2:/mnt/storage /mnt/storage
## Set fstab for automatic mounting at boot as mentioned earlier
Deploy qbittorrent via docker.
docker run -d \
--name=qbittorrent \
-e PUID=1000 \
-e PGID=1000 \
-e TZ=Asia/Shanghai \
-e WEBUI_PORT=8080 \
-p 8080:8080 \
-p 49152:49152 \
-p 49152:49152/udp \
-v /mnt/storage/qb-config:/config \
-v /mnt/media/qb-downloads:/downloads \
--restart unless-stopped \
lscr.io/linuxserver/qbittorrent:latest
The following points need to be noted in the above code:
- The default port is 6881, which generally does not need to be changed. Most PT sites ban port 6881, so it can be changed to 49152 or others.
- The path set in
-v /mnt/storage/qb-config:/config
must be created before using the command to deploy. - Replace the path
/mnt/storage/qb-config
in-v /mnt/storage/qb-config:/config
with the desired qbittorrent configuration path. All modifications will be saved, and if the container crashes, as long as the configuration is not lost, restarting the container will suffice. - The same modification is needed for the
/mnt/media/qb-downloads
part in-v /mnt/media/qb-downloads:/downloads
.
Tip
The key points during docker deployment are the ports and mount paths. Just find the image on docker hub and modify accordingly.
The subsequent text will not elaborate on the modification methods during docker deployment; the basic points are the same as those in this section.
k8s Deployment of qbittorrent#
Deploy qbittorrent through the kuboard panel.
Find a suitable namespace (you can manually create one) to create the workload.
Configure container information.
Mount storage.
Publish the application.
After saving, wait for the container logs to indicate success, and you can access it through the defined published port. For example, in the image above, port 30001 is mapped to the container's 8080 port, so you can access it via http://157.20.20.3:30001. The default account for qb is admin/adminadmin.
Basic Settings for qb#
In the upper navigation bar of the qb page, there is a small gear icon, which is the entry for qb settings.
- Regarding the Download option, in the save management section, you need to set the part in the red box in the image below. The first setting is related to sonarr and radarr; when performing hard links through sonarr and radarr, the folder will be modified. Setting it to manual allows you to take over the download folder settings. The second function is to extract and back up the downloaded file's torrent. Usually, downloaded media files do not come with torrent files, so this setting can back up the torrent to a specified folder.
- In the BitTorrent tab, if you are using the BT method to search for resources on sites like EZTV, RARGB, ACG.RIP, etc., checking "Enable DHT (decentralized network) to find more users" will yield better results. If you are using PT sites, you can uncheck it.
The role of the tracker server is to help users increase download speeds and search for more user nodes. Tracker servers can be copied and pasted from trackerslist.com.
- In the Advanced tab, checking the two tracker-related options helps connect to more tracker servers, thus increasing speed.
Scraping#
Regarding the scraping method, this solution uses sonarr and radarr to create hard links for the downloaded resource files. The principle is to rename the files so that media servers like emby and plex can connect to tmdb, imdb, and other internet movie databases to search for corresponding information, thus displaying the poster wall, synopsis, actor information, etc., in the media library. Among them, sonarr corresponds to TV series scraping, radarr corresponds to movie scraping, and sonarr and radarr can index public BT sites through jackett to achieve local searches for internet series, enabling qb to automatically download and scrape.
Expand
Deployment#
This section only introduces the docker deployment method. The deployment methods and notes for jackett, sonarr, and radarr are the same as the qb deployment method mentioned earlier. The k8s deployment method refers to the qb deployment method.
Tip
Remember to create the corresponding folders.
jackett#
docker run -d \
--name=jackett \
-e PUID=1000 \
-e PGID=1000 \
-e TZ=Asia/Shanghai \
-e AUTO_UPDATE=true \
-e RUN_OPTS= \
-p 9117:9117 \
-v /mnt/storage/jackett-congfig:/config \
-v /mnt/media:/downloads \
--restart unless-stopped \
lscr.io/linuxserver/jackett:latest
sonarr#
docker run -d \
--name=sonarr \
-e PUID=1000 \
-e PGID=1000 \
-e TZ=Asia/Shanghai \
-p 8989:8989 \
-v /mnt/storage/sonarr-config:/config \
-v /mnt/media/tv:/tv \
-v /mnt/media/anime:/anime \
-v /mnt/media/qb-downloads:/downloads \
--restart unless-stopped \
lscr.io/linuxserver/sonarr:latest
radarr#
docker run -d \
--name=radarr \
-e PUID=1000 \
-e PGID=1000 \
-e TZ=Asia/Shanghai \
-p 7878:7878 \
-v //mnt/media/radarr-config:/config \
-v /mnt/media/movies:/movies \
-v /mnt/media/qb-downloads:/downloads \
--restart unless-stopped \
lscr.io/linuxserver/radarr:latest
overseerr#
docker run -d \
--name=overseerr \
-e PUID=1000 \
-e PGID=1000 \
-e TZ=Asia/Shanghai \
-p 5055:5055 \
-v /mnt/storage/overseerr-config:/config \
--restart unless-stopped \
lscr.io/linuxserver/overseerr:latest
Settings#
jackett#
Access jackett via http://ip:9117 and click Add indexer to add.
sonarr#
Access sonarr via http://ip:8989,
In settings --> media management, set the media folder monitoring. Sonarr will automatically hard link media files from the downloaded series folder to this location.
In settings --> profiles, set the default language.
In index, set the default indexer, where you will need the API key from jackett. The added subscription index format is Torznab.
In settings --> connect, set the media center, where you will need to obtain the API key from emby. This part can wait until emby is deployed.
In settings --> download clients, set the connection to the qb downloader and create the corresponding monitoring directory.
In settings --> metadata, set the metadata retrieval information.
Once the above settings are complete, you can set up searches and monitoring for series in the Series section. It will also notify qb to download when monitoring is added, and after downloading, it will automatically hard link to the target folder for scraping.
radarr#
Radarr's basic settings are similar to sonarr, but radarr can be set to Chinese.
Log in via http://ip:7878, and in settings --> UI, you can set the page to Chinese.
In media management, adding "" can set the renaming to Chinese directories.
Overseerr#
Overseerr is a beautiful and powerful media management tool, which can be regarded as a combination of radarr and sonarr, managing them through API keys. After deployment, you will enter the guided setup page, where the API keys used can be found in the settings --> general section of sonarr and radarr.
Important points to note:
- This software relies on the plex server, so please deploy the plex server first.
- In general settings, choose to display the language as Chinese and the region as all.
- TV and anime can be added separately in the server.
- Media scraping heavily relies on TMDB access, so please set up a global proxy, for example, through a soft router for proxy access. If the proxy is set within the virtual machine where overseerr is located, please add
-e https_proxy=http://172.17.0.1:7890
during the docker deployment process; otherwise, overseerr will not be able to set the proxy.
Media Library (emby and plex)#
Expand
Refer to the deployment methods for emby and plex on DockerHub for reference. Here are examples:
emby#
docker run -d \
--name=emby \
-e UID=1000 \
-e GID=1000 \
-e TZ=Asia/Shanghai \
-p 8096:8096 \
-p 8920:8920 \
-v /mnt/storage/emby-config:/config \
-v /mnt/storage/emby-lib:/lib \
-v /mnt/media:/media \
--restart unless-stopped \
lscr.io/linuxserver/emby:latest
plex#
docker run -d \
--name plex \
--network=host \
--restart=always \
-p 32400:32400/tcp \
-p 8324:8324/tcp \
-p 32469:32469/tcp \
-p 1900:1900/udp \
-p 32410:32410/udp \
-p 32412:32412/udp \
-p 32413:32413/udp \
-p 32414:32414/udp \
-e TZ="Asia/Shanghai" \
-e PLEX_CLAIM="plex_clam_code" \
-e ADVERTISE_IP="http://ip:32400/" \
-e PUID=1000 \
-e PGID=1000 \
-v /mnt/storatge/plex/database:/config \
-v /mnt/storatge/plex/transcode:/transcode \
-v /mnt/media:/data \
plexinc/pms-docker
The plex_clam code can be obtained from the page: https://www.plex.tv/zh/claim/
Tip
Plex is not open-source media library software; its account management requires registration on the official website, and the plex_clam_code is obtained through the account.
Subtitle Section#
During the download process, some films may not have embedded subtitles, which can make understanding difficult if they are in a foreign language. Configuring a subtitle server can solve this problem.
Expand
Deployment#
docker run -d \
--name=bazarr \
-e PUID=1000 \
-e PGID=1000 \
-e TZ=Asia/Shanghai \
-p 6767:6767 \
-v /mnt/storage/bazarr-config:/config \
-v /mnt/media/movies:/movies \
-v /mnt/media/tv:/tv \
-v /mnt/media/anime:/anime \
--restart unless-stopped \
lscr.io/linuxserver/bazarr:latest
Settings#
The general settings in bazarr are basically the same as those in radarr and sonarr.
The difference is that there are three places where you need to set the API key. The API keys for sonarr and radarr can be obtained from their respective pages, while the subtitle download API key needs to be obtained by registering an account on the corresponding subtitle website. I am using 射手网 (伪) for this, with the registration link provided in the text.