I'm new here and don't know much about DLNA/UMS/media encoding, but do know my way around Linux a bit.

I am running a headless UMS server from a Raspberry pi 3b (Ubuntu 16.04), streaming to a Playstation 4 (most updated software version). I have 500+ movie files, the vast majority of which work flawlessly (much better than plex, I must say!!) with zero transcoding, as PS4 only accepts H.264 encoding and that is what most of my media is.

However, there are some .mp4 files which do not play on PS4 (mostly TV series) which are also seemingly H.264. With no transcoding, the PS4 says they do not play. When I try any of the available default transcoders (2 or so are shown on the Ps4 for every file), they still don't work. Specifically, the transcoders available by default are FFmpeg and MEncoder.

This is kind of perplexing to me because both files are written with x264, but obviously I'm missing some thing(s). I am not sure what kind of transcoding settings I should set on my server to have the server encode the stream properly for the PS4. Does anyone have experience or expertise in doing this? Or advice for where I should start?

You need to start UMS with the "trace" command line argument, and then post the log file (debug.log) from when you try to play one of the files that don't work. This will show the decision process.

Regarding transcoding, I'm pretty sure that the RPi isn't anywhere close to have enough oomph to do that. If you want to be able to do transcoding, you'll probably have to use a "real computer" with a reasonable CPU as your server.

I think the PS4 only supports up to level 4.2 H.264, which means that at 1080p (see reference https://planetcalc.com/3321/), files are only allowed 4 reference frames. The "bad" file is using 6 reference frames, and the person who encoded it seems to have manually set the profile flag to "High L4.1".
In other words it really is a bad file

If it is a scene release it will have been nuked and a fixed version released. There are lots of other more standard files out there to choose from, I would recommend looking for the mkv filetype instead of mp4

You can reencode the file with FFmpeg, but you have to figure out the correct parameters. You can find tons of examples of FFmpeg parameters online. It will probably take a while though, depending on the power of your computer of course.

Personally I recommend using x264 https://www.videolan.org/developers/x264.html since it has good defaults - you can simply set a CRF (Constant Rate Factor) and leave everything at its default settings, and you'll get a high quality file. For 1080p I'd go with CRF 18-22, with 18 being highest quality and 22 being lowest, but still barely noticeable.
You can then use MKVToolnix to mux (muxing is when you swap streams across files without re-encoding them) the new video stream with the old audio stream, and end up with a .mkv file that will work https://mkvtoolnix.download/

I use a tool that automates those steps called MeGUI but it is Windows-only

Just to clarify things, I'm using level 4.2 with 5 frames without problems, here is what can be played natively on PS4 (from Sony website)

Videos

When using a USB storage device, your video files need to be in a folder for your PS4™ system to recognize them.
You can view video recorded by a 360-degree omnidirectional camera (in equirectangular video format) on your PS VR. Press the OPTIONS button, and then select [VR Mode].