From Pfsense site:
"This project started in 2004 as a fork of the m0n0wall project, but focused towards full PC installations rather than the embedded hardware focus of m0n0wall.

I do NOT want IDS with zillions of false positives and daily hours of babysitting, nor do I want useless antivirus such as ClamAV - embedded or not. Kindly install those yourself if you find a need for them.

Hi
It will be great If you add IDS and Antivirus packages in default installation or a user can select custom packages during installation.

That is already possible with Menu>>Systems>>Packages.

Just install the packages you want and configure according to your needs, viz snort, HAVP, squid, squidguard, pfBlocker among others.

From Pfsense site:
"This project started in 2004 as a fork of the m0n0wall project, but focused towards full PC installations rather than the embedded hardware focus of m0n0wall.

Quoting something from almost the beginning of this century. A lot of water has flowed under <name your="" prefferred="" river="" here="">since then. With PHK's NanoBSD script commit to FreeBSD, it added a new dimension which became a part of pfSense now!</name>

There is a line between required features and base system bloat. Packages address needs beyond the boundary.

IDS and AV require frequent updates to stay current and functional. The base system stays stable for many months at a time. Packages allow those to be updated frequently and kept out of the base system.

I would not want to roll a whole new firmware release just because snort decided to change their rule format (again) or because clamav had a CVE or DB format change (again).

I think basic functions such as Squid and Unbound should be part of the feature set instead of packages that work sometimes.

Maybe time to fork off an embedded hardware version so Pfsense can move forward with more energy without this baggage?

Then you also shift the development burden of those away from the community helping with packages to the core team which is already busy keeping the main system in line. And see above, re: slow updates.

These features work best as packages. If they only "work sometime", then take it up with the package maintainers. Squid 2.7.x and SquidGuard work perfectly all the time in thousands of installs.

The quality of a function is not dependent on whether it's a package or in the base system.

I second jimp because it is always better to separate the base OS from additional whisltes and bells. jimp is right to point out that the base system remains stable for a longer period of time than the packages.

One can install additional whistles and bells on top of base OS as an when required. The most exciting thing about FreeBSD is PHK's NanoBSD scripts which pfSense seemed to have adopted to separate the OS from packages (although there doe snot seem to have adequate guidelines for packages (pbi in particular).

The only problem I found was when one wants to make changes to kernel level parameters.

PS: @jimp: is dev mailing list active? I am asking because I encountered some problems and posted that to the dev list a few days ago, and yet there does not seem any activity. ;-)

Plus Cisco just bought Sourcefire. Which means snort and clamav are now both Cisco products. Looks like we need to find replacements ASAP. :-)

Not a good news! :( However, checked seclists and sourcefire claimed that it shall remain open sourced, (http://seclists.org/snort/2013/q3/363). Since snort and clamav are GPLed, I guess they Cisco is required to release further changes, don't they?

@Zenny - dev list is hit and miss. Lots of us are quite busy and don't have a ton of time to respond to things, but depending on what it was, the Dev list isn't always the best place for it.

From Pfsense site:
"This project started in 2004 as a fork of the m0n0wall project, but focused towards full PC installations rather than the embedded hardware focus of m0n0wall.

I do NOT want IDS with zillions of false positives and daily hours of babysitting, nor do I want useless antivirus such as ClamAV - embedded or not. Kindly install those yourself if you find a need for them.

Just because some functionality is well integrated, tested and part of the distribution doesn't mean you have to use it and enable it.
There's plenty of functionality in pfSense that I don't use, and a bunch of packages that sort of bite each other in unpredictable way, because there are no "standard" packages and the various interactions are not part of the testing.

There may be other valid reasons not to include package <x>as part of the distribution, but the fact that n% of users won't use it or don't need it, isn't one of them.</x>

The quality of a function is not dependent on whether it's a package or in the base system.

Quite true, but the integration needs to be better. e.g. naming conventions: the base system names the menus usually BY FUNCTION, while packages tend to name the menues BY PROJECT NAME, which frankly is annoying.

Similarly, the packages as we have them now, do not signal in any way what is mutually exclusive, e.g. Dansguardian does not exclude HAVP and/or SquidGuard from being installed, which is something that could be solved with meta packages that are sorted by function, such that one can install only one IDS/IPS system, only one content filter, etc.

It's probably the majority of users that have one or more packages installed, but unfortunately, due to a lack of conventions and/or a system of enforcing these conventions, the fit and finish of the entire installation quickly takes a nose dive once a few packages are installed, which makes it feel inferior to other products, just because things get inconsistent, not because things really are inferior in substance.

Plus Cisco just bought Sourcefire. Which means snort and clamav are now both Cisco products. Looks like we need to find replacements ASAP. :-)

Suricata would seem to be a good choice, seems to also scale better and have other advantages, while still being able to work with Snort rules, if required.
ClamAV can be forked, if need be, but of course updating the rules may become an issue after a while.

Just because some functionality is well integrated, tested and part of the distribution doesn't mean you have to use it and enable it.
There's plenty of functionality in pfSense that I don't use, and a bunch of packages that sort of bite each other in unpredictable way, because there are no "standard" packages and the various interactions are not part of the testing.

There may be other valid reasons not to include package <x>as part of the distribution, but the fact that n% of users won't use it or don't need it, isn't one of them.</x>

Actually, it is a perfectly valid reason. We recently removed OLSRD and RIP (routed) from base for that very reason. They were better as packages, they were no longer used by enough people to justify keeping them in the base system.

Quite true, but the integration needs to be better. e.g. naming conventions: the base system names the menus usually BY FUNCTION, while packages tend to name the menues BY PROJECT NAME, which frankly is annoying.

That's no reason to put them in base. Just fix/change the menu names. But that's a Bikeshed discussion for another thread. Naming by function breaks the moment you have two packages that have the same function but different actual names, and you can't tell them apart.

And again, if a package doesn't work like you want, submit patches or convince the maintainer to fix/make changes. That does not have any relationship to it being in base.

Similarly, the packages as we have them now, do not signal in any way what is mutually exclusive, e.g. Dansguardian does not exclude HAVP and/or SquidGuard from being installed, which is something that could be solved with meta packages that are sorted by function, such that one can install only one IDS/IPS system, only one content filter, etc.

It could use some "conflicts" type checking but the examples you gave aren't always really conflicts. I'm sure someone has a valid reason for combining those in some weird way, and if we stopped them from being installed, something else would break.

It's probably the majority of users that have one or more packages installed, but unfortunately, due to a lack of conventions and/or a system of enforcing these conventions, the fit and finish of the entire installation quickly takes a nose dive once a few packages are installed, which makes it feel inferior to other products, just because things get inconsistent, not because things really are inferior in substance.

If you have some examples of this with non-beta/non-development packages, feel free to cite actual examples and not generalities that aren't really true. Lots of us run many packages with no problems. Certain specific packages may have issues, but again, that's a problem with the individual package's code that isn't going to magically get fixed by bringing it into the base system.

If you want better packages, then help the maintainers make them better, submit patches to make them better, or fund the maintainers to help them get better.

Actually, it is a perfectly valid reason. We recently removed OLSRD and RIP (routed) from base for that very reason. They were better as packages, they were no longer used by enough people to justify keeping them in the base system.

The biggest downside to me using a package as opposed to something built in by default is what happens when you upgrade a system. There is extra downtime while the package gets re-installed. With CARP enabled for HA and you upgrade your primary server it will take over after it reboots but before packages are done upgrading/reinstalled. This is one of the reasons I prefer not to use haproxy. Snort, openvpn client export, etc and non-critical packages like that are acceptable to come later but important routing and loadbalancing packages are critical functions.

A way to help with that scenario though would be if there was an easy way to manually disable CARP on the primary that would survive reboots. Better yet would be for the firewall to know when it is done with upgrades and just delay CARP failover back to primary automatically. I would actually prefer the manual method though to make sure that all the packages get upgraded properly before putting load on it. The automatic method though would just be a nice thing for people who don't want to worry about it but where hosted webserver,etc uptime is still very critical to them.

Actually, it is a perfectly valid reason. We recently removed OLSRD and RIP (routed) from base for that very reason. They were better as packages, they were no longer used by enough people to justify keeping them in the base system.

The biggest downside to me using a package as opposed to something built in by default is what happens when you upgrade a system. There is extra downtime while the package gets re-installed. With CARP enabled for HA and you upgrade your primary server it will take over after it reboots but before packages are done upgrading/reinstalled. This is one of the reasons I prefer not to use haproxy. Snort, openvpn client export, etc and non-critical packages like that are acceptable to come later but important routing and loadbalancing packages are critical functions.

A way to help with that scenario though would be if there was an easy way to manually disable CARP on the primary that would survive reboots. Better yet would be for the firewall to know when it is done with upgrades and just delay CARP failover back to primary automatically. I would actually prefer the manual method though to make sure that all the packages get upgraded properly before putting load on it. The automatic method though would just be a nice thing for people who don't want to worry about it but where hosted webserver,etc uptime is still very critical to them.

Which is still, technically, not a package system issue, but a general CARP issue.

Firmware upgrades, for most users, happen once or twice a year, or less often. A few minutes of downtime during a scheduled maintenance window for a firmware upgrade is not a huge issue for most people.

Yes, if you're tracking snapshots it's more often and more disruptive, but the fact of the matter is that is not how most people work.

Still, that's an issue to take up with the CARP procedures, it's not specific to things being a package really.

I agree with your reasoning. Upgrades don't happen often on production sites. When they happen for me I fail over to a backup site so it isn't such a big deal. I was somehow overlooking that seemingly obvious point.

I also failed to mention that main reason I prefer the built-in load balancer is because it doesn't drop connections during firewall fail-over where haproxy does. That again shouldn't happen often.

I also failed to mention that main reason I prefer the built-in load balancer is because it doesn't drop connections during firewall fail-over where haproxy does. That again shouldn't happen often.

The built-in balancer works because relayd works like NAT rules, so it fails over with the normal state table entries.

Until HAproxy grows a way to sync its internal connection states in a cluster-like fashion, there isn't anything we can do for that. There's a separate thread for that somewhere, I seem to recall posting about it in the last couple weeks. Usually web sessions are fleeting enough that it doesn't matter a ton that the states are gone.

More complex systems are more likely to break. I guess we don't need to discuss this.

Is IDS neccessary? For most of us, no.

Of course, everyone has his favouriet package "which should be in the base system". Yup, if it's in the base system it gives the user the illusion of real solid support. However, when thinking again about illusions and additonal complexity, I'd rather prefer the modular approach, where I can simply remove addons which begin to act up.

More complex systems are more likely to break. I guess we don't need to discuss this.

Is IDS neccessary? For most of us, no.

It all depends how you use pfSense. e.g. if I tried to program the firewall, it would be a fairly insane effort for a little home network, because there's no starting point, and the number and kinds of software people run these days isn't anything like the old days where you'd open up the mail, DNS and SSH ports, and shut everything else.
We have things like dropbox, drobo, back-to-my-mac, iCloud, email, DNS, VoIP, Skype, P2P, Tor, RDP, VNC, Apple Remote Desktop, remote server admin tools, push services, photo streams, proprietary protocols of ever more license demons, web apps, etc.
I'd end up with a mess of rules, and probably still a bunch of things that don't work quite well.

Or I can use SNORT's blocking features, which simply selectively block hosts for a few hours if they do things that look suspicious, and features like the ssh lock-out feature that block a host after a certain number of failed login attempts, and pfBlocker with some list subscription of known bad hosts in combination with something like Apple's application firewall, which isn't ports/protocol based, but decides which apps can accept incoming network traffic.

As things get more and more internet driven, old school firewall programming gets more and more difficult, and smart, behavior driven thread detection/blocking more important.

As such I think IDS/IPS along with crowd-sourced blocking lists are the future of firewalls, and while rule based firewalls work well for systems that are essentially shut out of the internet except for a very few carefully poked holes, that model doesn't work too well for the way most people use the net these days and in the future.

I think having pfsense start up with nothing, exactly as it does now, is the best way to go. It ensures maximum compatibility across a broad range of platforms. I think if you want a package, add a package is the correct mentality. SNORT especially, is something that some just MUST have and others should avoid like the plague.

Less bloat in the main system for things many users do not need (snort users are in the minority)

Most packages works fine as-is for most users

Smaller download sizes, less bandwidth used for hosting pfSense downloads and mirrors

Lots of other good things discussed here and elsewhere about the package system.

Cons for having things as a package

Making those 4 clicks to install the package can be tiresome and tough work the one time you have to do it. I'm surprised people haven't broken their fingers.

Some people want it pre-installed

Somehow magically better code and support because it's in the base system.

Things that have little or no bearing on whether or not it's a package:

The menu name (snort vs IDS/IPS vs some other confusing name)

How the package behaves during a firmware upgrade that happens for most people 1-2 times per year, if that.

How a package interacts with other features such as CARP

So, in other words:

Some valid complaints are not about the package itself, but about how packages in general are handled. This is a base system issue, and you can't "fix" that by moving a package into the base system. You're treating the symptom not curing the disease.

Some valid complaints are about the code of the packages themselves. This will not be solved by moving it to the base system. Someone still has to fix it. Bringing it into base would only happen if a package was already in good shape, or if someone funding getting it into good shape and other factors deemed it worthy. The same fixes would happen if it were kept a package and someone fixed the package. Hence, broken code is broken no matter where it is, and good code is good no matter where it is.

Help identify and fix issues with package handling in the base system (patches are always welcome, doubly so if they actually fix things in an elegant and useful way)

Keep using packages and make them even more awesome than they already are.

If anything arguments could be made for taking even more things out of the base system, though what we have now strikes a good default balance.

If you want a real debate about whether a package should be a package, or in the base system, take a closer look at a small, simple package such as sudo or blinkled. It's a much better debate than this one for large/complex add-ons. ;D

I think modularizing things into packages is per se a good thing, but having started with pfSense as a total noob not all that long time ago, I remember the confusions the packages create.

And that boils largely down to competing/conflicting packages that are not recognizable as such, and the vastly differing quality and level of support for various packages.

If there were some sort of curation process, e.g. official packages (things that are maintained by the pfSense team, and are just taken out of the base system for a more modular setup), qualified packages (very commonly used things that for license or other reasons are not part of the base system or official packages, but are tested and have a seal of approval that they work and are consistent with the UI and naming conventions), and user contributed packages (for which no assertion is made in regards to anything), that would be a step forward, particularly if one could restrict the list of packages shown in the "available packages" section to a certain quality level, e.g. only official packages, or qualified and official packages, or everything.

As long as there's a better way of handling package conflicts, and as long as package isn't synonym with less support, less quality, less consistency, etc. I'm all for modularizing things. However right now, I consistently get the feel that anything that's in a package is "second class citizen", and that's pretty much the main reason that I'd love to see everything that I consider important part of the base system.

You do see that Alpha, Beta, RC, Stable tag in the packages right? Are those tags confusing?
Also, in every disro of Unix like OS, packages are handled like that. I wouldn't want the "Microsoft" one stop shop o-code method anyway.
Sounds like you just don't like the way opensource projects work in general.
There are some pretty sweet commercial appliances you can buy for an arm and a leg.
Of course, no one can see the code most of the time. No idea if its protecting you or rooting you.
Actually, I have a pretty good idea… Huawei makes some pretty nice stuff. Install it liberally.

You do see that Alpha, Beta, RC, Stable tag in the packages right? Are those tags confusing?

No, but they turn out to be meaningless. These tags make a statement about the stability of the package, not about the quality of the integration, whether or not they are a modularized part of the official project, or some more or less supported/approved add-on

Also, in every disro of Unix like OS, packages are handled like that. I wouldn't want the "Microsoft" one stop shop o-code method anyway.

This wouldn't be the Microsoft model, more along the lines the Apple AppStore model, except more liberal: for a package to be either official (something that is done by the core project team, but put into a package to make things more modular) or qualified (third party contribution, but tested), it would have to fulfill certain standards e.g. the way their interface works, if it can use aliases properly, what naming conventions are used, etc.
If it doesn't do these things, it would be marked as unqualified user contributed package.

If you want a real debate about whether a package should be a package, or in the base system, take a closer look at a small, simple package such as sudo or blinkled. It's a much better debate than this one for large/complex add-ons. ;D

You do see that Alpha, Beta, RC, Stable tag in the packages right? Are those tags confusing?

Actually they are confusing. Take a look at haproxy, haproxy-full, haproxy-devel. They are all marked as release. Something about devel being release status seems a big odd :). I brought that up in another discussion and it was mentioned the tags are not maintained very well. I am not really trying to complain about it though. After a post on the forums I got it figured out. I am just trying to state that the tags are not very accurate it seems for some packages.

The RC/Beta/etc tags are largely meaningless. I'd love to fix that, but that means either personally testing each one or getting reliable feedback about the actual quality of a package.

Some, like sudo, are simple and probably could be release quality, but they are also very new and not many have given feedback to say it works or has issues. So to me it may be release, but in general it's "beta" since it's really not quite well-tested and reviewed.

My experience is that Stable means it will probably work, beta means alpha and alpha means don't make any freakin plans. You are going to be pulling your hair out for the next several hours if not days.

I am reiterating that it is always the best to separte base OS from the extra goodies (packages). It keeps simple, both development and system adminstration.

The demand from some of the users who might come from the GUI culture that a monolithic (including packages) pfSense shall deviate the focus from developers. Everyone's need is possible with the current package system (though I am finding pbi a bit exhaustive) on top of base OS. Else I suggest to play a bit on command line and add desired packages in jails to separate the OS from the base OS.

Alternatively, they can compile whatever they need from the pfSense source directly as per their own requirments than asking for the developers. I preferred this way maybe because I am comfortable with the command line since the last century. The exhaustive menus and options on a single page of GUI exhaust me, yet I am for GUIs for those in my team who does not want to follow the sharp learning curve of command line tools, mostly coming from windows and mac cultures (now I can add ubuntu users, too in this group).

DO NOT UPGRADE UNLESS IT BREAKS - *nix Philosophy No. 2

As far as the broken packages with frequent updates of pfSense base are concerned (which is preferred by most of the non-*nix users these days), I would like to remind *nix philosophy that do not upgrade if it does not break. However for snapshot releases you may need to do so, but before check the changelog. FYI, I am still on July 18 snapshot because it is working for my own needs. ;-)

SEPARATION OF PACKAGES - A Suggestion
pfSense 2.1 is alright as it is now, yet I suggest the developers of pfSense to separate the active packages from not-maintained packages to avoid confusion that it is not the problem with pfSense developers' part when the package breaks.

One of the shortcoming I perceive is that it appears to be version-agnostic (with regard to the version of the base system).

Yes, there is a text box which may say "stable platform: 2.0". What does this mean? Will it work on 2.0.0, or on 2.0.x as well? Will it run on 2.1.x? Why does it appear at all when I run an obviously incompatible version? Well, the last point might actually be a feature - you can see that somne function is potentially available if you chose to upgrade from, let's say 1.2 to 2.0.

However, things are a bit more complicated. Take vnstat2 for an example. It happily runs under 2.0.x and equally happily crashes under 2.1.x. There's a manual fix available somewhere here in this forum, for months by now, IIRC. However, integrating this fix into the official package will probably break vnstat2 on older pfSense relases. Defiinitely not the way to go!

I contemplated adding a package named like "vnstat2 for 2.1-RC0 and maybe further", but that seems like an extremely crappy workaround. Especially since a pfSense upgrade from 2.0 to 2.1 will not automagically upgrade "vnstat2" to "vnstat2 for 2.1-RC0 and maybe further".

A package system which maintains different versions of packages for different base system versions might solve this issue, but causes another issue: someone has to maintain the "comptability matrx" for each new relese. Not bloody likely.