Functional Design in Clojurehttps://clojuredesign.club/
Each week, we discuss a software design problem and how we might solve it using functional principles and the Clojure programming language.Fri, 5 Jun 2020 08:00:00 -0700en-usHugo -- gohugo.io2018-2019, Christoph Neumann and Nate JonesChristoph Neumann and Nate JonesEach week, we discuss a software design problem and how we might solve it using functional principles and the Clojure programming language.notech,software,design,functional,clojure,immutable,geek,nerd,development,nate,jones,christoph,neumannChristoph Neumann and Nate Jonesfeedback@clojuredesign.clubChristoph Neumann and Nate Jonesfeedback@clojuredesign.clubEach week, we discuss a software design problem and how we might solve it using functional principles and the Clojure programming language.Fri, 5 Jun 2020 08:00:00 -07001800Ep 076: Multiple Views on Juxthttps://clojuredesign.club/episode/076-multiple-views-on-juxt/
https://clojuredesign.club/episode/076-multiple-views-on-juxt/Fri, 05 Jun 2020 08:00:00 -0700 We take a turn with juxt, looking at all the ways it can help line up data.Each week, we discuss a different topic about Clojure and functional programming.

]]>Christoph Neumann and Nate JonesWe take a turn with juxt, looking at all the ways it can help line up data.7630:08We take a turn with juxt, looking at all the ways it can help line up data.Ep 075: Merge With Funhttps://clojuredesign.club/episode/075-merge-with-fun/
https://clojuredesign.club/episode/075-merge-with-fun/Fri, 22 May 2020 08:00:00 -0700 We focus in on merge-with, a powerful function for aggregating data.Each week, we discuss a different topic about Clojure and functional programming.

]]>Christoph Neumann and Nate JonesWe focus in on merge-with, a powerful function for aggregating data.7526:08We focus in on merge-with, a powerful function for aggregating data.Ep 074: Deploy, They Said. It'll Be Fun, They Said.https://clojuredesign.club/episode/074-deploy-they-said-itll-be-fun-they-said/
https://clojuredesign.club/episode/074-deploy-they-said-itll-be-fun-they-said/Fri, 15 May 2020 08:00:00 -0700 We survey the myriad ways we've used to launch our code into production, and laugh about the various complexities we've found.Each week, we discuss a different topic about Clojure and functional programming.

This week, the topic is: “Deploying Clojure.” We survey the myriad ways we’ve used to launch our code into production, and laugh about the various complexities we’ve found.

Selected quotes:

“How do you get closure on the development of your Clojure code?”

“What do you mean by production?”

“Tomcat goes to war.”

“That was back before containers.”

“Trade a little bit of money for a lot of sanity.”

“The problem with these things is the configuration you end up with is really simple after you’ve read 8 hours of documentation.”

“The number of lines of code does not indicate the amount of effort.”

“Every time you accept a new component into your system, you are also accepting its complexity.”

]]>Christoph Neumann and Nate JonesWe survey the myriad ways we've used to launch our code into production, and laugh about the various complexities we've found.7427:35We survey the myriad ways we've used to launch our code into production, and laugh about the various complexities we've found.Ep 073: Silent Socketshttps://clojuredesign.club/episode/073-silent-sockets/
https://clojuredesign.club/episode/073-silent-sockets/Fri, 17 Apr 2020 08:00:00 -0700 We worry about the health of our websockets and, after looking for help from the standards bodies, roll up our sleeves and handle it ourselves.Each week, we discuss a different topic about Clojure and functional programming.

]]>Christoph Neumann and Nate JonesWe worry about the health of our websockets and, after looking for help from the standards bodies, roll up our sleeves and handle it ourselves.7338:19We worry about the health of our websockets and, after looking for help from the standards bodies, roll up our sleeves and handle it ourselves.Ep 072: Sea of Socketshttps://clojuredesign.club/episode/072-sea-of-sockets/
https://clojuredesign.club/episode/072-sea-of-sockets/Fri, 03 Apr 2020 08:00:00 -0700 We switch to using a component to manage our websockets, enabling ease of development and future growth.Each week, we discuss a different topic about Clojure and functional programming.

This week, the topic is: “Organizing our websocket code.” We switch to using a component to manage our websockets, enabling ease of development and future growth.

Selected quotes:

“It feels like every time I use a global, a butterfly dies.”

“We don’t want to leave the REPL!”

“It’s important to keep the abstraction un-leaky.”

“When you lower the cognitive load of programming, that makes it more enjoyable.”

]]>Christoph Neumann and Nate JonesWe switch to using a component to manage our websockets, enabling ease of development and future growth.7232:11We switch to using a component to manage our websockets, enabling ease of development and future growth.Ep 071: Socket Synergyhttps://clojuredesign.club/episode/071-socket-synergy/
https://clojuredesign.club/episode/071-socket-synergy/Fri, 20 Mar 2020 08:00:00 -0700 We wander into the weeds, jumping through the myriad hoops required to deliver a simple notification.Each week, we discuss a different topic about Clojure and functional programming.

]]>Christoph Neumann and Nate JonesWe wander into the weeds, jumping through the myriad hoops required to deliver a simple notification.7129:10We wander into the weeds, jumping through the myriad hoops required to deliver a simple notification.Ep 070: Socket Circushttps://clojuredesign.club/episode/070-socket-circus/
https://clojuredesign.club/episode/070-socket-circus/Fri, 13 Mar 2020 08:00:00 -0700 We talk about spontaneously sending data from the server to the client to address our users' insecurities.Each week, we discuss a different topic about Clojure and functional programming.

This week, the topic is: “Websockets.” We talk about spontaneously sending data from the server to the client to address our users’ insecurities.

Selected quotes:

“Classic engineering solution, let’s just bend a couple of assumptions.”

“Reloading the web page is 1995-style integration.”

“Poll every tenth of a second? Yeah, that’ll work till we need to scale past 4 users.”

“In every engineering choice, there is always a tradeoff.”

]]>Christoph Neumann and Nate JonesWe talk about spontaneously sending data from the server to the client to address our users' insecurities.7029:43We talk about spontaneously sending data from the server to the client to address our users' insecurities.Ep 069: Stuck in the Webhttps://clojuredesign.club/episode/069-stuck-in-the-web/
https://clojuredesign.club/episode/069-stuck-in-the-web/Fri, 28 Feb 2020 08:00:00 -0700 We examine our history writing web handlers and talk about all the ways we've broken them.Each week, we discuss a different topic about Clojure and functional programming.

This week, the topic is: “Request handler pitfalls.” We examine our history writing web handlers and talk about all the ways we’ve broken them.

Selected quotes:

“Doing a pitfalls episode is easy, because you just need to look at your scars.”

“Letting exceptions escape is TMI for the web.”

“Multiple error handling middleware is ok. They’re in place, just in case.”

“Smaller doesn’t necessarily mean simpler.”

“A little bit of repetition can lead to a lot of clarity.”

“Java can turn just about anything into a string.”

]]>Christoph Neumann and Nate JonesWe examine our history writing web handlers and talk about all the ways we've broken them.6928:50We examine our history writing web handlers and talk about all the ways we've broken them.Ep 068: Static on the Linehttps://clojuredesign.club/episode/068-static-on-the-line/
https://clojuredesign.club/episode/068-static-on-the-line/Fri, 21 Feb 2020 08:00:00 -0700 We tease apart the layers involved in serving static assets and are surprised by how many we find.Each week, we discuss a different topic about Clojure and functional programming.

This week, the topic is: “Serving static assets.” We tease apart the layers involved in serving static assets and are surprised by how many we find.

Selected quotes:

“You’d think that serving up plain old files would be nice and simple.”

“MIME types are an explosion of complexity.”

“Simplicity is the only remedy for complexity. Abstraction is not a remedy for complexity.”

]]>Christoph Neumann and Nate JonesWe tease apart the layers involved in serving static assets and are surprised by how many we find.6824:20We tease apart the layers involved in serving static assets and are surprised by how many we find.Ep 067: Handling Handler's Handleshttps://clojuredesign.club/episode/067-handling-handlers-handles/
https://clojuredesign.club/episode/067-handling-handlers-handles/Fri, 14 Feb 2020 08:00:00 -0700 We manage to find a way to get our handlers the resources they need to get real work done.Each week, we discuss a different topic about Clojure and functional programming.

]]>Christoph Neumann and Nate JonesWe manage to find a way to get our handlers the resources they need to get real work done.6723:52We manage to find a way to get our handlers the resources they need to get real work done.Ep 066: Compose the Codechttps://clojuredesign.club/episode/066-compose-the-codec/
https://clojuredesign.club/episode/066-compose-the-codec/Fri, 31 Jan 2020 08:00:00 -0700 We reinvent the wheel, and along the way discover a few reasons why you might want to do so as well.Each week, we discuss a different topic about Clojure and functional programming.

]]>Christoph Neumann and Nate JonesWe reinvent the wheel, and along the way discover a few reasons why you might want to do so as well.6628:05We reinvent the wheel, and along the way discover a few reasons why you might want to do so as well.Ep 065: Stuck in the Middlehttps://clojuredesign.club/episode/065-stuck-in-the-middle/
https://clojuredesign.club/episode/065-stuck-in-the-middle/Fri, 24 Jan 2020 08:00:00 -0700 We find that the middle is a very good place to start when almost everything is composed functions.Each week, we discuss a different topic about Clojure and functional programming.

]]>Christoph Neumann and Nate JonesWe find that the middle is a very good place to start when almost everything is composed functions.6530:41We find that the middle is a very good place to start when almost everything is composed functions.Ep 064: Put Ring on Ithttps://clojuredesign.club/episode/064-put-ring-on-it/
https://clojuredesign.club/episode/064-put-ring-on-it/Fri, 17 Jan 2020 08:00:00 -0700 We focus on the bedrock abstraction for all Clojure web applications and marvel at its simplicity.Each week, we discuss a different topic about Clojure and functional programming.

This week, the topic is: “Ring, the foundation of Clojure HTTP” We focus on the bedrock abstraction for all Clojure web applications and marvel at its simplicity.

Selected quotes:

“Somebody wants to make a web app, and suddenly they’re up against all these layers.”

“The law of leaky abstractions is good to remember.”

“HTTP is just a function call over the Internet.”

“Your function gets a map, and it returns a map.”

]]>Christoph Neumann and Nate JonesWe focus on the bedrock abstraction for all Clojure web applications and marvel at its simplicity.6419:45We focus on the bedrock abstraction for all Clojure web applications and marvel at its simplicity.Ep 063: Web of Complexityhttps://clojuredesign.club/episode/063-web-of-complexity/
https://clojuredesign.club/episode/063-web-of-complexity/Fri, 10 Jan 2020 08:00:00 -0700 We launch a new series and immediately get tangled in the many layers that make up the web.Each week, we discuss a different topic about Clojure and functional programming.

This week, the topic is: “HTTP and the Web of Complexity.” We launch a new series and immediately get tangled in the many layers that make up the web.

Selected quotes:

“Clojure has several ways of getting to the web.”

“There really are a lot of layers.”

“There are a lot of bad requests out there.”

“For all the decodings, there is an encoding.”

]]>Christoph Neumann and Nate JonesWe launch a new series and immediately get tangled in the many layers that make up the web.6322:38We launch a new series and immediately get tangled in the many layers that make up the web.Ep 062: 2019 in Reviewhttps://clojuredesign.club/episode/062-2019-in-review/
https://clojuredesign.club/episode/062-2019-in-review/Fri, 03 Jan 2020 08:00:00 -0700 We look back at the past year and highlight our favorite episodes.Each week, we discuss a different topic about Clojure and functional programming.

]]>Christoph Neumann and Nate JonesWe look back at the past year and highlight our favorite episodes.6236:03We look back at the past year and highlight our favorite episodes.Ep 061: Transcendental Transformationshttps://clojuredesign.club/episode/061-transcendental-transformations/
https://clojuredesign.club/episode/061-transcendental-transformations/Fri, 27 Dec 2019 08:00:00 -0700 We unpack transducers and find a familiar pattern that enables freedom from collections.Each week, we discuss a different topic about Clojure and functional programming.

]]>Christoph Neumann and Nate JonesWe look at clojure.core.reducers and how it extracts performance by composing reducing functions.6026:40We look at clojure.core.reducers and how it extracts performance by composing reducing functions.Ep 059: Lining It Uphttps://clojuredesign.club/episode/059-lining-it-up/
https://clojuredesign.club/episode/059-lining-it-up/Fri, 13 Dec 2019 08:00:00 -0700 We examine the sequence abstraction and then ponder how it helps and hinders our data transformation.Each week, we discuss a different topic about Clojure and functional programming.

This week, the topic is: “Sequences.” We examine the sequence abstraction and then ponder how it helps and hinders our data transformation.

Selected quotes:

“There is only one filter, map, and reduce. They need to be able to interpret all collections.”

“Primitives only take you so far.”

“The first thing is the first thing!”

“The seq abstraction is foundational to Clojure core.”

“The into function pours a sequence back into a concrete data structure.”

]]>Christoph Neumann and Nate JonesWe examine the sequence abstraction and then ponder how it helps and hinders our data transformation.5932:06We examine the sequence abstraction and then ponder how it helps and hinders our data transformation.Ep 058: Reducing It Downhttps://clojuredesign.club/episode/058-reducing-it-down/
https://clojuredesign.club/episode/058-reducing-it-down/Fri, 06 Dec 2019 08:00:00 -0700 We take a long hard look at reduce and find the first of many generally useful nuggets inside.Each week, we discuss a different topic about Clojure and functional programming.

]]>Christoph Neumann and Nate JonesWe take a long hard look at reduce and find the first of many generally useful nuggets inside.5828:52We take a long hard look at reduce and find the first of many generally useful nuggets inside.Ep 057: Clojure/Conj 2019 Recaphttps://clojuredesign.club/episode/057-clojure-conj-2019-recap/
https://clojuredesign.club/episode/057-clojure-conj-2019-recap/Fri, 29 Nov 2019 08:00:00 -0700 We go through our notes and recall the most memorable talks from the Conj last week.Each week, we discuss a different topic about Clojure and functional programming.

]]>Christoph Neumann and Nate JonesWe go through our notes and recall the most memorable talks from the Conj last week.5734:51We go through our notes and recall the most memorable talks from the Conj last week.Ep 056: Opt-in Complexityhttps://clojuredesign.club/episode/056-opt-in-complexity/
https://clojuredesign.club/episode/056-opt-in-complexity/Fri, 22 Nov 2019 08:00:00 -0700 We discuss complexity and try to come up with a simple explanation for why Clojurians avoid it so ruthlessly.Each week, we discuss a different topic about Clojure and functional programming.

]]>Christoph Neumann and Nate JonesWe discuss complexity and try to come up with a simple explanation for why Clojurians avoid it so ruthlessly.5628:25We discuss complexity and try to come up with a simple explanation for why Clojurians avoid it so ruthlessly.Ep 055: Sets! What Are They Good For?https://clojuredesign.club/episode/055-sets-what-are-they-good-for/
https://clojuredesign.club/episode/055-sets-what-are-they-good-for/Fri, 15 Nov 2019 08:00:00 -0700 We examine one of the lesser used data structures in Clojure and talk about its unique characteristics and uses.Each week, we discuss a different topic about Clojure and functional programming.

This week, the topic is: “Sets! What are they good for?” We examine one of the lesser used data structures in Clojure and talk about its unique characteristics and uses.

Selected quotes:

“Sets aren’t going to get top billing.”

“Sets are really about uniqueness.”

“Sets let us calculate differences cheaply.”

“Clever is a word I used to like when I was a younger programmer.”

“Clever now equals hours of suffering!”

]]>Christoph Neumann and Nate JonesWe examine one of the lesser used data structures in Clojure and talk about its unique characteristics and uses.5528:25We examine one of the lesser used data structures in Clojure and talk about its unique characteristics and uses.Ep 054: The Forest in the Treeshttps://clojuredesign.club/episode/054-the-forest-in-the-trees/
https://clojuredesign.club/episode/054-the-forest-in-the-trees/Fri, 08 Nov 2019 08:00:00 -0700 We discuss three powerful libraries (Specter, Spectacles, and clojure.walk) and where they might fit into our Clojure programs.Each week, we discuss a different topic about Clojure and functional programming.

]]>Christoph Neumann and Nate JonesWe discuss three powerful libraries (Specter, Spectacles, and clojure.walk) and where they might fit into our Clojure programs.5421:58We discuss three powerful libraries (Specter, Spectacles, and clojure.walk) and where they might fit into our Clojure programs.Ep 053: How Can I Save My Data From Serialization?https://clojuredesign.club/episode/053-how-can-i-save-my-data-from-serialization/
https://clojuredesign.club/episode/053-how-can-i-save-my-data-from-serialization/Fri, 01 Nov 2019 08:00:00 -0700 We record our thoughts on the many trade-offs we have encountered preserving our data when it leaves our programs.Each week, we answer a different question about Clojure and functional programming.

]]>Christoph Neumann and Nate JonesWe record our thoughts on the many trade-offs we have encountered preserving our data when it leaves our programs.5330:36We record our thoughts on the many trade-offs we have encountered preserving our data when it leaves our programs.Ep 052: Functions! Functions! Functions!https://clojuredesign.club/episode/052-functions-functions-functions/
https://clojuredesign.club/episode/052-functions-functions-functions/Fri, 25 Oct 2019 08:00:00 -0700 We wonder how we could function without these critical building blocks, so we catagorize their varied uses.Each week, we discuss a different topic about Clojure and functional programming.

]]>Christoph Neumann and Nate JonesWe wonder how we could function without these critical building blocks, so we catagorize their varied uses.5237:27We wonder how we could function without these critical building blocks, so we catagorize their varied uses.Ep 051: Maps! Maps! Maps!https://clojuredesign.club/episode/051-maps-maps-maps/
https://clojuredesign.club/episode/051-maps-maps-maps/Fri, 18 Oct 2019 08:00:00 -0700 We discuss maps and their useful features, including a key distinction that we couldn't live without.Each week, we discuss a different topic about Clojure and functional programming.

]]>Christoph Neumann and Nate JonesWe discuss maps and their useful features, including a key distinction that we couldn't live without.5128:01We discuss maps and their useful features, including a key distinction that we couldn't live without.Ep 050: Parentheses! Parentheses! Parentheses!https://clojuredesign.club/episode/050-parentheses-parentheses-parentheses/
https://clojuredesign.club/episode/050-parentheses-parentheses-parentheses/Fri, 11 Oct 2019 08:00:00 -0700 We defend the lowly parentheses, and discuss the benefits of having this stalwart shepherd dutifully organizing our code.Each week, we discuss a different topic about Clojure and functional programming.

This week, our topic is: “Parentheses! Parentheses! Parentheses!” We defend the lowly parentheses, and discuss the benefits of having this stalwart shepherd dutifully organizing our code.

Selected quotes:

“I’m converting my visual cues into audio information.”

“Parentheses are like dust bunnies that have stacked up over time.”

“Pounds of parentheses.”

“The parenthesis is greatly misunderstood.”

“Parentheses make the grammar of the language incredibly simple.”

“I’ve never seen that in math, at least not in normal math.”

“If you have 15 close parentheses, you might have to do some refactoring.”

“Parentheses are the hugs your program needs.”

“Peace, Love, Parentheses.”

]]>Christoph Neumann and Nate JonesWe defend the lowly parentheses, and discuss the benefits of having this stalwart shepherd dutifully organizing our code.5027:17We defend the lowly parentheses, and discuss the benefits of having this stalwart shepherd dutifully organizing our code.Ep 049: Keywords! Keywords! Keywords!https://clojuredesign.club/episode/049-keywords-keywords-keywords/
https://clojuredesign.club/episode/049-keywords-keywords-keywords/Fri, 04 Oct 2019 08:00:00 -0700 We examine all the fascinating properties of keywords, how to use them, and why they're so much better than strings and enums.Each week, we discuss a different topic about Clojure and functional programming.

This week, our topic is: “Keywords! Keywords! Keywords!” We examine all the fascinating properties of keywords, how to use them, and why they’re so much better than strings and enums.

Selected quotes:

“The question of the day is: Why is there no question of the day?”

“Everything behind the scenes should get turn into integers because that is the good and true way.”

“A FAQ after the fact.”

“We forget about identical? because Clojure gives us value semantics!”

“Magic numbers you can spell with letters.”

“The ability of communicating with letters–something we humans have been developing for a while.”

“What is this sorcery?!”

“JSON. That’s not not a guy who likes strings.”

“Let’s blame JSON.”

“If you don’t have to recursively walk a tree, why recursively walk a tree?”

“A little bit of copying is better than a lot of depending.”

“The randomly placed last one wins!”

“Trust but verify.”

]]>Christoph Neumann and Nate JonesWe examine all the fascinating properties of keywords, how to use them, and why they're so much better than strings and enums.4925:25We examine all the fascinating properties of keywords, how to use them, and why they're so much better than strings and enums.Ep 048: Help! How Do I Fix My REPL?https://clojuredesign.club/episode/048-help-how-do-i-fix-my-repl/
https://clojuredesign.club/episode/048-help-how-do-i-fix-my-repl/Fri, 27 Sep 2019 08:00:00 -0700 We catalog the many ways we've broken our REPLs and talk through our strategies for getting back on track.Each week, we answer a different question about Clojure and functional programming.

This week, the question is: “Help! How do I fix my REPL?” We catalog the many ways we’ve broken our REPLs and talk through our strategies for getting back on track.

Selected quotes:

“The REPL is a wonderful, wonderful thing.”

“A cleansing restart.”

“What is my Clojure band name?”

“tools.namespace will help you find rename errors early.”

“Refresh will continue to work, as long as you continue to work.”

“Even a defonce can be def’d again. It’s a defonce, for now.”

]]>Christoph Neumann and Nate JonesWe catalog the many ways we've broken our REPLs and talk through our strategies for getting back on track.4827:03We catalog the many ways we've broken our REPLs and talk through our strategies for getting back on track.Ep 047: What Is "Nil Punning"?https://clojuredesign.club/episode/047-what-is-nil-punning/
https://clojuredesign.club/episode/047-what-is-nil-punning/Fri, 20 Sep 2019 08:00:00 -0700 We gaze into the nil and find a surprising number of things to talk about.Each week, we answer a different question about Clojure and functional programming.

This week, the question is: “What is ‘nil punning’?” We gaze into the nil and find a surprising number of things to talk about.

Selected quotes:

“The lowly, magnificent nil. Some people love it, some people hate it.”

“Null is the value you give your program if you want to see it die.”

“Nil is not null.”

“This function found nothing, and I passed that to the next function, and it found nothing in the nothing.”

“It’s amazing how much nothing you can find in nothing.”

“You can pull data out without fear.”

“What does a nil Cat look like?”

“A lot of arithmetic stuff is nil-intolerant.”

“No answer isn’t going to start becoming an answer later.”

]]>Christoph Neumann and Nate JonesWe gaze into the nil and find a surprising number of things to talk about.4728:09We gaze into the nil and find a surprising number of things to talk about.Ep 046: When Is Clojure Not the Right Tool for the Job?https://clojuredesign.club/episode/046-when-is-clojure-not-the-right-tool-for-the-job/
https://clojuredesign.club/episode/046-when-is-clojure-not-the-right-tool-for-the-job/Fri, 13 Sep 2019 08:00:00 -0700 We look at the varied forms that Clojure can assume and consider where it might not fit.Each week, we answer a different question about Clojure and functional programming.

]]>Christoph Neumann and Nate JonesWe look at the varied forms that Clojure can assume and consider where it might not fit.4625:06We look at the varied forms that Clojure can assume and consider where it might not fit.Ep 045: Why Have Derived Fields in Data When I Can Just Calculate Derived Data as Needed With a Function?https://clojuredesign.club/episode/045-why-have-derived-fields-in-data-when-i-can-just-calculate-derived-data-as-needed-with-a-function/
https://clojuredesign.club/episode/045-why-have-derived-fields-in-data-when-i-can-just-calculate-derived-data-as-needed-with-a-function/Fri, 06 Sep 2019 08:00:00 -0700 We take a focused look at the balance of using functions or derived fields and where each is preferable.Each week, we answer a different question about Clojure and functional programming.

This week, the question is: “Why have derived fields in data when I can just calculate derived data as needed with a function?” We take a focused look at the balance of using functions or derived fields and where each is preferable.

Selected quotes:

“Clojure has a gravity toward using the built-in data structures.”

“Immutability helps you handle concurrent updates to dependent fields without using locks.”

“If you add derived fields, you can query the data using information at different levels of abstraction.”

]]>Christoph Neumann and Nate JonesWe take a focused look at the balance of using functions or derived fields and where each is preferable.4520:57We take a focused look at the balance of using functions or derived fields and where each is preferable.Ep 044: What's So Different About Clojure's REPL?https://clojuredesign.club/episode/044-whats-so-different-about-clojures-repl/
https://clojuredesign.club/episode/044-whats-so-different-about-clojures-repl/Fri, 30 Aug 2019 08:00:00 -0700 We evaluate what a REPL really is and show that it's much more about the developer experience than simply calculating values.Each week, we answer a different question about Clojure and functional programming.

]]>Christoph Neumann and Nate JonesWe evaluate what a REPL really is and show that it's much more about the developer experience than simply calculating values.4428:15We evaluate what a REPL really is and show that it's much more about the developer experience than simply calculating values.Ep 043: What Is 'Faking' a Resource?https://clojuredesign.club/episode/043-what-is-faking-a-resource/
https://clojuredesign.club/episode/043-what-is-faking-a-resource/Fri, 23 Aug 2019 08:00:00 -0700 We talk about the virtues of faking and then outline several real techniques for getting work done.Each week, we answer a different question about Clojure and functional programming.

]]>Christoph Neumann and Nate JonesWe talk about the virtues of faking and then outline several real techniques for getting work done.4323:54We talk about the virtues of faking and then outline several real techniques for getting work done.Ep 042: What Does It Mean to Be 'Data-Oriented'?https://clojuredesign.club/episode/042-what-does-it-mean-to-be-data-oriented/
https://clojuredesign.club/episode/042-what-does-it-mean-to-be-data-oriented/Fri, 16 Aug 2019 08:00:00 -0700 We merge together different aspects of Clojure's data orientation and specify which of those help make development more pleasant.Each week, we answer a different question about Clojure and functional programming.

This week, the question is: “What does it mean to be ‘data-oriented’?” We merge together different aspects of Clojure’s data orientation, and specify which of those help make development more pleasant.

Selected quotes:

“Clojure has the corner on data.”

“Other languages have data too, it’s just locked in little cages.”

“Data is inert, it can’t harm you.”

“Because Clojure is expressed in its own data structures, and those structures are simple, that makes Clojure syntax simple.”

“Find a good way to represent the information that you want to work with, in a way that feels appropriate for the subject matter.”

“If you find a good way of representing your information, that representation tends to be pretty stable. All of the change is in the functions you use to work with it.”

]]>Christoph Neumann and Nate JonesWe merge together different aspects of Clojure's data orientation and specify which of those help make development more pleasant.4226:47We merge together different aspects of Clojure's data orientation and specify which of those help make development more pleasant.Ep 041: Why Do Clojurians Make Such a Big Deal About Immutability?https://clojuredesign.club/episode/041-why-do-clojurians-make-such-a-big-deal-about-immutability/
https://clojuredesign.club/episode/041-why-do-clojurians-make-such-a-big-deal-about-immutability/Fri, 09 Aug 2019 08:00:00 -0700 We cover several practical side effects of immutability and why we've become such big fans of data that doesn't let us down.Each week, we answer a different question about Clojure and functional programming.

This week, the question is: “Why do Clojurians make such a big deal about immutability?” We cover several practical side effects of immutability and why we’ve become such big fans of data that doesn’t let us down.

Selected quotes:

“Well, we don’t have Monads to talk about.”

“What good is a program, if you can’t change stuff!?”

“The memory cost of data structures is in proportion to the changes, not the users.”

“I can hang on to a reference to the old state and a reference to the new state very cheaply.”

“You can reduce comparison to referential equality.”

“Once you can efficiently save every version of the state, going back to a previous version is no big deal.”

“I love determinism. Determinism is trustable.”

“I can trust immutable data. And if I can trust it, then it can occupy a smaller part of my brain.”

]]>Christoph Neumann and Nate JonesWe cover several practical side effects of immutability and why we've become such big fans of data that doesn't let us down.4127:11We cover several practical side effects of immutability and why we've become such big fans of data that doesn't let us down.Ep 040: Should I Use Lein, Boot, or Tools.deps?https://clojuredesign.club/episode/040-should-i-use-lein-boot-or-tools-deps/
https://clojuredesign.club/episode/040-should-i-use-lein-boot-or-tools-deps/Fri, 02 Aug 2019 08:00:00 -0700 We assemble a list of build tool characteristics and talk about how each tool stacks up before giving our recommendations.Each week, we answer a different question about Clojure and functional programming.

This week, the question is: “Should I use lein, boot, or tools.deps?” We assemble a list of build tool characteristics and talk about how each tool stacks up before giving our recommendations.

Selected quotes:

“The best thing about boot is that you can use all the flexibility of Clojure code. The worst thing about boot is that you can use all the flexibility of Clojure code.”

“Tools.deps, the non-build build tool.”

“I have a file, it’s got Clojure in it, and I want to run it.”

“Tools.deps has enabled people to make build functions that stand alone. They don’t have any ceremony to become part of the tool. It’s just a main.”

]]>Christoph Neumann and Nate JonesWe assemble a list of build tool characteristics and talk about how each tool stacks up before giving our recommendations.4028:04We assemble a list of build tool characteristics and talk about how each tool stacks up before giving our recommendations.Ep 039: Why Use Clojure Over Another Functional Language?https://clojuredesign.club/episode/039-why-use-clojure-over-another-functional-language/
https://clojuredesign.club/episode/039-why-use-clojure-over-another-functional-language/Fri, 26 Jul 2019 08:00:00 -0700 We examine the different categories of functional programming languages and distill out what differentiates Clojure and why we prefer it.Each week, we answer a different question about Clojure and functional programming.

This week, the question is: “Why use Clojure over another functional language?”. We examine the different categories of functional programming languages and distill out what differentiates Clojure and why we prefer it.

Selected quotes:

“Running just one function when developing is not only allowed in Clojure, it’s encouraged and celebrated.”

“You don’t have to make the whole world (application) agree. You can work on just a part of it and then bring it back into the rest of the world when you want it to agree.”

“I would like some XML in my cake.”

“Oh, you were a hipster Scala user.”

“When I pull in code off clojars, it’s going to use the Clojure way, because there is a Clojure way.”

“If you can make all your abstractions with a simpler set of semantics, wouldn’t that be better than a broader set?”

“Multi-paradigm languages are inherently more complex. You really end up in the ‘good parts’ kind of problem. Scala, The Good Parts. Javascript, The Good Parts.”

“Code is about communicating with two things. The computer and the other developers. The computer can handle esoteric language features, but other developers will have a harder time with them.”

]]>Christoph Neumann and Nate JonesWe examine the different categories of functional programming languages and distill out what differentiates Clojure and why we prefer it.3925:17We examine the different categories of functional programming languages and distill out what differentiates Clojure and why we prefer it.Ep 038: How Do I Convince My Coworkers to Use Clojure?https://clojuredesign.club/episode/038-how-do-i-convince-my-coworkers-to-use-clojure/
https://clojuredesign.club/episode/038-how-do-i-convince-my-coworkers-to-use-clojure/Fri, 19 Jul 2019 08:00:00 -0700 We recall our own experiences evangelizing Clojure and give practical advice from the trenches.Each week, we answer a different question about Clojure and functional programming.

]]>Christoph Neumann and Nate JonesWe recall our own experiences evangelizing Clojure and give practical advice from the trenches.3824:53We recall our own experiences evangelizing Clojure and give practical advice from the trenches.Ep 037: What Advice Would You Give to Someone Getting Started With Clojure?https://clojuredesign.club/episode/037-what-advice-would-you-give-to-someone-getting-started-with-clojure/
https://clojuredesign.club/episode/037-what-advice-would-you-give-to-someone-getting-started-with-clojure/Fri, 12 Jul 2019 08:00:00 -0700 We trade off giving practical tips for intrepid learners reminisce about our own paths into Clojure.Each week, we answer a different question about Clojure and functional programming.

This week, the question is: “What advice would you give to someone getting started with Clojure?”. We trade off giving practical tips for intrepid learners while we reminisce about our own paths into Clojure.

Selected quotes:

“Reading other people’s code will change how you view the problem.”

“You’re going to get stuck, but that’s ok.

“REPL driven development is so fast, you’ll need to take a break from the whiplash.”

“Clojure doesn’t give you breaks, like other programming languages do with their compiles and restarts.”

“The best code to use is code that was written by somebody else, because it’s probably bug free.”

“The problem isn’t the code that I’m writing, the problem is with my approach.”

]]>Christoph Neumann and Nate JonesWe trade off giving practical tips for intrepid learners reminisce about our own paths into Clojure.3725:57We trade off giving practical tips for intrepid learners reminisce about our own paths into Clojure.Ep 036: Why Do You Recommend Clojure?https://clojuredesign.club/episode/036-why-do-you-recommend-clojure/
https://clojuredesign.club/episode/036-why-do-you-recommend-clojure/Fri, 05 Jul 2019 08:00:00 -0700 We take turns sharing our favorite reasons, and we can't help but have fun riffing on how enjoyable Clojure is to use.It’s summertime, and that means it’s time for something new. Each week, we will answer a different question about Clojure and functional programming.

This week, we’re starting off with “Why do you recommend Clojure?”. We take turns sharing our favorite reasons, and we can’t help but have fun riffing on how enjoyable Clojure is to use. Come along for the ride.

Selected quotes:

“Question everything else. Clojure is the answer.”

“Once you discover structural editing, you’ll never go back.”

“Data is inert. It can’t act.”

“When dealing with mutable data and side effects, it’s like trying to add a domino in the middle and make sure to not break the chain.”

“Most of the time, running with scissors is considered to be a bad thing.”

“Somebody in Java has done all that plumbing that you don’t really want to do.”

]]>Christoph Neumann and Nate JonesWe take turns sharing our favorite reasons, and we can't help but have fun riffing on how enjoyable Clojure is to use.3619:58We take turns sharing our favorite reasons, and we can't help but have fun riffing on how enjoyable Clojure is to use.Ep 035: Lifted Learningshttps://clojuredesign.club/episode/035-lifted-learnings/
https://clojuredesign.club/episode/035-lifted-learnings/Fri, 28 Jun 2019 08:00:00 -0700 Christoph and Nate lift concepts from the raw log-parsing series.Christoph and Nate lift concepts from the raw log-parsing series.

Reflecting on the lessons learned in the log series.

(01:15) Concept 1: We found Clojure to be useful for devops.

Everything is a web application these days,

“The only UIs in Devops are dashboards.”

For most of the series, our UI was our connected editor.

We grabbed a chunk of the log file and were fiddling with the data in short order.

We talk about connected editors in our REPL series, starting with Episode 12.

Being able to iteratively work on the log parsing functions in our editor was key to exploring the data in the log files.

(04:04) Concept 2: Taking a lazy approach is essential when working with a large data set.

Lazily going through a sequence is reminiscent of database cursors. You are at some point in a stream of data.

We ran into some initial downsides.

When using with-open, fully lazy processing results in an I/O error, because the file has been closed already.

Shouldn’t be too eager too early, because then the entire dataset will reside in memory.

Two kinds of functions: lazy and eager.

Lazy functions only take from a sequence as they need more values.

Eager functions consume the whole sequence before returning.

Ensure that only the last function in the processing chain is eager.

“It only takes one eager to get everybody unlazy.”

(08:38) Concept 3: Clojure helps you make your own lazy sequences using lazy-seq.

Clojure has a deep library of functions for making and processing lazy sequences.

We were able to make our own lazy sequences that could then be used with those functions.

Wrap the body in lazy-seq and return either nil (to indicate the end) or a sequence created by calling cons on a real value and a recursive call to itself.

(12:41) Concept 4: We work with information at different levels, and that forms an information hierarchy.

The data goes from bits to characters to lines, and then we get involved.

We move from lines on up to more meaningful entities. Parsed lines are maps that have richer information, and then errors are richer still.

Our parsers take a sequence and emit a new sequence that is at a higher level of information.

We first explored this concept in the Time series.

The transformations from one level to the next are all pure.

(14:53) Concept 5: Sometimes you have to go down before you can go up again another way.

We pre-abstracted a little bit, and only accepted lines that had all of the data we were looking for (time, log level, etc.).

Exceptions broke that abstraction, so we reworked our “parsed line” map to make the missing keys optional.

(15:54) Concept 6: Maps are flexible bags of dimensions. They are a set of attributes rather than a series of rigid slots that must be filled.

Functions only need to look at the parts of the map that they need.

Every time we amplify the data, we add a new set of dimensions.

Thanks to namespacing, all of these dimensions coexist peacefully.

Multiple levels of dimensions give you more to filter/map/reduce on.

Just because you distill, doesn’t mean you want to lose essence.

(21:09) Concept 7: Operating within a level of information is a different concern than lifting up to a higher level of information.

Within a level, functions aid in filtering and aggregating.

Between levels, functions recognize patterns and groupings to produce higher levels of information.

Make the purpose of the function clear in how you name it.

Separate functions that “lift” the data from functions that operate at the same level of information.

When exploring data, you don’t know where it will lead, so start by moving the data up a level in small steps.

]]>Christoph Neumann and Nate JonesChristoph finds exceptional log lines and takes a more literal approach.3426:44Christoph finds exceptional log lines and takes a more literal approach.Ep 033: Cake or Ice Cream? Yes!https://clojuredesign.club/episode/033-cake-or-ice-cream-yes/
https://clojuredesign.club/episode/033-cake-or-ice-cream-yes/Fri, 14 Jun 2019 08:00:00 -0700 Nate needs to parse two different errors and takes some time to compose himself.Nate needs to parse two different errors and takes some time to compose himself.

Previously, we were able to parse out errors and give the parsing function the ability to search as far into the future as necessary.

We did this by having the function take a sequence and return a sequence, managed by lazy-seq.

(01:30) New Problem: We need to correlate two different kinds of errors.

The developers looked at our list of sprinkle errors and they think that they’re caused by the 357 errors.

They have requested that we look at the entire log and generate a report of 357 and sprinkle errors, so we can tell if they’re correlated.

“When someone says, do I want cake or ice cream, the right answer is: yes, I want both!”

Before, we were only parsing out a single type of error and summarizing it, but now we need to parse out both types of errors.

If we try to parse both kinds of errors with the same function, we will quickly get ourselves into nested ifs or maybe an infinite cond. Perhaps a complex state machine with backtracking?

(05:55) Realization: Each error stands alone. Once you detect the beginning of a sprinkle error, you won’t need to look for a 357 error.

You can take each one in turn.

(06:30) Solution step 1: What if we had two functions, one for each type of error.

Each of these functions would take the entire sequence and tell us if there was an error at the beginning.

Previously, our function both recognized errors and handled the sequence generation. If we pull those apart, we can add parsing for more errors easily.

Each error parsing function would return nil if no error was found at the head of the sequence.

(08:46) Solution step 2: Create a function that uses the two detectors to find out what error is at the head of the sequence.

It takes the sequence, and wraps consecutive calls in an or block.

The or block will try each one in turn until one matches and then that is the result.

Each error’s parsing is in its own function, and the combining function serves as an inventory.

]]>Christoph Neumann and Nate JonesNate needs to parse two different errors and takes some time to compose himself.3321:22Nate needs to parse two different errors and takes some time to compose himself.Ep 032: Call Me Lazyhttps://clojuredesign.club/episode/032-call-me-lazy/
https://clojuredesign.club/episode/032-call-me-lazy/Fri, 07 Jun 2019 08:00:00 -0700 Christoph finds map doesn't let him be lazy enough.Christoph finds map doesn’t let him be lazy enough.

Last week, we were dealing with multi-line sprinkle errors.

We were able to get more context using partition.

(01:33) Problem: the component lines had to be adjacent.

Solution last week was to create larger partitions to hopefully get the rest of the error.

This became a magic number problem, guessing how far we had to look ahead.

“If there’s anything I’ve learned in my career, telling the future is one of the hardest things to do.’

What number should be big enough? 100? 1000?

(04:00) The other problem is that the function is handed a pre-selected set of lines.

The decision about how many lines is appropriate is made outside the function.

Wouldn’t it be nice if the function had control over how far to look ahead.

“The function can’t function.”

“Functions are all we’ve got in functional programming. Well, that and lists.”

It would be great if the function itself could take a sequence and look as far as it needs to.

How about handing the function the entire lazy sequence?

(05:52) Problem: Handing in the entire sequence means we can’t use map to convert lines into sprinkle errors anymore.

We can write a function that gives us just one sprinkle error from the sequence, but we want to convert the sequence into a sequence of all the sprinkle errors.

We’re going from something that operates on a subset of the sequence to something that operates on the entire sequence, which is too much control.

We need a way for it to look ahead but still

It’s no longer just working on a chunk of the sequence, but on the unbounded sequence itself.

We need to elevate it to the same power as other sequence operators, like map and filter.

We don’t, however, want the function to eagerly find all sprinkle errors in the sequence. It needs to be lazy.

(09:26) Solution 1: How can we just get one sprinkle error out?

If the first line isn’t the error start, recur with the tail until found.

Do a take-while to find the second half of the error.

When both found, return the value.

We need to terminate the search if we hit the end of the sequence, so we only continue if (seq lines) is not nil.

“There’s no sense in looking in an empty bucket.”

But we don’t want just one, we want the entire sequence.

It would be really nice to return the value when we find it and then wait to find the next one until it is requested.

Conceptually, we could tell the calling function an index of where to start looking for the next error.

(13:44) Solution: In Clojure, we keep our place using the lazy-seq function.

lazy-seq is a sequence, but it hasn’t been realized yet.

It’s like being able to hand back a value and a function to call for the next value.

When you find a value, you can cons it onto the head of an invocation of lazy-seq to make a new sequence.

Step 1. Wrap your entire function body in lazy-seq.

This is similar to using delay, because it wraps the code in something that will only be evaluated when it is first accessed.

Step 2. Ensure that the body obeys the contract. It must return either:

nil, which indicates that the sequence is complete.

a sequence, usually constructed by calling cons on a value and a call to lazy-seq.

Top of the body is a call to (when (seq lines) ..., to ensure that the sequence terminates when there is no data left.

Since the top of our function is lazy-seq, we can cons the found value onto a recursive call to the function.

In the recursive call, we must pass the next section of the sequence, so that when evaluated it will pick up at the right place.

If we don’t find the start of the error, we recurse with the rest of the sequence to try parsing from there.

This function will go through the sequence eagerly until it finds something.

Instead of operating on single elements in the sequence, we can take a sequence and produce a sequence, powered by lazy-seq.

With this capability, you can build a higher level sequence that consumes this sequence and produces a new summary, all done lazily.

]]>Christoph Neumann and Nate JonesNate finds that trouble comes in pairs.3122:49Nate finds that trouble comes in pairs.Ep 030: Lazy Does Ithttps://clojuredesign.club/episode/030-lazy-does-it/
https://clojuredesign.club/episode/030-lazy-does-it/Fri, 24 May 2019 08:00:00 -0700 Christoph's eagerness to analyze the big production logs shows him the value of being lazy instead.Christoph’s eagerness to analyze the big production logs shows him the value of being lazy instead.

Last time: going through the log file looking for the mysterious ‘code 357’.

“The error message that just made sense to the person who wrote it. At the time written. For a few days”

Back and forth with the dev team, but our devops sense was tingling.

Took a sample, fired up a REPL,

Ended up with a list of tuples:

First element: regexp to match

Second element: handler to transform matches into data

(02:00) It’s running slower and slower, the bigger the log file we analyze.

“This is a small 4-person tech company.” “Where we can use new technologies in the same decade that they were created?” “Yes!”

Problem: No one turned on log rotation! The log file is 7G and our application crashes.

“I think we should get lazy.”

“Work harder by getting lazier.”

“Haskell was lazy before it was cool!”

Each line contains all the information we need, so we can process them one at a time.

(4:30) Eager and lazy is like the difference between push and pull.

Giving someone a big bag of work to do, or having them grab more work as they finish.

The thing doing the work needs a little more to work on, so it pulls it in.

Clojure helps with this. It gives us an abstraction so we don’t have to see the I/O happening when we do our processing.

When your map of a lazy sequence needs more data, it gets it on demand.

Clojure core is built to support this lazy style of processing.

File I/O in Java is lazy in the same way. It reads data into a buffer and when that buffer is used up, more is read.

Lazy processing is like a bucket brigade. At the head, you pour out your bucket and the person next to you notices the empty bucket and fills it up. Then this is repeated down the line as each bucket demands to be filled.

(07:55) Let’s make our code lazy.

Current lines function slurps in the file and splits on newline.

Idea: Convert it to open the file and return a lazy sequence using line-seq.

The return value can be threaded through the rest of our pipeline.

Each step of our pipeline is lazy, and the start is lazy, so the whole process should be lazy.

“It’s lazy all the way.”

Problem: We run it, and BOOM, we get an I/O error.

What happened? We were using the with-open macro, which closes the file handle after the body is complete.

Since the I/O is delayed till we consume the sequence, when we start the file is already closed.

“The ability to pull more I/O out of the file has been terminated.”

“Nothing at all, which is a lot less useful than something.”

(12:29) Rather than having a lines function, why don’t we just bring that code into the summary function?

Entire process is wrapped in a with-open so that all steps including summary complete before the file is closed.

Takes a filename and returns an incident count by user.

It does all that we want, but it’s too chunky. We’re doing too much in the function.

We usually want to move I/O to the edges and this commingles it with our logic.

“We just invited I/O to come move into the middle of our living room.”

I/O and summary are in the same function, so to make a new summary, we have to duplicate everything.

We could split out the guts, extract the general and detailed parsing into a separate function. For reuse.

This means you are only duplicating the with-open and line-seq for each new summary.

(16:29) How can we stop duplicating the with-open? To separate that idiom into just one place.

If you can’t return the line-seq, is there a way we can hand in the logic we need all at once?

Idea: Make a function that does the with-open and takes a function.

“Let’s make it higher order.”

We hand in the “work to do” as a function.

“What should we call it? How about process. That’s nice and generic.”

We turn the problem of abstraction into the problem of writing a function that takes a line sequence and produces a result.

Any functions that take and produce a sequence, including those that we wrote, can be used to write this function.

Clojure gives us several ways of composing functions together, we’ll use ->> (the thread-last macro) in this case.

As we improve the vocabulary for working with lines, our ability to express the function gains power and conciseness.

Design tension: if there is something that needs to be done for every summary, it can be pushed into the process function.

The downside to that is that we sign up for that for every summary, and that might not be appropriate for the ones we haven’t written yet.

We opt for making it easier to express and compose the function passed in.

We can still make a function that takes a filename and returns a summary, but the way we construct that function is through composition of transforms.

We can pre-bake a few transforms into shorter names if we want to use them repeatedly.

(23:20) We will still run into the I/O error problem if we’re not careful.

The function that we pass to process needs to have an eager evaluation at the end.

If all we do is transform with lazy functions, the I/O won’t start before the list is returned.

group-by or frequencies will suffice, but if you don’t have one of those, reach for doall.

“You gotta give it a good swift kick with doall.”

Style point: doall at the beginning or at the end of the thread? We like it at the end.

(26:07) We have everything we need.

Lazy so we don’t pull in the entire file.

I/O sits in one function.

We have control over when we’re eager.

Message Queue discussion:

(26:38) Long-time listener Dave sent us a code sample!

An alternative implementation for parse-details that doesn’t use macros.

Top level is an or macro.

Inside the or, each regex is matched in a when-let, the body of which uses the matches to construct the detailed data.

If the regex fails to match, nil is returned and the or will move on to the next block.

We tend to think of or as only for booleans, but it works well for controlling program flow as well.

The code is very clean and concise. And it only uses Clojure core.

“Without dipping into macro-land… Not that there’s anything wrong with that.”

]]>Christoph Neumann and Nate JonesChristoph's eagerness to analyze the big production logs shows him the value of being lazy instead.3030:09Christoph's eagerness to analyze the big production logs shows him the value of being lazy instead.Ep 029: Problem Unknown: Log Lineshttps://clojuredesign.club/episode/029-problem-unknown-log-lines/
https://clojuredesign.club/episode/029-problem-unknown-log-lines/Fri, 17 May 2019 08:00:00 -0700 Nate is dropped in the middle of a huge log file and hunts for the source of the errors.Nate is dropped in the middle of a huge log file and hunts for the source of the errors.

We dig into the world of DonutGram.

“We are the devops.”

Problem: we start seeing errors in the log.

“The kind of error that makes total sense to whoever wrote the log line.”

(04:10) We want to use Clojure to characterize the errors.

Why not grep?

Well…we grep out the lines, count them, and we get 10,351 times. Is that a lot? Hard to say. We need more info.

(05:50) Developers think it’s safe to ignore, but it’s bugging us.

We want to come up with an incident count by user.

“Important thing we do a lot in the devops area: we see data and then we have to figure out, what does it mean?”

“What story is the data telling us?”

The “forensic data” versus “operating data” (Ep 022, 027)

“With forensic data, you’re trying to save the state of the imprint of the known universe over time.”

(07:40) The log file is the most common forensic data for applications.

To characterize, we need to manipulate the data at a higher level than grep.

First goal: make the data useful in Clojure

Get a nice sample from the production log, put it in a file, load up the REPL, and start writing code to parse it.

Need to parse out the data from the log file and turn it into something more structured.

(12:05) Let’s make a function, parse-line that separates out all the common elements for each line:

timestamp

log level

thread name

package name

freeform “message”

“A message has arrived from the developers, lovingly bundled in a long log line.”

(13:10) We can parse every line generically, but we need to parse the message and make sense of that.

“We want to lift up the unstructured data into structured data that we can map, filter, and reduce on.”

We will have different kinds of log lines, so we need to detect them and parse appropriately.

We want to amplify the map to include the details for the specific kind of log line.

We’ll use a kind field to identify which kind of log line it is.

There are two steps:

recognize which kind of log line the “message” is for

parse the data out of that message

“Maybe we’ll have a podcast that’s constantly about constantly. It’s not just juxt.”

(18:45) How do we do this concisely?

Let’s use cond:

flatten out all the cases

code to detect kind (the condition)

code to parse the details (the expression)

Can use includes? in the condition to detect the kind and re-matches to parse.

(20:00) Why not use the regex to detect the kind too?

Can avoid writing the regex twice by using a def.

It’s less code, but now we’re running the regex twice: once in the condition and once to parse.

We can’t capture the result of the test in cond. No “cond-let” macro.

We could write our own cond-let macro, but should we?

“When you feel like you should write a macro, you should step back and assess your current state of being. Am I tired? Did I have a bad day? Do I really need this?”

(24:05) New goals for our problem:

one regex literal

only run the regex once

keep the code that uses the matches close to the regex

Similar problem to “routes” for web endpoints: want the route definition next to the code that executes using the data for that route.

Feels like an index or table of options.

(25:20) Let’s make a “table”. A vector of regex and handler-code pairs.

We need a coffee mug: “Clojure. It’s just a list.”

The code can be a function literal that takes the matches and returns a map of parsed data.

Write a function parse-details which goes through each regex until one matches and then invokes the handler for that one. (See below.)

(30:15) Once we have higher-level data, it’s straight forward to filter and group-by to get our user count.

Once again, the goal is to take unstructured data and turn it into structured data.

“You have just up-leveled unstructured information into a sequence of structured information.”

Can slurp, split, map, filter, and then aggregate.

(32:10) What happens when we try to open a 10 GB log file?

Sounds like a problem for next week.

“Fixing production is always a problem for right now.”

“If a server falls over and no one outside of the devops team knows about it, did the server really fall over?”

]]>Christoph Neumann and Nate JonesNate is dropped in the middle of a huge log file and hunts for the source of the errors.2935:06Nate is dropped in the middle of a huge log file and hunts for the source of the errors.Ep 028: Fail Donuthttps://clojuredesign.club/episode/028-fail-donut/
https://clojuredesign.club/episode/028-fail-donut/Fri, 10 May 2019 08:00:00 -0700 Christoph has gigs of log data and he’s looking to Clojure for some help.Christoph has gigs of log data and he’s looking to Clojure for some help.

Introducing a new topic.

The last few weeks we focused on Twitter and automatically posting to it.

Surprised by how much there is to talk about in a focused problem.

“There will always be more problems for the world to solve.”

(01:53) Imagine if you will, the world of DonutGram!

A fictitious social network where people post their donut experiences.

This series is not about making DonutGram, but living DonutGram.

“Oh, the dark underbelly of application development: support.”

“Let’s take the shiny rock and flip it over and look at all the worms and bugs.”

We talked about forensic data before, and much of DevOps is about looking at that data.

You want to paint the most complete story. It’s a development problem too.

Most of the time, forensic data is written to a log file. It’s the first line of investigation.

Imagine all of the components of your application speaking into one pipe, and you now get to reconstruct what happened.

(05:59) Everything is rosy at DonutGram, but then users start having issues.

Users get the “Fail Donut” and start posting that.

We will be assuming the role of the heroic DevOps team.

There is quite a bit of data written to the log file, more and more with each bugfix.

We’re going to use Clojure to investigate this problem.

(08:50) What problems might we encounter?

Problem 1: Size

No one configured log rotation!

The log file is too large to load for analysis.

Problem 2: Unstructured data

Everything is a line or multiple lines.

We can build up layers of abstractions to gradually build understanding.

Problem 3: Non-linear data

We want to tell a story about what went wrong.

Many times, the pieces of that story are in different areas of the log file, and must be collated.

There can be multiple competing stories.

The parts of the story are like dominoes. How do you know if the third domino is important without knowing that the first two have fallen?

Problem 4: Alerts

“If you have a really good story to tell, how do you tell anyone about it?”

Alerts should be timely, depending on the audience.

Call out to the audience: do you have battle stories about processing log files? Let us know!

Clojure in this episode:

nil

]]>Christoph Neumann and Nate JonesChristoph has gigs of log data and he’s looking to Clojure for some help.2816:56Christoph has gigs of log data and he’s looking to Clojure for some help.Ep 027: Collected Contexthttps://clojuredesign.club/episode/027-collected-context/
https://clojuredesign.club/episode/027-collected-context/Fri, 03 May 2019 08:00:00 -0700 Nate and Christoph reflect on what they learned during the Twitter series.Nate and Christoph reflect on what they learned during the Twitter series.

6 months of podcast episodes!

Situated programs: Part of the real world. Affect and affected by the world around it.

(04:25) Concept 1: Was very helpful to focus on nailing down the algorithm before diving into the code.

]]>Christoph Neumann and Nate JonesNate and Christoph reflect on what they learned during the Twitter series.2733:20Nate and Christoph reflect on what they learned during the Twitter series.Ep 026: One Call to Rule Them Allhttps://clojuredesign.club/episode/026-one-call-to-rule-them-all/
https://clojuredesign.club/episode/026-one-call-to-rule-them-all/Fri, 26 Apr 2019 08:00:00 -0700 Christoph thinks goals are data, not function names.Christoph thinks goals are data, not function names.

We were talking about the twitter handle again.

Last week, we talked about faking. It’s not mocking.

The magic that makes it possible is using a protocol.

Switch out the real handle or the fake handle in component based on configuration.

“Yes, I do want to speak to the log file.”

“Sometimes, the log file gets lonely.”

(02:43) Christoph wants to talk about the protocol.

We made a function for each operation we did in Twitter.

Post tweet

Search

Get timeline

Each function took the information needed for that operation as individual parameters.

(04:32) Example: the AWS API wrapper that Cognitect released.

Other wrappers are enormous, with functions for each AWS functions.

Cognitect’s wrapper has just one: invoke. At least only one that gets work done.

Unconventional (aka “weird”).

Take a step back. What needs to happen to call a remote API?

We need to make an HTTP call to some end point.

“100% of the information that that server needs to get from us, we transmit as data.”

“The path, the headers, everything about it is data.”

“The function is in the URL and the URL is in the data. It’s all data.”

If you want the operations to be exposed as functions, you end up promoting the operation to be a function name, but when making the request you need to put the operation back into the data.

Benefit (not really): we get to write a lot of boilerplate code.

“Nothing like some boilerplate to get you warmed up in the morning.”

(09:15) Back to the Twitter handle, what if we just had one function?

“Making a function for each endpoint goes against what we talked about last week, which is making our context explicit.”

(09:55) First benefit of one function: Simplifies the worker half of our algorithm.

]]>Christoph Neumann and Nate JonesNate wants to experiment with the UI, but Twitter keeps getting the results.2525:16Nate wants to experiment with the UI, but Twitter keeps getting the results.Ep 024: You Are Here, but Why?https://clojuredesign.club/episode/024-you-are-here-but-why/
https://clojuredesign.club/episode/024-you-are-here-but-why/Fri, 12 Apr 2019 08:00:00 -0700 Christoph needs to test his logic, but he must pry it from the clutches of side effects.Christoph needs to test his logic, but he must pry it from the clutches of side effects.

Last week we ended up with “imperative mud”: lots of nested I/O and logic.

(01:45) Difficult to test when all the I/O and logic are mixed together.

]]>Christoph Neumann and Nate JonesChristoph needs to test his logic, but he must pry it from the clutches of side effects.2434:37Christoph needs to test his logic, but he must pry it from the clutches of side effects.Ep 023: Poster Childhttps://clojuredesign.club/episode/023-poster-child/
https://clojuredesign.club/episode/023-poster-child/Fri, 05 Apr 2019 08:00:00 -0700 Nate gets messy finding ingredients for his algorithm cake.Nate gets messy finding ingredients for his algorithm cake.

Last week we focused on how to determine what to post.

This week we focus on getting the data so we can compare it.

(01:55) Once again, we’ll use component to organize our app.

What components should we have?

Component 1: The worker that wakes up, checks the DB, checks Twitter, and posts as necessary.

]]>Christoph Neumann and Nate JonesChristoph questions his attempts to post to Twitter.2233:52Christoph questions his attempts to post to Twitter.Ep 021: Mutate the Internethttps://clojuredesign.club/episode/021-mutate-the-internet/
https://clojuredesign.club/episode/021-mutate-the-internet/Fri, 22 Mar 2019 08:00:00 -0700 Nate wants to tweet regularly, so he asks Clojure for some help.Nate wants to tweet regularly, so he asks Clojure for some help.

Problem: pre-author tweets so they can get posted automatically.

Want to make a “full stack” application this time.

Sounds complicated, what do we need?

Database of tweets: text to tweet and timestamp when it should be tweeted.

Frontend is a single-page application (SPA) that makes XHR (“AJAX”) requests to the backend.

Needs to be able to wake up and post.

Persistent process backend.

We will not cover all these parts.

“You have a problem, so you make a UI to solve your problem. Now you have 2 problems.” “More like 18 problems!”

We will focus on logic interacting with Twitter.

When should it post?

How does it know if a tweet has been posted?

What to do when Twitter returns an error?

Overarching theme: how do you deal with side-effects in a functional way?

Remember Ep 020: push side-effects and I/O to the edges.

Easy to fetch “current” time, but that’s a side effect!

“Just because it’s easy doesn’t mean it’s pure.”

If you make time a parameter, all of a sudden you can mess with it!

“If only real time was a parameter we could manipulate.”

Lots of fun to be had in the upcoming episodes.

”‘Start with the data’ is something we’ve come to again and again. If you can model the data, that’s a very good place to start.”

]]>Christoph Neumann and Nate JonesNate wants to tweet regularly, so he asks Clojure for some help.2122:28Nate wants to tweet regularly, so he asks Clojure for some help.Ep 020: Data Desserthttps://clojuredesign.club/episode/020-data-dessert/
https://clojuredesign.club/episode/020-data-dessert/Fri, 15 Mar 2019 00:00:00 -0800 Christoph and Nate discuss the flavor of pure data.Christoph and Nate discuss the flavor of pure data.

“The reduction of the good stuff.”

“We filter the points and reduce the good ones.”

Concept 1: To use the power of Clojure core, you give it functions as the
“vocabulary” to describe your data.

“predicate” function: produce truth values about your data

“view” or “extractor” function: returns a subset or calculated value from your data

“mapper” function: transforms your data into different data

“reduction” (or “reducer”) function: combines your data together

Concept 2: Don’t ignore the linguistic aspect of how you name your functions.

Reading the code can describe what it is doing.

Good naming is for humans. Clojure doesn’t care.

Concept 3: Transform the data source into a big “bag” data that is true
to structure and information of the source.

Source data describe the source information well and is not concerned with
the processing aspects.

Transform into data that is useful for processing.

Concept 4: Using loop + recur for data transform is a code smell.

Not composable: encourages shoving everything together in one place.

“End up with a ball of mud instead of a bag of data you can sift through.”

“You know what mud sticks to really well? More mud! It’s very cohesive! And
what couldn’t be better than cohesive programs!”

]]>Christoph Neumann and Nate JonesChristoph and Nate discuss the flavor of pure data.2028:40Christoph and Nate discuss the flavor of pure data.Ep 019: Dazed by Weak Weekshttps://clojuredesign.club/episode/019-dazed-by-weak-weeks/
https://clojuredesign.club/episode/019-dazed-by-weak-weeks/Fri, 08 Mar 2019 00:00:00 -0800 Nate wants to see more than data structures in a REPL.Nate wants to see more than data structures in a REPL.

Goal: see a text-based calendar of hours worked by day and week.

“With the power of the predicate, I summon thee from the bag of data!”

If data items share structure, you only need one predicate function, not separate functions per “type”.

Common pattern: filter then map, then reduce

Output:

Looks like a calendar with one line per week

Show daily total for each day

Show weekly total

Problem: how do we split the days into weeks?

“I don’t remember split-weeks in the Clojure cheat sheet.”

“What day does your week start on? My week starts on Tuesday.”

“I always assume that when a programmer says ‘most’, they mean ‘most of the people around me’.” “I actually think it just means ‘me’.”

“Anecdata: when you have two anecdotes, it makes data.”

Create higher-level summaries: entry → day → week

Sift through data at the level of your question.

Problem: A date identifies a “day”, but what identifies a week? A “week ID”, so to speak.

It would be nice if that week ID was ordered, so we could sort on it.

Idea: Use the date of the first day of the week

uniq, sortable, spans years

weeks that start on Tuesday will never have the same ID as weeks that start on Sunday

Might as well have the starting day of the week in the map too: :sunday, :tuesday, etc.

Convenient to have two different views for the same data in the same data structure

immutable data won’t change and get inconsistent.

it’s pre-calculated if you need to use it a lot vs just using a predicate “on the fly”.

]]>Christoph Neumann and Nate JonesNate wants to see more than data structures in a REPL.1929:56Nate wants to see more than data structures in a REPL.Ep 018: Did I Work Late on Tuesday?https://clojuredesign.club/episode/018-did-i-work-late-on-tuesday/
https://clojuredesign.club/episode/018-did-i-work-late-on-tuesday/Fri, 01 Mar 2019 00:00:00 -0800 Christoph wants to teach filter some vocabulary.Christoph wants to teach filter some vocabulary.

Continuing our discussion of rummaging through the big bag of data.

Mental shift for solving the problem:

Prior thinking: one function that has to look at all entries

New thinking: filter out irrelevant entries then reduce on just those

New mentality emphasizes the problem of “picking” over “combining”. Once you have the right set of entries, the reduce becomes trivial.

Idea: build up a “vocabulary” of picking.

“You build up the language to the level you want to use it at.”

A “predicate” is simply a function that gives you a truth value.

We want to create a set of predicates to use with filter.

By convention, predicates in Clojure end with ?. Eg. some?, contains?, every?

]]>Christoph Neumann and Nate JonesChristoph wants to teach filter some vocabulary.1828:51Christoph wants to teach filter some vocabulary.Ep 017: Data, at Your Servicehttps://clojuredesign.club/episode/017-data-at-your-service/
https://clojuredesign.club/episode/017-data-at-your-service/Fri, 22 Feb 2019 00:00:00 -0800 Nate finds it easier to get a broad view without a microscope.Nate finds it easier to get a broad view without a microscope.

After last week’s diversion into time math, we are back to the core problem this week.

Now we want a total by date.

Need to refactor the function to return the date in addition to minutes.

“We’re letting the code grow up into the problem.”

“Let’s let the problem pull the code out of us.”

First attempt

Use map to track running totals by day

As each new entry is encountered, update the total for that day in the map

New complication: Now we want a total for all work on Sundays.

The loop + recur approach is getting complicated!

More and more concerns all mixed together in one place

Closely ties the traversal of the data to the processing of the data

Better idea: use reduce. Just write “reducer” functions.

Simplify by ensuring data passed to reduce is already filtered.

“In imperative land, let’s take three different dimensions of consideration and shove them all together in this one zone.”

Motivating question for a solution: “How is this composable?”

“In Clojure you end up with really small functions because you end up composing them at the end.”

Ugly: the reducer for “work on Sundays” still has an if for throwing away data.

Better: add another filter to just pass through Sundays.

Best: minimal work in the reducer. Use map and filter to get the data in shape first.

Imperative thinking: what value do I need to operate on?

Functional thinking: how can I accurately represent the data present in the input?

After you have all the data at hand, you can summarize it however you want!

Why reducers? When you need to operate one step at a time: streaming data, game state, etc.

]]>Christoph Neumann and Nate JonesNate finds it easier to get a broad view without a microscope.1728:28Nate finds it easier to get a broad view without a microscope.Ep 016: When 8 - 1 = 6https://clojuredesign.club/episode/016-when-8-minus-1-equals-6/
https://clojuredesign.club/episode/016-when-8-minus-1-equals-6/Fri, 15 Feb 2019 00:00:00 -0800 Christoph discovers that time creates its own alternate universe.Christoph discovers that time creates its own alternate universe.

Continuing our exploration of “literate” time logs

We want a function to turn the timestamps into minutes.

Format: Fri Feb 08 2019 11:30-13:45

Keep it simple: extract times with a regex and use math

minutes in day = hours * 60 + minutes

total minutes = end minutes - starting minutes

Problem: What happens when we work past midnight? Negative minutes!

We decided to have only one date, so a time before the starting time must span midnight.

Format only allows for an activity to be <24 hours.

“If we end up doing any activity in our life longer than 24 hours, I think we should that we might have other problems.”

Easy Solution: If the end time is before start time, add 24 hours.

“When I get past any sort of simpler math, I just type it into my connected editor REPL because I know Clojure can do it faster than I can.”

Now we have a function to get minutes, want to add them all up.

Use loop and recur to iterate through the array and track the sum.

Oh wait, what about Daylight Savings Time?

“We all pretend that time is actually in the future.”

If it involves dates and times, we can’t just do simple math.

“If I do this the right way, I now have to open a whole new can of worms.”

Easy way out: write “doesn’t support DST” in the release notes and call it “user error”!

“Any time you have to be careful about something, you’re probably doing it wrong.”

Use a time API.

“The Clojure library is just a wrapper because nobody wants to reinvent this thing.”

Java time is immutable, so that works nicely with functional languages. No cloneing!

Lots of functions in the time API. Which ones do we need?

Our workflow: try out different expressions in a comment block in our connected editor to figure out the specific time functions we need.

“Local” dates and times don’t have time zones, but “zoned” dates and times do have them.

Need to create timestamps for accurate math: date + time + zone

In practice, when we use dates and times, they are implicitly connected to a time zone.

Your time zone is your alternate universe: it affects the meaning of your dates and times.

We added support for DST without changing the function signature.

But, how do we add up other totals? Looks like we’re going to need to change even more.

]]>Christoph Neumann and Nate JonesChristoph discovers that time creates its own alternate universe.1629:37Christoph discovers that time creates its own alternate universe.Ep 015: Finding the Timehttps://clojuredesign.club/episode/015-finding-the-time/
https://clojuredesign.club/episode/015-finding-the-time/Fri, 08 Feb 2019 00:00:00 -0800 Nate spends some time figuring out how to track his time.Nate spends some time figuring out how to track his time.

New problem: Want to track time using a text file.

Text files are great:

Keyboard oriented: in terminal, in editor

Something you can put under revision control

Need: date and time range

Format of the time stamp should be convenient for us to read and edit.

Let the computer do the work to convert the convenient notation to something usable!

Format: Fri Feb 08 2019 11:30-13:45

One timestamp per line. Oh wait, we need a description. Six minutes in and we’re already changing the requirements!

What are “literate” text files? “You can mix human words in amongst data the computer will use to make it more understandable for the humans.”

How to find the times? Attempt to match the time format within each line.

“It’s kind of like an inverse comment. Instead of every line being valid and you have to comment out lines you don’t want, if it’s in a known format, those are the lines we want and everything else is a comment.”

Can use line-seq with clojure.java.io/reader. That uses lazy I/O.

Potential Issue: lazy I/O can defer I/O errors to when you’re in the middle of processing the data.

Lazy I/O is less of an issue with files and more of an issue with network sockets.

]]>Christoph Neumann and Nate JonesNate spends some time figuring out how to track his time.1526:22Nate spends some time figuring out how to track his time.Episode 014: Fiddle With the REPLhttps://clojuredesign.club/episode/014-fiddle-with-the-repl/
https://clojuredesign.club/episode/014-fiddle-with-the-repl/Fri, 01 Feb 2019 00:00:00 -0800 Christoph gets some work done by fiddling around.Christoph gets some work done by fiddling around.

Editor integration is a massive change in how to use the REPL.

Put a comment block right underneath a function you are working on, and invoke the function with test arguments.

Can quickly make changes to code and try it out–all in the same place!

Problem: comment blocks littering the code.

“A good rule of thumb I’ve found is: when I have to use the world ‘careful’, it’s a big red flag. I’m doing something I should be able to prevent without having to be careful.”

Code in comment blocks can be helpful in the future.

”‘It’s not done yet.” What file is ever done? You’re always going to come back to it in 6 months.”

Easy to skip around between forms you’ve run before.

“It’s like a random access REPL history.”

Ah ha moment: make a separate file for the helpful comment blocks.

“[The comment blocks] are like the copilot of my development experience.” “They’re me helping me!” “You’re your own copilot!”

Separate file needed a namespace: the fiddle namespace was born.

“I’m just fiddling around.”

Put the “fiddle files” in the dev tree so it only gets loading in development.

]]>Christoph Neumann and Nate JonesChristoph gets some work done by fiddling around.1429:28Christoph gets some work done by fiddling around.Episode 013: Connect the REPLhttps://clojuredesign.club/episode/013-connect-the-repl/
https://clojuredesign.club/episode/013-connect-the-repl/Fri, 25 Jan 2019 00:00:00 -0800 Nate continues to explore the REPL by gluing it to his editor to see what happens.Nate continues to explore the REPL by gluing it to his editor to see what happens.

We revisit Tic-Tac-Toe and change it using our new REPL superpowers.

Let’s add request logging to the play endpoint: /play?row=0&col=1.

Need to add log/info calls in handler function for play endpoint: on-play.

Naive Christoph would add logging to the function, recompile, and run the jar.

Less naive Christoph would reload the namespace but still restart the web server.

Crazy idea: paste the function definition into the REPL!

Copy the whole defn for on-play

Go to the REPL window

Switch to the app.main namespace (where we put on-play)

Paste the function definition into the REPL

No restarting required! Each time we evalute the defn in the REPL, the new function replaces the old version immediately!

We can iterate on a single function quickly.

Clojure is Lisp, and Lisp is dynamic and isomorphic

Dynamic: can change definition on the fly, so the new function exists immediately after executing the definition

Isomorphic: everything is a “form”. No special “declaration syntax” vs “expression syntax”. A declaration is a form just like any other expression.

“Clojure is dynamic to the end. Every statement is modifying the running application, and because every statement has access to the entire application, all the way down to the definitions, you can redefine on the fly.”

Declarations are just ordinary macros (def, defn, defmacro, etc), not special syntax only the compiler understands.

A declaration macro updates the enclosing namespace dynamically on the fly!

“You just replaced the wheels on the car while you were driving.”

“I can replace the steering wheel and my hands don’t even realize they’re using a new steering wheel.”

“Running the form is what brings that thing dynamically into existence or replaces the one that was there.”

“It’s the way you can speed your development up because you can replace these functions on the fly.”

Problem: you have to copy and paste between the editor and the REPL.

Idea: use terminal automation to help with the copy + paste!

“Yet another example of solving a problem you probably shouldn’t even have.”

Pre-Lisp thinking: “The right way to do software is to compile it from grains of sand each and every time, and then run it from the beginning.”

]]>Christoph Neumann and Nate JonesNate continues to explore the REPL by gluing it to his editor to see what happens.1332:10Nate continues to explore the REPL by gluing it to his editor to see what happens.Episode 012: Embrace the REPLhttps://clojuredesign.club/episode/012-embrace-the-repl/
https://clojuredesign.club/episode/012-embrace-the-repl/Fri, 18 Jan 2019 00:00:00 -0800 Christoph complicated development by misunderstanding the REPL.Christoph complicated development by misunderstanding the REPL.

We go back to the roots of Christoph’s experience with Clojure…

How do I run Clojure code?

The REPL is fun for evaluation, but how do you run a “real” program?

Experience in other languages: edit the file, compile, rerun the program

Mentality: get through the edit, compile, run loop as fast as possible!

Autobuilder is the logical end.

“Where’s my autobuilder in Clojure?!”

The REPL model is fundamentally different.

“[The REPL] is like cutting the Gordian Knot. You change the problem and that’s how you solve it.”

“The reason why the problem I wanted solved, wasn’t ‘solved’, is because nobody has that problem because they do things differently here.”

Tactic 1: edit, save, restart the REPL

“Restarting the REPL isn’t super slow, but let’s just say it’s not instantaneous.”

Tactic 2: edit, save, run (use 'the.namespace :reload) in the REPL

“Now I have a REPL history of use statements!”

Problems:

forgetting to reload dependent namespaces

loading dependencies in the wrong order

old definitions built up

Big Problem: function renames left the old version around, so accidental calls using the old name produced no errors and ran old behavior!

Back to quitting the REPL to clean out the cruft. Ugh!

Discovered clojure.tools.namespace! Reloads everything and cleans out the cruft!

]]>Christoph Neumann and Nate JonesChristoph complicated development by misunderstanding the REPL.1225:33Christoph complicated development by misunderstanding the REPL.Episode 011: The Convention of Configurationhttps://clojuredesign.club/episode/011-the-convention-of-configuration/
https://clojuredesign.club/episode/011-the-convention-of-configuration/Fri, 11 Jan 2019 00:00:00 -0800 Nate is worried about the hardcoded credentials in the code.Nate is worried about the hardcoded credentials in the code.

It’s episode 011 on 01/11. It’s binary!

“As a developer, you quickly learn what’s under your control and what’s not. Very little is under your control.”

]]>Christoph Neumann and Nate JonesNate is worried about the hardcoded credentials in the code.1128:47Nate is worried about the hardcoded credentials in the code.Episode 010: From Mud to Brickshttps://clojuredesign.club/episode/010-from-mud-to-bricks/
https://clojuredesign.club/episode/010-from-mud-to-bricks/Fri, 04 Jan 2019 00:00:00 -0800 Christoph can't stand the spaghetti mess in main. Time to refactor.Christoph can’t stand the spaghetti mess in main. Time to refactor.

Main function does too much. We want cohesive pieces that each have one job.

Two part problem:

Too much in the main loop.

Main starts and then never stops. Have to exit the repl to restart.

We need 1) separate parts that can be 2) started and stopped.

Main should just focus on the code needed for command line invocation.

Let’s get the parts component-ized!

“It’s one thing that I’ve become more sure of in my career is that no application is actually done.” “Useful apps only get more complex.”

Internal dependencies are different than external dependencies (“libraries”).

Many internal dependencies create high coupling throughout the code.

“Once everything starts touching everything you have to understand everything to understand anything.”

Like functions use parameters to limit scope, a component is the next level up and uses resource dependences to limit coupling.

Each component implements a clear behavior (interface) and can be a resource to other components.

Can understand component’s behavior via its interface (and docs) without reading all the code–just like understanding a function through it’s signature and docs.

We use the “Component” library to make components in Clojure–has REPL integration too.

Components to make:

web server

polling and fetching loop

the core.async channel used between them

The Lifecycle interface allows a component to be started and stopped.

start must return a reference to the “started” state

stop must return a reference to the “stopped” state

Gotcha: don’t return a nil!

To use Lifecycle, you’ll need to make your component a “record”.

Two main goals of Component:

allow stateful components

define dependencies between components

Surprise! Any reference can be a “component” as a non-lifecycle dependency.

Write a function new-system which returns the component “system map”

Mind your names. Make the system map key match a component’s dependent field.

“Component is a convenient way of being able to specify all those dependencies in a concise map, so you know this is the intersection of all of my application together.”

A component should be able to be understood alone.

“Component is like giving you application parts as function parameters. Just like when making a function, you don’t worry about how something gets passed in as a parameter.”

You still need to understand each of the parts, but you don’t have to worry about where the part came from.

At the top level, you can see all the parts together and how they are connected.

Immutability gives you bulkheads between each of your components so you can safely reason about them separately.

Use component.repl to start and stop the whole system without restarting the REPL!

Need some tooling to keep the main thread from exit. Can use promise, deref and a shutdown handler (see below).

“We can keep each ball of mud it its own little basket so all the mud doesn’t ooze together.”

]]>Christoph Neumann and Nate JonesChristoph can't stand the spaghetti mess in main. Time to refactor.1028:35Christoph can't stand the spaghetti mess in main. Time to refactor.Episode 009: Channeling Re-Searchhttps://clojuredesign.club/episode/009-channeling-re-search/
https://clojuredesign.club/episode/009-channeling-re-search/Fri, 28 Dec 2018 00:00:00 -0800 Nate can't decide what to watch on Twitter, and the app restarts are killing him.Nate can’t decide what to watch on Twitter, and the app restarts are killing him.

The Twitterthon continues. “It’s an infinite stream of Twitter.”

Nate wants to watch #clojurescript too.

Just change the search string to "#clojure OR #clojurescript"?

Should avoid hardcoding the value so we don’t have to recompile every time.

Command line argument still requires a restart.

Let’s use a curl UI (like Ep 004)!

Wait, what?! Why curl?

Can send something into the running process

Can separate out the client

Builds on top of HTTP, so there are lots of tools for testing (like curl itself)

Use a URL query string: http://localhost:8000/search?The+search+string

“It’s a true query string in the truest sense of the term ‘query.’”

“It is actually using the thing for the thing it was meant to be.”

How do we get the query string from the webserver thread to the polling loop thread?

“This is a perfect case for a global.” Oh wait, that’s mutation.

How should we structure the app? The main function must:

start the thread for the web server

start the thread for the polling loop

Specific problem: the /search route handler function needs a way to send the new query string to the polling loop.

With mutation: share an atom and give the handler and the polling loop a reference to it.

No mutation? Use a core.async channel.

A channel allows a thread to pass data to another thread.

With no channel buffer, it will synchronize two threads (“rendezvous”).

(We lie. There is no peek function.)

Problem: polling thread is stuck waiting on the channel, so it stops polling.

Solution: Use alt!! to simultaneously listen to the “new search” channel and a timeout channel.

What is a timeout channel? A channel that closes after n msecs have passed.

New problem: the cache (for de-duplicating) has stale content after changing the query.

Solution: same logic that adopts the new search term will reset the cache. (See Ep 007.)

Polling loop structure:

fetch

process results

block while listening to the channels

if new search string, recur with new cache and new search string

otherwise, recur with same cache and search string

Only want the fetch in one part of the loop.

Don’t even need curl. Just type in the URL for the backend on your phone.

core.async lets threads coordinate without having to know about each other!

]]>Christoph Neumann and Nate JonesNate can't decide what to watch on Twitter, and the app restarts are killing him.926:47Nate can't decide what to watch on Twitter, and the app restarts are killing him.Episode 008: Twitter, Platedhttps://clojuredesign.club/episode/008-twitter-plated/
https://clojuredesign.club/episode/008-twitter-plated/Fri, 21 Dec 2018 00:00:00 -0800 Christoph tries to figure out why Twitter stopped talking about Clojure.Christoph tries to figure out why Twitter stopped talking about Clojure.

“Are you twitterpated?”

Building on where we left off last episode.

Runs and just stops working.

“I was pretty sure it stopped working because people on Twitter just stopped talking about Clojure. After about a day of that, I realized people were talking about Clojure, and I just wasn’t seeing it.”

The auth token expired! What should we do?

Why should the main loop have to deal with getting a new auth token?

“The Twitter wrapper should be concerned with all of the warts and complexities of dealing with Twitter.”

“What problems should bubble up, and which ones shouldn’t?”

The wrapper should handle the retry.

It’s like a kitchen in a restaurant. What are the steps of fulfilling an order? The customer doesn’t care.

“There’s a side-effect: the freezer mutates.”

The wrapper gets to worry about all the steps:

turning the order into the specific request for the kitchen

do the I/O to fetch and fulfill the request

the “input transform” takes the mass of data and picks out the relevant parts

the “internal” version is returned

“Like all good metaphors, they stretch to the point where they break, like a rubber band.”

If the handle is mutable, then retry can just update the handle with the new token.

A mutable handle does allow the wrapper to control the concern.

The “handle” is the state of the wrapper. The term “handle” comes from I/O libraries.

Instead of mutation, have the search function return [updated-handle, result].

search can catch an auth exception, retry, and return a new auth handle.

Instead of search retrying, the fetcher can do it! Then it works for all kinds of requests.

Better yet, leave fetch simple, and have a fetch-with-retry function that uses fetch.

Can have even more policy functions like, fetch-with-retry-forever.

“Keep calm, and assoc on.”

“I’m never going to miss another Clojure tweet. I’m going to read them all!”

Clojure in this episode:

loop, recur

try, catch

atom

assoc

]]>Christoph Neumann and Nate JonesChristoph tries to figure out why Twitter stopped talking about Clojure.823:07Christoph tries to figure out why Twitter stopped talking about Clojure.Episode 007: Input Overflowhttps://clojuredesign.club/episode/007-input-overflow/
https://clojuredesign.club/episode/007-input-overflow/Fri, 14 Dec 2018 00:00:00 -0800 Nate just wants to print the tweets, but the input wants to take over.Nate just wants to print the tweets, but the input wants to take over.

Got tweet fetching working last time. Now we want to print out the tweets.

API returns a lot information

“We should probably not use pprint as our output mechanism.”

Using the uniqueness filtering from last time

The goal is NOT to write a generic Twitter wrapper.

Goal is to write a wrapper that just gives us what our application needs.

“I don’t want to spread the complexity of Twitter all over my app. I want to hide it inside of the wrapper.”

]]>Christoph Neumann and Nate JonesNate just wants to print the tweets, but the input wants to take over.726:34Nate just wants to print the tweets, but the input wants to take over.Episode 006: All Wrapped Up in Twitterhttps://clojuredesign.club/episode/006-all-wrapped-up-in-twitter/
https://clojuredesign.club/episode/006-all-wrapped-up-in-twitter/Fri, 07 Dec 2018 00:00:00 -0800 Christoph tries to get a handle on his #clojure tweet-stream habit.Christoph tries to get a handle on his #clojure tweet-stream habit.

NOT tic-tac-toe

Follow #clojure tweet stream and see it print out in the terminal

“We like reinventing things.”

“The terminal is the best UI out there.”

Does Twitter have an API?

Websocket? Nope! Requires a big “E” plan: “enterprise”.

PubSub? Nope! Not from Twitter.

Alas, we must poll the /search/tweets.json endpoint

Problem: we’re going to keep getting results we’ve already seen

Avoid duplicates? Let’s use core cache.

Once again, we use loop and recur for our main loop

Time for an API wrapper, but what does the wrapper do?

HTTP POST form-encoded parameters

Ack! 401 Access Denied

“An important step in any API exploration is your first 401 response.”

“OAuth?” “Oh….auth…”

Meet OAuth, the API bouncer.

Make an auth function to call the OAuth endpoint and get an auth token

Have auth return a “handle” with the auth token. Other wrapper functions will need handle.

Need to keep handle around. Put that in the app state too.

Let the exceptions fly!

“Exceptions are an exceptionally accepted way of handling exceptional circumstances.”

“I caught what you meant.”

Make a fetch function that does the I/O work.

Create a search function that takes handle and query

Look for pure logic and move it into its own function, then it’s easy to test.

Transform args to search into a “request description” and have fetch operate on that.

“Twitch, I mean Twitter. You know, that Internet thing that starts with t-w-i-t.”

]]>Christoph Neumann and Nate JonesChristoph tries to get a handle on his #clojure tweet-stream habit.625:03Christoph tries to get a handle on his #clojure tweet-stream habit.Episode 005: Tracking, for the Winhttps://clojuredesign.club/episode/005-tracking-for-the-win/
https://clojuredesign.club/episode/005-tracking-for-the-win/Fri, 30 Nov 2018 00:00:00 -0800 Nate tries to figure out who actually won this never-ending game of tic-tac-toe.Nate tries to figure out who actually won this never-ending game of tic-tac-toe.

Tic-tac-toe is “just boring enough, just interesting enough.”

How do we know who won the game? Inspect the board.

If you can track progress toward the win, you check for the win quickly

“Tic-tac-toe at scale!”

Tracer bullet: go simple, just examine the 8 options for winning

“In that case, nil has won…which is nobody.”

Keep detection logic out of the high-level winner function–should read like a process description of steps.

Make new “verbs” like row-winner and column-winner and use those.

“You’re just adding new verbs to raise Clojure up to the level of your problem. You can speak about your problem using those verbs.”

Let’s make it faster! Need incremental detection to be efficient.

Tracking structure with win condition totals

keys like: [playercaseindex]

value is a counter for that case

eg. { [:x :row 0] 1, [:y :row 0] 0, [:x :diag] 2, ...}

The win tracker is a “nested model” of the game state

Put the tracker it its own namespace app.game.tracker

Use [:x 1 0] as the play

Nested updates: (update game-state :tracking tracker/update [:x 1 0])

How do we handle diagonals? Not all moves will increment those totals.

“If you see a word that’s a higher level concept, it allows you to stay at that higher level and be able to view the algorithm instead of viewing the implementation. That’s the point of lifting up all these little functions.”

]]>Christoph Neumann and Nate JonesNate tries to figure out who actually won this never-ending game of tic-tac-toe.526:19Nate tries to figure out who actually won this never-ending game of tic-tac-toe.Episode 004: Atomic Curlshttps://clojuredesign.club/episode/004-atomic-curls/
https://clojuredesign.club/episode/004-atomic-curls/Fri, 23 Nov 2018 00:00:00 -0800 Christoph tries to make tic-tac-toe work over the Internet and discovers the power of the atom.Christoph tries to make tic-tac-toe work over the Internet and discovers the power of the atom.

Let’s get a web framework, build a UI, hook that up to an HTTP API, throw in some websockets for notifications!

“We’d end up with our first 12 hour podcast episode.”

A “tracer bullet”: get something functioning to see the important parts of the solution.

Can choose when to replace important parts by something more “production worthy”.

“Let’s just ditch all the complexity of having a UI and just use curl.”

curl will print the text response. We have a terminal UI!

“We’re extending the command line out to the web!”

“I just keep curling every day or every other day until it’s my turn.”

“It’s your morning curls!”

How do we handle web requests? What is a route?

Routes: /new, /show, /play?row=0&col=1

“Super simple URL API. Who needs REST?”

Let’s run this on port 1337 to hide it from the Internet.

One shared game state stored in an atom. It’s the only game in town!

Use an atom instead of a database to cut complexity while problem solving.

The ! (“bang”) in swap! and reset! indicates you’re causing a side effect.

Handler’s sole job: take web requests and alter the game state using the game model.

Function called by swap! should be pure. Don’t throw exceptions!

Dilemma: How much do you put inside the function called by “swap!”?

For errors, the transaction function can:

Return unchanged reference. Use “triage” function to diagnose.

Have an “error” attribute in the state and set an “outcome” code like :success,:invalid-coordinate, etc.

Tracer bullet shows lots of ways to complex-ify this into “Tic-Tac-Toe, Enterprise Edition”

]]>Christoph Neumann and Nate JonesChristoph tries to make tic-tac-toe work over the Internet and discovers the power of the atom.422:40Christoph tries to make tic-tac-toe work over the Internet and discovers the power of the atom.Episode 003: Tic-Tac-REPLhttps://clojuredesign.club/episode/003-tic-tac-repl/
https://clojuredesign.club/episode/003-tic-tac-repl/Fri, 16 Nov 2018 00:00:00 -0800 Nate tries to turn the tic-tac-toe game engine into a real application he can play with a friend.Nate tries to turn the tic-tac-toe “game engine” into a real application he can play with a friend.

Let’s play the game!

How do you keep track of the game state as it changes?

Bucket brigade the reference to the future loop using recur

The game loop: read input, evaluate, print out new board, loop.

“It has occurred to me that we are basically writing a REPL.”

“We have the tic-tac-REPL”

How do you get input from the user? How to you make sure it’s right?

“It keeps harassing the non-compliment user until they type the right thing in”

Input loop: read, validate, loop on error, return on success

Keep the logic pure! Separate out the parsing and validation functions.

“Much better to tuck it away in a function!”

Sequence the pure parts with a minimalist function that does the I/O.

“I don’t like having ovens present because they’re hard to put in my test cases.”

Unit test those pure parts. (No one likes to be mocked.)

I/O is a side effect!

“Every time I redefine one of those things I feel like I’m reaching down into the bowels of Clojure and doing something moderately illegal.”

Using keywords as error codes is nifty

“You don’t have any different kinds of nil. You just have one. It’s the nuh-uh.”

Can use a tuple with the first element always being a keyword and the second being data for the “details”

Clojure in this episode:

read-line

string/split

swap! and reset!

loop and recur

let vs loop

keywords

nil punning

]]>Christoph Neumann and Nate JonesNate tries to turn the tic-tac-toe game engine into a real application he can play with a friend.323:09Nate tries to turn the tic-tac-toe game engine into a real application he can play with a friend.Episode 002: Tic-Tac-Toe, State in a Rowhttps://clojuredesign.club/episode/002-tic-tac-toe-state-in-a-row/
https://clojuredesign.club/episode/002-tic-tac-toe-state-in-a-row/Fri, 09 Nov 2018 00:00:00 -0800 Christoph tries to make tic-tac-toe and gets stuck on immutability.Christoph tries to make tic-tac-toe and gets stuck on immutability.

The REPL calculator

Let’s make a game!

How do you keep track of the game board?

How do you update the game board if you can’t change anything?

OO told me the “right” way to encapsulate, now what do I do?

“Nine lines of let block, and one line of actual function.”

“The reference bucket brigade”

reductions wants to blow your mind

Multiple universes of tic-tac-toe, and they’re all in a row!

Time travel, for free!

Clojure in this episode:

assoc

assoc-in

->

reduce

reductions

]]>Christoph Neumann and Nate JonesChristoph tries to make tic-tac-toe and gets stuck on immutability.220:00Christoph tries to make tic-tac-toe and gets stuck on immutability.Episode 001: Why, Oh Why?https://clojuredesign.club/episode/001-why-oh-why/
https://clojuredesign.club/episode/001-why-oh-why/Fri, 02 Nov 2018 00:00:00 -0800 Nate and Christoph try to figure out how to make a podcast.Nate and Christoph try to figure out how to make a podcast.

Who are we?

What are we doing?

What will we talk about?

Lots of (with-meta podcast ...)

Clojure in this episode:

nil

]]>Christoph Neumann and Nate JonesNate and Christoph try to figure out how to make a podcast.113:44Nate and Christoph try to figure out how to make a podcast.