With every new tool, framework or methodology comes along, developers with their insatiable appetite or thirst for knowledge and power, they will find ways to make dedicated time to learn how they work and how they’re planning to use them as part of the day to day job.
And functional programming (FP) - a newish paradigm has been permeating through the scenes of developer community for some time; everything from Haskell, Elixir, React, AWS Lambdas to Clojure etc.
Or, at least it’s yet to make establish some norms within the community…
But I must digress.
Some Refresher Points
reduce are the default go-to tools for expressing their FP-ness in all over their front-end codebase.
You’ve probably seen those patterns numerous of times via google searches or countless Medium or FreeCodeCamp tutorial blogpost or what not.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31
In the JS world, this is how we roll our F.M.Rs mojos. 👩💻👨💻😎
Now, let’s head the Python fence, and see how they’re handle things over there.
Disclaimer: Before I proceed further, I just want to say, beforehand, at the time of this writing, as as the entire online community is facing a major unprecedented challenge in our lifetimes and we want all doing the best to treat things with great caution, the examples I’ll be using next may cause upset to the readers and I do not mean to cause any grave amount of discontentment amongst. The purpose of my blog is to purely demonstrate my learnings over the past few weeks since start of the global pandemic situation, and wish to reiterate my platform will be used for this purpose only. I wish to apologise in advance.
Lately, I have been dabbling on how Python does well with its own
reduce functions approach, and I couldn’t help to look at the following datasets provided from the Dr John Hopkins University’s Github repo on latest covid19 cases.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45
Their data were originally in CSV formatter so I wrote my own little Python script that feeds on these CSV files and have them fully JSONified and from here, I can see immediate pattern going on here.
It reveals that a lot daily cases are collected from every state and provinces in each country and are individually reported for number of cases that are categorised as either confirmed, recovered and death cases. With this, my immediate thought is this would be a good case to place Python FP’s exercise into good use.
To start off, I quickly whipped up my own mini API app using Flask/Falcon - (which doesn’t really matter to be honest)
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38
In this example, I made my resource endpoint
covid19 and it expects
file_id as the main parameter_id used to query a specific json file to be fetched from the server.
In order to make the right (and exact) search for the specific covid19 json file, I decided to add some a bit regex expression here just to make sure
file_id matches the date format my json files are named ie
mm-dd-yyyy. That way I will have Python exception handler to capture the error should the
file_id failed to meet the regex pattern matching requirement, raised the ValueError as that will halt entire get resource API operation.
Once the file_id matching completes, then we can start making the
call (as below) and fetches correct json file off from the server,
1 2 3 4 5 6
Once the data is fetched upon returning from the
fetch_json_data call, I start building out my list of unique countries by using List and Set combo. With this new list, it will be used later which will be traversing the list of countries data of covid19 cases that matches with the dict’s key
Country/Region . It has two types of keys because John Hopkins University dataset had revised the data set representations in between the early cases of Coronavirus in January 2020 and Mar 2020.
Beforehand, I need to determine covid19 schema dictionary I want to extract from each item in the array using
get_covid19_schema like so.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24
See how I’m using
map function by passing both a function and an array iterator as parameters? This will tell the map operator to perform such function on each iteratee of the array iterator and return the consequence of the actions as a result. According to its API docs, the
map output signature comes as a Map object. Meaning that it’s only a proxy result and we need intermediary function call that will be responsible for converting it into actual list of each element that has been applied either a list, set or tuple. Which in this case, it’s going to be a list of my dictionary results.
This is fascinating as coming from the JS world, we don’t need intermediary function calls when we perform
map functions as JS itself does not come with complex data structures (ES6 Sets anyone 🤔?) that Python has. Arrays are the most basic and the most popular structure that JS Developers used by default so it’s easy to see why we didn’t need to worry getting proxying result set that Python has to come grips with… 😶
Later on, now that we got the list of
new_data of countries that comes with different state/provinces that has different covid19 cases data, the next job is to make an aggregate of the total number of covid19 cases for each category from each unique country, which I get from my
unique_countries list. Each of this country’s aggregate numbers take into this form.
1 2 3 4 5 6
Then, I put them altogether into one big list as a payload response to the client side.
To achieve this goal, I thought ahead of using
reduce for this…
And this is what I came up with.
1 2 3 4 5 6 7 8 9
There’s a lot of happening here - which I’ll explain.
- I start looping through each country
ucfrom a list of unique countries
- For each country I pick, I want to filter the
new_datalist of countries cases based on the selected country and their individual covid19 cases. With each returned
uccases, I needed to convert into a list. This is because in the data set, there are some countries which have several number of provinces/states (ie China, Russia or USA) where their total number of cases are defined by the geographical distinction such as this example.
- Because of this, I need to be aggregating these same covid19 cases from each province/state of the same country to get the complete total number of cases for the country.
- Once the total cases for that country is found, I will add it to the
- Go back step 1 and repeat the process all over again until you reach the end of the
The above snippet says with
new_datalist , I apply
filter function by passing
filter_by_country callback that is responsible for filtering out countries in the same list, To filter the data correctly, I rely on the closures upon
uc (which is gathered from
unique_countries looping call earlier and its current element (x) that’s being interated upon.
This is equivalent to JS snippet below.
Notice they’re conceptually the same, but the syntax between the two clearly differs.
And notice we put
list as a wrapper around the
filter function? Apparently,
filter also returns a Map object, just like
map example earlier. You can return in any type of data structure you want as well.
1 2 3 4 5 6
With that mind, we take
filtered_data_by_uc and pipe that into our next action step
1 2 3 4 5
Here, we use
reduce method to take in the
filtered_data_by_uc list and take each country’s number of covid19 cases to aggregrate them (using
sum_up_totals callback) in get the total sums of confirmed, recovered and death cases, producing into a standalone dict object for that particular country.
Here’s the implementation for
1 2 3 4 5 6 7 8
reduce function signature is pretty much the same with
map counterparts ie using lambdas, closures etc - with the minor difference is that it doesn’t return a list or any similar iterable object, but rather as a single value (or an accumulator if you like) after applying the lambda function as above.
This comes off as expected as any FP developers from Haskell, Scala, Erlang, Clojure communities etc would tell you that’s how they live and breathe writing this kind of code - like a boss 😎.
reduce function, you can also return an iterable object as an accumulator result as well.
Which let me to think I didn’t need the for-loop to append individual unique countries’ aggregate results into
I could simply rewrite to this.
1 2 3 4 5 6 7
With the introduction of new
sum_up_totals_each_country lambda function, I’ve basically moved
filter_by_country filtering function inside that function where (by using closures and callbacks) it can gain access to the list of
unique_countries and the raw
new_data to performing the list filtering from there.
Once the filtering is complete, using the
acc which is a proxy to
new_data_with_sums, I start to traverse the same filtered list to accumulate the total cases results for a country that has multiple states and provinces and have it appended to the
Here’s the implementation of
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22
And the rest is history.
Notice how the Python’s very own built-in functional tools (
Python doesn’t do that by design for its own set of iterables when doing functional programming. Thus I always wondered why I had to write the code above by wrapping functions one on top of another… which lead me to have the fact that Python’s design philosophy was driven in an imperative-style way for a long time. It’s never meant to go to the FP route, so says this blog post from the original Python creator himself. Upon reading these, that may explain the deep culture of imperative style coding Pythonistas have been doing for a very long time using tools like itertools and list comprehensions for more efficient data looping in place.
I’m pretty much excited of what lies ahead for Python as a whole and many others. I can’t wait to start using more of these tools set in future projects. (Scala anyone? 😌😉)
Till then, Happy Coding, (and remember: stay safe and do your social distance-coding rules)!