Good Data Practices¶
Of course, in the process of working with gerrychain, it is inevitable
that you will want to examine some data. While there are a lot of great
ways to store and deal with data, there are some ways that we have found
more helpful than others which we would like to share here.
Important
In general, the techniques described here are not appropriate for storing the entire Markov chain of redistricting plans. Instead, the methods described below are primarily aimed at the collection and analysis of statistical information about each of the partitions in the chain.
In the event that you would like to store each partition in the chain in its entirety, we would recommend that you make use of our Binary-Ensemble package.
Writing Data to JSONL¶
One of the most common methods for storing data is to write the relevant information to a JSONL file. Here is an example with our Pennsylvania data:
from gerrychain import Graph, MarkovChain, Partition
from gerrychain.accept import always_accept
from gerrychain.constraints import contiguous
from gerrychain.proposals import build_recom_proposal_fn
from gerrychain.updaters import Tally, cut_edges
chain = MarkovChain(
total_steps=20,
rng=42,
)
graph = Graph.from_json("./PA_VTDs.json")
chain.initial_partition = Partition(
graph,
assignment="2011_PLA_1",
)
chain.add_updaters(
{
"population": Tally("TOT_POP", alias="population"),
"area": Tally("area", alias="area"), # We can only do this since PA_VTDs.json
# has an "area" attribute for each node
"cut_edges": cut_edges,
}
)
ideal_population = sum(chain.initial_partition["population"].values()) / len(
chain.initial_partition
)
chain.proposal_fn = build_recom_proposal_fn(
pop_col="TOT_POP",
pop_target=ideal_population,
epsilon=0.01,
)
chain.add_constraint(contiguous)
chain.acceptance_fn = always_accept
And now we can run the chain and write the data to a JSONL file:
import json
with open("PA_output.jsonl", "w") as f:
for i, partition in enumerate(chain):
data = {
"step": i,
"populations": partition["population"],
"areas": partition["area"],
"n_cut_edges": len(partition["cut_edges"]),
}
# Add newline character to separate entries in JSONL file
f.write(json.dumps(data) + "\n")
This will produce output with lines of the form:
{"step": 0, "populations": {"3": 706653, "10": 706992, "9": 702500, "5": 695917, "15": 705549, "6": 705782, "11": 705115, "8": 705689, "4": 705669, "18": 705847, "12": 706232, "17": 699133, "7": 712463, "16": 699557, "14": 705526, "13": 705028, "2": 705689, "1": 705588}, "areas": {"3": 1.0871722918594986, "10": 2.367083752509999, "9": 1.579113333589498, "5": 3.0122633409220008, "15": 0.35732152655850036, "6": 0.23906899201449974, "11": 0.949621240640999, "8": 0.19927536179150002, "4": 0.4185125039540002, "18": 0.5691588362529991, "12": 0.6009789760809999, "17": 0.48479405839200057, "7": 0.23842544605850016, "16": 0.28336540997449977, "14": 0.06036624468650007, "13": 0.04260779136050022, "2": 0.02065452186049993, "1": 0.02454134236900001}, "n_cut_edges": 2361}
which is a bit easier to read with some formatting:
{
"step": 0,
"populations": {
"3": 706653,
"10": 706992,
"9": 702500,
"5": 695917,
"15": 705549,
"6": 705782,
"11": 705115,
"8": 705689,
"4": 705669,
"18": 705847,
"12": 706232,
"17": 699133,
"7": 712463,
"16": 699557,
"14": 705526,
"13": 705028,
"2": 705689,
"1": 705588
},
"areas": {
"3": 1.0871722918594986,
"10": 2.367083752509999,
"9": 1.579113333589498,
"5": 3.0122633409220008,
"15": 0.35732152655850036,
"6": 0.23906899201449974,
"11": 0.949621240640999,
"8": 0.19927536179150002,
"4": 0.4185125039540002,
"18": 0.5691588362529991,
"12": 0.6009789760809999,
"17": 0.48479405839200057,
"7": 0.23842544605850016,
"16": 0.28336540997449977,
"14": 0.06036624468650007,
"13": 0.04260779136050022,
"2": 0.02065452186049993,
"1": 0.02454134236900001
},
"n_cut_edges": 2361
}
This method has a few advantages:
The data is easy to read
In the event that the run is interrupted (which happens more often than we would like), the data is still saved up to the point of interruption.
The data can then be read back in with something like
import json
with open("PA_output.jsonl", "r") as f:
for line in f:
data = json.loads(line)
# Do something with the data
print(f"data['step'] = {data['step']}")
data['step'] = 0
data['step'] = 1
data['step'] = 2
data['step'] = 3
data['step'] = 4
data['step'] = 5
data['step'] = 6
data['step'] = 7
data['step'] = 8
data['step'] = 9
data['step'] = 10
data['step'] = 11
data['step'] = 12
data['step'] = 13
data['step'] = 14
data['step'] = 15
data['step'] = 16
data['step'] = 17
data['step'] = 18
data['step'] = 19
Pandas DataFrames¶
Another method that can be particularly useful when experimenting with different redistricting ensembles is to store the data in a pandas dataframe.
import pandas as pd
district_data = []
# Rebuild the chain to replay the same seeded trajectory. Iterating the
# original chain again would continue its RNG stream.
replay_chain = MarkovChain(
total_steps=20,
rng=42,
)
replay_chain.initial_partition = Partition(
graph,
assignment="2011_PLA_1",
)
replay_chain.add_updaters(
{
"population": Tally("TOT_POP", alias="population"),
"area": Tally("area", alias="area"),
"cut_edges": cut_edges,
}
)
replay_chain.proposal_fn = chain.proposal_fn
replay_chain.add_constraint(contiguous)
replay_chain.acceptance_fn = always_accept
for i, partition in enumerate(replay_chain):
for district_name in partition["population"].keys():
population = partition["population"][district_name]
area = partition["area"][district_name]
n_cut_edges = len(partition["cut_edges"])
district_data.append((i, district_name, population, area, n_cut_edges))
df = pd.DataFrame(
district_data, columns=["step", "district_name", "population", "area", "n_cut_edges"]
)
The utility of this method is shown in the ability to use dataframe views to easily filter and manipulate the data. For example, if we wanted to look at the data for step 11, we could write something like:
df[df["step"] == 11]
| step | district_name | population | area | n_cut_edges | |
|---|---|---|---|---|---|
| 198 | 11 | 3 | 706653 | 1.087172 | 2387 |
| 199 | 11 | 10 | 701165 | 1.157217 | 2387 |
| 200 | 11 | 9 | 705053 | 1.883993 | 2387 |
| 201 | 11 | 5 | 701197 | 2.297773 | 2387 |
| 202 | 11 | 15 | 705549 | 0.357322 | 2387 |
| 203 | 11 | 6 | 707653 | 0.242983 | 2387 |
| 204 | 11 | 11 | 699095 | 2.587204 | 2387 |
| 205 | 11 | 8 | 709934 | 0.229182 | 2387 |
| 206 | 11 | 4 | 709683 | 0.400408 | 2387 |
| 207 | 11 | 18 | 704663 | 0.122458 | 2387 |
| 208 | 11 | 12 | 706232 | 0.600979 | 2387 |
| 209 | 11 | 17 | 699133 | 0.484794 | 2387 |
| 210 | 11 | 7 | 699538 | 0.018366 | 2387 |
| 211 | 11 | 16 | 699557 | 0.283365 | 2387 |
| 212 | 11 | 14 | 706710 | 0.507068 | 2387 |
| 213 | 11 | 13 | 706169 | 0.104609 | 2387 |
| 214 | 11 | 2 | 711357 | 0.144892 | 2387 |
| 215 | 11 | 1 | 705588 | 0.024541 | 2387 |