{
"cells": [
{
"cell_type": "markdown",
"id": "dfd33995",
"metadata": {},
"source": [
"# Good Data Practices\n",
"\n",
"\n",
"\n",
"Of course, in the process of working with `gerrychain`, it is inevitable\n",
"that you will want to examine some data. While there are a lot of great\n",
"ways to store and deal with data, there are some ways that we have found\n",
"more helpful than others which we would like to share here.\n",
"\n",
"> Important\n",
">\n",
"> In general, the techniques described here are not appropriate\n",
"> for storing the entire Markov chain of redistricting plans. Instead, the\n",
"> methods described below are primarily aimed at the collection and analysis\n",
"> of statistical information about each of the partitions in the chain.\n",
">\n",
"> In the event that you would like to store each partition in the chain in\n",
"> its entirety, we would recommend that you make use of our\n",
"> [Binary-Ensemble](https://binary-ensemble.readthedocs.io) package."
]
},
{
"cell_type": "markdown",
"id": "6390c8e6",
"metadata": {},
"source": [
"## Writing Data to JSONL\n",
"\n",
"\n",
"
\n",
"\n",
"One of the most common methods for storing data is to write the\n",
"relevant information to a JSONL file. Here is an example with our\n",
"Pennsylvania data:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "4425cdad",
"metadata": {},
"outputs": [],
"source": [
"from gerrychain import Graph, MarkovChain, Partition\n",
"from gerrychain.accept import always_accept\n",
"from gerrychain.constraints import contiguous\n",
"from gerrychain.proposals import build_recom_proposal_fn\n",
"from gerrychain.updaters import Tally, cut_edges\n",
"\n",
"chain = MarkovChain(\n",
" total_steps=20,\n",
" rng=42,\n",
")\n",
"\n",
"graph = Graph.from_json(\"./PA_VTDs.json\")\n",
"\n",
"chain.initial_partition = Partition(\n",
" graph,\n",
" assignment=\"2011_PLA_1\",\n",
")\n",
"\n",
"chain.add_updaters(\n",
" {\n",
" \"population\": Tally(\"TOT_POP\", alias=\"population\"),\n",
" \"area\": Tally(\"area\", alias=\"area\"), # We can only do this since PA_VTDs.json\n",
" # has an \"area\" attribute for each node\n",
" \"cut_edges\": cut_edges,\n",
" }\n",
")\n",
"\n",
"ideal_population = sum(chain.initial_partition[\"population\"].values()) / len(\n",
" chain.initial_partition\n",
")\n",
"\n",
"chain.proposal_fn = build_recom_proposal_fn(\n",
" pop_col=\"TOT_POP\",\n",
" pop_target=ideal_population,\n",
" epsilon=0.01,\n",
")\n",
"chain.add_constraint(contiguous)\n",
"chain.acceptance_fn = always_accept"
]
},
{
"cell_type": "markdown",
"id": "162e2870",
"metadata": {},
"source": [
"And now we can run the chain and write the data to a JSONL file:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "c09aac39",
"metadata": {},
"outputs": [],
"source": [
"import json\n",
"\n",
"with open(\"PA_output.jsonl\", \"w\") as f:\n",
" for i, partition in enumerate(chain):\n",
" data = {\n",
" \"step\": i,\n",
" \"populations\": partition[\"population\"],\n",
" \"areas\": partition[\"area\"],\n",
" \"n_cut_edges\": len(partition[\"cut_edges\"]),\n",
" }\n",
" # Add newline character to separate entries in JSONL file\n",
" f.write(json.dumps(data) + \"\\n\")"
]
},
{
"cell_type": "markdown",
"id": "41450aef",
"metadata": {},
"source": [
"This will produce output with lines of the form:\n",
"\n",
"```console\n",
"{\"step\": 0, \"populations\": {\"3\": 706653, \"10\": 706992, \"9\": 702500, \"5\": 695917, \"15\": 705549, \"6\": 705782, \"11\": 705115, \"8\": 705689, \"4\": 705669, \"18\": 705847, \"12\": 706232, \"17\": 699133, \"7\": 712463, \"16\": 699557, \"14\": 705526, \"13\": 705028, \"2\": 705689, \"1\": 705588}, \"areas\": {\"3\": 1.0871722918594986, \"10\": 2.367083752509999, \"9\": 1.579113333589498, \"5\": 3.0122633409220008, \"15\": 0.35732152655850036, \"6\": 0.23906899201449974, \"11\": 0.949621240640999, \"8\": 0.19927536179150002, \"4\": 0.4185125039540002, \"18\": 0.5691588362529991, \"12\": 0.6009789760809999, \"17\": 0.48479405839200057, \"7\": 0.23842544605850016, \"16\": 0.28336540997449977, \"14\": 0.06036624468650007, \"13\": 0.04260779136050022, \"2\": 0.02065452186049993, \"1\": 0.02454134236900001}, \"n_cut_edges\": 2361}\n",
"```\n",
"\n",
"which is a bit easier to read with some formatting:\n",
"\n",
"```json\n",
"{\n",
" \"step\": 0,\n",
" \"populations\": {\n",
" \"3\": 706653,\n",
" \"10\": 706992,\n",
" \"9\": 702500,\n",
" \"5\": 695917,\n",
" \"15\": 705549,\n",
" \"6\": 705782,\n",
" \"11\": 705115,\n",
" \"8\": 705689,\n",
" \"4\": 705669,\n",
" \"18\": 705847,\n",
" \"12\": 706232,\n",
" \"17\": 699133,\n",
" \"7\": 712463,\n",
" \"16\": 699557,\n",
" \"14\": 705526,\n",
" \"13\": 705028,\n",
" \"2\": 705689,\n",
" \"1\": 705588\n",
" },\n",
" \"areas\": {\n",
" \"3\": 1.0871722918594986,\n",
" \"10\": 2.367083752509999,\n",
" \"9\": 1.579113333589498,\n",
" \"5\": 3.0122633409220008,\n",
" \"15\": 0.35732152655850036,\n",
" \"6\": 0.23906899201449974,\n",
" \"11\": 0.949621240640999,\n",
" \"8\": 0.19927536179150002,\n",
" \"4\": 0.4185125039540002,\n",
" \"18\": 0.5691588362529991,\n",
" \"12\": 0.6009789760809999,\n",
" \"17\": 0.48479405839200057,\n",
" \"7\": 0.23842544605850016,\n",
" \"16\": 0.28336540997449977,\n",
" \"14\": 0.06036624468650007,\n",
" \"13\": 0.04260779136050022,\n",
" \"2\": 0.02065452186049993,\n",
" \"1\": 0.02454134236900001\n",
" },\n",
" \"n_cut_edges\": 2361\n",
"}\n",
"```\n",
"\n",
"This method has a few advantages:\n",
"\n",
"1. The data is easy to read\n",
"2. In the event that the run is interrupted (which happens more often than\n",
" we would like), the data is still saved up to the point of interruption.\n",
"\n",
"The data can then be read back in with something like"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "6e431a08",
"metadata": {},
"outputs": [],
"source": [
"import json\n",
"\n",
"with open(\"PA_output.jsonl\", \"r\") as f:\n",
" for line in f:\n",
" data = json.loads(line)\n",
" # Do something with the data\n",
" print(f\"data['step'] = {data['step']}\")"
]
},
{
"cell_type": "markdown",
"id": "fc0e31ec",
"metadata": {},
"source": [
"## Pandas DataFrames\n",
"\n",
"Another method that can be particularly useful\n",
"when experimenting with different redistricting ensembles\n",
"is to store the data in a pandas dataframe."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "3ffb69fc",
"metadata": {},
"outputs": [],
"source": [
"import pandas as pd\n",
"\n",
"district_data = []\n",
"\n",
"# Rebuild the chain to replay the same seeded trajectory. Iterating the\n",
"# original chain again would continue its RNG stream.\n",
"replay_chain = MarkovChain(\n",
" total_steps=20,\n",
" rng=42,\n",
")\n",
"replay_chain.initial_partition = Partition(\n",
" graph,\n",
" assignment=\"2011_PLA_1\",\n",
")\n",
"replay_chain.add_updaters(\n",
" {\n",
" \"population\": Tally(\"TOT_POP\", alias=\"population\"),\n",
" \"area\": Tally(\"area\", alias=\"area\"),\n",
" \"cut_edges\": cut_edges,\n",
" }\n",
")\n",
"replay_chain.proposal_fn = chain.proposal_fn\n",
"replay_chain.add_constraint(contiguous)\n",
"replay_chain.acceptance_fn = always_accept\n",
"\n",
"for i, partition in enumerate(replay_chain):\n",
" for district_name in partition[\"population\"].keys():\n",
" population = partition[\"population\"][district_name]\n",
" area = partition[\"area\"][district_name]\n",
" n_cut_edges = len(partition[\"cut_edges\"])\n",
" district_data.append((i, district_name, population, area, n_cut_edges))\n",
"\n",
"df = pd.DataFrame(\n",
" district_data, columns=[\"step\", \"district_name\", \"population\", \"area\", \"n_cut_edges\"]\n",
")"
]
},
{
"cell_type": "markdown",
"id": "ac5dbd62",
"metadata": {},
"source": [
"The utility of this method is shown in the ability to use dataframe\n",
"views to easily filter and manipulate the data. For example, if\n",
"we wanted to look at the data for step 11, we could write something\n",
"like:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "810a2bb1",
"metadata": {},
"outputs": [],
"source": [
"df[df[\"step\"] == 11]"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
}
},
"nbformat": 4,
"nbformat_minor": 5
}