Arsthbt

I have a large dict src (up to 1M items) and I would like to take N (typical values would be N=10K-20K) items, store them in a new dict dst and leave only the remaining items in src. It doesn't matter which N items are taken. I'm looking for the fastest way to do it on Python 3.6 or 3.7.

Fastest approach I've found so far:

src = i: i ** 3 for i in range(1000000)

# Taking items 1 by 1 (~0.0059s)
dst = 
while len(dst) < 20000:
 item = src.popitem()
 dst[item[0]] = item[1]

Is there anything better? Even a marginal gain would be good.

A simple comprehension inside dict will do:

dict(src.popitem() for _ in range(20000))

Here you have the timing tests

setup = """
src = i: i ** 3 for i in range(1000000)

def method_1(d):
 dst = 
 while len(dst) < 20000:
 item = d.popitem()
 dst[item[0]] = item[1]
 return dst

def method_2(d):
 return dict(d.popitem() for _ in range(20000))
"""
import timeit
print("Method 1: ", timeit.timeit('method_1(src)', setup=setup, number=1))

print("Method 2: ", timeit.timeit('method_2(src)', setup=setup, number=1))

Results:

Method 1: 0.007701821999944514
Method 2: 0.004668198998842854

I found this approach slightly faster (-10% speed) using dictionary comprehension that consumes a loop using range that yields & unpacks the keys & values

dst = key:value for key,value in (src.popitem() for _ in range(20000))

on my machine:

your code: 0.00899505615234375
my code: 0.007996797561645508

so about 12% faster, not bad but not as good as not unpacking like Netwave simpler answer

This approach can be useful if you want to transform the keys or values in the process.

This seems a bit faster still, though I'm not sure why:

from itertools import islice
def method_4(d):
 result = dict(islice(d.items(), 20000))
 for k in result: del d[k]
 return result

Compared to other versions, using Netwave's testcase:

Method 1: 0.004459443036466837 # original
Method 2: 0.0034434819826856256 # Netwave
Method 3: 0.002602717955596745 # chepner
Method 4: 0.001974945073015988 # this answer

It's a bit baffling to me, as one could expect the extra iteration and lookup to be slower. Hopefully I haven't made some embarrassing mistake.