Allowed memory size exhausted in Doctrine and ini_set doesn't work

I am importing a lot of files in a querry, problem is I got a PHP error telling me I exhausted all the memories.
But this problem is occuring only on the production server, not in my local.

I already tried to do a

ini_set('memory_limit', '-1');

and a

set_time_limit ( 0 );

And my code is

public function mediachooserAction(Request $request, $origin = null, $type = null, $term = null)
{
    ini_set('memory_limit', '-1');
    if ($this->container->has('profiler')) {
        $this->container->get('profiler')->disable();
    }

    $entities = $this->getRepository('AppBundle:Media')->findMedias($origin, $type, $term);

    $pagerfanta = new Pagerfanta(new ArrayAdapter($entities));
    $pagerfanta->setMaxPerPage(20);
    if ($request->query->has('page')) {
        $pagerfanta->setCurrentPage($request->query->get('page'));
    }

    return $this->render('backend/ckeditor/mediachooser/index.html.twig', [
        'entities'      => $pagerfanta,
        'origin'        => $origin,
        'type'          => $type,
        'originChoices' => Media::getOriginChoices(),
        'typeChoices'   => Media::getTypeChoices(),
    ]);
}

But this isn't helping.

Do you know how I can fix this issue without doing any optimisation in my program ?