snipe-it/app/Http/Controllers/Api/ImportController.php

190 lines
7.3 KiB
PHP
Raw Normal View History

<?php
namespace App\Http\Controllers\Api;
use App\Helpers\Helper;
use App\Http\Controllers\Controller;
use App\Http\Requests\ItemImportRequest;
use App\Http\Transformers\ImportsTransformer;
use App\Models\Asset;
use App\Models\Company;
use App\Models\Import;
use Artisan;
use Illuminate\Support\Facades\Request;
use Illuminate\Support\Facades\Session;
use Illuminate\Support\Facades\Storage;
Importer mapping - v1 (#3677) * Move importer to an inline-template, allows for translations and easier passing of data from laravel to vue. * Pull the modal out into a dedicated partial, move importer to views/importer. * Add document of CSV->importer mappings. Reorganize some code. Progress. * Add header_row and first_row to imports table, and process upon uploading a file * Use an expandable table row instead of a modal for import processing. This should allow for field mapping interaction easier. * Fix import processing after moving method. * Frontend importer mapping improvements. Invert display so we show found columns and allow users to select an importer field to map to. Also implement sample data based on first row of csv. * Update select2. Maintain selected items properly. * Backend support for importing. Only works on the web importer currently. Definitely needs testing and polish. * We no longer use vue-modal plugin. * Add a column to track field mappings to the imports table. * Cleanup/rename methods+refactor * Save field mappings and import type when attempting an import, and repopulate these values when returning to the page. * Update debugbar to fix a bug in the debugbar code. * Fix asset tag detection. Also rename findMatch to be a bit clearer as to what it does. Remove logging to file of imports for http imports because it eats an incredible amouint of memory. This commit also moves imports out of the hardware namespace and into their own webcontroller and route prefix, remove dead code from AssetController as a result. * Dynamically limit options for select2 based on import type selected, and group them by item type. * Add user importer. Still need to implement emailing of passwords to new users, and probably test a bit more. This also bumps the memory limit for web imports up as well, I need to profile memory usage here before too long. * Query the db to find user matches rather than search the array. Performance is much much better. * Speed/memory improvements in importers. Move to querying the db rather than maintaining an array for all importers. Also only store the id of items when we import, rather than the full model. It saves a decent amount of memory. * Remove grouping of items in select2 With the values being set dynamically, the grouping is redundant. It also caused a regression with automatically guessing/matching field names. This is starting to get close. * Remove debug line on every create. * Switch migration to be text field instead of json field for compatibility with older mysql/mariadb * Fix asset import regression matching email address. * Rearrange travis order in attempt to fix null settings. * Use auth::id instead of fetching it off the user. Fixes a null object reference during seeding.
2017-06-21 16:37:37 -07:00
use League\Csv\Reader;
use Symfony\Component\HttpFoundation\File\Exception\FileException;
class ImportController extends Controller
{
/**
* Display a listing of the resource.
*
* @return \Illuminate\Http\Response
*/
public function index()
{
$this->authorize('import');
$imports = Import::latest()->get();
return (new ImportsTransformer)->transformImports($imports);
}
/**
Importer mapping - v1 (#3677) * Move importer to an inline-template, allows for translations and easier passing of data from laravel to vue. * Pull the modal out into a dedicated partial, move importer to views/importer. * Add document of CSV->importer mappings. Reorganize some code. Progress. * Add header_row and first_row to imports table, and process upon uploading a file * Use an expandable table row instead of a modal for import processing. This should allow for field mapping interaction easier. * Fix import processing after moving method. * Frontend importer mapping improvements. Invert display so we show found columns and allow users to select an importer field to map to. Also implement sample data based on first row of csv. * Update select2. Maintain selected items properly. * Backend support for importing. Only works on the web importer currently. Definitely needs testing and polish. * We no longer use vue-modal plugin. * Add a column to track field mappings to the imports table. * Cleanup/rename methods+refactor * Save field mappings and import type when attempting an import, and repopulate these values when returning to the page. * Update debugbar to fix a bug in the debugbar code. * Fix asset tag detection. Also rename findMatch to be a bit clearer as to what it does. Remove logging to file of imports for http imports because it eats an incredible amouint of memory. This commit also moves imports out of the hardware namespace and into their own webcontroller and route prefix, remove dead code from AssetController as a result. * Dynamically limit options for select2 based on import type selected, and group them by item type. * Add user importer. Still need to implement emailing of passwords to new users, and probably test a bit more. This also bumps the memory limit for web imports up as well, I need to profile memory usage here before too long. * Query the db to find user matches rather than search the array. Performance is much much better. * Speed/memory improvements in importers. Move to querying the db rather than maintaining an array for all importers. Also only store the id of items when we import, rather than the full model. It saves a decent amount of memory. * Remove grouping of items in select2 With the values being set dynamically, the grouping is redundant. It also caused a regression with automatically guessing/matching field names. This is starting to get close. * Remove debug line on every create. * Switch migration to be text field instead of json field for compatibility with older mysql/mariadb * Fix asset import regression matching email address. * Rearrange travis order in attempt to fix null settings. * Use auth::id instead of fetching it off the user. Fixes a null object reference during seeding.
2017-06-21 16:37:37 -07:00
* Process and store a CSV upload file.
*
* @param \Illuminate\Http\Request $request
* @return \Illuminate\Http\Response
*/
public function store()
{
$this->authorize('import');
if (!config('app.lock_passwords')) {
$files = Request::file('files');
$path = config('app.private_uploads').'/imports';
$results = [];
$import = new Import;
foreach ($files as $file) {
if (!in_array($file->getMimeType(), array(
'application/vnd.ms-excel',
'text/csv',
'application/csv',
'text/x-Algol68', // because wtf CSV files?
'text/plain',
'text/comma-separated-values',
'text/tsv'))) {
$results['error']='File type must be CSV. Uploaded file is '.$file->getMimeType();
return response()->json(Helper::formatStandardApiResponse('error', null, $results['error']), 500);
}
//TODO: is there a lighter way to do this?
if (! ini_get("auto_detect_line_endings")) {
ini_set("auto_detect_line_endings", '1');
}
$reader = Reader::createFromFileObject($file->openFile('r')); //file pointer leak?
$import->header_row = $reader->fetchOne(0);
//duplicate headers check
$duplicate_headers = [];
for($i = 0; $i<count($import->header_row); $i++) {
$header = $import->header_row[$i];
if(in_array($header, $import->header_row)) {
$found_at = array_search($header, $import->header_row);
if($i > $found_at) {
//avoid reporting duplicates twice, e.g. "1 is same as 17! 17 is same as 1!!!"
//as well as "1 is same as 1!!!" (which is always true)
//has to be > because otherwise the first result of array_search will always be $i itself(!)
array_push($duplicate_headers,"Duplicate header '$header' detected, first at column: ".($found_at+1).", repeats at column: ".($i+1));
}
}
}
if(count($duplicate_headers) > 0) {
return response()->json(Helper::formatStandardApiResponse('error',null, implode("; ",$duplicate_headers)), 500); //should this be '4xx'?
}
// Grab the first row to display via ajax as the user picks fields
$import->first_row = $reader->fetchOne(1);
$date = date('Y-m-d-his');
$fixed_filename = str_slug($file->getClientOriginalName());
try {
$file->move($path, $date.'-'.$fixed_filename);
} catch (FileException $exception) {
$results['error']=trans('admin/hardware/message.upload.error');
if (config('app.debug')) {
$results['error'].= ' ' . $exception->getMessage();
}
return response()->json(Helper::formatStandardApiResponse('error', null, $results['error']), 500);
}
$file_name = date('Y-m-d-his').'-'.$fixed_filename;
$import->file_path = $file_name;
$import->filesize = filesize($path.'/'.$file_name);
$import->save();
$results[] = $import;
}
$results = (new ImportsTransformer)->transformImports($results);
return [
Importer mapping - v1 (#3677) * Move importer to an inline-template, allows for translations and easier passing of data from laravel to vue. * Pull the modal out into a dedicated partial, move importer to views/importer. * Add document of CSV->importer mappings. Reorganize some code. Progress. * Add header_row and first_row to imports table, and process upon uploading a file * Use an expandable table row instead of a modal for import processing. This should allow for field mapping interaction easier. * Fix import processing after moving method. * Frontend importer mapping improvements. Invert display so we show found columns and allow users to select an importer field to map to. Also implement sample data based on first row of csv. * Update select2. Maintain selected items properly. * Backend support for importing. Only works on the web importer currently. Definitely needs testing and polish. * We no longer use vue-modal plugin. * Add a column to track field mappings to the imports table. * Cleanup/rename methods+refactor * Save field mappings and import type when attempting an import, and repopulate these values when returning to the page. * Update debugbar to fix a bug in the debugbar code. * Fix asset tag detection. Also rename findMatch to be a bit clearer as to what it does. Remove logging to file of imports for http imports because it eats an incredible amouint of memory. This commit also moves imports out of the hardware namespace and into their own webcontroller and route prefix, remove dead code from AssetController as a result. * Dynamically limit options for select2 based on import type selected, and group them by item type. * Add user importer. Still need to implement emailing of passwords to new users, and probably test a bit more. This also bumps the memory limit for web imports up as well, I need to profile memory usage here before too long. * Query the db to find user matches rather than search the array. Performance is much much better. * Speed/memory improvements in importers. Move to querying the db rather than maintaining an array for all importers. Also only store the id of items when we import, rather than the full model. It saves a decent amount of memory. * Remove grouping of items in select2 With the values being set dynamically, the grouping is redundant. It also caused a regression with automatically guessing/matching field names. This is starting to get close. * Remove debug line on every create. * Switch migration to be text field instead of json field for compatibility with older mysql/mariadb * Fix asset import regression matching email address. * Rearrange travis order in attempt to fix null settings. * Use auth::id instead of fetching it off the user. Fixes a null object reference during seeding.
2017-06-21 16:37:37 -07:00
'files' => $results,
];
}
return response()->json(Helper::formatStandardApiResponse('error', null, trans('general.feature_disabled')), 500);
}
/**
* Processes the specified Import.
*
* @param int $import_id
* @return \Illuminate\Http\Response
*/
public function process(ItemImportRequest $request, $import_id)
{
$this->authorize('import');
// Run a backup immediately before processing
if ($request->has('run-backup')) {
\Log::debug('Backup manually requested via importer');
Artisan::call('backup:run');
} else {
\Log::debug('NO BACKUP requested via importer');
}
$errors = $request->import(Import::find($import_id));
Importer mapping - v1 (#3677) * Move importer to an inline-template, allows for translations and easier passing of data from laravel to vue. * Pull the modal out into a dedicated partial, move importer to views/importer. * Add document of CSV->importer mappings. Reorganize some code. Progress. * Add header_row and first_row to imports table, and process upon uploading a file * Use an expandable table row instead of a modal for import processing. This should allow for field mapping interaction easier. * Fix import processing after moving method. * Frontend importer mapping improvements. Invert display so we show found columns and allow users to select an importer field to map to. Also implement sample data based on first row of csv. * Update select2. Maintain selected items properly. * Backend support for importing. Only works on the web importer currently. Definitely needs testing and polish. * We no longer use vue-modal plugin. * Add a column to track field mappings to the imports table. * Cleanup/rename methods+refactor * Save field mappings and import type when attempting an import, and repopulate these values when returning to the page. * Update debugbar to fix a bug in the debugbar code. * Fix asset tag detection. Also rename findMatch to be a bit clearer as to what it does. Remove logging to file of imports for http imports because it eats an incredible amouint of memory. This commit also moves imports out of the hardware namespace and into their own webcontroller and route prefix, remove dead code from AssetController as a result. * Dynamically limit options for select2 based on import type selected, and group them by item type. * Add user importer. Still need to implement emailing of passwords to new users, and probably test a bit more. This also bumps the memory limit for web imports up as well, I need to profile memory usage here before too long. * Query the db to find user matches rather than search the array. Performance is much much better. * Speed/memory improvements in importers. Move to querying the db rather than maintaining an array for all importers. Also only store the id of items when we import, rather than the full model. It saves a decent amount of memory. * Remove grouping of items in select2 With the values being set dynamically, the grouping is redundant. It also caused a regression with automatically guessing/matching field names. This is starting to get close. * Remove debug line on every create. * Switch migration to be text field instead of json field for compatibility with older mysql/mariadb * Fix asset import regression matching email address. * Rearrange travis order in attempt to fix null settings. * Use auth::id instead of fetching it off the user. Fixes a null object reference during seeding.
2017-06-21 16:37:37 -07:00
$redirectTo = "hardware.index";
switch ($request->get('import-type')) {
case "asset":
$redirectTo = "hardware.index";
break;
case "accessory":
$redirectTo = "accessories.index";
break;
case "consumable":
$redirectTo = "consumables.index";
break;
case "component":
$redirectTo = "components.index";
break;
case "license":
$redirectTo = "licenses.index";
break;
Importer mapping - v1 (#3677) * Move importer to an inline-template, allows for translations and easier passing of data from laravel to vue. * Pull the modal out into a dedicated partial, move importer to views/importer. * Add document of CSV->importer mappings. Reorganize some code. Progress. * Add header_row and first_row to imports table, and process upon uploading a file * Use an expandable table row instead of a modal for import processing. This should allow for field mapping interaction easier. * Fix import processing after moving method. * Frontend importer mapping improvements. Invert display so we show found columns and allow users to select an importer field to map to. Also implement sample data based on first row of csv. * Update select2. Maintain selected items properly. * Backend support for importing. Only works on the web importer currently. Definitely needs testing and polish. * We no longer use vue-modal plugin. * Add a column to track field mappings to the imports table. * Cleanup/rename methods+refactor * Save field mappings and import type when attempting an import, and repopulate these values when returning to the page. * Update debugbar to fix a bug in the debugbar code. * Fix asset tag detection. Also rename findMatch to be a bit clearer as to what it does. Remove logging to file of imports for http imports because it eats an incredible amouint of memory. This commit also moves imports out of the hardware namespace and into their own webcontroller and route prefix, remove dead code from AssetController as a result. * Dynamically limit options for select2 based on import type selected, and group them by item type. * Add user importer. Still need to implement emailing of passwords to new users, and probably test a bit more. This also bumps the memory limit for web imports up as well, I need to profile memory usage here before too long. * Query the db to find user matches rather than search the array. Performance is much much better. * Speed/memory improvements in importers. Move to querying the db rather than maintaining an array for all importers. Also only store the id of items when we import, rather than the full model. It saves a decent amount of memory. * Remove grouping of items in select2 With the values being set dynamically, the grouping is redundant. It also caused a regression with automatically guessing/matching field names. This is starting to get close. * Remove debug line on every create. * Switch migration to be text field instead of json field for compatibility with older mysql/mariadb * Fix asset import regression matching email address. * Rearrange travis order in attempt to fix null settings. * Use auth::id instead of fetching it off the user. Fixes a null object reference during seeding.
2017-06-21 16:37:37 -07:00
case "user":
$redirectTo = "users.index";
break;
}
if ($errors) { //Failure
return response()->json(Helper::formatStandardApiResponse('import-errors', null, $errors), 500);
}
//Flash message before the redirect
Session::flash('success', trans('admin/hardware/message.import.success'));
return response()->json(Helper::formatStandardApiResponse('success', null, ['redirect_url' => route($redirectTo)]));
}
/**
* Remove the specified resource from storage.
*
* @param int $import_id
* @return \Illuminate\Http\Response
*/
public function destroy($import_id)
{
$this->authorize('create', Asset::class);
if ($import = Import::find($import_id)) {
try {
// Try to delete the file
[WIP] Added #5957 - Flysystem support (#6262) * Added AWS url to example env * Upgrader - added check for new storage path and attempt to move * Ignore symlink * Updated paths for models * Moved copy methods * Added AWS_URL support For some reasin, Flysystem was generating the wrong AWS url (with a region included) * Switch to Flysystem for image uploads * Nicer display of image preview * Updated image preview on edit blades to use Flysystem * Twiddled some more paths * Working filesystems config * Updated Asset Models and Departments to use Flysystem * Janky workaround for differing S3/local urls/paths * Try to smartly use S3 as public disk if S3 is configured * Use public disk Storage options for public files * Additional transformer edits for Flysystem * Removed debugging * Added missing use Storage directive * Updated seeders to use Flysystem * Default logo * Set a default width We can potentially override this in settings later * Use Flysystem for logo upload * Update downloadFile to use Flysystem * Updated AssetFilesController to use Flysystem * Updated acceptance signatures to use Flysystem * Updated signature view to use Flysystem This isn’t working 100% yet * Use Flysystem facade for displaying asset image * Set assets path Should clean all these up when we’re done here * Added Rackspace support for Flysystem * Added Flysystem migrator console command * Added use Storage directive for categories * Added user avatars to Flysystem * Added profile avatar to Flysystem * Added the option to delete local files with the migrator * Added a check to prevent people from trying to move from local to local * Fixed the selectlists for Flysystem * Fixed the getImageUrl method to reflect Flysystem * Fixed AWS copy process * Fixed models path * More selectlist updates for Flysystem * Updated example .envs with updated env variable names * *sigh* * Updated non-asset getImageUrl() methods to use Flysystem * Removed S3 hardcoding * Use Flysystem in email headers * Fixed typo * Removed camera support from asset file upload We’ll find a way to add this in later (and add that support to all of the other image uploads as well) * Fixed path for categories * WIP - Switched to standard handleImages for asset upload. This is currently broken as I refact the handleImages method. Because the assets store/create methods use their own Form Request, the handleImages method doesn’t exist in that Form Request so it wil error now. * Fixed css URL error * Updated Debugbar to latest version (#6265) v3.2 adds support for Laravel 5.7 * Fixed: Missing CSS file in basic.blade.php (#6264) * Fixed missing CSS file in basic.blade.php * Added * Changed stylesheet import for authorize.blade.php * Updated composer lock * Added AWS_BUCKET_ROOT as env variable * Use nicer image preview for logo upload * Removed AssetRequest form request * Removed asset form request, moved custom field validation into model * Added additional help text for logo upload * Increased the size of the image resize - should make this a setting tho * Few more formatting tweaks to logo section of branding blade preview * Use Flysystem for asset/license file uploads * Use Flysystem for removing images from models that have been deleted * Enable backups to use Flysystem This only handles part of the problem. This just makes it so we can ship files to S3 if we want, but does not account for how we backup files that are hosted on S3 * Use Flysystem to download license files * Updated audits to use Flysystem
2018-09-29 21:33:52 -07:00
Storage::delete('imports/'.$import->file_path);
$import->delete();
return response()->json(Helper::formatStandardApiResponse('success', null, trans('admin/hardware/message.import.file_delete_success')));
} catch (\Exception $e) {
// If the file delete didn't work, remove it from the database anyway and return a warning
$import->delete();
return response()->json(Helper::formatStandardApiResponse('warning', null, trans('admin/hardware/message.import.file_not_deleted_warning')));
}
}
}
}