RazinAleks/SO-Python_QA-Networking_and_APIs_class
432
1[{"Question":"First of all, I can't use any executables. I need to do this in pure Python but sadly requests and BS4 doesn't support JS pages and Selenium needs a webdriver which is an executable.\nDoes anyone knows \/ Is there any way to scrape a JS Rendered page using purely Python and it's modules without having to run any exe?\nI'm not asking for exact solutions, only for the method and modules, if it's \neven possible.\nThank you for reading this and thank you for any constructive comments!\nHave a nice day!\nFor the full context: I'm trying to run a web-scraping script on a daily basis on a cloud that doesn't allows running any exes. Tried it with Selenium and PhantomJS but got a no permission error.","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":51,"Q_Id":51965133,"Users Score":0,"Answer":"Sorry for the misunderstanding, just noticed that the free version of the cloud didn't give permission for these. I got a paid one and it now works like a charm.","Q_Score":1,"Tags":"python,python-3.x,web-scraping","A_Id":51965854,"CreationDate":"2018-08-22T10:46:00.000","Title":"Any way to scrape a JS Rendered page without using executables in Python?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I get this error when I try to import wxPython from python 3.7. I have google around but to no luck. Any help would be appreciated.\n\nTraceback (most recent call last):\n File \"C:\/Users\/STEVE\/Desktop\/Python Files\/Chat Bot\/Joyla\/joyla.py\", line 3, in \n import wx\n File \"C:\\Users\\STEVE\\AppData\\Local\\Programs\\Python\\Python37-32\\lib\\site-packages\\wx__init__.py\", line 17, in \n from wx.core import *\n File \"C:\\Users\\STEVE\\AppData\\Local\\Programs\\Python\\Python37-32\\lib\\site-packages\\wx\\core.py\", line 12, in \n from ._core import *\n ImportError: DLL load failed: The specified module could not be found.","AnswerCount":1,"Available Count":1,"Score":0.1973753202,"is_accepted":false,"ViewCount":742,"Q_Id":51976677,"Users Score":1,"Answer":"I had this same problem with the same exact error message. It turned out that I have mistakenly installed the 32 bit version of python instead of the 64 bit version.\nThe python web site will trick you - if you just use the download link on the top page it will give you the 32 bit version. You have to go to the downloads page, then windows, then look for the 64 bit version - \"Windows x86-64 executable installer\". \nThe actual file name of the 64 bit download is\n \"python-3.7.0-amd64.exe\". \nIf you get the 32 bit version the file name will be \n \"python-3.7.0.exe\".\nAnother way to check, after installing, is to open a python console (or a command prompt and type \"python\" to open the python command line).\nIf you have the 32 bit version it will say:\n Python 3.7.0 (v3.7.0:1bf9cc5093, Jun 27 2018, 04:06:47) \n [MSC v.1914 32 bit (Intel)] on win32\nIf you have the 64 bit version it will say:\n Python 3.7.0 (v3.7.0:1bf9cc5093, Jun 27 2018, 04:59:51) \n [MSC v.1914 64 bit (AMD64)] on win32","Q_Score":0,"Tags":"python,dll,wxpython,importerror","A_Id":52228847,"CreationDate":"2018-08-23T00:01:00.000","Title":"ImprtError when importing wxPython from Python3.7","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"Is it possible to manipulate existing svg files in python? I know the svgwrite package but it only creates new svg files... I just want to add 3 numbers to an existing svg file. It should be only an update showing the old svg file with these 3 new numbers.","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":1445,"Q_Id":51982876,"Users Score":0,"Answer":"No. As with other text files, you need to read the file to memory, modify and then write the modified file to disk.","Q_Score":1,"Tags":"python,svg,svgwrite","A_Id":51983709,"CreationDate":"2018-08-23T09:46:00.000","Title":"Manipulating SVG-files with python","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"how to install twilio via pip?\nI tried to install twilio python module\nbut i can't install it \ni get following error \nno Module named twilio\nWhen trying to install twilio\npip install twilio\nI get the following error.\npyopenssl 18.0.0 has requirement six>=1.5.2, but you'll have six 1.4.1 which is incompatible.\nCannot uninstall 'pyOpenSSL'. It is a distutils installed project and thus we cannot accurately determine which files belong to it which would lead to only a partial uninstall.\ni got the answer and installed\npip install --ignore-installed twilio\nbut i get following error\n\nCould not install packages due to an EnvironmentError: [Errno 13] Permission denied: '\/Library\/Python\/2.7\/site-packages\/pytz-2018.5.dist-info'\nConsider using the `--user` option or check the permissions.\n\ni have anaconda installed \nis this a problem?","AnswerCount":2,"Available Count":1,"Score":-0.0996679946,"is_accepted":false,"ViewCount":2822,"Q_Id":51985401,"Users Score":-1,"Answer":"step1:download python-2.7.15.msi\nstep 2:install and If your system does not have Python added to your PATH while installing\n\"add python exe to path\"\nstep 3:go C:\\Python27\\Scripts of your system\nstep4:in command prompt C:\\Python27\\Scripts>pip install twilio\nstep 5:after installation is done >python command line\n import twilio\nprint(twilio.version)\nstep 6:if u get the version ...you are done","Q_Score":2,"Tags":"python,module,installation,twilio","A_Id":53578904,"CreationDate":"2018-08-23T12:03:00.000","Title":"How to install twilio via pip","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":1,"System Administration and DevOps":0,"Web Development":0},{"Question":"I'm trying to retrieve the objects\/items (server name, host name, domain name, location, etc...) that are stored under the saved quote for a particular Softlayer account. Can someone help how to retrieve the objects within a quote? I could find a REST API (Python) to retrieve quote details (quote ID, status, etc..) but couldn't find a way to fetch objects within a quote.\nThanks!\nBest regards,\nKhelan Patel","AnswerCount":2,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":71,"Q_Id":51988650,"Users Score":0,"Answer":"Thanks Albert getRecalculatedOrderContainer is the thing I was looking for.","Q_Score":0,"Tags":"python,rest,ibm-cloud,ibm-cloud-infrastructure","A_Id":52010150,"CreationDate":"2018-08-23T14:53:00.000","Title":"How to retrieve objects from the sotlayer saved quote using Python API","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I am trying to set up Google OAuth 2.0 on my homepage.\nOther things seem to be well set up, but CallBack url is a problem.\nI'm using \"https\" and also entered my callback url that starts with https in Google Oauth 2.0 console, but OAuth is still trying to callback configured with http url. How do I fix it?\nIf I go directly into the callback redirection url starting with https, it works fine.","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":70,"Q_Id":51996810,"Users Score":0,"Answer":"I solved this problem myself.\nSimply added it in the settings.py was solution.\nACCOUNT_DEFAULT_HTTP_PROTOCOL = \"https\"\nThat's all.","Q_Score":0,"Tags":"django,python-3.6,google-oauth,django-allauth","A_Id":52008496,"CreationDate":"2018-08-24T02:53:00.000","Title":"Google OAuth 2.0 keep trying to callback using \"http\" url","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"If requests.Session() can handle cookies and does almost everything that app.test_client() does. Then why use the app.test_client()?","AnswerCount":2,"Available Count":1,"Score":0.1973753202,"is_accepted":false,"ViewCount":3506,"Q_Id":52028124,"Users Score":2,"Answer":"test_client is already prebuilt into flask, this makes it easier for people to quickly test their programs. Both the requests utility and test_client server the same functionality, so the usage is just based on personal preference.","Q_Score":11,"Tags":"python,python-3.x,flask","A_Id":52045779,"CreationDate":"2018-08-26T16:21:00.000","Title":"Why use Flask's app.test_client() over requests.Session() for testing","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I have a directed bigraph that contains directed and undirected edges.\nnetworkx_Graph = nx.read_adjlist('graph.txt', create_using=nx.DiGraph())\nI was able to find the number of directed edges using: len(list(networkx_Graph.in_edges(data=False)))\nBut I am trying to find the number of undirected edges.\nThis is quite easy using the python package snap, but I am not finding anything like this in networkx's documentation?\nWhat is the networkx equivalent of snap.CntUniqDirEdges()?","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":372,"Q_Id":52030344,"Users Score":0,"Answer":"You have to convert the graph to an undirected graph the calculate the size. NetworkX does not have a function that does this in one shot like snap does.\n\nnetworkx_Graph.to_undirected().size()","Q_Score":0,"Tags":"python,graph,networkx","A_Id":52031095,"CreationDate":"2018-08-26T21:19:00.000","Title":"How can I find the number of unique undirected edges of a graph using networkx?","Data Science and Machine Learning":1,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I have an AWS Lambda (python) function that is set to trigger on S3 Object Create. What I am trying to do is push the created file to a network drive through our VPC but I am not entirely sure how to configure this or the python code to map a shared drive through the VPC. Am I thinking about this in the wrong way?","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":941,"Q_Id":52046518,"Users Score":0,"Answer":"Step 1: Accessing the file (Object created event) from Lambda Python Code can be done through Boto3 Library. AWS SDK for Python\nStep 2: Configuring your lambda function for VPC - to be done on the Console or through CLI when you create the lambda function. On the Lamda Function Details Page, there is a \"Network\" block of UI containing a drop down called \"VPC\". VPC End Points will be listed here and to be selected.","Q_Score":0,"Tags":"python,amazon-s3,aws-lambda,amazon-vpc","A_Id":52073184,"CreationDate":"2018-08-27T20:25:00.000","Title":"How can I download a file from S3 to local network drive using lambda?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I am facing token expire issue every 20 to 40 mins but actual time is one hour but I need a token validity one day.\nPlease help me.","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":146,"Q_Id":52054877,"Users Score":0,"Answer":"This is not possible to change the token validity period with AWS Cognito User Pools. Currently, its fixed to 1 hour.","Q_Score":0,"Tags":"python-3.x,aws-lambda,aws-cognito","A_Id":52055046,"CreationDate":"2018-08-28T09:48:00.000","Title":"How to increase the AWS Cognito Access Token Validity one hour to one day in Amazon Cognito User Pools","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"Is there a way to provide a single URL for a user to download all the content from an S3 path?\nOtherwise, is there a way to create a zip with all files found on an S3 path recursively?\nie. my-bucket\/media\/123\/*\nEach path usually has 1K+ images and 10+ videos.","AnswerCount":3,"Available Count":1,"Score":0.1973753202,"is_accepted":false,"ViewCount":3130,"Q_Id":52064420,"Users Score":3,"Answer":"There's no built-in way. You have to download all files, compact them \"locally\", re-upload it, and then you'll have a single URL for download.","Q_Score":1,"Tags":"python,amazon-web-services,amazon-s3,aws-lambda","A_Id":52064465,"CreationDate":"2018-08-28T18:43:00.000","Title":"Create a zip with files from an AWS S3 path","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I need some assistance with migrating with Heroku as I've added git+https:\/\/github.com\/Rapptz\/discord.py@rewrite#egg=discord.py[voice] into my requirements.txt file and I'm confused as when I do that, it doesn't change to rewrite, even if I still import discord. I have changed all my code to rewrite, like for example: bot.say to ctx.send. All of that is done, but when I place git+https:\/\/github.com\/Rapptz\/discord.py@rewrite#egg=discord.py[voice] into my requirements.txt file, it still thinks it's async. Please help as I tried so much just to get this working and I can't seem to find a way.\nThanks in advance.","AnswerCount":1,"Available Count":1,"Score":0.1973753202,"is_accepted":false,"ViewCount":710,"Q_Id":52075155,"Users Score":1,"Answer":"LOL wait woops I just had to add yarl<1.2 to requirements.txt","Q_Score":0,"Tags":"python,asynchronous,heroku,discord,discord.py-rewrite","A_Id":52083610,"CreationDate":"2018-08-29T10:24:00.000","Title":"How do I change from Discord.py async to rewrite while using Heroku?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I want to store data about tweets in the database in raw format and figured out that you can pull out the jsob from tweepy.Status for this purpose like this:\nstatus._json\nHow can I parse json back to the tweepy.Status object?","AnswerCount":2,"Available Count":1,"Score":0.2913126125,"is_accepted":false,"ViewCount":913,"Q_Id":52077085,"Users Score":3,"Answer":"I've found non-elegant solution for my problem. All you need is that:\ntweepy.Status().parse(None, status_json)\nwhere None should be tweepy.api.API object, but it not nedeed for parsing at all.\n\nYou can also compare the result with the original status for self-check. In my case this has True result:\ntweepy.Status().parse(None, status_json) == status","Q_Score":1,"Tags":"python,json,tweepy","A_Id":52077086,"CreationDate":"2018-08-29T12:08:00.000","Title":"How to parse JSON string to tweepy.Status object?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":1,"System Administration and DevOps":0,"Web Development":0},{"Question":"I have given a download file url in parse method, what is the default location for that files and Is there any possibility to change that downloaded directory?","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":162,"Q_Id":52082742,"Users Score":0,"Answer":"The response contents in parse(self, response) method are not files and are not stored on disk. The response is a python object and is stored in memory. \nScrapy only downloads contents if HTTP_CACHE_ENABLED setting is set and it will cache all pages in <project_root>\/.scrapy directory.","Q_Score":0,"Tags":"python,scrapy","A_Id":52088635,"CreationDate":"2018-08-29T17:04:00.000","Title":"what is default downloaded path in scrapy?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"so I'm having the weirdest problem! I have a python script which was working just fine and accessing a local mongodb and all was good.. then I wanted to add a new feature but I wanted to try it first so I copied my script into a new file and when I tried to run it it didn't access to mongodb and it kept on giving me this error pymongo: [Errno 10013] An attempt was made to access a socket in a way forbidden by its access permissions\n BUT the old script is still working just fine!!\nI searched every were google could have taken me and tried everything I know but the problem is still there all new scripts are giving me this error and all old ones are working just fine.. \ncan anyone please help me?","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":501,"Q_Id":52087994,"Users Score":1,"Answer":"the problem was with the antivirus.. I have comodo antivirus and it has been blocking the scripts,, I unblocked them and now they work :)","Q_Score":0,"Tags":"python,mongodb,dbaccess","A_Id":52088266,"CreationDate":"2018-08-30T01:16:00.000","Title":"pymongo: [Errno 10013] An attempt was made to access a socket in a way forbidden by its access permissions","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"Python: 2.7.9\nSelenium 2.53.6 \n\nWe drive Selenium from Python, and it works great.\nIssue is that when we hit this line:\ndriver = webdrive.Firefox()\nThe Windows Firewall pops up, asking us to give permission to python.exe\nWe notice that we can ignore the firewall prompt, and everything seems to work OK.\nCan anyone tell us:\n\nWhy something in python land (selenium specifically) is opening a port\nThe port open that triggers the firewall prompt is clearly NOT required, (since we can ignore the prompt, and web drier still works). What is the explanation for this? \nHow we can , in code, suppress the firewall prompt? (e.g. by perhaps only allowing the engine to bind to 127.0.0.1 rather than the device IP?)","AnswerCount":1,"Available Count":1,"Score":-0.1973753202,"is_accepted":false,"ViewCount":630,"Q_Id":52102843,"Users Score":-1,"Answer":"Why something in python land (selenium specifically) is opening a port\nIs this required?\n\nyour Python selenium code must communicate with a standalone server (geckodriver) over HTTP. This can be a local or remote connection depending on your configuration... but you must allow this socket connection to use selenium with Firefox.","Q_Score":1,"Tags":"python,selenium","A_Id":52107370,"CreationDate":"2018-08-30T17:53:00.000","Title":"Selenium: Why does initializing the Firefox webdriver from Python trigger a Windows Firewall alert?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I installed selenium and downloaded chromedriver.exe\nWhen i run the code in my gitbash terminal then its working but not working when I run a python script in visual studio code.\nOn internet it say to put the file in the path but i don't know much about it. Where should i place the chromedriver.exe?","AnswerCount":4,"Available Count":2,"Score":0.0,"is_accepted":false,"ViewCount":3951,"Q_Id":52111479,"Users Score":0,"Answer":"I use Anaconda for which i placed chromedriver.exe in the following\nC:\\Users\\AppData\\Local\\Continuum\\anaconda3\\Scripts","Q_Score":1,"Tags":"python,selenium,google-chrome,selenium-webdriver,selenium-chromedriver","A_Id":52137040,"CreationDate":"2018-08-31T08:31:00.000","Title":"Where to place ChromeDriver while executing tests through selenium in python?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I installed selenium and downloaded chromedriver.exe\nWhen i run the code in my gitbash terminal then its working but not working when I run a python script in visual studio code.\nOn internet it say to put the file in the path but i don't know much about it. Where should i place the chromedriver.exe?","AnswerCount":4,"Available Count":2,"Score":0.049958375,"is_accepted":false,"ViewCount":3951,"Q_Id":52111479,"Users Score":1,"Answer":"driver=webdriver.Chrome(executable_path=r'C:\\Users\\littl\\Downloads\\chromedriver_win32\\chromedriver.exe')","Q_Score":1,"Tags":"python,selenium,google-chrome,selenium-webdriver,selenium-chromedriver","A_Id":57448535,"CreationDate":"2018-08-31T08:31:00.000","Title":"Where to place ChromeDriver while executing tests through selenium in python?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I am using python and boto to assume an AWS IAM role. I want to see what policies are attached to the role so i can loop through them and determine what actions are available for the role. I want to do this so I can know if some actions are available instead of doing this by calling them and checking if i get an error. However I cannot find a way to list the policies for the role after assuming it as the role is not authorised to perform IAM actions.\nIs there anyone who knows how this is done or is this perhaps something i should not be doing.","AnswerCount":1,"Available Count":1,"Score":0.1973753202,"is_accepted":false,"ViewCount":209,"Q_Id":52119306,"Users Score":1,"Answer":"To obtain policies, your AWS credentials require permissions to retrieve the policies.\nIf such permissions are not associated with the assumed role, you could use another set of credentials to retrieve the permissions (but those credentials would need appropriate IAM permissions).\nThere is no way to ask \"What policies do I have?\" without having the necessary permissions. This is an intentional part of AWS security because seeing policies can reveal some security information (eg \"Oh, why am I specifically denied access to the Top-Secret-XYZ S3 bucket?\").","Q_Score":0,"Tags":"python,amazon-web-services,aws-sdk,boto3,amazon-iam","A_Id":52123513,"CreationDate":"2018-08-31T16:13:00.000","Title":"How to list available policies for an assumed AWS IAM role","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I am using Twilio to send and receive SMS messages from a Python application. The issue is that their tutorials use ngrok as a way to get through the firewall but I don't want to have to run ngrok every time I run my app and the URL changes every time ngrok runs so I have to change the webhook url on Twilio every time. Is there a better way around this? Is this something that requires a server?","AnswerCount":2,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":1159,"Q_Id":52123648,"Users Score":2,"Answer":"There are two options that you have.\n\nThe paid option of ngrok allows you to set a persistent url so that you don't have to chance the webhook url on Twilio each time.\nIf you have a server, then you would also be able to set a persistent url to your server.\n\nUnfortunately, the free version of ngrok does not allow you to set a persistent url.","Q_Score":0,"Tags":"python,twilio,webhooks","A_Id":52123715,"CreationDate":"2018-08-31T23:30:00.000","Title":"How to generate fixed webhook url?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I am kinda newby in python thus the question. \nI am trying to create a simple http web server that can receive chunked data from a post request. \nI have realized later that once a request sends a headers with chunked data, the Content-length headers will be set to zero, thus reading the sent data with 'request.get_data()' will fail.\nIs there another way of reading the chunked data?\nThe request I receive does give me the data length in the 'X-Data-Length' headers.","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":519,"Q_Id":52137499,"Users Score":0,"Answer":"Did you write both of the js upload file code and the flask in backend to handle upload request? If not then you will need some help with js to upload it.\nOne way to achieve chucked data upload is:\n\nChucked that file in the frontend with js. Give it some headers in the request for the total size, number of the chunk, chunk size... and send each chuck in a separate POST request (You can use dropzone.js for example, they will do the job for you, just need to config the params) \nIn the backend, create an upload API which will read the request headers and merge the file chunks back together","Q_Score":0,"Tags":"python-2.7,flask","A_Id":52137640,"CreationDate":"2018-09-02T13:29:00.000","Title":"Flask unable to receive chunked data","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I have some code on my mac in the latest version of python idle 3, that collects certain data from a csv file that gets sent to myself and prints out the output in the terminal. I want to create a webpage that has a button or link that a user clicks and it runs the code and prints out the output of my program.\nEventually i want to be able to create a website with multiple links that can do the same operation.\nWill i need to create an sql database? If so how?...","AnswerCount":2,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":876,"Q_Id":52156530,"Users Score":0,"Answer":"From the sound of it, you want to use a webpage as a user interface for your python script. Unfortunately without utilising a server-side language this is not possible.\nMultiple options exist for reacting to a button press on the server side, with PHP being the most well known, but solutions using only python do exist, such as Flask.\nIf you're just after a local GUI for your script, simpler options exist within python such as Tk.","Q_Score":0,"Tags":"html,database,python-3.x,web","A_Id":52156697,"CreationDate":"2018-09-03T21:54:00.000","Title":"How to run a python script when a hyperlink is clicked from a webpage?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"Is it possible to send additional metadata with a Celery 'FAILURE' state?\nCurrently I'm only finding it possible to send exception data and nothing else. Ideally I'd like to send with it an array of extra information that can be picked up by my frontend.","AnswerCount":2,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":1346,"Q_Id":52163940,"Users Score":0,"Answer":"I don't think so. However, you can access the task's traceback property to get the stacktrace, does that help?","Q_Score":2,"Tags":"python,celery","A_Id":52164731,"CreationDate":"2018-09-04T10:14:00.000","Title":"Additional Metadata on Celery 'FAILURE'","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I have a question here. I have been using Python's Etree to parse XML and do syntax checking on it. The problem I am having is it will throw an error when it is unable to parse the XML, but it is not good about indicating where the mistake was actually first made. I realized what I kind of need is to be able to enforce a rule that says there is to be no '>' in the text of an XML element (which for my XML purposes is correct and sound). Is there a way to tell Etree to do this when parsing the XML? I know there is libxml, but if I am to use a library that doesn't come by default with Python 2.75, then I will need the source code as I am not allowed to install additional Python libraries where I work. So, an answer to the question about enforcing no '>' in the text of an XML element, and some suggestions on how to spot the line where a mistake is first made in an XML document; such as forgetting the opening '<' in a closing XML tag. Any help would be much appreciated! Thanks.","AnswerCount":1,"Available Count":1,"Score":0.1973753202,"is_accepted":false,"ViewCount":90,"Q_Id":52170669,"Users Score":1,"Answer":"I'm not sure about your headline question. Why do you want to enforce a rule that \">\" does not appear in text, since there is no such rule in XML?\nIf you're not happy with the diagnostics you're getting from an XML parser then the only real option is to try a different parser (though do check that you are extracting all the information that's available - I don't know Python's ETree, but some parsers hide the diagnostics in obscure places).\nBut there are some limitations. If a start tag is missing, then no parser is going to be able to tell you where it should have been; it can only tell you where the unmatched end tag is. So asking it to tell you \"where the mistake was first made\" is asking too much.","Q_Score":0,"Tags":"python,xml,parsing,syntax","A_Id":52174016,"CreationDate":"2018-09-04T16:38:00.000","Title":"Python XML syntax checking - Enforce NO '>' in the text of an element","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":1,"System Administration and DevOps":0,"Web Development":0},{"Question":"I am new to programming and my first task, which should be really simple, is to automate a proccess, where I log in in a website with my credentials, I click on some buttons and that is it. I am using Python 3.6 on windows 10.\nI am trying to do it using the webbot module, which so far has come really handy but I have one big problem.\nThe standard browser for webbot is Google Chrome and thus the site always opens with Chrome. I need to open the site with Internet Explorer.\nI have set IE as default browser, but nothing changed, Chrome would still open.\nI deleted Chrome, but then when I would run the programm nothing would happen.\nI checked the init.py file and the drivers folder of the module and I think that this module can only work with chrome.\nIs it possible to use IE or does this mean that this package does not support this browser at all?\nWhich alternatives would you suggest?\nEdit: If I am not mistaken Selenium does not support IE11 on windows 10, so that is not an option, unless I am mistaken.\nThanks in advance","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":1106,"Q_Id":52178568,"Users Score":0,"Answer":"There is no support for another browser other than Chrome (as far as the webbot module is concerned).","Q_Score":0,"Tags":"python-3.x,web-scraping","A_Id":53250134,"CreationDate":"2018-09-05T06:36:00.000","Title":"Python - Using a browser other than Chrome with webbot package","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I have a rather odd situation where I will have multiple interfaces connected to the same network. When I receive a broadcast or multicast message, I would like to know what interface it came from. It there a way to do that in C or ideally Python?\nThanks!","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":65,"Q_Id":52196479,"Users Score":0,"Answer":"The most obvious one would be to bind several sockets, each to one interface only - do not listen to 0.0.0.0.","Q_Score":2,"Tags":"python,c,sockets,network-programming,udp","A_Id":52196631,"CreationDate":"2018-09-06T04:23:00.000","Title":"How do you know which interface a broadcast message came from?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I'm using Selenium to scrape table data from a website. I found that I can easily iterate through the rows to get the information that I need using xcode. Does selenium keep hitting the website every time I search for an object's text by xcode? Or does it download the page first and then search through the objects offline?\nIf the former is true does is there a way to download the html and iterate offline using Selenium?","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":43,"Q_Id":52249090,"Users Score":1,"Answer":"Selenium uses a Web Driver, similar to your web browser. Selenium will access\/download the web page once, unless you've wrote the code to reload the page.\nYou can download the web page and access it locally in selenium. For example you could get selenium to access the web page \"C:\\users\\public\\Desktop\\index.html\"","Q_Score":0,"Tags":"python,selenium","A_Id":52249127,"CreationDate":"2018-09-09T22:24:00.000","Title":"What happens when scraping items from a website table using Selenium?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I have a list of about 100,000 URLs saved in my computer. ( that 100,000 can very quickly multiply into several million.) For every url, i check that webpage and collect all additional urls on that page, but only if each additional link is not already in my large list. The issue here is reloading that huge list into memory iteratively so i can consistently have an accurate list. where the amount of memory used will probably very soon become much too much, and even more importantly, the time it takes inbetween reloading the list gets longer which is severely holding up the progress of the project.\nMy list is saved in several different formats. One format is by having all links contained in one single text file, where i open(filetext).readlines() to turn it straight into a list. Another format i have saved which seems more helpful, is by saving a folder tree with all the links, and turning that into a list by using os.walk(path).\nim really unsure of any other way to do this recurring conditional check more efficiently, without the ridiculous use of memory and loadimg time. i tried using a queue as well, but It was such a benefit to be able to see the text output of these links that queueing became unneccesarily complicated. where else can i even start?","AnswerCount":3,"Available Count":2,"Score":0.0665680765,"is_accepted":false,"ViewCount":54,"Q_Id":52253181,"Users Score":1,"Answer":"The main issue is not to load the list in memory. This should be done only once at the beginning, before scrapping the webpages. The issue is to find if an element is already in the list. The in operation will be too long for large list.\nYou should try to look into several thinks; among which sets and pandas. The first one will probably be the optimal solution.\nNow, since you thought of using a folder tree with the urls as folder names, I can think of one way which could be faster. Instead of creating the list with os.walk(path), try to look if the folder is already present. If not, it means you did not have that url yet. This is basically a fake graph database. To do so, you could use the function os.path.isdir(). If you want a true graph DB, you could look into OrientDB for instance.","Q_Score":0,"Tags":"python","A_Id":52253298,"CreationDate":"2018-09-10T07:43:00.000","Title":"Iteratively Re-Checking a Huge list","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":1,"System Administration and DevOps":0,"Web Development":0},{"Question":"I have a list of about 100,000 URLs saved in my computer. ( that 100,000 can very quickly multiply into several million.) For every url, i check that webpage and collect all additional urls on that page, but only if each additional link is not already in my large list. The issue here is reloading that huge list into memory iteratively so i can consistently have an accurate list. where the amount of memory used will probably very soon become much too much, and even more importantly, the time it takes inbetween reloading the list gets longer which is severely holding up the progress of the project.\nMy list is saved in several different formats. One format is by having all links contained in one single text file, where i open(filetext).readlines() to turn it straight into a list. Another format i have saved which seems more helpful, is by saving a folder tree with all the links, and turning that into a list by using os.walk(path).\nim really unsure of any other way to do this recurring conditional check more efficiently, without the ridiculous use of memory and loadimg time. i tried using a queue as well, but It was such a benefit to be able to see the text output of these links that queueing became unneccesarily complicated. where else can i even start?","AnswerCount":3,"Available Count":2,"Score":0.0,"is_accepted":false,"ViewCount":54,"Q_Id":52253181,"Users Score":0,"Answer":"Have you considered mapping a table of IP addresses to URL? Granted this would only work if you are seeking unique domains vs thousands of pages on the same domain. The advantage is you would be dealing with a 12 integer address. The downside is the need for additional tabulated data structures and additional processes to map the data.","Q_Score":0,"Tags":"python","A_Id":52253636,"CreationDate":"2018-09-10T07:43:00.000","Title":"Iteratively Re-Checking a Huge list","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":1,"System Administration and DevOps":0,"Web Development":0},{"Question":"So there are variants of this question - but none quite hit the nail on the head.\nI want to run spyder and do interactive analysis on a server. I have two servers , neither have spyder. They both have python (linux server) but I dont have sudo rights to install packages I need.\nIn short the use case is: open spyder on local machine. Do something (need help here) to use the servers computation power , and then return results to local machine.\nUpdate:\nI have updated python with my packages on one server. Now to figure out the kernel name and link to spyder.\nLeaving previous version of question up, as that is still useful.\nThe docker process is a little intimidating as does paramiko. What are my options?","AnswerCount":2,"Available Count":1,"Score":0.0996679946,"is_accepted":false,"ViewCount":32710,"Q_Id":52283535,"Users Score":1,"Answer":"I did a long search for something like this in my past job, when we wanted to quickly iterate on code which had to run across many workers in a cluster. All the commercial and open source task-queue projects that I found were based on running fixed code with arbitrary inputs, rather than running arbitrary code. \nI'd also be interested to see if there's something out there that I missed. But in my case, I ended up building my own solution (unfortunately not open source). \nMy solution was: \n1) I made a Redis queue where each task consisted of a zip file with a bash setup script (for pip installs, etc), a \"payload\" Python script to run, and a pickle file with input data.\n2) The \"payload\" Python script would read in the pickle file or other files contained in the zip file. It would output a file named output.zip.\n3) The task worker was a Python script (running on the remote machine, listening to the Redis queue) that would would unzip the file, run the bash setup script, then run the Python script. When the script exited, the worker would upload output.zip.\nThere were various optimizations, like the worker wouldn't run the same bash setup script twice in a row (it remembered the SHA1 hash of the most recent setup script). So, anyway, in the worst case you could do that. It was a week or two of work to setup.\nEdit: \nA second (much more manual) option, if you just need to run on one remote machine, is to use sshfs to mount the remote filesystem locally, so you can quickly edit the files in Spyder. Then keep an ssh window open to the remote machine, and run Python from the command line to test-run the scripts on that machine. (That's my standard setup for developing Raspberry Pi programs.)","Q_Score":20,"Tags":"python,docker,server,spyder,fabric","A_Id":52285027,"CreationDate":"2018-09-11T20:09:00.000","Title":"Run Spyder \/Python on remote server","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":1,"Web Development":0},{"Question":"I have created and REST API using Flask-RESTPlus. I am getting response in json and its fine. Now I have a new requirement to specify Response content type as csv or json. \nI checked API doc there is nothing mentioned !!\nIs it possible to get reponse in csv using Flask-RESTPlus ??","AnswerCount":1,"Available Count":1,"Score":0.3799489623,"is_accepted":false,"ViewCount":586,"Q_Id":52325100,"Users Score":2,"Answer":"I was able to get output in csv . It took some time though\n\n@api.representation('text\/csv')\ndef data_csv(data, code, headers):\n '''Get result in csv '''\n resp = make_response(convert_data(data), code)\n resp.headers.extend(headers)\n return resp","Q_Score":1,"Tags":"python,csv,flask-restplus","A_Id":52396235,"CreationDate":"2018-09-14T04:43:00.000","Title":"Get response from Flask-RESTPlus in csv","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I've been working on scrapy for 3 months. for extracting selectors I use simple response.css or response.xpath..\nI'm asked to switch to ItemLoaders and use add_xpath add_css etc.\nI know how ItemLoaders work and ho convinient they are but can anyone compare these 2 w.r.t efficiency? which way is efficient and why ??","AnswerCount":2,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":191,"Q_Id":52330140,"Users Score":0,"Answer":"Item loaders do exactly the same thing underneath that you do when you don't use them. So for every loader.add_css\/add_xpath call there will be responce.css\/xpath executed. It won't be any faster and the little amount of additional work they do won't really make things any slower (especially in comparison to xml parsing and network\/io load).","Q_Score":1,"Tags":"python,python-3.x,scrapy,css-selectors","A_Id":52332084,"CreationDate":"2018-09-14T10:30:00.000","Title":"Scrapy: Difference between simple spider and the one with ItemLoader","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I need Twitter tweet button below every blog post.\nHow do I make {% pageurl %} return an absolute URL of that specific blog post?\nThanks","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":510,"Q_Id":52405554,"Users Score":1,"Answer":"Instead of {% pageurl my_page %}, use {{ my_page.full_url }}.","Q_Score":0,"Tags":"python,django,wagtail","A_Id":52406374,"CreationDate":"2018-09-19T12:09:00.000","Title":"How to make pageurl return an absolute url","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"Suppose I have multiple mongodbs like mongodb_1, mongodb_2, mongodb_3 with same kind of data like employee details of different organizations.\nWhen user triggers GET request to get employee details from all the above 3 mongodbs whose designation is \"TechnicalLead\". then first we need to connect to mongodb_1 and search and then disconnect with mongodb_1 and connect to mongodb_2 and search and repeat the same for all dbs.\nCan any one suggest how can we achieve above using python EVE Rest api framework.\nBest Regards,\nNarendra","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":50,"Q_Id":52418721,"Users Score":0,"Answer":"First of all, it is not a recommended way to run multiple instances (especially when the servers might be running at the same time) as it will lead to usage of the same config parameters like for example logpath and pidfilepath which in most cases is not what you want.\nSecondly for getting the data from multiple mongodb instances you have to create separate get requests for fetching the data. There are two methods of view for the model that can be used:\n\nquery individual databases for data, then assemble the results for viewing on the screen.\nQuery a central database that the two other databases continously update.","Q_Score":0,"Tags":"python,mongodb,eve","A_Id":52872010,"CreationDate":"2018-09-20T06:14:00.000","Title":"How to search for all existing mongodbs for single GET request","Data Science and Machine Learning":0,"Database and SQL":1,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I'm studying Python for 4\/5 months and this is my third project built from scratch, but im not able to solve this problem on my own.\nThis script downloads 1 image for each url given.\nIm not able to find a solution on how to implement Thread Pool Executor or async in this script. I cannot figure out how to link the url with the image number to the save image part. \nI build a dict of all the urls that i need to download but how do I actually save the image with the correct name?\nAny other advise?\nPS. The urls present at the moment are only fake one.\nSynchronous version:\n\n\n import requests\n import argparse\n import re\n import os\n import logging\n\n from bs4 import BeautifulSoup\n\n\n parser = argparse.ArgumentParser()\n parser.add_argument(\"-n\", \"--num\", help=\"Book number\", type=int, required=True) \n parser.add_argument(\"-p\", dest=r\"path_name\", default=r\"F:\\Users\\123\", help=\"Save to dir\", )\n args = parser.parse_args()\n\n\n\n logging.basicConfig(format='%(asctime)s - %(name)s - %(levelname)s - %(message)s',\n level=logging.ERROR)\n logger = logging.getLogger(__name__) \n\n\n def get_parser(url_c): \n url = f'https:\/\/test.net\/g\/{url_c}\/1'\n logger.info(f'Main url: {url_c}')\n responce = requests.get(url, timeout=5) # timeout will raise an exeption\n if responce.status_code == 200:\n page = requests.get(url, timeout=5).content\n soup = BeautifulSoup(page, 'html.parser')\n return soup\n else:\n responce.raise_for_status()\n\n\n def get_locators(soup): # take get_parser\n # Extract first\/last page num\n first = int(soup.select_one('span.current').string)\n logger.info(f'First page: {first}')\n last = int(soup.select_one('span.num-pages').string) + 1\n\n # Extract img_code and extension\n link = soup.find('img', {'class': 'fit-horizontal'}).attrs[\"src\"]\n logger.info(f'Locator code: {link}')\n code = re.search('galleries.([0-9]+)\\\/.\\.(\\w{3})', link)\n book_code = code.group(1) # internal code \n extension = code.group(2) # png or jpg\n\n # extract Dir book name\n pattern = re.compile('pretty\":\"(.*)\"')\n found = soup.find('script', text=pattern)\n string = pattern.search(found.text).group(1)\n dir_name = string.split('\"')[0]\n logger.info(f'Dir name: {dir_name}')\n\n logger.info(f'Hidden code: {book_code}')\n print(f'Extension: {extension}')\n print(f'Tot pages: {last}')\n print(f'')\n\n return {'first_p': first, \n 'last_p': last, \n 'book_code': book_code, \n 'ext': extension, \n 'dir': dir_name\n }\n\n\n def setup_download_dir(path, dir): # (args.path_name, locator['dir'])\n # Make folder if it not exist\n filepath = os.path.join(f'{path}\\{dir}')\n if not os.path.exists(filepath):\n try:\n os.makedirs(filepath)\n print(f'Directory created at: {filepath}')\n except OSError as err:\n print(f\"Can't create {filepath}: {err}\") \n return filepath \n\n\n def main(locator, filepath):\n for image_n in range(locator['first_p'], locator['last_p']):\n url = f\"https:\/\/i.test.net\/galleries\/{locator['book_code']}\/{image_n}.{locator['ext']}\"\n logger.info(f'Url Img: {url}')\n responce = requests.get(url, timeout=3)\n if responce.status_code == 200:\n img_data = requests.get(url, timeout=3).content \n else: \n responce.raise_for_status() # raise exepetion \n\n with open((os.path.join(filepath, f\"{image_n}.{locator['ext']}\")), 'wb') as handler:\n handler.write(img_data) # write image\n print(f'Img {image_n} - DONE')\n\n\n if __name__ == '__main__':\n try:\n locator = get_locators(get_parser(args.num)) # args.num ex. 241461\n main(locator, setup_download_dir(args.path_name, locator['dir'])) \n except KeyboardInterrupt:\n print(f'Program aborted...' + '\\n')\n\n\nUrls list:\n\n\n def img_links(locator):\n image_url = []\n for num in range(locator['first_p'], locator['last_p']):\n url = f\"https:\/\/i.test.net\/galleries\/{locator['book_code']}\/{num}.{locator['ext']}\"\n image_url.append(url)\n logger.info(f'Url List: {image_url}') \n return image_url","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":353,"Q_Id":52430038,"Users Score":0,"Answer":"I found the solution in the book fluent python. Here the snippet:\n\n def download_many(cc_list, base_url, verbose, concur_req):\n counter = collections.Counter()\n with futures.ThreadPoolExecutor(max_workers=concur_req) as executor:\n to_do_map = {}\n for cc in sorted(cc_list):\n future = executor.submit(download_one, cc, base_url, verbose)\n to_do_map[future] = cc\n done_iter = futures.as_completed(to_do_map)\n if not verbose:\n done_iter = tqdm.tqdm(done_iter, total=len(cc_list))\n for future in done_iter:\n try:\n res = future.result()\n except requests.exceptions.HTTPError as exc:\n error_msg = 'HTTP {res.status_code} - {res.reason}'\n error_msg = error_msg.format(res=exc.response)\n except requests.exceptions.ConnectionError as exc:\n error_msg = 'Connection error'\n else:\n error_msg = ''\n status = res.status\n if error_msg:\n status = HTTPStatus.error\n counter[status] += 1\n if verbose and error_msg:\n cc = to_do_map[future]\n print('*** Error for {}: {}'.format(cc, error_msg))\n return counter","Q_Score":2,"Tags":"python,python-3.x,asynchronous,python-multithreading,imagedownload","A_Id":52735044,"CreationDate":"2018-09-20T17:05:00.000","Title":"python asyncronous images download (multiple urls)","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"So I'm making a school project and I have to make a python server and android client. So I've already created a server using websockets with python 3 (I'm not using Flask or socket.io and etc, just regular websockets) and a client in android studio. I ran everything locally and it works great! \nSo now I want to go up in a level a little bit, I want to host my server on Heroku. I tried but I have some problems with that... As I mentioned, I'm using only websockets and not Flask and that means that I need to specify a host ip\/url and a port. But when I host the server on Heroku it says that \"address already in use\".\nDoes anyone know how solve it?","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":1539,"Q_Id":52477103,"Users Score":5,"Answer":"You can host in heroku by using the ip \"0.0.0.0\", and get the port from the env variable called \"PORT\". On the client you can connect to the websocket server using \"wss:\/\/yourherokuapp.herokuapp.com\/0.0.0.0\". Make sure on your Procfile your script is running as a web process type.","Q_Score":5,"Tags":"python,heroku,websocket","A_Id":54756879,"CreationDate":"2018-09-24T10:14:00.000","Title":"How to host websocket app python server on Heroku without Flask?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I'm exploring ODL and mininet and able to run both and populate the network nodes over ODL and I can view the topology via ODL default webgui.\nI'm planning to create my own webgui and to start with simple topology view. I need advise and guideline on how I can achieve topology view on my own webgui. Plan to use python and html. Just a simple single page html and python script. Hopefully someone could lead me the way. Please assist and thank you.","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":162,"Q_Id":52493515,"Users Score":0,"Answer":"If a web GUI for ODL would provide value for you, please consider working to contribute that upstream. The previous GUI (DLUX) has recently been deprecated because no one was supporting it, although it seems many people were using it.","Q_Score":0,"Tags":"python,html,api,web,opendaylight","A_Id":52532208,"CreationDate":"2018-09-25T08:12:00.000","Title":"How to view Opendaylight topology on external webgui","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"My script scans the price of certain products on nike.com and will keep scraping the price of the products till it goes on sale and at that point it will create multiple instances to login into multiple accounts to purchase the product. \nI already have the function of scraping the website and checking out the product made but I want to know should I use multiprocessing or multithreading to execute the logging in and checking out process?\nWhich will be more efficient at handling multiple instances of the web-automation process? I'm using selenium headless in firefox if that helps.","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":353,"Q_Id":52507999,"Users Score":0,"Answer":"Threads are much faster to create, have a smaller memory footprint since they share memory and can communicate by just updating variables, since they all have access to the same memory space.\nI personally like to you the multiprocessing.dummy module that let's you handle threads with the multiprocessing api which is very handy.","Q_Score":0,"Tags":"python,multithreading,selenium,multiprocessing,webautomation","A_Id":52508193,"CreationDate":"2018-09-25T23:42:00.000","Title":"Multithreading or Multiprocessing for a webautomation bot in python?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":1,"System Administration and DevOps":0,"Web Development":0},{"Question":"I would like to share my internal IP between two devices on a network (using python 3).\nLet's say I have my phone, and my computer. Both connected to the same network. I need to run a client and server script to connect the two but in order to do that, my phone (client) needs the ip of the computer (server).\nThe IP of the computer changes all the time (school wifi, nothing I can do about it) and even so I would like this to work instantly when connected to a new router without having to manually enter the IP.\nOne more thing, due to the huge amounts of devices on the network, mapping every device and finding the computer name to get the IP will take too long for its purpose.","AnswerCount":2,"Available Count":2,"Score":1.2,"is_accepted":true,"ViewCount":50,"Q_Id":52513050,"Users Score":0,"Answer":"In case anyone was wondering. I take this question as unsolvable, but, in order to solve my issue, I have set my computer to upload its internal IP to a server each time it connects to a new network. My phone then reads the server.","Q_Score":0,"Tags":"python,python-3.x,networking,ip","A_Id":52564276,"CreationDate":"2018-09-26T08:15:00.000","Title":"Exchange internal ip blindly using python","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I would like to share my internal IP between two devices on a network (using python 3).\nLet's say I have my phone, and my computer. Both connected to the same network. I need to run a client and server script to connect the two but in order to do that, my phone (client) needs the ip of the computer (server).\nThe IP of the computer changes all the time (school wifi, nothing I can do about it) and even so I would like this to work instantly when connected to a new router without having to manually enter the IP.\nOne more thing, due to the huge amounts of devices on the network, mapping every device and finding the computer name to get the IP will take too long for its purpose.","AnswerCount":2,"Available Count":2,"Score":0.0,"is_accepted":false,"ViewCount":50,"Q_Id":52513050,"Users Score":0,"Answer":"Please use DNS for the purpose, or assign static addresses to your devices, and use the defined static addresses in your scripts.","Q_Score":0,"Tags":"python,python-3.x,networking,ip","A_Id":52534087,"CreationDate":"2018-09-26T08:15:00.000","Title":"Exchange internal ip blindly using python","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"How can i import Robot Keywords file from Python script and execute keywords from that file and generate report like Robot html report.\nThanks","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":229,"Q_Id":52516100,"Users Score":0,"Answer":"You cannot import a robot keyword file from a python script, unless that script is using the low level robot API to run a test. \nRobot keywords can only be used from within a running robot test.","Q_Score":0,"Tags":"python,robotframework","A_Id":52516514,"CreationDate":"2018-09-26T10:55:00.000","Title":"How can i import Robot Keywords file from Python script and execute keywords from that file and generate report","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"This might be a silly question. When use sessions with Python to request a file from a website behind a login, can the website detect that you are logging in via a script? How common is it for websites to detect this? I tried looking this up but couldn't find an answer so if this is a repeat question could you point me to some info so I can find my answer?\nThanks in advance!","AnswerCount":2,"Available Count":2,"Score":1.2,"is_accepted":true,"ViewCount":83,"Q_Id":52526787,"Users Score":0,"Answer":"You should change the user agent. But other than that I don\u2019t think its detectable. \nYou can change the user agent by setting a custom header e.g. requests.get(url, headers= {\"user-agent\": \"Mozilla\/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit\/537.36 (KHTML, like Gecko) Chrome\/60.0.3112.113 Safari\/537.36\"}","Q_Score":1,"Tags":"python,server,webserver","A_Id":52526825,"CreationDate":"2018-09-26T22:21:00.000","Title":"Can a web server detect a login from a script?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"This might be a silly question. When use sessions with Python to request a file from a website behind a login, can the website detect that you are logging in via a script? How common is it for websites to detect this? I tried looking this up but couldn't find an answer so if this is a repeat question could you point me to some info so I can find my answer?\nThanks in advance!","AnswerCount":2,"Available Count":2,"Score":0.0,"is_accepted":false,"ViewCount":83,"Q_Id":52526787,"Users Score":0,"Answer":"Nothing is a silly question when you don't have the answer.\nThat being said, every request adds a header by default called the user agent, this can be a multitude of different things but is primarily used to detect what type of device or browser thenuser is connecting to the site with.\nThis includes Python requests! Python sends a user agent Python (version number) in its user agent header.\nIt is very common for webmasters to block these user agents, however, it is extremely simple to spoof the user agent header by changing it prior to sending the request itself.\nYou should look into header customisation of requests. \nIt is also very common for people writing scripts that connect to webserver to hide, spoof or otherwise obfuscate their user agent, but there are plenty more headers send by default which are also used to block unwanted traffic.\nHope this helps","Q_Score":1,"Tags":"python,server,webserver","A_Id":52526889,"CreationDate":"2018-09-26T22:21:00.000","Title":"Can a web server detect a login from a script?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"What is the BeautifulSoup method for retrieving text from an element. I haveprices = soup.find_all('p', class_='price_color') and want to retrieve the text from prices. I tried p = prices.get_text() but got an error stating: 'ResultSet' object has no attribute 'text'","AnswerCount":1,"Available Count":1,"Score":0.3799489623,"is_accepted":false,"ViewCount":36,"Q_Id":52529065,"Users Score":2,"Answer":"find_all returns a ResultSet object and you can iterate that object using a for loop.\nYou can try something such as:\nfor values in soup.find_all('p', class_='price_color'):\n print values.text","Q_Score":0,"Tags":"python,text,beautifulsoup","A_Id":52529129,"CreationDate":"2018-09-27T03:37:00.000","Title":"BeautifulSoup method to retrieve the text from an element","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I'm working on a scraper project and one of the goals is to get every image link from HTML & CSS of a website. I was using BeautifulSoup & TinyCSS to do that but now I'd like to switch everything on Selenium as I can load the JS.\nI can't find in the doc a way to target some CSS parameters without having to know the tag\/id\/class. I can get the images from the HTML easily but I need to target every \"background-image\" parameter from the CSS in order to get the URL from it. \nex: background-image: url(\"paper.gif\");\nIs there a way to do it or should I loop into each element and check the corresponding CSS (which would be time-consuming)?","AnswerCount":1,"Available Count":1,"Score":0.1973753202,"is_accepted":false,"ViewCount":84,"Q_Id":52558105,"Users Score":1,"Answer":"You can grab all the Style tags and parse them, searching what you look.\nAlso you can download the css file, using the resource URL and parse them.\nAlso you can create a XPATH\/CSS rule for searching nodes that contain the parameter that you're looking for.","Q_Score":1,"Tags":"python,css,selenium,web-scraping,selenium-chromedriver","A_Id":52559637,"CreationDate":"2018-09-28T15:04:00.000","Title":"Selenium Python: How to get css without targetting a specific class\/id\/tag","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I have one very general question and another a bit more specific. \nLet's start with the general one as it is strictly connected with the second one. \nSo I'd like to create a website (internal, not available on the internet) with a form that validates user input and if it doesn't meet certain criteria- it cannot be submitted. \nDoes it make sense to create this website (and form) with python django? I mean- obviously it has a lot of sense but there is something else I am trying to puzzle out here:\nWill it make possible to get user input from this website form and pass it to python selenium script? \nI am trying to figure this out as I'd like to improve ordering process with which I work. It is done with a web application provided by an external provider (all things that happen in this application are automatized with python selenium).\nFor the time being I have to read order form (mostly scanned paper forms, pdfs and faxes) put data to excel and then get them to python selenium script but I would like to stop using excel (as it can be really messy). I have to use excel as an intermediary as I don't have a working OCR (so I am an OCR in this case :)) plus current order forms are very different from one another (and standardization is not an option). \nIs it even possible? Is the way I am thinking about anywhere near common sense? Maybe there is an easier method? Any ideas? Thanks for all advices and suggestions.","AnswerCount":1,"Available Count":1,"Score":0.537049567,"is_accepted":false,"ViewCount":107,"Q_Id":52595166,"Users Score":3,"Answer":"Pretty broad, but definitely possible. I would use Flask personally. You can pass any data easily from a form to python and execute any python code on that data with a simple Flask website.","Q_Score":0,"Tags":"python,django,selenium","A_Id":52595231,"CreationDate":"2018-10-01T16:22:00.000","Title":"Web user input form and passing its contents to python selenium","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I have a python script that pulls information from an API (JSON format) and then executes a series of commands and computations based on the API data. I want the computation to run only when there is new data available. So my question is: what is the best way to detect the availability of new data in the API?\nMy current idea is to just pull all the data once every day. Hash the entire thing and compare the hash-numbers. The problem is that python doesn't want to hash a dicitionary object. Any suggestions?","AnswerCount":2,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":256,"Q_Id":52607902,"Users Score":0,"Answer":"you can convert data to string and then hash the result. \nyou can use json.dumps() to convert","Q_Score":0,"Tags":"python,hash","A_Id":52607988,"CreationDate":"2018-10-02T12:00:00.000","Title":"Python: best way to detect when API updates?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":1,"System Administration and DevOps":0,"Web Development":0},{"Question":"I have a script that uses a lot of headless Selenium automation and looped HTTP requests. It's very important that I implement a threading\/worker queue for this script. I've done that.\nMy question is: Should I be using multi-thread or multi-process? Thread or ProcessPool? I know that:\n\"If your program spends more time waiting on file reads or network requests or any type of I\/O task, then it is an I\/O bottleneck and you should be looking at using threads to speed it up.\"\nand...\n\"If your program spends more time in CPU based tasks over large datasets then it is a CPU bottleneck. In this scenario you may be better off using multiple processes in order to speed up your program. I say may as it\u2019s possible that a single-threaded Python program may be faster for CPU bound problems, it can depend on unknown factors such as the size of the problem set and so on.\"\nWhich is the case when it comes to Selenium? Am I right to think that all CPU-bound tasks related to Selenium will be executed separately via the web driver or would my script benefit from multiple processes?\nOr to be more concise: When I thread Selenium in my script, is the web driver limited to 1 CPU core, the same core the script threads are running on?","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":1504,"Q_Id":52617726,"Users Score":4,"Answer":"Web driver is just a driver, a driver cannot drive a car without a car. \nFor example when you use ChromeDriver to communicate with browser, you are launching Chrome. And ChromeDriver itself does no calculation but Chrome does.\nSo to clarify, webdriver is a tool to manipulate browser but itself is not a browser.\nBased on this, definitely you should choose thread pool instead of process pool as it is surely an I\/O bound problem in your python script.","Q_Score":7,"Tags":"python,python-3.x,multithreading,selenium,concurrency","A_Id":52618724,"CreationDate":"2018-10-02T23:27:00.000","Title":"Concurrency and Selenium - Multiprocessing vs Multithreading","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":1,"System Administration and DevOps":0,"Web Development":0},{"Question":"I was trying to do a weather api related python program and initially while running the dependencies there was a error that occurred which reads as \n'No module named 'citipy' error'. \nBut i used from citipy import city command initially and even installed citipy using pip install citipy and upgraded it too. \nThe error still persists. Please help.","AnswerCount":4,"Available Count":3,"Score":1.2,"is_accepted":true,"ViewCount":4976,"Q_Id":52638844,"Users Score":2,"Answer":"I was able to solve it by just changing the kernel. I don't know how kernel affects when your trying to import a module, but it worked for me.","Q_Score":0,"Tags":"python,api,openweathermap","A_Id":52750167,"CreationDate":"2018-10-04T04:06:00.000","Title":"ModuleNotFoundError: No module named 'citipy' error","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":1,"System Administration and DevOps":0,"Web Development":0},{"Question":"I was trying to do a weather api related python program and initially while running the dependencies there was a error that occurred which reads as \n'No module named 'citipy' error'. \nBut i used from citipy import city command initially and even installed citipy using pip install citipy and upgraded it too. \nThe error still persists. Please help.","AnswerCount":4,"Available Count":3,"Score":0.049958375,"is_accepted":false,"ViewCount":4976,"Q_Id":52638844,"Users Score":1,"Answer":"I had initially installed citipy through git bash, but when I ran my code in jupyter notebook it did not run. To solve the problem I had to install citipy through anaconda prompt (which is how I open my jupyter notebook) and I think this is how both the citipy and your code are in the same kernel.","Q_Score":0,"Tags":"python,api,openweathermap","A_Id":55272731,"CreationDate":"2018-10-04T04:06:00.000","Title":"ModuleNotFoundError: No module named 'citipy' error","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":1,"System Administration and DevOps":0,"Web Development":0},{"Question":"I was trying to do a weather api related python program and initially while running the dependencies there was a error that occurred which reads as \n'No module named 'citipy' error'. \nBut i used from citipy import city command initially and even installed citipy using pip install citipy and upgraded it too. \nThe error still persists. Please help.","AnswerCount":4,"Available Count":3,"Score":0.0,"is_accepted":false,"ViewCount":4976,"Q_Id":52638844,"Users Score":0,"Answer":"It happened the same to me, and I noticed that I skipt to activate pythondata from anaconda prompt","Q_Score":0,"Tags":"python,api,openweathermap","A_Id":69748249,"CreationDate":"2018-10-04T04:06:00.000","Title":"ModuleNotFoundError: No module named 'citipy' error","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":1,"System Administration and DevOps":0,"Web Development":0},{"Question":"I am running multiple scrapers using the command line which is an automated process.\nPython : 2.7.12\nScrapy : 1.4.0\nOS : Ubuntu 16.04.4 LTS\nI want to know how scrapy handles the case when \n\nThere is not enough memory\/cpu bandwidth to start a scraper.\nThere is not enough memory\/cpu bandwidth during a scraper run.\n\nI have gone through the documentation but couldn't find anything.\nAnyone answering this, you don't have to know the right answer, if you can point me to the general direction of any resource you know which would be helpful, that would also be appreciated","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":95,"Q_Id":52643398,"Users Score":1,"Answer":"The operating system kills any process that tries to access more memory than the limit.\nApplies to python programs too. and scrapy is no different.\nMore often than not, bandwidth is the bottleneck in scraping \/ crawling applications.\nMemory would only be a bottleneck if there is a serious memory leak in your application.\nYour application would just be very slow if CPU is being shared by many process on the same machine.","Q_Score":0,"Tags":"python,python-2.7,memory-management,scrapy","A_Id":52643555,"CreationDate":"2018-10-04T09:28:00.000","Title":"How does scrapy behave when enough resources are not available","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"Folks, I have a scraping script that I need to run on specific times for live info, but I can't have my computer on me all day. So I thought about running it on an online interpreter, but repl.it doesn't have webdriver and the other I found didn't neither. Could you help me with that?\nThanks","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":694,"Q_Id":52693693,"Users Score":0,"Answer":"I'm not sure, but I don't guess if you can do it on a free online interpreter!\nYou can buy a server and use that, You can SSH to it anytime you want, or even better, You can develop a micro web service using Flask or something else to report the data you need!\nOther way I can think of is let your computer be online 24\/7 and use smtplib to email yourself the data in an interval!","Q_Score":0,"Tags":"python,selenium,selenium-webdriver,pythoninterpreter,repl.it","A_Id":52693719,"CreationDate":"2018-10-07T23:09:00.000","Title":"How to run Selenium with Webdriver on Online Python interpreters?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I would like to pull messages from a RabbitMQ queue, wrap them in a object and dispatch for some kind of processing. Ofcourse I could iteratively do that until the queue is empty, but I was wondering if there is any other way (some flag of some kind) or a neater way.","AnswerCount":1,"Available Count":1,"Score":0.1973753202,"is_accepted":false,"ViewCount":202,"Q_Id":52699030,"Users Score":1,"Answer":"RabbitMQ does not support batches of messages, so you do indeed need to consume each message individually. \nMaybe an alternative would be to batch the messages yourself by publishing one large message with all the required content.","Q_Score":0,"Tags":"python-2.7,rabbitmq,pika","A_Id":52726069,"CreationDate":"2018-10-08T09:18:00.000","Title":"Is it possible to pull all messages from a RabbitMQ queue at once?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":1,"System Administration and DevOps":0,"Web Development":0},{"Question":"What options do I set for chromedriver so that the web server cannot tells the browser is manually launched or programming launched using Selenium?\nThanks,","AnswerCount":1,"Available Count":1,"Score":0.1973753202,"is_accepted":false,"ViewCount":67,"Q_Id":52712046,"Users Score":1,"Answer":"The Webserver you try to access has no way of knowing how the browser has been launched. It can only detect (or rather, guess) that it's an automated browser when said browser shows atypical behavior for a human (e.g. makes loads of requests per seconds, clicks 2 things with no delay whatsoever). Therefor it doesn't matter how you launch the browser - just how you use it.","Q_Score":1,"Tags":"python,selenium,selenium-webdriver","A_Id":52712071,"CreationDate":"2018-10-09T01:17:00.000","Title":"Launching Chromedriver using Selenium","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"We are trying to order a 128 subnet. But looks like it doesn't work, get an error saying Invalid combination specified for ordering a subnet. The same code works to order a 64 subnet. Any thoughts how to order a 128 subnet? \n\nnetwork_mgr = SoftLayer.managers.network.NetworkManager(client)\nnetwork_mgr.add_subnet(\u2018private\u2019, 128, vlan_id, test_order=True)\n\n\nTraceback (most recent call last):\n File \"subnet.py\", line 11, in <module>\n result = nwmgr.add_subnet('private', 128, vlan_id, test_order=True)\n File \"\/usr\/local\/lib\/python2.7\/site-packages\/SoftLayer\/managers\/network.py\", line 154, in add_subnet\n raise TypeError('Invalid combination specified for ordering a'\nTypeError: Invalid combination specified for ordering a subnet.","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":52,"Q_Id":52714654,"Users Score":0,"Answer":"Currently it seems not possible to add 128 ip subnet into the order, the package used by the manager to order subnets only allows to add subnets for: 64,32,16,8,4 (capacity),\nIt is because the package that does not contain any item that has 128 ip addresses subnet, this is the reason why you are getting the error Exception you provided.\nYou may also verify this through the Portal UI, if you can see 128 ip address option through UI in your account, please update this forum with a screenshot.","Q_Score":0,"Tags":"python-2.7,ibm-cloud-infrastructure","A_Id":52730451,"CreationDate":"2018-10-09T06:31:00.000","Title":"SoftLayer API: order a 128 subnet","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I am attempting to install Ambari Server 2.7.1, which depends on python 2.7. From my understanding, the default Centos 7 Python is 2.7.5, which \"should\" be fine, but when I go to install the Ambari Server using yum, it fails stating:\n\"Error: Package: ambari-server-2.7.1.0-169.x86_64 (ambari-2.7.1.0) Requires: python-xml\".\nWhen I search the yum repos I have installed (Base, CR, Debuginfo, Fasttrack, Sources, Vault, EPEL, HDP, HDF, AMBARI, and mysql57-community), I cannot find python-xml anywhere, but from searching google, found that it should be part of base Python 2.7.\nI have also tried \"yum clean all\" and this has no effect on the problem.\nWhat am I missing?","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":426,"Q_Id":52725799,"Users Score":0,"Answer":"Well, I am not sure there will be an acceptable answer to this...I spun up an identical base image on my private Openstack cloud and installed Ambari server with no issues...the only difference I could think of between installs was the order in which I installed components. For example, I installed Ambari server followed by MySQL the second go-around, whereas I installed MySQL first, the first time I tried it.\nI could not find anything in the log files that gave me any clues as to what happened...but I do have both instances running in my cloud in the hopes of troubleshooting it further.","Q_Score":0,"Tags":"python,centos7,yum","A_Id":52745159,"CreationDate":"2018-10-09T16:48:00.000","Title":"Centos 7 missing python-xml package","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I've written a small flask rest api application and a related client library that uses requests to interface with the api. And now I'm writing test cases using pytest. The tests directly against the flask app run fine using the built in test client.\nHowever, now I'm trying to run tests against the flask app through the client library, and it's failing with errors like:\n\nInvalidSchema(\"No connection adapters were found for '%s'\" % url)\n\nAs I understand, I can separately mock out the requests calls, but is there a way I can test the client library directly against the flask application?","AnswerCount":1,"Available Count":1,"Score":0.1973753202,"is_accepted":false,"ViewCount":247,"Q_Id":52729749,"Users Score":1,"Answer":"If you test client library it better to choose mocks your API.\nBut if you want to test client(library) <-> server(flask) integration you need to make some preparation of environment. Like configure client, start server on the same host and port. And then run the tests.","Q_Score":0,"Tags":"python,flask,python-requests,pytest","A_Id":52731357,"CreationDate":"2018-10-09T21:43:00.000","Title":"Testing a flask application with a client that uses requests","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I am using Ubuntu and I have a code which needs to install various libraries. I mostly use python2.7, but due to some libraries this code uses, I had to move to python3, these libraries are \nimport urllib.request\nimport urllib.parse\nimport urllib.error\nBut then there was one library import which I could not import in python3, after searching on Google, I found that moving to python3.7 will solve the issue, and it did solve. Import was\nfrom json import JSONDecodeError\nBut now I have the issue of 'from retrying import retry'......After installing it with pip and pip3, I could import it in Python2.7 and python3, but I am failing to import it in python3.7.....\nSo, basically I am jumping across python versions to import the libraries required to run the code of an ex company employee....Please guide me how to import \"retrying\" in python3.7 or any way I can just install all the below ones in one python version\nimport urllib.request\nimport urllib.parse\nimport urllib.error\nfrom json import JSONDecodeError\nfrom retrying import retry","AnswerCount":1,"Available Count":1,"Score":0.1973753202,"is_accepted":false,"ViewCount":287,"Q_Id":52741688,"Users Score":1,"Answer":"I installed python3.5, and everything worked fine in it. Weird it did not work in python3.7, anyways my issue is resolved.","Q_Score":0,"Tags":"python,python-3.x,python-2.7,ubuntu,installation","A_Id":52758487,"CreationDate":"2018-10-10T13:44:00.000","Title":"Unable to install package \"retrying\" in python 3.7","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":1,"System Administration and DevOps":0,"Web Development":0},{"Question":"I am using python and getting this error.\n\nimport telegram\ntelegram.Bot(token = '###############')\n\nWhen I run this, appears: \n\"AttributeError: module 'telegram' has no attribute 'Bot'\"\nAny ideas how to solve this?","AnswerCount":4,"Available Count":1,"Score":0.049958375,"is_accepted":false,"ViewCount":15267,"Q_Id":52749629,"Users Score":1,"Answer":"Note that your file name (.py) does not the same with your package name.","Q_Score":6,"Tags":"python,api,telegram,attributeerror","A_Id":57895181,"CreationDate":"2018-10-10T22:29:00.000","Title":"AttributeError: module 'telegram' has no attribute 'Bot'","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I've tried many different things to fix the problem, but when I use from selenium import webdriver, I always get ImportError: No module named selenium\nSelenium is definitely installed in c:\/Python27\/Scripts using pip install selenium and also tried -m pip install -U selenium\nAny suggestions? I've tried everything I could find on SO.","AnswerCount":3,"Available Count":3,"Score":1.2,"is_accepted":true,"ViewCount":6609,"Q_Id":52751053,"Users Score":2,"Answer":"@Ankur Singh answered my question. I needed to go run conda install -c clinicalgraphics selenium in an Anaconda Prompt window before importing Selenium","Q_Score":1,"Tags":"python,selenium,selenium-webdriver,jupyter-notebook,jupyter","A_Id":52752048,"CreationDate":"2018-10-11T01:57:00.000","Title":"Python: Unable to import Selenium using Jupyter","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":1,"System Administration and DevOps":0,"Web Development":0},{"Question":"I've tried many different things to fix the problem, but when I use from selenium import webdriver, I always get ImportError: No module named selenium\nSelenium is definitely installed in c:\/Python27\/Scripts using pip install selenium and also tried -m pip install -U selenium\nAny suggestions? I've tried everything I could find on SO.","AnswerCount":3,"Available Count":3,"Score":0.0665680765,"is_accepted":false,"ViewCount":6609,"Q_Id":52751053,"Users Score":1,"Answer":"I had the same issue using Python 3.7 and pip install selenium --user solved it for me.","Q_Score":1,"Tags":"python,selenium,selenium-webdriver,jupyter-notebook,jupyter","A_Id":62248876,"CreationDate":"2018-10-11T01:57:00.000","Title":"Python: Unable to import Selenium using Jupyter","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":1,"System Administration and DevOps":0,"Web Development":0},{"Question":"I've tried many different things to fix the problem, but when I use from selenium import webdriver, I always get ImportError: No module named selenium\nSelenium is definitely installed in c:\/Python27\/Scripts using pip install selenium and also tried -m pip install -U selenium\nAny suggestions? I've tried everything I could find on SO.","AnswerCount":3,"Available Count":3,"Score":0.0,"is_accepted":false,"ViewCount":6609,"Q_Id":52751053,"Users Score":0,"Answer":"Installation success still error?\nLocation problem so do this:-\nCheck where selenium package and file.py located. If both are in different location, move the selenium package to the location where file.py located.","Q_Score":1,"Tags":"python,selenium,selenium-webdriver,jupyter-notebook,jupyter","A_Id":68063740,"CreationDate":"2018-10-11T01:57:00.000","Title":"Python: Unable to import Selenium using Jupyter","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":1,"System Administration and DevOps":0,"Web Development":0},{"Question":"I am trying to parse pricing list json files for some aws services. After parsing I am randomly picking a key from key list to get the data. Currently my code loads the json files one at a time, which takes time. I would like to get some suggestions on how I can speed up this process.","AnswerCount":2,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":118,"Q_Id":52751533,"Users Score":0,"Answer":"Ended up creating a database on redis server.","Q_Score":0,"Tags":"json,python-3.x,amazon-web-services","A_Id":53009841,"CreationDate":"2018-10-11T03:02:00.000","Title":"Need suggestions for fast and efficient way to parse AWS Pricing List json files","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":1,"System Administration and DevOps":0,"Web Development":0},{"Question":"started the test in Selenium in addition in my browser I manually clicked and\/or entered the data in the fields.\nIs it possible to save the actions that I made manually - actions logs?\nI want to know what the user's actions during manual test.","AnswerCount":4,"Available Count":3,"Score":0.0,"is_accepted":false,"ViewCount":264,"Q_Id":52792860,"Users Score":0,"Answer":"I suggest you using Selenium with third-party framework like Robotframework.\nIt will be easier to observe actions with those behavior driven test framework.\nAnd they will also help you to capture screenshot while error occurs.","Q_Score":0,"Tags":"java,python,selenium,webdriver","A_Id":52792957,"CreationDate":"2018-10-13T12:22:00.000","Title":"Manual actions in Selenium","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"started the test in Selenium in addition in my browser I manually clicked and\/or entered the data in the fields.\nIs it possible to save the actions that I made manually - actions logs?\nI want to know what the user's actions during manual test.","AnswerCount":4,"Available Count":3,"Score":0.0,"is_accepted":false,"ViewCount":264,"Q_Id":52792860,"Users Score":0,"Answer":"S\u0142awomir, why don't You start record all actions from beginning, after launch webbrowser. For FF55+ there is an addon Katalon Recorder. You can record all steps and than export all actions to Java, Pyton code. Than just copy code from exporter, and You can use it in your webdriver tests.","Q_Score":0,"Tags":"java,python,selenium,webdriver","A_Id":52854082,"CreationDate":"2018-10-13T12:22:00.000","Title":"Manual actions in Selenium","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"started the test in Selenium in addition in my browser I manually clicked and\/or entered the data in the fields.\nIs it possible to save the actions that I made manually - actions logs?\nI want to know what the user's actions during manual test.","AnswerCount":4,"Available Count":3,"Score":0.0,"is_accepted":false,"ViewCount":264,"Q_Id":52792860,"Users Score":0,"Answer":"Below example\n\nSimple example\nDuring the test Selenium opens a page example.com. Next Selenium click on link to example.com\/login. On the login page Selenium enters username and password -- it is correct test\n\nExample with manual actions\nDuring the test Selenium opens a page example.com. Now manually I click on link example.com\/about (but on About page isn't link to example.com\/login Next Selenium try to click on link to example.com\/login, but can't because link to example.com\/login doesn't exist. -- test failed\n\nTest failed because I make manual action, so I want to log all manually actions","Q_Score":0,"Tags":"java,python,selenium,webdriver","A_Id":52860596,"CreationDate":"2018-10-13T12:22:00.000","Title":"Manual actions in Selenium","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I'm building a PWA with django\/python on the server-side and vue on the client-side and want to use firebase as a database as well as make use of the firebase authentication.\nAfter some thorough research I realised that I had to make a few choices. \nQuestion 1: Authentication \n\nI can do authentication on the client-side or server-side. Which one would be best (more secure) ?\n\nQuestion 2: Database\n\nIn terms of CRUDS I am a bit conflicted. Do I write all my data to firestore from the client-side?\nDo I rather use api's to communicate with my backend and then write data to firestore from the backend? What are the security implications of doing this?\n\nShould I just use both in terms of context? If there are no security implications I would do my authentication client-side and my CRUDS from the server-side. I think I would also have to check authentication to write to the database from the backend.","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":2897,"Q_Id":52793661,"Users Score":5,"Answer":"Authentication of a user's credentials should always happen on a server, as it can't be securely done on the client's computer. What Firebase Authentication allows however, is that the authentication runs on Google's servers, while you control it from a simple client-side API call.","Q_Score":7,"Tags":"python,django,firebase,vue.js,firebase-authentication","A_Id":52793728,"CreationDate":"2018-10-13T13:54:00.000","Title":"Firebase (client-side vs server-side)","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"How to access a folder object inside S3 bucket. How can I access a folder inside S3 bucket using python boto3.\nCode is working for a folder in S3 bucket but to for folders inside S3 bucket","AnswerCount":3,"Available Count":1,"Score":0.1325487884,"is_accepted":false,"ViewCount":4249,"Q_Id":52805052,"Users Score":2,"Answer":"If I understand you correctly.. I had this issue in my python3 script. Basically you need to pass the to the boto3 function an s3 bucket and the file name. Make this file name include the folder, with the forward slash separating them. Instead of passing just the file name and trying to pass the folder as a separate parameter. \nSo if you have MyS3Bucket and you want to upload file.txt to MyFolder inside MyS3Bucket, then pass the file_name=\u201cMyFolder\u201d+\u201d\/\u201c+\u201dfile.txt\u201d as a parameter to the upload function.\nLet me know if you need a code snippet. \nEven if you don\u2019t have the folder in the S3 bucket, boto3 will create it for you on the fly. This is cool because you can grant access in s3 based on a folder, not just the whole bucket at once. \nGood luck!","Q_Score":0,"Tags":"python,amazon-web-services,amazon-s3,boto3","A_Id":55203529,"CreationDate":"2018-10-14T17:03:00.000","Title":"How to upload file to folder in aws S3 bucket using python boto3","Data Science and Machine Learning":0,"Database and SQL":1,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"For example in python if I\u2019m sending data through sockets could I make my own encryption algorithm to encrypt that data? Would it be unbreakable since only I know how it works?","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":42,"Q_Id":52806028,"Users Score":1,"Answer":"Yes you can. Would it be unbreakable? No. This is called security through obscurity. You're relying on the fact that nobody knows how it works. But can you really rely on that?\nSomeone is going to receive the data, and they'll have to decrypt it. The code must run on their machine for that to happen. If they have the code, they know how it works. Well, at least anyone with a lot of spare time and nothing else to do can easily reverse engineer it, and there goes your obscurity.\nIs it feasable to make your own algorithm? Sure. A bit of XOR here, a bit of shuffling there... eventually you'll have an encryption algorithm. It probably wouldn't be a good one but it would do the job, at least until someone tries to break it, then it probably wouldn't last a day.\nDoes Python care? Do sockets care? No. You can do whatever you want with the data. It's just bits after all, what they mean is up to you.\nAre you a cryptographer? No, otherwise you wouldn't be here asking this. So should you do it? No.","Q_Score":0,"Tags":"python,encryption","A_Id":52806168,"CreationDate":"2018-10-14T18:54:00.000","Title":"Is it possible to make my own encryption when sending data through sockets?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I want to retrieve the text data for a specific twitter account and save it for a ML project about text generation, is this possible using the Twitter API?","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":164,"Q_Id":52807824,"Users Score":0,"Answer":"Yes. It is possible. Try tweepy. It is a wrapper for Twitter API.","Q_Score":0,"Tags":"python,text,twitter","A_Id":52807837,"CreationDate":"2018-10-14T23:02:00.000","Title":"How do\/can I retrieve text data from a Twitter account in Python","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"Scrapy Crawlera was working just well in my Windows machine, but it gets error 111 when I run it in my linux server. Why is that?\nWhen I use curl, I got this error:\ncurl: (7) Failed connect to proxy.crawlera.com:8010; Connection refused","AnswerCount":2,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":2658,"Q_Id":52866297,"Users Score":1,"Answer":"It turned out when dealing with ports, CPanel (or maybe Linux?) blocks ports by default if it is not whitelisted in the firewall. I opened it via WHM since I use CPanel, and everything works fine now.","Q_Score":1,"Tags":"python,web-scraping,scrapy,screen-scraping,crawlera","A_Id":52882604,"CreationDate":"2018-10-18T02:53:00.000","Title":"Connection was refused by other side: 111: Connection refused. when using Scrapy Crawlera in a linux server","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I have a python script that takes in a .csv file and performs some data cleaning and some more advanced operations and gives back a .csv file as its output.\nIs there a library I can use to build a webpage and host it on some server for users to be able to upload the input .csv file into it and be able to download the output .csv file?","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":57,"Q_Id":52885542,"Users Score":0,"Answer":"I would say your best choice if you are new to web development, is to start learning Flask and the basics of \"How the Web Works\". There are other libraries such as Django or Cherrypy as well. Flask is a web framework that can run a server side application (a.k.a. back-end) and it's relatively simple out of the box.\nHowever, when you say \"running python script on webpage\" and \"build a webpage\", that's not exactly how it works. You will build your webpage (a.k.a. the client-side application or the front-end) using HTML, CSS and possibly JavaScript. Web browsers can't run python.","Q_Score":0,"Tags":"python","A_Id":52885671,"CreationDate":"2018-10-19T03:34:00.000","Title":"Running python script on webpage","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"So my directory currently looks something like this \nsrc\n \\aws_resources\n \\s3\n s3file.py\n util.py\ntest\n mytest.py\nI'm running the mytest.py file and it essentially imports the s3 file, while the s3 file imports the util.py file\nMy mytest.py says \nfrom src.aws_resources.s3 import *\n\nbut when I run the test file I get an error \nImportError: No module named aws_resources.util\n\nI tried adding \nfrom src.aws_resources import util to mytest.py but I still get the same error. Any suggestions would be appreciated.","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":672,"Q_Id":52952756,"Users Score":0,"Answer":"Your current working directory is test, which means any import line will look into the test folder to find modules, if not found, go to the other paths existing in your sys.path (e.g. PYTHONPATH) to find.\nHere are my two assumptions:\n1.) src is not part of your PYTHONPATH\n2.) In your s3file.py you have this line (or similar): from aws_resources.util import *\nSince aws_resources is not a valid folder under test, and it couldn't be found under any of your sys.path, the Python interpreter couldn't find the module and rasied ImportError.\nYou can try one of the following:\n1.) Restructure your folder structure so that everything can be immediately referenced from your current working directory.\n2.) Handle the from aws_resources.util import * line in your s3file.py if you are importing it from another working directory.\n3.) Add src into your PYTHONPATH, or move it to a folder identified in PATH\/PYTHONPATH so you can always reference back to it.\nThe approach really depends on your use case for src and how often it gets referenced elsewhere.","Q_Score":0,"Tags":"python,python-2.7,import","A_Id":52952937,"CreationDate":"2018-10-23T15:28:00.000","Title":"\"No module named \" found when importing file that imports another file","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I am using updatedItem() function which is either inserting or updating values in Dynamodb. If values are updating I want to fetch those items and invoke a new lambda function. How can I achieve this?","AnswerCount":2,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":330,"Q_Id":52985493,"Users Score":0,"Answer":"Don't think previously accepted answer works. The return Attributes never returned the partition\/sort keys, whether an item was updated or created.\nWhat worked for me was to add ReturnValues: UPDATED_OLD. If the returned Attributes is undefined, then you know that an item was created.","Q_Score":1,"Tags":"python-2.7,amazon-web-services,aws-lambda,amazon-dynamodb","A_Id":71245645,"CreationDate":"2018-10-25T09:08:00.000","Title":"Find whether the value has been updated or inserted in Dynamodb?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I have a client that reads messages from RabbitMq and sent it to a server, the IP of the server is specified in the message, I want the sending and receiving parts to be asynchronous because I will be dealing with multiple servers and I don't want the code to hang and wait for a response. by asynchronous I mean the send and receive work in parallel. any suggestion ? thanks","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":54,"Q_Id":52991421,"Users Score":0,"Answer":"You can use WebSockets for this. The send and receive part are event-driven and work in parallel.\nIf you already have sending and receiving mechanism and you just want them to be in parallel, have a look at greenlets in python.","Q_Score":0,"Tags":"python-3.x,sockets,asynchronous,rabbitmq","A_Id":52991521,"CreationDate":"2018-10-25T14:11:00.000","Title":"Asynchronous send\/receive client","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I am doing a project where I want to build a model that estimates the number of retweets at a given time for a given tweet bearing a certain keyword, so that when a new tweet with the same keyword comes in, I can track its number of retweets and see if there is any anomaly. For that, on top of collecting a large number of tweets with that certain keyword for modeling, I need to know for each of the tweets what was the number of retweets on day 1, day 2, etc (the unit of time is arbitrary here, can be in days or in minutes) since it was created. \nI've done some research on stackoverflow, but I have not seen a solution for this particular problem. I understand that twitter API allows you to search for tweets with keywords, but it only gives you the tweets' current number of retweets but not the historical performance. \nI would really appreciate it if anyone can point me to the right direction. Thank you very much!","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":270,"Q_Id":53068167,"Users Score":0,"Answer":"There's no API method in any of Twitter's API tiers (standard, premium or enterprise) that would enable you to do this. You'd need to have already been listening for the Tweets, record the Tweet IDs, check them every day for total engagement numbers, and then record the differential every day.","Q_Score":1,"Tags":"python,api,twitter,web-scraping","A_Id":53071904,"CreationDate":"2018-10-30T15:49:00.000","Title":"Get Historical Tweet Metadata from Twitter API in Python","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I want to connect two computers and send messages between them,and I'm not sure why the socket module doesn't work for me.So is there any other ways to connect two computers?","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":200,"Q_Id":53075044,"Users Score":2,"Answer":"If you wanna solve this problem,\nFirst thing , you should check two computer's network connecting, \nin terminal you can use one computer of yours to typing ping ${target_computer_address}and check terminal's echo to make sure two computer network connection is working.\nSecond thing , you can use python to open a tcp port to listen and print recv data in screen ,and in other computer telnet previous computer ip:port just typing someting, make sure python can print you's typing character correct.","Q_Score":0,"Tags":"python,python-3.x,connection","A_Id":53075143,"CreationDate":"2018-10-31T01:28:00.000","Title":"How to connect two computers using python","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I have a script that uploads files to Google Drive. I want to upload python files. I can do it manually and have it keep the file as .py correctly (and it's previewable), but no matter what mimetypes I try, I can't get my program to upload it correctly. It can upload the file as a .txt or as something GDrive can't recognize, but not as a .py file. I can't find an explicit mimetype for it (I found a reference for text\/x-script.python but it doesn't work as an out mimetype).\nDoes anyone know how to correctly upload a .py file to Google Drive using REST?","AnswerCount":2,"Available Count":1,"Score":-0.0996679946,"is_accepted":false,"ViewCount":690,"Q_Id":53096788,"Users Score":-1,"Answer":"Also this is a valid Python mimetype: text\/x-script.python","Q_Score":2,"Tags":"python,rest,google-drive-api,mime-types","A_Id":71426330,"CreationDate":"2018-11-01T07:22:00.000","Title":"What Is the Correct Mimetype (in and out) for a .Py File for Google Drive?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I'd like to post a video from url to my Facebook page. Is there any way for it, I already be able to post Images but seems like it's no way for video?","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":206,"Q_Id":53112641,"Users Score":0,"Answer":"I'd like to post a video from url to my Facebook page. Is there any way for it, I already be able to post Images but seems like it's no way for video?\n\nThat is correct, videos can not be uploaded by specifying a video URL.\nYou need to perform an actual HTTP upload in multipart\/form-data format.","Q_Score":0,"Tags":"python,django,facebook,facebook-graph-api","A_Id":53114362,"CreationDate":"2018-11-02T04:22:00.000","Title":"Post video to Facebook Page with Facebook SDK","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I have Python 3.7.1 installed on Windows 7 from www.python.org.\nWhen I want to \"pip install pylab\" I receive the following message:\n\"pip is configured with locations that require tls\/ssl however the ssl module in python is not available\".\nPlease help me to overcome this problem.\nMany thanks","AnswerCount":5,"Available Count":5,"Score":0.1973753202,"is_accepted":false,"ViewCount":18293,"Q_Id":53137700,"Users Score":5,"Answer":"I also had the same problem. But solved it by just adding path variables in evt variables \nSo check for the following paths \n1. Anaconda root \n2. Anaconda\/Scripts\n3. Anaconda\/Library\/bin\nthe 3rd one was not added and after adding all these evt variables do restart your PC. and the error will be resolved.","Q_Score":10,"Tags":"python,ssl","A_Id":54933554,"CreationDate":"2018-11-04T04:18:00.000","Title":"ssl module in python is not available Windows 7","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I have Python 3.7.1 installed on Windows 7 from www.python.org.\nWhen I want to \"pip install pylab\" I receive the following message:\n\"pip is configured with locations that require tls\/ssl however the ssl module in python is not available\".\nPlease help me to overcome this problem.\nMany thanks","AnswerCount":5,"Available Count":5,"Score":0.1586485043,"is_accepted":false,"ViewCount":18293,"Q_Id":53137700,"Users Score":4,"Answer":"What worked for me is:\n conda activate in command prompt and then use pip","Q_Score":10,"Tags":"python,ssl","A_Id":55987478,"CreationDate":"2018-11-04T04:18:00.000","Title":"ssl module in python is not available Windows 7","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I have Python 3.7.1 installed on Windows 7 from www.python.org.\nWhen I want to \"pip install pylab\" I receive the following message:\n\"pip is configured with locations that require tls\/ssl however the ssl module in python is not available\".\nPlease help me to overcome this problem.\nMany thanks","AnswerCount":5,"Available Count":5,"Score":0.0,"is_accepted":false,"ViewCount":18293,"Q_Id":53137700,"Users Score":0,"Answer":"Activating conda (from condabin folder: activate.bat) and then from the python.exe folder \"python.exe -m pip install siphon\"","Q_Score":10,"Tags":"python,ssl","A_Id":57626871,"CreationDate":"2018-11-04T04:18:00.000","Title":"ssl module in python is not available Windows 7","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I have Python 3.7.1 installed on Windows 7 from www.python.org.\nWhen I want to \"pip install pylab\" I receive the following message:\n\"pip is configured with locations that require tls\/ssl however the ssl module in python is not available\".\nPlease help me to overcome this problem.\nMany thanks","AnswerCount":5,"Available Count":5,"Score":0.0,"is_accepted":false,"ViewCount":18293,"Q_Id":53137700,"Users Score":0,"Answer":"I had the same issue because of corporate firewall settings.\nIf you are on windows go to \"Internet Properties\" ---> \"Connection\" ---> \"LAN Settings\" and check the address (if it is a wpad.dat file, download it and look for \"ProxyPort\" and \"ProxyServer\").\nThen try :\npip --proxy http:\/\/user:password@ProxyServer:ProxyPort install module","Q_Score":10,"Tags":"python,ssl","A_Id":58202904,"CreationDate":"2018-11-04T04:18:00.000","Title":"ssl module in python is not available Windows 7","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I have Python 3.7.1 installed on Windows 7 from www.python.org.\nWhen I want to \"pip install pylab\" I receive the following message:\n\"pip is configured with locations that require tls\/ssl however the ssl module in python is not available\".\nPlease help me to overcome this problem.\nMany thanks","AnswerCount":5,"Available Count":5,"Score":1.0,"is_accepted":false,"ViewCount":18293,"Q_Id":53137700,"Users Score":7,"Answer":"According to your comment with the PATH variable you are missing the right entries to this so it cannot find the libraries and so on. \nIf you reinstall python in the first screen you have an option \u2018Add Python x.x.x to PATH\u2019 enable this option and then retry what you did.","Q_Score":10,"Tags":"python,ssl","A_Id":53174338,"CreationDate":"2018-11-04T04:18:00.000","Title":"ssl module in python is not available Windows 7","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I am trying to push some files up to s3 with the AWS CLI and I am running into an error:\nupload failed: ... An HTTP Client raised and unhandled exception: unknown encoding: idna\nI believe this is a Python specific problem but I am not sure how to enable this type of encoding for my python interpreter. I just freshly installed Python 3.6 and have verified that it being used by powershell and cmd. \n$> python --version\n Python 3.6.7\nIf this isn't a Python specific problem, it might be helpful to know that I also just freshly installed the AWS CLI and have it properly configured. Let me know if there is anything else I am missing to help solve this problem. Thanks.","AnswerCount":5,"Available Count":2,"Score":1.2,"is_accepted":true,"ViewCount":6262,"Q_Id":53144254,"Users Score":8,"Answer":"I had the same problem in Windows.\nAfter investigating the problem, I realized that the problem is in the aws-cli installed using the MSI installer (x64). After removing \"AWS Command Line Interface\" from the list of installed programs and installing aws-cli using pip, the problem was solved.\nI also tried to install MSI installer x32 and the problem was missing.","Q_Score":12,"Tags":"python-3.x,amazon-s3,aws-cli","A_Id":54337242,"CreationDate":"2018-11-04T18:53:00.000","Title":"AWS CLI upload failed: unknown encoding: idna","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I am trying to push some files up to s3 with the AWS CLI and I am running into an error:\nupload failed: ... An HTTP Client raised and unhandled exception: unknown encoding: idna\nI believe this is a Python specific problem but I am not sure how to enable this type of encoding for my python interpreter. I just freshly installed Python 3.6 and have verified that it being used by powershell and cmd. \n$> python --version\n Python 3.6.7\nIf this isn't a Python specific problem, it might be helpful to know that I also just freshly installed the AWS CLI and have it properly configured. Let me know if there is anything else I am missing to help solve this problem. Thanks.","AnswerCount":5,"Available Count":2,"Score":-0.0798297691,"is_accepted":false,"ViewCount":6262,"Q_Id":53144254,"Users Score":-2,"Answer":"Even I was facing same issue. I was running it on Windows server 2008 R2. I was trying to upload around 500 files to s3 using below command.\n\naws s3 cp sourcedir s3bucket --recursive --acl\n bucket-owner-full-control --profile profilename\n\nIt works well and uploads almost all files, but for initial 2 or 3 files, it used to fail with error: An HTTP Client raised and unhandled exception: unknown encoding: idna\nThis error was not consistent. The file for which upload failed, it might succeed if I try to run it again. It was quite weird.\nTried on trial and error basis and it started working well.\nSolution:\n\nUninstalled Python 3 and AWS CLI.\nInstalled Python 2.7.15\nAdded python installed path in environment variable PATH. Also added pythoninstalledpath\\scripts to PATH variable. \nAWS CLI doesnt work well with MS Installer on Windows Server 2008, instead used PIP. \n\nCommand: \n\npip install awscli\n\nNote: for pip to work, do not forget to add pythoninstalledpath\\scripts to PATH variable.\nYou should have following version:\nCommand: \n\naws --version\n\nOutput: aws-cli\/1.16.72 Python\/2.7.15 Windows\/2008ServerR2 botocore\/1.12.62\nVoila! The error is gone!","Q_Score":12,"Tags":"python-3.x,amazon-s3,aws-cli","A_Id":53693183,"CreationDate":"2018-11-04T18:53:00.000","Title":"AWS CLI upload failed: unknown encoding: idna","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"Im writing a webapplication, where im trying to display the connected USB devices. I found a Python function that does exactly what i want but i cant really figure out how to call the function from my HTML code, preferably on the click of a button.","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":76,"Q_Id":53152383,"Users Score":0,"Answer":"simple answer: you can't. the code would have to be run client-side, and no browser would execute potentially malicious code automatically (and not every system has a python interpreter installed). \nthe only thing you can execute client-side (without the user taking action, e.g. downloading a program or browser add-on) is javascript.","Q_Score":0,"Tags":"python,html,python-2.7","A_Id":53152455,"CreationDate":"2018-11-05T10:20:00.000","Title":"Calling a Python function from HTML","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"We have several microservices on Golang and Python, On Golang we are writing finance operations and on Python online store logic, we want to create one API for our front-end and we don't know how to do it.\nI have read about API gateway and would it be right if Golang will create its own GraphQL server, Python will create another one and they both will communicate with the third graphql server which will generate API for out front-end.","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":832,"Q_Id":53159897,"Users Score":2,"Answer":"I do not know much details about your services, but great pattern I successfully used on different projects is as you mentioned GraphQL gateway. \nYou will create one service, I prefer to create it in Node.js where all requests from frontend will coming through. Then from GraphQL gateway you will request your microservices. This will be basically your only entry point into the backend system. Requests will be authenticated and you are able to unify access to your data and perform also some performance optimizations like implementing data loader's caching and batching to mitigate N+1 problem. In addition you will reduce complexity of having multiple APIs and leverage all the GraphQL benefits. \nOn my last project we had 7 different frontends and each was using the same GraphQL gateway and I was really happy with our approach. There are definitely some downsides as you need to keep in sync all your frontends and GraphQL gateway, therefore you need to be more aware of your breaking changes, but it is solvable with for example deprecated directive and by performing blue\/green deployment with Kubernetes cluster. \nThe other option is to create the so-called backend for frontend in GraphQL. Right now I do not have enough information which solution would be best for you. You need to decide based on your frontend needs and business domain, but usually I prefer GraphQL gateway as GraphQL has great flexibility and the need to taylor your API to frontend is covered by GraphQL capabilities. Hope it helps David","Q_Score":2,"Tags":"python,api,go,graphql","A_Id":53173290,"CreationDate":"2018-11-05T18:11:00.000","Title":"How to create Graphql server for microservices?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I have created a lambda that iterates over all the files in a given S3 bucket and deletes the files in S3 bucket. The S3 bucket has around 100K files and I am selecting and deleting the around 60K files. I have set the timeout for lambda to max (15 minutes) timeout value. The lambda is consistently returning \"network error\" after few minutes though it seems to run in the background for sometime even after the error is returned. How can get around this?","AnswerCount":2,"Available Count":1,"Score":0.0996679946,"is_accepted":false,"ViewCount":14183,"Q_Id":53180543,"Users Score":1,"Answer":"I was testing another function and this error came up as a result. Reading a little in the documentation I found that I activated the throttle option, that it reduces the amount of rate for your function.\nThe solution is to create another function and see if the throttle is making that error.","Q_Score":4,"Tags":"python,amazon-web-services,amazon-s3,aws-lambda,boto3","A_Id":65430231,"CreationDate":"2018-11-06T21:47:00.000","Title":"AWS Lambda : Calling the invoke API action failed with this message: Network Error","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I need a browser in Selenium to work as if it's maximized, so that website that I process perceived it as maximized, but I want it really to be minimized, so that I could work in parallel while the script executes. The simple: driver.maximize_window() just maximizes windows.\nSo, is there a way for a window of browser to be maximized for website, but in reality minimized?","AnswerCount":4,"Available Count":2,"Score":0.049958375,"is_accepted":false,"ViewCount":554,"Q_Id":53189353,"Users Score":1,"Answer":"If you want to work while WebDriver is executed, you can created a virtual instance (with e.g. VirtualBox or HyperV) an run test suite in VM.","Q_Score":0,"Tags":"python,selenium,webdriver","A_Id":53191171,"CreationDate":"2018-11-07T12:19:00.000","Title":"Is there 'as if' maximized mode of browser in Webdriver Selenium, but in reality minimized?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I need a browser in Selenium to work as if it's maximized, so that website that I process perceived it as maximized, but I want it really to be minimized, so that I could work in parallel while the script executes. The simple: driver.maximize_window() just maximizes windows.\nSo, is there a way for a window of browser to be maximized for website, but in reality minimized?","AnswerCount":4,"Available Count":2,"Score":0.0,"is_accepted":false,"ViewCount":554,"Q_Id":53189353,"Users Score":0,"Answer":"Maximized is just a browser size from the site perspective. Set the browser size to the screen resolution of your desktop and minimize the browser.","Q_Score":0,"Tags":"python,selenium,webdriver","A_Id":53191735,"CreationDate":"2018-11-07T12:19:00.000","Title":"Is there 'as if' maximized mode of browser in Webdriver Selenium, but in reality minimized?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"my program have one server socket and multiple client socket,\nso, what i want do is When the server close(shutdown),change one of the client sockets to server socket.\nis there a way to do this?","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":457,"Q_Id":53200594,"Users Score":0,"Answer":"what i want do is When the server close(shutdown),change one of the\n client sockets to server socket. is there a way to do this?\n\nNo -- once a TCP socket is connected, it can only be used for that one TCP connection, and once the TCP connection has been broken (e.g. by the server exiting), then all you can do with the socket is close it.\nHowever, that doesn't mean you can't have a high-availability chat system like the one you are envisioning. What you can do is have your clients \"plan ahead\" by gathering the IP addresses of all the other clients in advance (either through the server, or if all the clients are on the same LAN, perhaps through broadcast or multicast UDP packets). That way, if\/when the server goes away, the clients all have another IP address handy that they can automatically try to connect to (using a new TCP socket).\nNote that you'll need to make sure that the program running at that IP address is accepting incoming TCP connections (server-style); you also might want to specify some kind of rule so that all of the clients will reconnect to the same IP address (e.g. sort the IP addresses numerically and have all the clients try to connect to the smallest IP address in the list, or something like that).","Q_Score":1,"Tags":"python,sockets","A_Id":53202099,"CreationDate":"2018-11-08T02:14:00.000","Title":"Python Socket programming change server socket","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I'd like to fetch mails from a server, but I also want to control when to delete them.\nIs there a way to do this?\nI know this setting is very usual in mail clients, but it seems this option is not well supported by POPv3 specification and\/or server implementations.\n(I'm using python, but I'm ok with other languages\/libraries, Python's poplib seems very simplistic)","AnswerCount":2,"Available Count":1,"Score":0.1973753202,"is_accepted":false,"ViewCount":964,"Q_Id":53231525,"Users Score":2,"Answer":"POP3 by design downloads and removes mail from a server after it's successfully fetched. If you don't want that, then use the IMAP protocol instead. That protocol has support to allow you to delete mail at your leisure as opposed to when it's synced to your machine.","Q_Score":2,"Tags":"python,pop3,poplib","A_Id":53231559,"CreationDate":"2018-11-09T18:34:00.000","Title":"Fetch mails via POP3, but keep them on the server","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"Is there any way to get all games that user has played while using Discord?\nSo based on that I can give them specific roles.\nThanks!","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":1082,"Q_Id":53233805,"Users Score":1,"Answer":"You'd have to store them, there is no history of games played available in the API","Q_Score":1,"Tags":"python,discord.py-rewrite","A_Id":53254574,"CreationDate":"2018-11-09T22:01:00.000","Title":"How to get all games that user has played while using Discord with discord.py","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I'm looking for a method to connect client(web browser) to server(without external IP) using p2p.\nAs client-side language i would like to use javascript.\nI was reading about WebRTC Peer-to-peer but i don't know if it only works with two clients(javascript) or if i can use some other language ( PHP, Python, Node.js ).\nI know about signaling, STUN and TURN servers. I have server with external IP address so it won't be a problem.\nMy question is what programming language can i use on the server?","AnswerCount":3,"Available Count":2,"Score":0.0665680765,"is_accepted":false,"ViewCount":1119,"Q_Id":53253718,"Users Score":1,"Answer":"I added to Andrey Suglobov's answer: The client does not receive the WebRTC packets from the server because it doesn't have an external IP. In order to solve this problem, you have to configure it to communicate via the TURN server in the middle.\n[WebRTC server] \u2194 [TURN] \u2194 [NAT] \u2194 [client]\nGenerally, the client uses JavaScript because it's a browser. But WebRTC is a specification that supports P2P on the web. If supporting this specification, it does not matter what programming language you use.\nThank you.","Q_Score":3,"Tags":"javascript,python,node.js,p2p,peer","A_Id":53273494,"CreationDate":"2018-11-11T22:05:00.000","Title":"Peer-to-peer Javascript & something","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I'm looking for a method to connect client(web browser) to server(without external IP) using p2p.\nAs client-side language i would like to use javascript.\nI was reading about WebRTC Peer-to-peer but i don't know if it only works with two clients(javascript) or if i can use some other language ( PHP, Python, Node.js ).\nI know about signaling, STUN and TURN servers. I have server with external IP address so it won't be a problem.\nMy question is what programming language can i use on the server?","AnswerCount":3,"Available Count":2,"Score":1.2,"is_accepted":true,"ViewCount":1119,"Q_Id":53253718,"Users Score":0,"Answer":"Probably found an answer.\nI can use javascript server side in node.js","Q_Score":3,"Tags":"javascript,python,node.js,p2p,peer","A_Id":53262995,"CreationDate":"2018-11-11T22:05:00.000","Title":"Peer-to-peer Javascript & something","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I'm pretty much stuck right now.\nI wrote a parser in python3 using the python-docx library to extract all tables found in an existing .docx and store it in a python datastructure.\nSo far so good. Works as it should.\nNow I have the problem that there are hyperlinks in these tables which I definitely need! Due to the structure (xml underneath) the docx library doesn't catch these. Neither the url nor the display text provided. I found many people having similar concerns about this, but most didn't seem to have 'just that' dilemma.\nI thought about unpacking the .docx and scan the _ref document for the corresponding 'rid' and fill the actual data I have with the links found in the _ref xml.\nEither way it seems seriously weary to do it that way, so I was wondering if there is a more pythonic way to do it or if somebody got good advise how to tackle this problem?","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":438,"Q_Id":53269311,"Users Score":0,"Answer":"You can extract the links by parsing xml of docx file. \nYou can extract all text from the document by using document.element.getiterator()\nIterate all the tags of xml and extract its text. You will get all the missing data which python-docx failed to extract.","Q_Score":0,"Tags":"python,xml,hyperlink,docx","A_Id":57556934,"CreationDate":"2018-11-12T20:04:00.000","Title":"Extracting URL from inside docx tables","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I want a help in robotics stuff..\nMy Question is ... : \nHow to connect my robotic car ( car is having raspberry pi as controller) with my computer via internet .. so the I can control the car from my computer's keybord.. \nPreviously i was using VNC and made a python tkinter script (stored at raspberry pi) and by the help of vnc i control the car but it was not good .. \nMost of the time the when i press the key, function works after sometime and worst thing was that it stores all the commands in a queue or buffer .. \nSo realtime operation was not happenping ( like: if i press forward arrow key for 2 seconds, it evoked moveForward() 20 times which is equal to 2 meters forward move and takes 4 seconds to travel .. BUT after that if i press right arrow key then it evokes moveRight() .. the worst part is it will execute after finishing the moveForward() stored in a queue i.e after 4 seconds .. and not on real time)\nIs there any way to control\/give command to raspberry pi on real time and not in a queue manner via socketing or other thing ?\nnote : i have a static ip address with specific port open and it has to be done over internet.","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":72,"Q_Id":53275066,"Users Score":0,"Answer":"The appearance of your car might mainly relate to the whole system response time. The Raspberry Pi may be not fast enough. If there is no necessary, analog signal may on real time.","Q_Score":0,"Tags":"python,sockets,raspberry-pi,real-time,remote-connection","A_Id":53275756,"CreationDate":"2018-11-13T06:30:00.000","Title":"Remote connection between Raspberry pi and other computer through python over internet on real time","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I'm trying to get browser network logs using selenium to debug request\/responses. Could you please help me to find out a way.\nAnd I'm using selenium 3.14.0 and latest Chrome browser.","AnswerCount":5,"Available Count":1,"Score":0.0399786803,"is_accepted":false,"ViewCount":22344,"Q_Id":53286828,"Users Score":1,"Answer":"Try selenium-wire, I think this is a better way which also provides undetected-chromedriver against bot detection.","Q_Score":8,"Tags":"python,selenium,selenium-webdriver,selenium-chromedriver","A_Id":67264765,"CreationDate":"2018-11-13T17:49:00.000","Title":"How to get browser network logs using python selenium","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I am trying to automate the editing, compiling and reading of results of MQL4 file using python, are there any tools like selenium but targeted for UI?","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":175,"Q_Id":53295473,"Users Score":0,"Answer":"What works for me is two things:\n\nFirst option: clear the strategy tester logs, use comments in the code with specific string format, copy the logs and parse the data from the comments using python.\nSecond option: parse the expert advisor report using Python, from HTML to pandas dataframe for further processing.","Q_Score":0,"Tags":"python,python-3.x,automation,build-automation,mql4","A_Id":62120583,"CreationDate":"2018-11-14T08:02:00.000","Title":"How to edit, compile and read MQL4 files using Python?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I am just starting the process of learning to access APIs using Python. Initially I presumed OAuth User IDs were generated or supplied by the API owner somehow. Would it actually be fabricated by myself, using alphanumerics and symbols? This is just for a personal project, I wouldn't have a need for a systematic creation of IDs. The APIs I am working with are owned by TDAmeritrade.","AnswerCount":1,"Available Count":1,"Score":0.1973753202,"is_accepted":false,"ViewCount":54,"Q_Id":53306198,"Users Score":1,"Answer":"OAuth is has no capabilities for any type of provisioning of users or credentials.\nOAuth 2.0 provides for the Delegation of Authorization\n\nBy the Resource Owner (User)\nto the OAuth Client (Application)\nfor Access to a Resource Server (Website or Api or ???)","Q_Score":1,"Tags":"python,api,oauth","A_Id":53307184,"CreationDate":"2018-11-14T17:56:00.000","Title":"Creation of OAuth User IDs","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I mounted my drive using this : \nfrom google.colab import drive\ndrive.mount('\/content\/drive\/')\nI have a file inside a folder that I want the path of how do I determine the path? \nSay the folder that contains the file is named 'x' inside my drive","AnswerCount":4,"Available Count":2,"Score":1.0,"is_accepted":false,"ViewCount":18640,"Q_Id":53355510,"Users Score":6,"Answer":"The path as parameter for a function will be \/content\/drive\/My Drive\/x\/the_file, so without backslash inside My Drive","Q_Score":7,"Tags":"python,google-colaboratory","A_Id":56513444,"CreationDate":"2018-11-17T20:57:00.000","Title":"How to determine file path in Google colab?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I mounted my drive using this : \nfrom google.colab import drive\ndrive.mount('\/content\/drive\/')\nI have a file inside a folder that I want the path of how do I determine the path? \nSay the folder that contains the file is named 'x' inside my drive","AnswerCount":4,"Available Count":2,"Score":1.2,"is_accepted":true,"ViewCount":18640,"Q_Id":53355510,"Users Score":9,"Answer":"The path will be \/content\/drive\/My\\ Drive\/x\/the_file.","Q_Score":7,"Tags":"python,google-colaboratory","A_Id":53357067,"CreationDate":"2018-11-17T20:57:00.000","Title":"How to determine file path in Google colab?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I am writing a Python script to monitor the status of a certain websites and report if any error via email notification.\nI am able to test the http status of the websites and websphere console urls.\nSince the admin (console) is on DMGR , my code is able to check the status of only DMGR but not the nodes inside the DMGR ,whenever few env is down .. only node goes down. I need a way to monitor the node's status as well.\nAny help would be appreciated.\nThanks","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":71,"Q_Id":53378777,"Users Score":0,"Answer":"You application will be deployed to the JVMs on the nodes. \nIs your application configured to be accessible via the Node hostname [check virtualHost config]?\nif yes - thats the url, you need to check.","Q_Score":0,"Tags":"python-3.x,websphere,monitor,email-notifications","A_Id":53408409,"CreationDate":"2018-11-19T16:20:00.000","Title":"Websphere -check the status of nodes from a browser for automation purpose | Python","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"It just keeps failing when I do sudo apt-get upgrade because it failed to upgrade some package that based on Python3. The error was: undefined symbol: XML_SetHashSalt. I'd been searching around for solutions but there is no such answer on StackOverflow.\nThen at the end, I found an answer on not very related question mention that the library path for libexpat.so.1 pointing to \/usr\/local\/lib\/ may cause the issue. So I try to rename libexpat.so.1 to libexpat.so.1-bk then it works.\nSo I just re-post it here, hope it helps for anyone facing it.","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":1034,"Q_Id":53405066,"Users Score":0,"Answer":"It seems that you have broken your system.\nIf you are using apt, the \/usr\/local\/ should never be used.\nIf you are using \/usr\/local\/, do not use apt to manage your installation.","Q_Score":1,"Tags":"python-3.x","A_Id":53405117,"CreationDate":"2018-11-21T03:58:00.000","Title":"Apt-get error \"undefined symbol: xml_sethashsalt\" when install\/update python3","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":1,"Web Development":0},{"Question":"I am trying to find out ways to secure REST API communication. The following are the items considered.\n\nEnabled token based authentication using JWT\nEnabled HTTPS and restrict requests over HTTP\n\nOur GUI interacts with service layer APIs which are secured as described above. All the requests from GUI contains sensitive information and hence the data has to be secured at every place.\nIn short, our GUI is secured with role based authentication and APIs are secured as above. Still I feel the communication between GUI and services are not secure enough.\n\nIs the payload from GUI secure enough? or should the payload also be encrypted from GUI itself?\n\nIf this is not the right place to ask this question, I am happy to move it to right place. \nPlease advice!\nThank you in advance","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":3632,"Q_Id":53405327,"Users Score":1,"Answer":"What I understood from the post is that your GUI is secured role based and the api is secured using token and https.\nApart from these, as I understand you app is too sensitive, in that case I would do the following to add some extra layer of security.\n\nAdd a two step verfication for the GUI just to ensure that right person is logged in all time.\nEncrypt the data (i.e. the payload) may be using the public\/private key.In this case the server side needs to be changed a bit as it needs to decrypt the request .\nMake sure your token has a lifetime and expires after certain time.\n\nLet me know if you are looking for something else.","Q_Score":2,"Tags":"python,angularjs,rest,security,encryption","A_Id":53405466,"CreationDate":"2018-11-21T04:35:00.000","Title":"How to secure REST API payload while in transit?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"We are developing an application on Google Cloud that makes use of Cloud Functions in Python. We're looking at developing a generic helper library that many of our functions will import.\nThe problem with this is if the helper library is changed in any way, all our functions will need to be redeployed.\nI'm trying to find a way to host (for want of a better word) our helper library (for example in Google Cloud Storage), and somehow import it into the main.py files, such that any changes to the helper library can be made without having to redeploy the functions. Is this possible at all?","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":104,"Q_Id":53448833,"Users Score":3,"Answer":"This is not supported with the provided tools. You can only invoke code that was deployed with the function. There is no \"dynamic\" loading of code over the internet.\nAlso, in my opinion, this is a pretty bad idea, because your functions could break in a very profound way if there's a problem during loading of the remote code, or you accidentally (or someone maliciously) push something wrong. You're going to be better off getting all your code and libraries together at once, test it all at once, and deploy it all at once.\nYou're free to try to implement something yourself, but I strongly advise against it.","Q_Score":1,"Tags":"python,python-3.x,google-cloud-platform,google-cloud-functions","A_Id":53449175,"CreationDate":"2018-11-23T14:52:00.000","Title":"Is there a way to import a python helper library from a deployed Google Cloud Function, outside the function?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":1,"System Administration and DevOps":0,"Web Development":0},{"Question":"Situation\nI wrote a simple program in python. It's a simple socket chatting program. In my program, the client just connect to an address (ip, port) and send a message, while at this time the server is ready and receives the message and prints it. I can assure the program is correct, since I tried on my computer.\nI have a VM instance on Google Cloud Platform, which I can operate through ssh, a simple way provided by google cloud. I can make sure the server is working.\nProblem\nI start a simple tcp server, python program on my google cloud server. Then I start my client program on my computer. But I get this error:\n\nConnectionRefusedError: [WinError 10061] No connection could be made because the target machine actively refused it\n\nor equivalently in Chinese:\n\nConnectionRefusedError: [WinError 10061] \u7531\u4e8e\u76ee\u6807\u8ba1\u7b97\u673a\u79ef\u6781\u62d2\u7edd\uff0c\u65e0\u6cd5\u8fde\u63a5\u3002\n\nHow do I solve this problem and connect to my google cloud server?\nI guess maybe the firewall refused my computer's connection, but have no idea how to solve it.","AnswerCount":2,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":3164,"Q_Id":53454732,"Users Score":4,"Answer":"This error means that your program is not listening on 0.0.0.0 port XYZ.\nCheck to see if your program is instead listening on localhost. If it is change to 0.0.0.0 which means all available networks. localhost means do not listen on any network interfaces and only accept connections from inside the computer.\nThen double check the port number.\nTo see if you have something listening run this command (Linux): netstat -at\nLook for a line with your port XYZ.\nWhen you start your program, make sure that it does not error when creating the listener. If you are trying to use a port number below 1024, you will need to lauch the program with root privileges.","Q_Score":1,"Tags":"python,server,google-cloud-platform,google-compute-engine,firewall","A_Id":53455102,"CreationDate":"2018-11-24T02:40:00.000","Title":"Can't connect to my google cloud VM instance through tcp using python","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":1,"Web Development":0},{"Question":"I have connected zapier to a webhook I am listening too, which sends a JSON file into my s3 bucket. \nI have some python code that I want to execute when a new file is uploaded into the bucket, in real time over the file. \nWhat is the best way to 'listen' for the upload of this file into the s3 bucket?","AnswerCount":3,"Available Count":1,"Score":0.1325487884,"is_accepted":false,"ViewCount":2916,"Q_Id":53471129,"Users Score":2,"Answer":"David here, from the Zapier Platform team. \nSeems like you've already found your answer, which is great. I just wanted to plug Zapier as an option (since you had mentioned you're already using it). Our S3 integration has a \"new file in bucket\" trigger, which you can combine with any other step (such as a Python Code step). Additionally, you could skip the middleman and structure your zap as:\n\nsome trigger\nAdd file to S3\nRun Python\n\nAnd not need to worry about webhooks at all.\n\u200bLet me know if you've got any other questions!","Q_Score":4,"Tags":"python,amazon-s3,webhooks,zapier","A_Id":53488253,"CreationDate":"2018-11-25T19:34:00.000","Title":"Best way to 'listen' to s3 bucket for new file?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I have test that executes for 6 hours. After 2 hours, my driver slows down due to nature of Chrome browser. Solution is to close browser and restart it. I found out that doing driver.quit() helps in performance due to some internal memory that is being used which causes test to become slow. I am wondering is there option to use driver.quit() without closing drivers because i need cookies that were generated in that session as well as not killing Python script that is being ran at that moment.","AnswerCount":1,"Available Count":1,"Score":0.1973753202,"is_accepted":false,"ViewCount":1274,"Q_Id":53544235,"Users Score":1,"Answer":"The purpose of driver.quit() is to close all browser windows and terminate WebDriver session. So no, you cannot use driver.quit() without closing drivers - that's what it does.\nIn my opinion, you should look at why at all you have this issue:\n\nIs there really a reason to run 6 hours of testing within the same session? Of course there could be special circumstances, but good practice is to cut entire test collection into independent sets, where each set can run on its own, on \"clean\" environment (i.e. new browser session). Not only this will prevent the problem you are facing, but also improves reliability of the tests (i.e. domino effect when one tests messes all future test executions), ability to debug (imagine you have a problem with a test that runs on hour #3, and the problem is not reproducible when you run it alone, or you can't run it alone), and flexibility of the executions.\nWhy does the browser need to be restarted after 2 hours? No, it's not a \"nature of Chrome\". It's a bug somewhere - memory leak, or something else. It might be worth investigating what it is about. Because you can stop tests after 2 hours, but are you going to tell your users to not use your application for more than 2 hours? Even if it's a bug in Selenium driver, it might be worth reporting it to selenium developers for your, and everyone else's benefit.","Q_Score":0,"Tags":"python,selenium,selenium-webdriver,selenium-chromedriver","A_Id":53544604,"CreationDate":"2018-11-29T17:10:00.000","Title":"Selenium webdriver closing without session","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I have a script that I use to send HTTP requests and Websocket data and I had to use an external program as I couldn't find a Websocket library that supports SOCKS5 proxies. \nSo, I've found about Proxifier and tried running my Python script with a rule that I have written in Proxifier but it didn't yield and good result. \nOther programs seemed to work fine, chrome.exe managed to go through the proxy, and I have no idea why my Python script won't go...","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":847,"Q_Id":53570121,"Users Score":0,"Answer":"Same issues with proxifier.\nSeems reason in portable version of application.\nTried standard version and everything works fine.\nHope this helps.","Q_Score":0,"Tags":"python,python-3.x,proxy","A_Id":55796085,"CreationDate":"2018-12-01T11:02:00.000","Title":"Running a Python script through Proxifier","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"i can connect locally to my mongodb server with the address 0.0.0.0\/0. However, when I deploy my code to the cloud I get the error deploy to google cloud function.\ngoogle cloud function with python 3.7 (beta)\natlas mongo db\npython lib:\n-pymongo\n-dnspython\nError: function crashed. Details:\nAll nameservers failed to answer the query _mongodb._tcp.**-***.gcp.mongodb.net. IN SRV: Server ***.***.***.*** UDP port 53 answered SERVFAIL\nTraceback (most recent call last): File \"\/env\/local\/lib\/python3.7\/site-packages\/pymongo\/uri_parser.py\", line 287, in _get_dns_srv_hosts results = resolver.query('_mongodb._tcp.' + hostname, 'SRV') File \"\/env\/local\/lib\/python3.7\/site-packages\/dns\/resolver.py\", line 1132, in query raise_on_no_answer, source_port) File \"\/env\/local\/lib\/python3.7\/site-packages\/dns\/resolver.py\", line 947, in query raise NoNameservers(request=request, errors=errors) dns.resolver.NoNameservers: All nameservers failed to answer the query _mongodb._tcp.**mymongodb**-r091o.gcp.mongodb.net. IN SRV: Server ***.***.***.*** UDP port 53","AnswerCount":2,"Available Count":1,"Score":0.3799489623,"is_accepted":false,"ViewCount":3909,"Q_Id":53576199,"Users Score":4,"Answer":"finally after stuck 2 day, goblok banget semaleman\njust change connection \nfrom \n\nSRV connection string (3.6+ driver)\n\nto \n\nStandard connection string (3.4+ driver)\n\nmongodb:\/\/<USERNAME>:<PASSWORD>@<DATABASE>-shard-00-00-r091o.gcp.mongodb.net:27017,<COLLECTION>-shard-00-01-r091o.gcp.mongodb.net:27017,<COLLECTION>-shard-00-02-r091o.gcp.mongodb.net:27017\/test?ssl=true&replicaSet=<COLLECTION>-shard-0&authSource=admin&retryWrites=true\nor you can see your connection string in atlas mongodb.\nidk why can`t connect with srv connection string in google cloud functions, maybe not suppot now, or just misconfiguration.","Q_Score":3,"Tags":"python,mongodb,google-cloud-functions,mongodb-atlas","A_Id":53582632,"CreationDate":"2018-12-02T00:09:00.000","Title":"All nameservers failed to answer UDP port 53 Google cloud functions python 3.7 atlas mongodb","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":1,"Web Development":0},{"Question":"So instead of having data-item-url=\"https:\/\/miglopes.pythonanywhere.com\/ra%C3%A7%C3%A3o-de-c%C3%A3o-purina-junior-10kg\/\"\nit keeps on appearing \ndata-item-url=\"http:\/\/localhost\/ra%C3%A7%C3%A3o-de-c%C3%A3o-purina-junior-10kg\/\"\nhow do i remove the localhost so my snipcart can work on checkout?","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":88,"Q_Id":53587049,"Users Score":0,"Answer":"Without more details of where this tag is coming from it's hard to know for sure... but most likely you need to update your site's hostname in the Wagtail admin, under Settings -> Sites.","Q_Score":0,"Tags":"localhost,wagtail,pythonanywhere,snipcart","A_Id":53600830,"CreationDate":"2018-12-03T03:23:00.000","Title":"data-item-url is on localhost instead of pythonanywhere (wagtail + snipcart project)","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I have a Scrapy crawler and I want to rotate the IP so my application will not be blocked. I am setting IP in scrapy using request.meta['proxy'] = 'http:\/\/51.161.82.60:80' but this is a VM's IP. My question is can VM or Machine's IP be used for scrapy or I need a proxy server?\nCurrently I am doing this. This does not throw any error but when I get response from http:\/\/checkip.dyndns.org it is my own IP not updated IP which I set in meta. That is why I want to know if I do need proxy server.","AnswerCount":2,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":64,"Q_Id":53645854,"Users Score":0,"Answer":"Definitely you need a proxy server. meta data is only a field in the http request. the server side still knows the public ip that really connecting from the tcp connection layer.","Q_Score":0,"Tags":"python,scrapy,web-crawler","A_Id":53647528,"CreationDate":"2018-12-06T06:38:00.000","Title":"Can VM \/ Machine IP be used instead of Proxy Server for Scrapy","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"Python's built-in HTTP clients don't have many features, so even the Python docs recommend using requests. But there's also urllib3, which requests, itself uses, and they share some core developers, making me think they're more complementary than competing.\nWhen would I use urllib3 instead of requests? What features does requests add on top of urllib3?","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":48,"Q_Id":53662099,"Users Score":0,"Answer":"Requests was conducted on the basis of urlib3 encapsulation.\nSince Requests are already an encapsulated library, many functions can be simplified. For example: timeout setting, proxy setting, file upload, get cookies, etc.\nHowever,requests can only be used directly and cannot be invoked asynchronously, so requests are slow.\nSo, if you're writing small spiders that don't require much speed, consider using requests.","Q_Score":2,"Tags":"python,python-requests,urllib3","A_Id":53662465,"CreationDate":"2018-12-07T01:38:00.000","Title":"Choosing between Python HTTP clients urllib3 and requests","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I have a weighted Graph using networkx and the topology is highly meshed. I would like to extract a number of paths between two nodes with distance minimization.\nTo clarify, the dijkstra_path function finds the weighted shortest path between two nodes, I would like to get that as well as the second and third best option of shortest weighted paths between two nodes.\nI tried using all_simple_paths and then ordering the paths in distance minimization order but it is extremely time consuming when the network is meshed with 500 nodes or so.\nAny thoughts on the matter? Thank you for your help!","AnswerCount":1,"Available Count":1,"Score":0.1973753202,"is_accepted":false,"ViewCount":270,"Q_Id":53674393,"Users Score":1,"Answer":"Try networkx's shortest_simple_paths.","Q_Score":0,"Tags":"python-3.x,networkx,dijkstra","A_Id":53677838,"CreationDate":"2018-12-07T17:35:00.000","Title":"Descending order of shortest paths in networkx","Data Science and Machine Learning":1,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"The website is running using node server. I want to create a python flask rest api to connect to pop up chat window of this website.My python flask is running in 8085 port and node server is running in 8082 port.\nin python flask app.py \n\n@app.route('\/') \n def hello_world():\n return render_template('popupchat.html') \n@app.route('\/chat',methods=[\"POST\"]) def chat():\n\nthis popupchat.html is pointing to website pupupchat window. and there is one bind.js script having $.post(\"\/chat\" , if i want to connect this pop up chat window running in node server to python flask server , how i will connect .\ni appreciate your suggestions","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":949,"Q_Id":53711272,"Users Score":0,"Answer":"If you are just debugging try to adjust the port on the side of the chat window. You can tell the chat window to reach a certain port by altering the url you are trying to reach. Example:\nhttp:\/\/your_ip:your_port\/your_route or in your case http:\/\/your_ip:8085\/your_route\nIf you are already deployed please talk to your admin since this might be depending on your server.","Q_Score":1,"Tags":"javascript,python,html,flask-restful","A_Id":53712290,"CreationDate":"2018-12-10T18:07:00.000","Title":"python flask running in one port connected to UI running in node server different port","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"HIHello everyone\nI have a Raspberry Pi that contain some data,\non another side, I have a server with a server, also there is a webpage connecting with the same server.\nwhat do I need? \n1- the Raspberry Pi must send its data to a server \n2-if the user set some data to the database by webpage the Raspberry Pi must get this data \n3-Raspberry Pi must listen to the database to know if there is an update or not.\nwhat the best way to do these points, is there any protocol for IoT can do these? \nI need any hint :) \nthank u for all.","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":483,"Q_Id":53715817,"Users Score":0,"Answer":"Try connecting your RPi with your server through any means of local area connection, it could be a connection through a wifi network or using a lan cable or even through serial ports. If successful in this attempt then you can access disk drive folders inside the server. In the webpage running on server ,make it listen for certain values and write the status to a text file or an database. Then make RPi to continuously monitor those file for data updation and make it work according to it.","Q_Score":0,"Tags":"php,python,database,raspberry-pi,iot","A_Id":53722341,"CreationDate":"2018-12-11T00:40:00.000","Title":"how set\/get data from Raspberry Pi to server","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"My goal is to add 10M contacts to telegram.\nHow can we add contacts to telegram using telegram API? \nI have tried using telethon in which I batched 500 contacts in one request. However, telegram responded all such requests with all contacts in retry_contacts and none were imported.\nI have also found out a solution to convert the txt file of\n10M contacts to csv file and import them using an android app.\nBut this takes approx 10 mins for 10k contacts. So this won't be\na good idea for adding 10M contacts.\nAny other method for having this done is also welcomed.","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":9550,"Q_Id":53730525,"Users Score":9,"Answer":"This is not possible. Telegram has deliberately set limits on the number of contacts you can add. Initially you can add about 5000 contacts and after that you can add about 100 more every day. This is because of security not decreasing their API load. If you could add 10M numbers, you could easily map @usernames to numbers which is against Telegram privacy policy.\nIn my experience, the best practical option is to add an array of 10 numbers each time using telethon's ImportContactsRequest, until you get locked. Then try 24 hours later again until you get locked again, and so on. This is the fastest solution and due to Telegram restrictions, if you only have 1 SIM card, it takes around 274 years to add 10M contacts.","Q_Score":6,"Tags":"python,automation,contacts,telegram,telethon","A_Id":53736422,"CreationDate":"2018-12-11T18:46:00.000","Title":"How to add Millions of contacts to telegram?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I am creating a REST API in python and currently, there is a route to a list of the resource instructors:\nGET \/instructors\nBut, I also have another client, used as a CRM for users who have an admin-role.\nFor the client, I want to display a list of instructors but with more fields and different properties.\nMy first thought is also to have the route:\nGET \/instructors \nThis obviously conflicts with the route above. \nWhat is the best name for this route?\nGET \/instructors\/admin\nor\nGET \/admin\/instructors\nor \nGET \/instructors?admin=True\nI am not sure how to approach this.\nThanks in advance.","AnswerCount":2,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":938,"Q_Id":53787342,"Users Score":0,"Answer":"I am glad that we are finally talking about the naming conventions this totally depends on personal preference and use cases and how your project has been designed so i will keep here my views ..\nLike you said all of the above seem to be good but how i would do is\n\nGET\/Instructors\/all \nGET\/Instructors\/admin \nGET\/Instructors\/any other special role\n\nyou may use queries when something specific has to be done with these roles again as in \nGET\/Instructors\/all?credibility=PHD \nsomething like the above its never a good idea to show every thing on just parent calls like GET\/Instructor as you said firstly it creates confusion and makes the readability of your endpoints difficult with time when the complexity of your application increases.","Q_Score":0,"Tags":"python,rest,api","A_Id":53787444,"CreationDate":"2018-12-14T21:50:00.000","Title":"Rest api naming conventions for consumer and admin","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I'm trying to use zeep library to provide soap call in python code and its okay it works when I try to run with python. Then, I'm trying to use jython to run this code (i need jython because next step will be on the server that uses jython to compile) and when I try to install lxml for jython it gives me this error:\nerror:Compiling extensions is not supported on Jython\nWhen I search for this situation, I found that jython doesn't support c based libraries. \nSo, there is a solution with jython-jni bridge but I couldn't understand how to be.\nIs there another solution? Or can you give me an obvious example?","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":323,"Q_Id":53815005,"Users Score":0,"Answer":"I couldn't achieve to implement jni but i created a new layer between jython and server.I mean, i made a REST call from jython compiler and this call listens my server for soap call and it worked.","Q_Score":0,"Tags":"java,python,lxml,jython","A_Id":53870455,"CreationDate":"2018-12-17T12:12:00.000","Title":"Jython can not use lxml library","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I am working on socket programming in python. I am a bit confused with the concept of s.listen(5) and multithreading.\nAs I know, s.listen(5) is used so that the server can listen upto 5 clients.\nAnd multithreading is also used so that server can get connected to many clients.\nPlease explain me in which condition we do use multithreading?\nThanks in advance","AnswerCount":2,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":1236,"Q_Id":53880028,"Users Score":0,"Answer":"As I know, s.listen(5) is used so that the server can listen upto 5 clients.\n\nNo. s.listen(5) declares a backlog of size 5. Than means that the listening socket will let 5 connection requests in pending stated before they are accepted. Each time a connection request is accepted it is no longer in the pending backlog. So there is no limit (other than the server resources) to the number of accepted connections.\nA common use of multithreading is to start a new thread after a connection has been accepted to process that connection. An alternative is to use select on a single thread to process all the connections in the same thread. It used to be the rule before multithreading became common, but it can lead to more complex programs","Q_Score":0,"Tags":"python-3.x,multithreading,sockets","A_Id":53887480,"CreationDate":"2018-12-21T06:15:00.000","Title":"server.listen(5) vs multithreading in socket programming","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":1,"System Administration and DevOps":0,"Web Development":0},{"Question":"I am working on a tool for my company created to get data from our Facebook publications. It has not been working for a while, so I have to get all the historical data from June to November 2018. \nMy two scripts (one that get title and type of publication, and the other that get the number of link clicks) are working well to get data from last pushes, but when I try to add a date range in my Graph API request, I have some issues:\n\nthe regular query is [page_id]\/posts?fields=id,created_time,link,type,name\nthe query for historical data is [page_id]\/posts?fields=id,created_time,link,type,name,since=1529280000&until=1529712000, as the API is supposed to work with unixtime\nI get perfect results for regular use, but the results for historical data only shows video publications in Graph API Explorer, with a debug message saying:\n\n\nThe since field does not exist on the PagePost object.\n\nSame for \"until\" field when not using \"since\". I tried to replace \"posts\/\" with \"feed\/\" but it returned the exact same result...\nDo you have any idea of how to get all the publications from a Page I own on a certain date range?","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":356,"Q_Id":53883849,"Users Score":0,"Answer":"So it seems that it is not possible to request this kind of data unfortunately, third party services must be used...","Q_Score":0,"Tags":"python,facebook,facebook-graph-api,unix-timestamp","A_Id":53889179,"CreationDate":"2018-12-21T11:15:00.000","Title":"Date Range for Facebook Graph API request on posts level","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"import random\nimport asyncio\nimport json\nimport aiohttp\nimport sys\nimport urllib\nfrom lxml.html.soupparser import parse\nfrom aiohttp import ClientSession\nfrom threading import Thread\n\ndef ttest():\n async def fetch(url, session):\n headers = {\n 'Host': 'example.com'\n }\n cookies2 = {\n 'test': 'test'\n }\n\n data = '{\"test\":\"test\"}'\n async with session.post(url, data=data, headers=headers, cookies=cookies2) as response:\n return await response.read()\n async def bound_fetch(sem, url, session):\n async with sem:\n html = await fetch(url, session)\n print(html)\n\n\n async def run(r):\n url = \"https:\/\/test.com\"\n tasks = []\n sem = asyncio.Semaphore(1000)\n async with aiohttp.ClientSession() as session:\n for i in range(r):\n task = asyncio.ensure_future(bound_fetch(sem, url, session))\n tasks.append(task)\n responses = asyncio.gather(*tasks)\n await responses\n\n number = 1\n loop = asyncio.get_event_loop()\n future = asyncio.ensure_future(run(number))\n loop.run_until_complete(future)\n\nttest()\n\nThis is the error: TypeError: _request() got an unexpected keyword argument 'cookies'\nI want use cookies like you see in the code, but i can not, can anyone help me?","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":4119,"Q_Id":53892470,"Users Score":1,"Answer":"The feature was added on aiohttp GitHub master but not released yet.\nPlease either install aiohttp from GitHub or wait for a while for aiohttp 3.5 release.\nI hope to publish it in a few days.","Q_Score":3,"Tags":"python-3.x,python-asyncio,aiohttp","A_Id":53894565,"CreationDate":"2018-12-22T01:54:00.000","Title":"TypeError: _request() got an unexpected keyword argument 'cookies' (aiohttp)","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"As a legacy from the previous version of our system, I have around 1 TB of old video files on AWS S3 bucket. Now we decided to migrate to AWS Media Services and all those files should be moved to MediaStore for the access unification.\nQ: Is there any way to move the data programmatically from S3 to MediaStore directly?\nAfter reading AWS API docs for these services, the best solution I've found is to run a custom Python script on an intermediate EC2 instance and pass the data through it.\nAlso, I have an assumption, based on pricing, data organization and some pieces in docs, that MediaStore built on top of S3. That's why I hope to find a more native way to move the data between them.","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":376,"Q_Id":53897584,"Users Score":0,"Answer":"I've clarified this with AWS support. There is no way to transfer files directly, although, it's a popular question and, probably, will be implemented.\nNow I'm doing this with an intermediate EC2 server, a speed of internal AWS connections between this, S3 and MediaStore is quite good. So I would recommend this way, at least, for now.","Q_Score":0,"Tags":"python,amazon-web-services,amazon-s3,amazon-ec2,aws-mediastore","A_Id":53904676,"CreationDate":"2018-12-22T16:52:00.000","Title":"Move objects from AWS S3 to MediaStore","Data Science and Machine Learning":0,"Database and SQL":1,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"When I am running a Python micro-service in a dockerized or kubernetes container it works just fine. But with Istio service mesh, it is not working.\nI have added ServiceEntry for two of my outbound external http apis. It seems I can access the url content form inside the container using curl command which is inside service mesh. So, I think the service entries are fine and working. \nBut when I try from the micro-service which uses xml.sax parser in Python, it gives me the upstream connect error or disconnect\/reset before headers though the same application works fine without Istio.\nI think it is something related to Istio or Envoy or Python. \nUpdate: I did inject the Istio-proxy side-car. I have also added ServiceEntry for external MySQL database and mysql is connected from the micro-service.","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":1036,"Q_Id":53898206,"Users Score":2,"Answer":"I have found the reason for this not working. My Python service is using xml.sax parser library to parse xml form the internet, which is using the legacy urllib package which initiate http\/1.0 request. \nEnvoy doesn't support http\/1.0 protocol version. Hence, it is not working. I made the workaround by setting global.proxy.includeIPRanges=\"10.x.0.1\/16\" for Istio using helm. This actually bypass the entire envoy proxy for all outgoing connections outside the given ip ranges.\nBut I would prefer not to globally bypass Istio.","Q_Score":0,"Tags":"python,kubernetes,istio,envoyproxy","A_Id":53969796,"CreationDate":"2018-12-22T18:17:00.000","Title":"Istio egress gives \"upstream connect error or disconnect\/reset before headers\" errors from python micro-service","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I\u2019m extracting the SSL certificate from a website using the socket + ssl library in python. My understanding that it connects using the preferred method used by the server. \nUsing this method I\u2019m able to identify what version of SSL is used to connect, but I also need to identify whether the website supports SSL v3, in the case when the default connection is TLS. \nIs there a way to identify this information without manually testing multiple SSL connections?","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":84,"Q_Id":53900577,"Users Score":0,"Answer":"I don't think sites advertise what they support. Rather, it's negotiated between client and server.\nYou could use the excellent server tester at www.ssllabs.com. It will try lots of configurations and report what the server in question supports. (Hopefully the site doesn't support SSL v3!)","Q_Score":0,"Tags":"python,ssl,openssl,ssl-certificate","A_Id":53900613,"CreationDate":"2018-12-23T01:40:00.000","Title":"Testing SSL v3 Support in Python","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I need to extract the text just after strong tag from html page given below? how can i do it using beautiful soup. It is causing me problem as it doesn't have any class or id so only way to select this tag is using text.\n{strong}Name:{\/strong} Sam smith{br}\nRequired result\nSam smith","AnswerCount":1,"Available Count":1,"Score":-0.1973753202,"is_accepted":false,"ViewCount":77,"Q_Id":53914377,"Users Score":-1,"Answer":"Thanks for all your answers but i was able to do this by following:\nb_el = soup.find('strong',text='Name:')\nprint b_el.next_sibling\nThis works fine for me. This prints just next sibling how can i print next 2 sibling is there anyway ?","Q_Score":0,"Tags":"python,web-scraping,beautifulsoup","A_Id":53914744,"CreationDate":"2018-12-24T13:58:00.000","Title":"extracting text just after a particular tag using beautifulsoup?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I'm creating documents from web scraping in python and uploading them to Firestore.\nTo do so I'm adding them to a dictionary and uploading them from a for loop in python, one by one(ideally would be better to upload the collection at once, but that doesn't seem an option). I want to use batches, however they have the 500 limit per batch and I need to do more than 100,000 operations. The operations are merely set() operations and a couple of update()\nIs there a function to know the current size of the batch so I can reinitialize it?\nWhat is the best way to use batches for more than 500 operations in python?","AnswerCount":3,"Available Count":1,"Score":0.1973753202,"is_accepted":false,"ViewCount":1725,"Q_Id":53921109,"Users Score":3,"Answer":"The maximum number of operations in a Batch is 500. If you need more operations, you'll need multiple batches.\nThere is no API to determine the current number of operations in a Batch. If you need that, you will have to track it yourself.","Q_Score":3,"Tags":"python,firebase,google-cloud-firestore","A_Id":53921780,"CreationDate":"2018-12-25T09:39:00.000","Title":"How to batch more than 500 operations in python for firestore?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"if someone find my telegram chat ID (instead of my phone number or username), what can do with that?\nis it dangerous?!\nis it a big deal that someone can find my ID? should I worry about it?","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":3297,"Q_Id":53924548,"Users Score":2,"Answer":"If someone with a (user)bot wants to send you a message, they can do that via the userID only. But only if they have \"seen\" you. Seeing you is considering their client to receive a message from you, be it in a private message or group.\nBots can only send you a message if you have started it in a private chat, otherwise they can only send you a message in a group they share with you.\nSo there is no real risk of people knowing your ID.","Q_Score":2,"Tags":"telegram,telegram-bot,python-telegram-bot,telepot","A_Id":54479855,"CreationDate":"2018-12-25T18:02:00.000","Title":"what can be done by a telegram chat ID?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I am using vpython library in spyder. After importing the library when I call simple function like print('x') or carry out any assignment operation and execute the program, immediately a browser tab named localhost and port address opens up and I get the output in console {if I used print function}. \nI would like to know if there is any option to prevent the tab from opening and is it possible to make the tab open only when it is required.\nPS : I am using windows 10, chrome as browser, python 3.5 and spyder 3.1.4.","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":238,"Q_Id":53943901,"Users Score":0,"Answer":"There is work in progress to prevent the opening of a browser tab when there are no 3D objects or graph to display. I don't know when this will be released.","Q_Score":1,"Tags":"python,spyder,vpython","A_Id":53982995,"CreationDate":"2018-12-27T10:57:00.000","Title":"Vpython using Spyder : how to prevent browser tab from opening?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":1,"System Administration and DevOps":0,"Web Development":0},{"Question":"I have created a chatbot using AWS lex and Lambda. Bot works as expected. I have store the slot data into sessionAttributes. The issue I am facing is when I communicate with bot from my website and if I open another tab of my site, it does't show the previous chat which happened in older tab(here both tabs are open).\nOn every new tab chat starts from start.\nRequirement is to continue from where it was left in previous tab.\nAm I missing any flow here ? I have gone though AWS doc but didn't get any clear picture to do the same. Any example of the same will help better.","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":116,"Q_Id":53946681,"Users Score":0,"Answer":"You need to store the chat into some database of your own. On page load, you need to fetch the chat of current session or current user (based on your requirement).\n That way even if the user refreshes the page or open up a new tab, he will be able to see the chat he already had with the chatbot.","Q_Score":0,"Tags":"python,amazon-web-services,aws-lambda,chatbot,amazon-lex","A_Id":53955733,"CreationDate":"2018-12-27T14:32:00.000","Title":"maintain aws lex chat communication on all pages of website","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I'm attempting to automate a website interaction that requires that a USB key be inserted, an alert box then asks you to verify, and after hitting okay it opens a local program that requires a pin be input to activate the key. I'm looking for a way to have the bot enter the pin into the external program. Is there a library that can automate this for me?","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":120,"Q_Id":53948970,"Users Score":0,"Answer":"I ended up using a keyboard input library to take advantage of the fact that windows changes window focus to the security window that opens, so now it emulates keyboard input to press enter, enter the pin, and press enter again.","Q_Score":0,"Tags":"python,selenium,internet-explorer","A_Id":54115153,"CreationDate":"2018-12-27T17:52:00.000","Title":"SSL USB Key Authentication in Selenium","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"so, I've tried to send a javascript XMLHttpRequest on a personally written http server that uses python's core functionality (i.e. socket and regex). took me a while but I finally had it going. so, I tested it out and after debugging the regex for parsing http POST and GET requests, I tested it out through the python console and it worked fine.\nin short, the http server receives a GET loopback request with personal identification, and reads whatever text was sent to it as data. \na tts loopback server.\nI wanted to do it because asides from selenium that honestly seemed like the only way for me to read random text from the screen using javascript (I could create a browser extension that works alongside it). I already created something for parsing html, so that's not the problem. later I wanted to extend the application and create a GUI using java for generally tts'ing files so I could listen to them while programming.\nthe problem was that although the socket was bound to port 80 on the loopback interface (127.0.0.1), when I sent an XMLHttpRequest to localhost, the server was not responding. I checked for incoming connections and there were none. from the terminal it worked fine though.","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":52,"Q_Id":53971452,"Users Score":0,"Answer":"if anyone else is wondering, no, it's not possible. unless it bypasses CORS restriction. (sadly). If anyone wants to do something similar, he has to either bypass CORS restrictions OR if you're building with python you can just use selenium and create a \"custom\" browser extension.","Q_Score":0,"Tags":"javascript,python,http,browser","A_Id":53972892,"CreationDate":"2018-12-29T16:49:00.000","Title":"javascript on browser send xmlhttprequest onto loopback server","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I am trying to use Scrapy shell to try and figure out the selectors for zone-h.org. I run scrapy shell 'webpage' afterwards I tried to view the content to be sure that it is downloaded. But all I can see is a dash icon (-). It doesn't download the page. I tried to enter the website to check if my connection to the website is somehow blocked, but it was reachable. I tried setting user agent to something more generic like chrome but no luck there either. The website is blocking me somehow but I don't know how can I bypass it. I digged through the the website if they block crawling and it doesn't say it is forbidden to crawl it. Can anyone help out?","AnswerCount":2,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":621,"Q_Id":53988585,"Users Score":0,"Answer":"Can you use scrapy shell \"webpage\" on another webpage that you know works\/doesn't block scraping?\nHave you tried using the view(response) command to open up what scrapy sees in a web browser?\nWhen you go to the webpage using a normal browser, are you redirected to another, final homepage?\n- if so, try using the final homepage's URL in your scrapy shell command\nDo you have firewalls that could interfere with a Python\/commandline app from connecting to the internet?","Q_Score":1,"Tags":"python,scrapy,web-crawler","A_Id":53989874,"CreationDate":"2018-12-31T14:33:00.000","Title":"Scrapy shell doesn't crawl web page","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I am currently scraping a website for work to be able to sort the data locally, however when I do this the code seems to be incomplete, and I feel may be changing whilst I scroll on the website to add more content. Can this happen ? And if so, how can I ensure I am able to scrape the whole website for processing?\nI only currently know some python and html for web scraping, looking into what other elements may be affecting this issue (javascript or ReactJS etc). \nI am expecting to get a list of 50 names when scraping the website, but it only returns 13. I have downloaded the whole HTML file to go through it and none of the other names seem to exist in the file, i.e. why I think the file may be changing dynamically","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":58,"Q_Id":54008539,"Users Score":0,"Answer":"Yes, the content of the HTML can be dynamic, and Javascript loading should be the most essential . For Python, scrapy+splash maybe a good choice to get started. \nDepending on how the data is handled, you can have different methods to handle dyamic content HTML","Q_Score":0,"Tags":"python,web-scraping","A_Id":54008662,"CreationDate":"2019-01-02T14:59:00.000","Title":"The HTML code I scrape seems to be incomplete in comparison to the full website. Could the HTML be changing dynamically?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I have a Flask app that uses selenium to get data from a website. I have spent 10+ hours trying to get heroku to work with it, but no success. My main problem is selenium. with heroku, there is a \"buildpack\" that you use to get selenium working with it, but with all the other hosting services, I have found no information. I just would like to know how to get selenium to work with any other recommended service than heroku. Thank you.","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":777,"Q_Id":54058237,"Users Score":0,"Answer":"You need hosting service that able to install Chrome, chromedriver and other dependencies. Find for Virtual Private hosting (VPS), or Dedicated Server or Cloud Hosting but not Shared hosting.","Q_Score":1,"Tags":"python,selenium,heroku,hosting","A_Id":54059218,"CreationDate":"2019-01-06T02:49:00.000","Title":"How does selenium work with hosting services?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I am doing a sentiment analysis project about local people's attitudes toward the transportation service in Hong Kong. I used the Twitter API to collect the tweets. However, since my research target is the local people in Hong Kong, tweets posted from, for instance, travelers should be removed. Could anyone give me some hints about how to extract tweets posted from local people given a large volume of Twitter data? My idea now is to construct a dictionary which contains traveling-related words and use these words to filter the tweets. But it may seem not to work\nAny hints and insights are welcomed! Thank you!","AnswerCount":1,"Available Count":1,"Score":0.3799489623,"is_accepted":false,"ViewCount":42,"Q_Id":54058996,"Users Score":2,"Answer":"There are three main ways you can do this.\n\nLanguage. If the user is Tweeting in Cantonese - or another local language - there is less chance they are a traveller compared to, say, Russian.\nUser location. If a user has a location present in their profile, you can see if it is within Hong Kong.\nUser timezone. If the user's timezone is the same as HK's timezone, they may be a local.\n\nAll of this is very fuzzy.","Q_Score":0,"Tags":"python,twitter,web-crawler,sentiment-analysis,social-media","A_Id":54060976,"CreationDate":"2019-01-06T05:49:00.000","Title":"How to extract tweets posted only from local people?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I am using select() to check if there is data in a socket with a timeout of 5 seconds. I'm willing to know if calling select() block only the thread or the whole program, if there is no data in the socket, select() will block for 5 seconds only the thread so the rest of the program can run freely or block everything until timeout?","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":49,"Q_Id":54059866,"Users Score":0,"Answer":"Yes, select will only block the current thread. The other threads can work freely.","Q_Score":0,"Tags":"python,select,block","A_Id":54059877,"CreationDate":"2019-01-06T08:30:00.000","Title":"Does calling Select() block only the thread or the whole program?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":1,"System Administration and DevOps":0,"Web Development":0},{"Question":"I am using selenium to scrape information from a website.\nThe page is behind a login so I can't provide an example - but basically I need to gather data on approx. 800 fields on the one page.\nCurrently it boils down to me navigating to the correct page and then running \nfor i in driver.find_elements_by_xpath('\/\/*[@id]'):\n some_list.append(i.get_attribute('textContent'))\nMy question is;\n\nDoes using get_attribute place any impact on the responding server?\n\nOr is the full page 'cached' and then I am simply reading the values that are already loaded?\nJust want to make sure I'm being kind to the other party and not doing 800 calls for get_attribute!\nThanks.","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":50,"Q_Id":54084584,"Users Score":5,"Answer":"get_attribute is retrieving data from sources already downloaded. You are not making requests to a web server when you execute that command","Q_Score":1,"Tags":"python,selenium,web-scraping","A_Id":54085021,"CreationDate":"2019-01-08T02:44:00.000","Title":"Selenium - How much am I hitting the server?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I am a Python developer. I wanted to create some online shopping stores which will be fully customized, Database will Mysql, Redis for caching, mail-gun for mailing, AWS hosting and Theme may be customized. I am confused in both platforms Magento and Shopify. Please Help Which have the best integration with python.","AnswerCount":2,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":201,"Q_Id":54103457,"Users Score":0,"Answer":"Yes Rehan is correct to go with the magento framework","Q_Score":0,"Tags":"python,magento,flask,e-commerce,shopify","A_Id":64007871,"CreationDate":"2019-01-09T04:54:00.000","Title":"Magento or Shopify which is best for integration with Python or which provide best APIs for Python","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I have apps deployed in AWS, like elastic search and ec2 instances inside VPC. Is there any service I can use to lookup the type of service running on the IP address from my VPC log. All my components are inside VPC I have the vpc log to get the ip address , Mostly it's all private ipv4 addresses.\nAny API in python or Java will be helpful.","AnswerCount":1,"Available Count":1,"Score":0.1973753202,"is_accepted":false,"ViewCount":1384,"Q_Id":54140167,"Users Score":1,"Answer":"There is no service to identify the service. However you can get an idea on what that IP is associated with via aws ec2 describe-network-interfaces","Q_Score":1,"Tags":"java,python,amazon-web-services,amazon-ec2,vpc","A_Id":54144184,"CreationDate":"2019-01-11T03:52:00.000","Title":"How to identify AWS component by IP address lookup?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"In my logs. I'm getting a large number of connections from a local IP address (which changes every time I restart my application. Is this anything to worry about or should I just block internal IPs from making requests to my server","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":63,"Q_Id":54152468,"Users Score":0,"Answer":"Turns out it's because Heroku uses an internal routing system meaning that connections appear to the server as if they had originated from an internal, private ip address","Q_Score":0,"Tags":"python,heroku,server,twisted","A_Id":54154957,"CreationDate":"2019-01-11T18:56:00.000","Title":"Heroku - Private IP address making requests to server","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I am logging to a website with valid credentials, but if my network changes or even device gets changed (within same network); it redirects to Authentication page where I have to put the access code, which I received via email. I want to skip this authentication page and navigate to other pages to continue my process. \nExpected result - Home page of the site\nActual result - Secure Access Code page","AnswerCount":1,"Available Count":1,"Score":0.1973753202,"is_accepted":false,"ViewCount":1260,"Q_Id":54180659,"Users Score":1,"Answer":"When you initialise your driver you can configure the browser to load your chrome profile, that is if your using chrome. This may allow you to bypass the authentication page if you have had a previous login with this profile. Not sure if this will work but it worth a shot.","Q_Score":0,"Tags":"python,selenium-webdriver","A_Id":54180917,"CreationDate":"2019-01-14T11:29:00.000","Title":"2 factor authentication handling in selenium webdriver with python","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I have a web-socket client which sends audio binary data in the request and receive them as a response from the web-socket server. I am using pyaudio to read binary audio data from (file\/microphone) which I then sending to the server. Then as the response I receive another binary audio data from the server. The question is can I use my recently opened pyaudio reading stream to play receiving audio in real time or I better need to create another pyaudio stream (Have two streams where one is responsible for binary data reading and another for binary data writing)?","AnswerCount":2,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":1404,"Q_Id":54183268,"Users Score":1,"Answer":"be aware of the fact, that, as far as I know, reading stream data is kind of generator process. Once you read it, you lose the data - In other words: based on your chunk, you basicly by every 'read' move a pointer that grabs your binary data.\nAnswer\nWhy dont you create 2 threads with 2 streams? Dont be afraid of streams. You can initialize as many as you want.\n\n1 thread receives binary data and push them into stream(from client to your sound device)\n2 thread receive data from your input stream (in binary form from your mic) and you push them to your client?\n\nI am working now little bit with PyAudio and streaming is quite interesting but hard to understand from programming point of view. You can actually create 2 output streams into your headphones and somewhere in a hardware on the way to your headphones the streams sum up themselves so you can simply listen two sounds in the same time. \nNOTE:\nAlso I wanted to say that you dont have to be worry about using threads. The streams works in batch, not in realtime. Whether it is read or write, it works in the way, that you have binary data which you push in a stream and you are done. Hardware accepts the binary data, stream them and only after it finishes, then the stream asks for another data. So if you have sample_rate 44100 and chunk 22050(just example), your loop will be just 0.5s. So you dont have to be even worry about overflows, too much data to handle or your threads getting crazy. \nIn fact, the moment you push data to stream, your python waits for your hardware to finish the job. Its very leightweight.","Q_Score":1,"Tags":"python,audio,audio-streaming,pyaudio","A_Id":54183504,"CreationDate":"2019-01-14T14:20:00.000","Title":"Using one pyaudio stream for both data reading and writing","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"After I launch Locust without the web UI:\n$ locust -f locust_files\/my_locust_file.py --no-web -c 1000 -r 100\nis it possible to change the number of users or hatch rate programmatically during the execution?","AnswerCount":3,"Available Count":3,"Score":1.2,"is_accepted":true,"ViewCount":1247,"Q_Id":54202932,"Users Score":1,"Answer":"No.. that is not possible.. Locust requires the number of virtual users and hatch rate to be defined at test startup.","Q_Score":1,"Tags":"python,locust","A_Id":54203228,"CreationDate":"2019-01-15T16:25:00.000","Title":"In Locust, can I modify the number of users and hatch rate after I start the test?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"After I launch Locust without the web UI:\n$ locust -f locust_files\/my_locust_file.py --no-web -c 1000 -r 100\nis it possible to change the number of users or hatch rate programmatically during the execution?","AnswerCount":3,"Available Count":3,"Score":0.0665680765,"is_accepted":false,"ViewCount":1247,"Q_Id":54202932,"Users Score":1,"Answer":"Warning: unsupported method\nStart locust in the usual way and investigate the calls made by the browser to the endpoints exposed by locust.\nE.g. the call to update the user count is a simple POST to the \/swarm endpoint with the desired locust count and hatch rate:\ncurl \"http:\/\/localhost:8089\/swarm\" -X POST -H \"Content-Type: application\/x-www-form-urlencoded\" --data \"locust_count=10&hatch_rate=1\"","Q_Score":1,"Tags":"python,locust","A_Id":54782803,"CreationDate":"2019-01-15T16:25:00.000","Title":"In Locust, can I modify the number of users and hatch rate after I start the test?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"After I launch Locust without the web UI:\n$ locust -f locust_files\/my_locust_file.py --no-web -c 1000 -r 100\nis it possible to change the number of users or hatch rate programmatically during the execution?","AnswerCount":3,"Available Count":3,"Score":0.0,"is_accepted":false,"ViewCount":1247,"Q_Id":54202932,"Users Score":0,"Answer":"1) If we want to increase the number of users during test:\nRun the same test in parallel with additional number of users\n2) If we want to decrease the number of users during test:\na) Run the second test with required number of users \nb) At the same time stop the first test\nBoth options can be automated with python or even bash scripts.\nDirty hack, but I think this will result desirable effect completely.","Q_Score":1,"Tags":"python,locust","A_Id":59071475,"CreationDate":"2019-01-15T16:25:00.000","Title":"In Locust, can I modify the number of users and hatch rate after I start the test?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I want to install some packages on the server which does not access to internet. so I have to take packages and send them to the server. But I do not know how can I install them.","AnswerCount":4,"Available Count":1,"Score":-0.0996679946,"is_accepted":false,"ViewCount":17437,"Q_Id":54213323,"Users Score":-2,"Answer":"Download the package from website and extract the tar ball.\nrun python setup.py install","Q_Score":0,"Tags":"python,pip,setup.py,installation-package","A_Id":54213344,"CreationDate":"2019-01-16T08:53:00.000","Title":"Install python packages offline on server","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":1,"System Administration and DevOps":0,"Web Development":0},{"Question":"I have a large text file of URLs (>1 million URLs). The URLs represent product pages across several different domains.\nI'm trying to parse out the SKU and product name from each URL, such as:\n\nwww.amazon.com\/totes-Mens-Mike-Duck-Boot\/dp\/B01HQR3ODE\/\n\n\ntotes-Mens-Mike-Duck-Boot\nB01HQR3ODE\n\nwww.bestbuy.com\/site\/apple-airpods-white\/5577872.p?skuId=5577872\n\n\napple-airpods-white\n5577872\n\n\nI already have the individual regex patterns figured out for parsing out the two components of the URL (product name and SKU) for all of the domains in my list. This is nearly 100 different patterns.\nWhile I've figured out how to test this one URL\/pattern at a time, I'm having trouble figuring out how to architect a script which will read in my entire list, then go through and parse each line based on the relevant regex pattern. Any suggestions how to best tackle this?\nIf my input is one column (URL), my desired output is 4 columns (URL, domain, product_name, SKU).","AnswerCount":2,"Available Count":1,"Score":0.0996679946,"is_accepted":false,"ViewCount":176,"Q_Id":54319452,"Users Score":1,"Answer":"While it is possible to roll this all into one massive regex, that might not be the easiest approach. Instead, I would use a two-pass strategy. Make a dict of domain names to the regex pattern that works for that domain. In the first pass, detect the domain for the line using a single regex that works for all URLs. Then use the discovered domain to lookup the appropriate regex in your dict to extract the fields for that domain.","Q_Score":0,"Tags":"python,regex,python-3.x","A_Id":54319613,"CreationDate":"2019-01-23T03:02:00.000","Title":"Parsing list of URLs with regex patterns","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":1,"System Administration and DevOps":0,"Web Development":0},{"Question":"I have just started myself with AWS cloud automations and have been using python boto3 for automations. I find boto3 is convenient for me becoz im not good with using AWS CLI commands using inside shell script for automations. My question is for AWS cloud automation, is boto3 superior to AWS CLI commands ? or whats is the advantage that python boto3 i having over AWS CLI commands or vice versa ?","AnswerCount":2,"Available Count":1,"Score":0.2913126125,"is_accepted":false,"ViewCount":1641,"Q_Id":54338549,"Users Score":3,"Answer":"If you can use boto3, then that is the far-superior choice. It gives you much more ability to supplement the AWS API calls with additional logic, such as filtering results with. It is also easier to chain API calls, such as making one call for a list of resources, then making follow-up calls to describe each resources in detail.\nThe AWS CLI is very convenient for one-off commands or simple automation, but things get tricky when using --filter and --query commands.","Q_Score":1,"Tags":"python,amazon-web-services,boto3,aws-cli","A_Id":54340510,"CreationDate":"2019-01-24T02:29:00.000","Title":"Advantage of AWS SDK boto3 over AWS CLI commands","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I'm a beginner to Python and trying to start my very first project, which revolves around creating a program to automatically fill in pre-defined values in forms on various websites.\nCurrently, I'm struggling to find a way to identify web elements using the text shown on the website. For example, website A's email field shows \"Email:\" while another website might show \"Fill in your email\". In such cases, finding the element using ID or name would not be possible (unless I write a different set of code for each website) as they vary from website to website.\nSo, my question is, is it possible to write a code where it will scan all the fields -> check the text -> then fill in the values based on the texts that are associated with each field?","AnswerCount":1,"Available Count":1,"Score":0.1973753202,"is_accepted":false,"ViewCount":81,"Q_Id":54350822,"Users Score":1,"Answer":"It is possible if you know the markup of the page, and you can write code to parse this page. In this case you should use xpath, lxml, beautiful soup, selenium etc. You can look at many manuals on google or youtube, just type \"python scraping\"\nBut if you want to write a program able to understand random page on a random site and understand what it should do, it is very difficult, it's a complex task with using machine learning. I guess this task is completely not for beginners.","Q_Score":1,"Tags":"python,python-3.x,selenium-webdriver","A_Id":54351000,"CreationDate":"2019-01-24T16:05:00.000","Title":"Filling forms on different websites using Selenium and Python","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I am new to Selenium. The web interface of our product pops up a EULA agreement which the user has to scroll down and accept before proceeding. This happens ONLY on initial login using that browser for that user. \nI looked at the Selenium API but I am unable to figure out which one to use and how to use it.\nWould much appreciate any suggestions in this regard.\nI have played around with the IDE for Chrome but even over there I don't see anything that I can use for this. I am aware there is an 'if' command but I don't know how to use it to do something like:\nif EULA-pops-up:\n Scroll down and click 'accept'\nproceed with rest of test.","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":348,"Q_Id":54354067,"Users Score":0,"Answer":"You may disable the EULA if that is an option for you, I am sure there is a way to do it in registries as well :\nC:\\Program Files (x86)\\Google\\Chrome\\Application there should be a file called master_preferences.\nOpen the file and setting:\nrequire_eula to false","Q_Score":0,"Tags":"python,selenium","A_Id":54354319,"CreationDate":"2019-01-24T19:31:00.000","Title":"How to handle EULA pop-up window that appears only on first login?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"My app is using a third-party API. It polls that API regularly at several endpoints. In also makes some additional calls to the API based on user's interaction with the app. The API is very slow, most requests take well over a second. The API is very flaky - timeouts are common, 500 errors are common, session key often randomly expires (even when defined \"keep_alive\" endpoint is called regularly). There is no option to use another API.\nWhat would be the best practices for dealing with such an API?\nHow to disable concurrent requests to this API on the requests level. So if one request is waiting for a response - the second request is not initiated? This should be done on \"per-domain\" basis, other requests to other domains should still be done concurrently.\nAny other settings to toggle with requests to make it easier to deal with such an API?","AnswerCount":2,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":399,"Q_Id":54365153,"Users Score":2,"Answer":"If your main problem is to serialize calls to that API in a multi-threaded (or multi process) application, a simple way would be to wrap it into a new module and consistenly use locking in that module.\nIf different clients can use a web API concurrently and you need to serialize the requests for performance reasons, you could imagine a dedicated serializing proxy. Just use above method in the proxy.","Q_Score":2,"Tags":"python,api,concurrency,python-requests","A_Id":54365613,"CreationDate":"2019-01-25T12:14:00.000","Title":"Python tips for working with an unstable `API`","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I'm setting up a webapp with a frontend and a backend that communicates with the frontend soley through RESTful methods. How do I make sure that the backend endpoints are only accessed by my own frontend, and not anyone else? I cannot find much information on this.","AnswerCount":2,"Available Count":1,"Score":1.0,"is_accepted":false,"ViewCount":14150,"Q_Id":54369416,"Users Score":8,"Answer":"Look into CORS. And make sure your server only allows access to specific origins.\n\nOn the backend - check if the X-Requested-With header is present in the request and set to XMLHttpRequest. Without a proper CORS handshake this header will be absent.\n\n\nThat being said, this will only protect your API from being used by other front-end apps or from being accessed directly from a browser address bar - because browsers respect CORS. People can still forge requests programmatically\/CLI and set headers to whatever they want.\nSo this is not actually \"securing\" just a way to prevent abuse & hotlinking","Q_Score":34,"Tags":"python,security,falconframework","A_Id":64536791,"CreationDate":"2019-01-25T16:39:00.000","Title":"How to secure own backend API which serves only my frontend?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I have a bunch of list of links which I'm doing a specific function on each link, the function takes about 25 sec, I use selenium to open each and get the page source of it then do my function, however whenever I build the program and cancel the build, I will have to start all over again.\nNote:I get links from different webs sitemap.\nIs there a way to save my progress and continue it later on?","AnswerCount":3,"Available Count":1,"Score":-0.0665680765,"is_accepted":false,"ViewCount":797,"Q_Id":54370412,"Users Score":-1,"Answer":"You should save the links in a text file. You should also save the index numbers in another text file, probably initializing with 0.\nIn your code, you can then loop through the links using something like:\nfor link in links[index_number:]\nAt the end of every loop, add the index number to the text file holding the index numbers. This would help you continue from where you left off.","Q_Score":0,"Tags":"python,list,selenium","A_Id":54370470,"CreationDate":"2019-01-25T17:46:00.000","Title":"Continuing where I left off Python","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I'm working on Twitter Streaming api to get live tweets data.\nI can print that data to console. But what I want is to save the data to a file and that data shouldn't be older than 5 minutes. \nHow do I continuously roll the file that holds the data from last 5 minutes as we can do for log files.\nAt the same time the file should be accessible for reading.\nIs there any way to do that in Python? \nI haven't come across such thing where we can mention such duration for which the file can hold specific data.","AnswerCount":3,"Available Count":1,"Score":-0.0665680765,"is_accepted":false,"ViewCount":56,"Q_Id":54376092,"Users Score":-1,"Answer":"Save the data in a file, with the actual time and check to see if the actual time is different by 5 min. Use time. Or use the sleep function and erase old data each 5 min.","Q_Score":0,"Tags":"python,file,text-files,tweepy,twitter-streaming-api","A_Id":54376116,"CreationDate":"2019-01-26T06:05:00.000","Title":"How do I create a file that doesn't contain information older than 5 minutes?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":1,"System Administration and DevOps":0,"Web Development":0},{"Question":"is there a way to get the IP of the connected client? (And if possible the port it uses). \nI tried client_socket.getsockname() but it gave me my IP address.","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":168,"Q_Id":54378675,"Users Score":0,"Answer":"You have to use the socket.getpeername() method.","Q_Score":1,"Tags":"python,python-2.7,sockets,python-sockets","A_Id":54965585,"CreationDate":"2019-01-26T13:09:00.000","Title":"Get IP of connected client in python","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"How can i use paramiko after it give me this error \"paramiko\\kex_ecdh_nist.py:39: CryptographyDeprecationWarning: encode_point has been deprecated on EllipticCurvePublicNumbers and will be removed in a future version. Please use EllipticCurvePublicKey.public_bytes to obtain both compressed and unco\"","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":288,"Q_Id":54385866,"Users Score":0,"Answer":"I had to disable gathering_facts to get past the warning. For some reason Ansible was getting stuck gathering facts.\nAfter disabling gathering facts, I still get this warning, but Ansible is able to continue execution.\nyaml file \ngather_facts: no","Q_Score":2,"Tags":"python-3.x","A_Id":56402605,"CreationDate":"2019-01-27T07:09:00.000","Title":"How can i use paramiko after it give me this error encode_point has been deprecated on EllipticCurvePublicNumbers","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I have built a small program that listens to a port on my laptop and an other program on my phone that connects to that port on my laptop. Now this all works fine but i was wondering if that same module could be used with external ip adresses. If it doesn't work with external ip's, is there a preinstalled module that can work with external ip's?","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":58,"Q_Id":54386753,"Users Score":1,"Answer":"If you forward that port in your network bridge (probably your all-in-one router) then yes! Simply listen on IPv4 address 0.0.0.0 or IPv6 address :: (maybe 0:0:0:0:0:0:0:0%0) to ensure that you're listening on all available IP addresses, instead of just localhost, and you're good to go.","Q_Score":2,"Tags":"python-3.x,ip-address","A_Id":54387231,"CreationDate":"2019-01-27T09:31:00.000","Title":"Can I use the python 'socket' module for listening on my external ip? if no, is there a preinstalled module which can?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"The situation:\nI am working on a senior design project that involves calculating the source of a signal by correlating audio from many devices that are all on the same WIFI network. The devices exchange information using REST apis.\nThe architecture is master slave, where the master unit will request audio from all of the slave units. Right now, the slave units need the IP of the master unit. Then they say 'hello' to the master unit who stores their IP, location etc in a list.\nWhat I think I want:\nI would like the slave units to have some way of automatically discover the master unit's IP. I don't think I really care about security. What is the best way to do this?\nIs there an idiomatic way to do this?\nI think I might just not have the correct words to google\nSolutions I have considered:\n1. Assign static IP to all (or just master unit).\n - not ideal because it would only work on one router\n - not slick\n\nMaster unit listens on hard-coded port and minions post to broadcast IP. \n\n\nMay not work on all routers \ndoesn't seem elegant","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":30,"Q_Id":54391482,"Users Score":0,"Answer":"Master unit listens on hard-coded port and minions post to broadcast IP.\n\n\nYes, using a well known port to rendezvous on is the standard way to solve this problem.\nI would turn your approach around a bit. There's more minions than masters, so master should do the broadcasting. A minion might send one (or a handful) of broadcasts upon power-on, to encourage an immediate reply from master. But as hours and days go by, the master should be the one primarily responsible for keeping the population in sync.\nA minion should remember IP of most-recent-master, and try unicasting to that upon startup.\nConsider using a packet format along these lines: {magic_number, version, start_time, num_minions, optional_list_of_minions}\nThe list of minions would include {ip_addr, time_of_last_transaction}, and would be empty if the list no longer fits within some limited size UDP packet. Minions could detect master reboot by noticing that start_time changed, and would re-connect soon after reboot. Do take care to randomly jitter your delay timers, so we won't see a thundering herd of minions.","Q_Score":0,"Tags":"python-3.x,networking,raspbian","A_Id":54392530,"CreationDate":"2019-01-27T18:28:00.000","Title":"Automatic device discovery (python or anything I can run on raspbian)","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I have written a code in python using youtube data API v3 where I am listing playlists created by a user. I have followed the code suggested by youtube API it runs well in local machine but when I deployed the same code on the server, It runs till authentication but after OAuth, it displays Internal server error in the browser instead of the result.\nis there any major changes that I am missing. Please help me with the issue.\nMy code in not running after oauth2callback function runs.","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":78,"Q_Id":54418572,"Users Score":1,"Answer":"I solved the error,\nI was deploying without a secure connection and that was raising an error. also redirect URL must be 'https' that was another problem.","Q_Score":0,"Tags":"python,google-oauth,url-redirection,youtube-data-api,google-authentication","A_Id":54624602,"CreationDate":"2019-01-29T10:11:00.000","Title":"Internal server error after I try to host code on server","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"Scrapy has several points\/places where allowed processing scraped data: spider, items and spider middlewares. But I don't understand where I should do it right. I can process some scraped data in all these places. Could you explain to me differences between them in detail?\nFor example: downloader middleware returns some data to the spider (number, short string, url, a lot of HTML, list and other). And what and where i should do with them? I understand what to do, but is not clear where to do it...","AnswerCount":2,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":347,"Q_Id":54421455,"Users Score":0,"Answer":"I will try to explain in order\nSpider is the one where you decide which URLs to make requests to\nDownloadMiddleware has a process_request method which is called before a request to URL is made, and it has process_response method which is called once response from that URL is received\nPipeline is the thing where data is sent when you yield a dictionary from your Spider","Q_Score":2,"Tags":"python,scrapy,scrapy-spider,scrapy-pipeline","A_Id":54435005,"CreationDate":"2019-01-29T12:47:00.000","Title":"In which file\/place should Scrapy process the data?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I get the MessageError: RangeError: Maximum call stack size exceeded when I try to upload a json file and I assume the problem is the size.\nI'm trying to upload a 187Kb json file on Google Colaboratory using the function files.upload(), but it gives me the error \"MessageError: RangeError: Maximum call stack size exceeded.\" When I try to upload manually on the sidebar, it just keeps loading endlessly. Is the another way to upload this file?","AnswerCount":2,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":5718,"Q_Id":54430763,"Users Score":0,"Answer":"What I did in response to this error message since I did not want it to count against my Google Drive quota is I uploaded the file in the left pane (labeled \"Files\") of the Google Colab notebook. The drawback to this is that the file will be gone when the notebook is recycled, however it does get the job avoiding both upload and Google Drive caps.","Q_Score":2,"Tags":"python,google-colaboratory","A_Id":60836386,"CreationDate":"2019-01-29T22:46:00.000","Title":"Google Colaboratory - Maximum call stack size exceeded","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":1,"System Administration and DevOps":0,"Web Development":0},{"Question":"I have never made an executable application before but from what I have read its pretty easy, using py2exe to generate the exe.\nBut I have a GUI that uses Selenium to scrape data from a backend (No I can not use API calls). How do I add chromedriver to the executable? Also, would all the imports go along when using a compiler?","AnswerCount":2,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":399,"Q_Id":54468218,"Users Score":0,"Answer":"When you compile a .py file into an .exe (from my personal experience) all of the imports are included.\nI would personally suggest using pyinstaller. I had quite a few problems using py2exe and as a beginner I found pyinstaller much more user-friendly and easier to troubleshoot.\nAs compiling a file does not alter the .py file, I would suggest getting it to a fully working state and trying it. If it doesn't appear to work or if some of the imports are lost, we can troubleshoot with the error code.","Q_Score":1,"Tags":"python,selenium,selenium-chromedriver,exe,executable","A_Id":54468382,"CreationDate":"2019-01-31T19:48:00.000","Title":"Python Selenium GUI to an executable","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I'm trying to install the requests library so I can make HTTP requests. I've tried looking for resources online but I can't seem to find tutorials for installing this library on MacOS.","AnswerCount":3,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":28458,"Q_Id":54484536,"Users Score":0,"Answer":"You can use pip to install requests library. Try these steps to setup pip and install requests library in macOS:\n\nsudo easy_install pip\npip install requests","Q_Score":1,"Tags":"python,python-3.x,python-requests","A_Id":54484665,"CreationDate":"2019-02-01T17:40:00.000","Title":"Installing requests python library on MacOS","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I'm getting the following error with my 3 node DAX Cluster:\nFailed to retrieve endpoints\nTraceback (most recent call last):\nMy setup:\n- Private Lambdas\n- Python 3.6\n- amazon-dax-client\n- Config settings - timeout 300ms, max retries 5\n- vpc has 3 subnets\n- The DAX cluster has 3 x r4.large nodes\nThis error happens intermittently - it still \"works\" most of the time but it is alarming given how often it happens.","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":193,"Q_Id":54581755,"Users Score":0,"Answer":"The DAX Python client up to and including 1.0.6 is a bit too sensitive when checking the health of cluster nodes and considers them unhealthy when they're not. The errors are benign, but annoying. This will be fixed in an upcoming release.","Q_Score":0,"Tags":"python,amazon-dynamodb-dax","A_Id":54582303,"CreationDate":"2019-02-07T20:28:00.000","Title":"Amazon Dax Failed to retrieve endpoints Traceback (most recent call last):","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"Was trying to execute python script in jenkins, that make HTTP request with help of 'requests' module, but got stuck with following error:\nImportError: No module named requests\nBuild step 'Custom Python Builder' marked build as failure\nFinished: FAILURE","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":4697,"Q_Id":54634554,"Users Score":0,"Answer":"You need to install the dependencies of the script, normally in setup.py or requirements.txt.\nIn the case of requirements.txt run: pip install -r requirements.txt\nIn the case of setup.py run: pip install .\nYou should do this in the job that is running the script.\nIf neither of these files exist pip install requests will include the particular dependency you are missing in your question.","Q_Score":0,"Tags":"python,jenkins,python-requests","A_Id":54634665,"CreationDate":"2019-02-11T16:07:00.000","Title":"Jenkins: ImportError: No module named requests","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I am trying to create an iframe inside a web content (no matter where exactly need to be inserted). The only way that I need to do this, is just by making it dynamically and also by using Python3+ version. The iframe's information is not important either. Id, class name and more can be random or custom. It is not necessary to insert specific data. Also, I don't need an answer about the height and the width. I already have solved it.\nWhat do I mean dynamically? I mean that if someone clicks on a button then I need to insert this iframe. If someone clicks on another button then I need to delete it, and so on.","AnswerCount":2,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":545,"Q_Id":54670433,"Users Score":0,"Answer":"Sounds like HTML \/ Javascript challenge. Why do you think it is related to python?","Q_Score":2,"Tags":"python,python-3.x,iframe","A_Id":54672807,"CreationDate":"2019-02-13T12:39:00.000","Title":"Q: How can I create dynamically an iframe inside a web content using Python?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"Hello stackoverflow people!\nI would like to discuss and see what's the better approach to my problem.\nI have an application that send files to clients using multiple protocols (FTP(S), SFTP, S3, EMail).\nThere's a celery task per directory. A directory can be send to multiple clients and can be send to multiple destinations. e.g. dir1 -> client1 -> FTP and EMail (2 tasks, fine to run in parallel), dir2 -> client1 AND client2 -> same FTP hostname, different remote directories (2 tasks, not fine to run in parallel).\nThis is working fine, however I'm causing client network congestion sometimes, due to multiple connections from multiple workers to the same destination, some clients don't know (or want to implement) QOS.\nI would like to have a logic that don't allow tasks connecting to the same protocol or hostname running at the same time. Per example, a directory that is being send to 2 x S3 buckets, should run once, after it finished the second will start. Or two different directories that is being send to the same FTP server.\nMy initial idea is to implement a celery_worker queue remote control. One queue for each account, protocol. And setup workers with concurrency 1 listening on the queues.\nWondering if any of you had a similar challenge and how did you workaround it.","AnswerCount":1,"Available Count":1,"Score":0.1973753202,"is_accepted":false,"ViewCount":201,"Q_Id":54688207,"Users Score":1,"Answer":"Your proposed solution is rather brittle (you really shouldn't rely on celery concurrency settings to control\/prevent concurrent execution) and will probably not solve all the potential race conditions (for example if taskA and taskB are on different queues but need to access a resource that doesn't support concurrent access).\nThere are quite a couple recipes (from rather informal advises to full-blown libs like celery-once) to prevent concurrent execution of one given task. They don't directly solve your own problem but basically the principle is the same: have some shared lock mechanism that the tasks communicate with - try to acquire the lock, only run once they get it, and of course release it. If you're using Redis as result backend, it's rather low read\/write cost and it's 'expire' feature can be really helpful, but you can also just use your SQL database.","Q_Score":1,"Tags":"python,python-3.x,celery,celery-task","A_Id":54690143,"CreationDate":"2019-02-14T10:28:00.000","Title":"Celery task \/ worker assignment logic","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":1,"Web Development":0},{"Question":"Some time ago I ve written a Python script, using poplib library, which retrieves messages from my pop3 email account. Now I would like to use it to retrieve emails from different mail server which works with IMAP. It works well, but only to retrieve messages from Inbox. Is there any way to also get emails from other folders like Spam, Sent etc? I know I could use imaplib and rewrite the script, but my questions is if it's possible to obtain that with poplib.","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":554,"Q_Id":54706093,"Users Score":4,"Answer":"No.\nPOP is a single folder protocol. It is very simple and was not designed for multiple folders.\nYou will need to use IMAP or other advanced protocols to access additional folders.","Q_Score":2,"Tags":"python,email,imaplib,poplib","A_Id":54719050,"CreationDate":"2019-02-15T09:21:00.000","Title":"Checking folders different than inbox using poplib in Python","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"Was asking if I could do a search on the entire eclipse library which includes my closed projects. I did a file search(Ctrl + H) but they only shows results for projects that are open.","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":157,"Q_Id":54724905,"Users Score":0,"Answer":"When a project is closed, it can no longer be changed in the Workbench and its resources no longer appear in the Workbench, but they do still reside on the local file system. Closed projects require less memory. Also, since they are not examined during builds, closing a project can improve build time.\nA closed project is visible in the Package Explorer view but its contents cannot be edited using the Eclipse user interface. Also, an open project cannot have dependency on a closed project. The Package Explorer view uses a different icon to represent a closed project.","Q_Score":0,"Tags":"python,eclipse,search,project","A_Id":54724960,"CreationDate":"2019-02-16T15:55:00.000","Title":"How to search through eclipse closed projects?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I am trying to run some automatic scripts on a webpage with selenium and python, but the problem I'm having is that the webpage is loaded using document.write().\nI have to find some elements, but they are not shown because when I view the source, it is shown as document.write(A lot of JS) instead of the html.\nHow can I do it so that I can view the HTML source code? I know there is the function driver.execute_script(), but I have to specify the script to run, and I don't think it will work.\nThe page is correctly rendered, only problem is the source cannot be parsed.","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":349,"Q_Id":54781641,"Users Score":0,"Answer":"As it turns out after some digging into the code, Selenium does the search in the rendered final view, but the problem was not the document.write(), but the fact that the field I was looking for was in an iframe, which selenium could not find on the default frame. \nAll I had to do was search through the iframes and find the ones that I needed.","Q_Score":0,"Tags":"javascript,python,selenium-webdriver,document.write","A_Id":54800939,"CreationDate":"2019-02-20T08:14:00.000","Title":"Selenium find element with document.write","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I have a WebApp on Flask+gunicorn+nginx. I need to send 200 requests from my other server at the same time to a WebApp , save the response and its speed. Also I need to send Json POST in this 200 requests.\nHow to do it right?\nUse python script or CURL?","AnswerCount":2,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":123,"Q_Id":54783844,"Users Score":0,"Answer":"I suggest you to use \"Postman\" for any kind of API testing. It is one the best tools available in the market for API testing , monitoring, documenting, and also sharing the results(as well as test scripts) for free.\nIf you don't want to use any other tool, then i suggest you to use a python script.","Q_Score":0,"Tags":"python,curl,flask,server,gunicorn","A_Id":54784754,"CreationDate":"2019-02-20T10:16:00.000","Title":"How to load server with python?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I am making APIs.\nI'm using CentOS for web server, and another windows server 2016 for API server.\nI'm trying to make things work between web server and window server.\nMy logic is like following flow.\n1) Fill the data form and click button from web server\n2) Send data to windows server\n3) Python script runs and makes more data\n4) More made data must send back to web server\n5) Web server gets more made datas\n6) BAMM! Datas append on browser!\nI had made python scripts.\nbut I can't decide how to make datas go between two servers..\nShould I use ajax Curl in web server?\nI was planning to send a POST type request by Curl from web server to Windows server.\nBut I don't know how to receipt those datas in windows server.\nPlease help! Thank you in advance.","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":213,"Q_Id":54799927,"Users Score":1,"Answer":"First option: (Recommended)\nYou can create the python side as an API endpoint and from the PHP server, you need to call the python API.\nSecond option:\nYou can create the python side just like a normal webpage and whenever you call that page from PHP server you pass the params along with HTTP request, and after receiving data in python you print the data in JSON format.","Q_Score":0,"Tags":"php,python,curl","A_Id":54800059,"CreationDate":"2019-02-21T05:36:00.000","Title":"Run python script by PHP from another server","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"Is there any way to directly send files from one API to another FTP server without downloading them to local in Python 3.\nCurrently we downloading from one API to local and then sending it to FTP server, want to avoid that hop from data flow by directly sending files to server.","AnswerCount":3,"Available Count":3,"Score":0.0,"is_accepted":false,"ViewCount":1136,"Q_Id":54804532,"Users Score":0,"Answer":"One of the options would be having another API function (TransferFile, ...), which will transfer data from API server to FTP site. Then you just call that API method from your code without downloading data to the local server.","Q_Score":1,"Tags":"python,api,ftp","A_Id":54804765,"CreationDate":"2019-02-21T10:14:00.000","Title":"Upload file Directly to FTP Server in Python","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"Is there any way to directly send files from one API to another FTP server without downloading them to local in Python 3.\nCurrently we downloading from one API to local and then sending it to FTP server, want to avoid that hop from data flow by directly sending files to server.","AnswerCount":3,"Available Count":3,"Score":0.0,"is_accepted":false,"ViewCount":1136,"Q_Id":54804532,"Users Score":0,"Answer":"The FTP protocol has provision for initiating a data transfert between two remote hosts from a third party client. This is called the proxy mode. Unfortunately, most servers disable it for security reasons, because it used to be a very efficient way for DOS attacks.\nIf you have control on both servers and if both use FTP and if they are not publicly exposed, this can be very efficient.\nIn any other use case, the data will have to pass through the client. The best that can be done is to open both connections and transfer data to the target host as soon as it has been received from the source without storing it on disk.","Q_Score":1,"Tags":"python,api,ftp","A_Id":54804881,"CreationDate":"2019-02-21T10:14:00.000","Title":"Upload file Directly to FTP Server in Python","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"Is there any way to directly send files from one API to another FTP server without downloading them to local in Python 3.\nCurrently we downloading from one API to local and then sending it to FTP server, want to avoid that hop from data flow by directly sending files to server.","AnswerCount":3,"Available Count":3,"Score":0.0,"is_accepted":false,"ViewCount":1136,"Q_Id":54804532,"Users Score":0,"Answer":"you can use byte data of the file(it will store as in-memory) and pass that to another API.","Q_Score":1,"Tags":"python,api,ftp","A_Id":54804658,"CreationDate":"2019-02-21T10:14:00.000","Title":"Upload file Directly to FTP Server in Python","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"For a connection is to be made, a bounded socket should be listening for clients. Client needs to know both ip address and port. For bounding a socket why we need an ip address of the server itself when the program(which listens for clients) itself is running on the server?","AnswerCount":1,"Available Count":1,"Score":0.1973753202,"is_accepted":false,"ViewCount":15,"Q_Id":54829284,"Users Score":1,"Answer":"Simply because a server has multiple addresses, at least the loopback one at 127.0.0.1 (IP v4) and one per physical network interfaces. For example a corporate proxy has commonly two interfaces, one on the internal network and one on the public one. Most have a third one for the DMZ. Being member of different networks, those interfaces must have different addresses. And it make sense to open some services on only one interface.\nBut you also can use the ANY address (0.0.0.0 in IPv4) that means to accept connections on any interface.","Q_Score":0,"Tags":"python,tcp","A_Id":54829438,"CreationDate":"2019-02-22T14:30:00.000","Title":"why socket binding in server needs its ip address >","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I'm trying to make a discord bot, and I read that I need to have an older version of Python so my code will work. I've tried using \"import discord\" on IDLE but an error message keeps on coming up. How can I use Python 3.6 and keep Python 3.7 on my Windows 10 computer?","AnswerCount":3,"Available Count":2,"Score":0.0,"is_accepted":false,"ViewCount":323,"Q_Id":54852821,"Users Score":0,"Answer":"Install in different folder than your old Python 3.6 then update path\nUsing Virtualenv and or Pyenv\nUsing Docker\n\nHope it help!","Q_Score":2,"Tags":"python,discord","A_Id":54853052,"CreationDate":"2019-02-24T14:21:00.000","Title":"how can I use python 3.6 if I have python 3.7?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":1,"System Administration and DevOps":0,"Web Development":0},{"Question":"I'm trying to make a discord bot, and I read that I need to have an older version of Python so my code will work. I've tried using \"import discord\" on IDLE but an error message keeps on coming up. How can I use Python 3.6 and keep Python 3.7 on my Windows 10 computer?","AnswerCount":3,"Available Count":2,"Score":0.0,"is_accepted":false,"ViewCount":323,"Q_Id":54852821,"Users Score":0,"Answer":"Just install it in different folder (e.g. if current one is in C:\\Users\\noob\\AppData\\Local\\Programs\\Python\\Python37, install 3.6. to C:\\Users\\noob\\AppData\\Local\\Programs\\Python\\Python36).\nNow, when you'll want to run a script, right click the file and under \"edit with IDLE\" will be multiple versions to choose. Works on my machine :)","Q_Score":2,"Tags":"python,discord","A_Id":54853010,"CreationDate":"2019-02-24T14:21:00.000","Title":"how can I use python 3.6 if I have python 3.7?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":1,"System Administration and DevOps":0,"Web Development":0},{"Question":"I have a folder containing multiple images, i want to write these images as a video file on Google drive. Is there any way to achieve this?\nI cannot write the images to video file in local system and then upload to Google Drive, because of space constraint.","AnswerCount":1,"Available Count":1,"Score":0.1973753202,"is_accepted":false,"ViewCount":39,"Q_Id":54863190,"Users Score":1,"Answer":"The google drive api is a file store api. It contains information about the files it contains and it will allow you to upload new files. To some extent it can covert one file type to another. For example it can covert an excel file to a google sheets file and back and forth.\nThe google drive api does not have the ablity to allow you to upload two images and have them coverted into a video.\nYou will need to encode the video locally on your machine and then upload the video after.","Q_Score":1,"Tags":"python-3.x,google-drive-api","A_Id":54864817,"CreationDate":"2019-02-25T09:36:00.000","Title":"Write a set of images to a Video file on Google Drive","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":1,"Web Development":0},{"Question":"While running pip install InstagramAPI im getting the following error.\n\n\"networkx 2.1 has requirement decorator>=4.1.0, but you'll have\n decorator 4.0.11 which is incompatible\"\n\nSo I uninstalled decorator and reinstalled it using pip install decorator==4.1.0. I confirmed with pip list the version of decorator. I then tried to pip install InstagramApi I got the same error \n\n\"networkx 2.1 has requirement decorator>=4.1.0, but you'll have\n decorator 4.0.11 which is incompatible.\"\n\nand my decorator module was regressed to version 4.0.11!!!\nSomeone please explain whats going on here. Thank you","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":94,"Q_Id":54875301,"Users Score":0,"Answer":"Why don't you use a virtual python environment (the python package) It should take care of these Dependency problems.","Q_Score":0,"Tags":"python,instagram-api","A_Id":54875371,"CreationDate":"2019-02-25T21:57:00.000","Title":"InstagramAPi install error with decorator version","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":1,"System Administration and DevOps":0,"Web Development":0},{"Question":"I am running Eclipse Che v6.18.1 in Google Chrome on MacBook Pro OS v10.10.5 (Yosemite). Eclipse Che workspace runs in a Docker container.\nHow can I open a new browser tab (in the same browser window as Eclipse Che) from within a Python code executed in Eclipse Che? \nSo not a new Google Chrome instance from within a Docker container (much too slow) but a new tab in already existing browser window on the host machine.\nIn Eclipse Che it is possible to preview an HTML file in the project Workspace (right-click => Preview). Then the HTML file opens in the next tab to the Eclipse Che IDE. How could I use that feature from within a Python code to open a new browser tab?","AnswerCount":3,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":569,"Q_Id":54882713,"Users Score":0,"Answer":"Are you trying to open a preview window similar to the sample nodejs Yeoman application? Or are you trying to open a new tab from the source code in someone's browser?\nIf it's the latter, then I do not think it is possible (or a good idea!) - Che does not run in a security context that will allow it to instruct the browser to open a new tab or window.","Q_Score":1,"Tags":"python,eclipse-che,codenvy","A_Id":55222067,"CreationDate":"2019-02-26T09:52:00.000","Title":"How can I open a new browser tab from Python in Eclipse Che?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I am able to parse from file using this method: \nfor event, elem in ET.iterparse(file_path, events=(\"start\", \"end\")):\nBut, how can I do the same with fromstring function? Instead of from file, xml content is stored in a variable now. But, I still want to have the events as before.","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":319,"Q_Id":54891949,"Users Score":0,"Answer":"From the documentation for the iterparse method:\n\n...Parses an XML section into an element tree incrementally, and\n reports what\u2019s going on to the user. source is a filename or file\n object containing XML data...\n\nI've never used the etree python module, but \"or file object\" says to me that this method accepts an open file-like object as well as a file name. It's an easy thing to construct a file-like object around a string to pass as input to a method like this.\nTake a look at the StringIO module.","Q_Score":2,"Tags":"python","A_Id":54892048,"CreationDate":"2019-02-26T18:30:00.000","Title":"Elementree Fromstring and iterparse in Python 3.x","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":1,"System Administration and DevOps":0,"Web Development":0},{"Question":"I am automating our mobile app on iOS and Android. When I get the search results(list of elements), I want to iterate through those all elements matching my xpath.\nThe problem is that - Appium returns only those elements which are visible in the viewport, which makes sense. However, I would like to get all elements matching my xpath\/locator strategy, although the elements are not in viewport. To get further set of elements, I have to scroll to those elements and get them into viewport.\nIs there any configuration provided by appium, to enable this feature? Or will I have to continue scrolling to those elements before accessing?","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":57,"Q_Id":54912357,"Users Score":0,"Answer":"You need to handle scrolling on your own.\nOn Android, Appium can make a snapshot of what is currently in the viewport.\nYou can get a list of elements and iterate them, then scroll by screen hight and get another list of elements, iterate them. Repeat it until the new list is empty - make sure you don't get same elements twice.\nOn iOS, it's more tricky: the driver will return you elements including the ones not in the viewport. For reliable interaction, I suggest scrolling to each element.","Q_Score":0,"Tags":"python-2.7,appium","A_Id":54922735,"CreationDate":"2019-02-27T18:36:00.000","Title":"How to get all items matching xpath, which are not in viewport?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I have 10 different IP cameras that I need to access in a FLASK server.\nI would like to know the best way to do that.\nCurrently, I have a dictionary that uses an ID to map to a VideoCapture object. A client sends the cam ID to the server and the server accesses the video capture object and returns a captured frame via the read function.\nSo technically I have 10 different objects of VideoCapture. Another method that I have used is, that upon getting camera ID, if the current cam ID is different from the received cam ID, then the video cap object is replaced with a new one.\nMy question is that is opening 10 video captures at one time fine? My server should be time sensitive. Does opening 10 captures congest the network? If yes then should I stick to the one object approach that always creates a new object on ID change or is there any better way to do this? Thank you.","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":1328,"Q_Id":54919482,"Users Score":1,"Answer":"The first way you used is OK. For every camera, you should keep one capture object. It will work faster than replacing one capture object with multiple connections. If you open RTSP connection, then it will not congest the network until you start reading frames. So you can go with the first way.\nOpening and then releasing one capture object for multiple connections will slow down the speed because in every new connection it needs time to access the camera.","Q_Score":1,"Tags":"python,opencv","A_Id":54919941,"CreationDate":"2019-02-28T06:13:00.000","Title":"OpenCV Capturing multiple RTSP Streams - Python","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"My Spyne server WSDL shows \n<soap:address location=\"http:\/\/some-domain.com>\nI want it to show https instead of http.\nBasically, <soap:address location=\"https:\/\/some-domain.com>\nNotice the difference lies in http and https.\nHow do I tell my Spyne server to address this?\nHave gone through the docs multiple times but could not figure it out.\nThanks!","AnswerCount":2,"Available Count":1,"Score":-0.0996679946,"is_accepted":false,"ViewCount":274,"Q_Id":54920093,"Users Score":-1,"Answer":"WSDL url is constructed from the first request.\nUpon starting the server, request the wsdl from the secure domain and it should work.","Q_Score":0,"Tags":"python,soap,https,spyne","A_Id":54932502,"CreationDate":"2019-02-28T07:00:00.000","Title":"Spyne SOAP server WSDL file cannot show https","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I have to automate a task which involves lots of google searching, which I am doing through selenium and python. After 20 searches google says suspicious activity detected and gives a reCaptcha to prove I am not a robot. I have tried other ways (like changing profile) but still the same problem.\nHow to get rid of it?","AnswerCount":2,"Available Count":1,"Score":0.0996679946,"is_accepted":false,"ViewCount":1817,"Q_Id":54940720,"Users Score":1,"Answer":"I solved this by rotating a decent pool of proxies with an inner load balancer, switching user agent and use captcha solving APIs where appropriate. Having a good amount of clean IP addresses and using them wisely has the biggest impact so far.","Q_Score":1,"Tags":"python,selenium,google-chrome,selenium-chromedriver,recaptcha","A_Id":55326670,"CreationDate":"2019-03-01T08:32:00.000","Title":"Google search using selenium causing suspicious network traffic and shows reCaptcha","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I have a set of xml files representing rules manager rules. There are several elements that map to tables or table-attribute pairs.\nI would like to discover all the elements that affect intersecting sets of tables or table-attributes.\nI can work in java or possibly python, or command line Windows\/Mac. Can someone suggests a tool or approach? Thanks so much!","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":28,"Q_Id":54945282,"Users Score":0,"Answer":"Experiment with xmlStarlet\nTry Saxon for cross file xpath queries\nBoth can work from the command line","Q_Score":0,"Tags":"java,python","A_Id":55133607,"CreationDate":"2019-03-01T13:04:00.000","Title":"Generate relationship data","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I'm trying to download files from a website using Python 3.\nDirect parsing of the URL doesn't work because the URL forwards to the login page everytime, where you need to login using the Google Login button, which forwards to Google.\nIs there a way to sign in and download the files using Python script? Maybe by implementing cookies in some way?\nThanks in advance.","AnswerCount":2,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":192,"Q_Id":54974983,"Users Score":0,"Answer":"You can use selenium, which can automatically fill the login form for you.","Q_Score":0,"Tags":"python,python-3.x,web-scraping,download,google-signin","A_Id":54975349,"CreationDate":"2019-03-03T23:48:00.000","Title":"Python - Download files from Website with Google Login","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I'm starting to learn telegram bot. My plan was to have a bot that daily sends you at a specific time a message, but also I wanted the option to manually poll the bot for getting that daily message, or a random one. Right now I have a bot running on pythonanywhere that can respond to the 2 commands, but what about sending the user the daily message at some time? \nWhat's the way to go, create a channel and then schedule a task on my webhook to daily send the message to the channel, or store all chat-id in my service and talk to them everytime? The first one seems obviously better but I was wondering if there's some trick to make everything works in the bot.","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":131,"Q_Id":54981185,"Users Score":0,"Answer":"For now I managed to create a channel, make the bot an admin and scheduled a task on the webhook to daily send a message as admin to the channel","Q_Score":0,"Tags":"python,telegram,telegram-bot","A_Id":55000584,"CreationDate":"2019-03-04T10:22:00.000","Title":"Telegram: Store chat id or Channel with bot inside?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I'm trying to count words which their visibility are visible in browser.\nI'm using Scrapy to get link and parse theme with Selector.\nProblem is I can only count all texts in spite of their visibility (hidden, in menu, in blockquote...) and the searching sites is a list of url (not the same structure)\nDo you have any suggestion?","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":54,"Q_Id":54994471,"Users Score":0,"Answer":"Scrapy just give you the page source(ctrl+u) to get the rendered page you have to use the Selenium or splash my splash is little speed compare to the Selenium but Selenium give you full control","Q_Score":0,"Tags":"python,web-scraping,scrapy","A_Id":55007667,"CreationDate":"2019-03-05T02:15:00.000","Title":"How to imitate the browser to find & count text","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I want to send a text file from one serial device(slcan0) to another serial device(slcan1) can this operation be performed in SocketCAN? The serial CAN device I am using is CANtact toolkit. Or can the same operation be done in Python-can?","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":485,"Q_Id":55017807,"Users Score":0,"Answer":"When you want to send a text file over the CAN bus, you have to decide which CAN-ID you want so sent for sending and receiving.\nMost likely your text file is larger than 8 bytes, so you would have to use a higher level protocol on CAN.\nISO-TP will allow 4095 of data in one message.\nIf this is still not enough, you would have to invent another protocol for sending and receiving the data. E.g. first send the length of data, then send the data in chunks of 4095 bytes.\nOnce you have figured this out, it does not really matter whether you use SocketCAN, Python-CAN, pyvit or anything else.","Q_Score":0,"Tags":"python,socketcan,python-can","A_Id":55027926,"CreationDate":"2019-03-06T07:34:00.000","Title":"How to send and receive a file in SocketCAN or Python-can?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"The context here is simple, there's a lambda (lambda1) that creates a file asynchronously and then uploads it to S3.\nThen, another lambda (lambda2) receives the soon-to-exist file name and needs to keep checking S3 until the file exists.\nI don't think S3 triggers will work because lambda2 is invoked by a client request\n1) Do I get charged for this kind of request between lambda and S3? I will be polling it until the object exists\n2) What other way could I achieve this that doesn't incur charges?\n3) What method do I use to check if a file exists in S3? (just try to get it and check status code?)","AnswerCount":2,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":783,"Q_Id":55031936,"Users Score":0,"Answer":"Let me make sure I understand correctly. \n\nClient calls Lambda1. Lambda1 creates a file async and uploads to S3\nthe call to lambda one returns as soon as lambda1 has started it's async processing. \nClient calls lambda2, to pull the file from s3 that lambda1 is going to push there. \n\nWhy not just wait for Lambda one to create the file and return it to client? Otherwise this is going to be an expensive file exchange.","Q_Score":0,"Tags":"python-3.x,amazon-web-services,amazon-s3,aws-lambda,boto3","A_Id":55032258,"CreationDate":"2019-03-06T20:52:00.000","Title":"Long polling AWS S3 to check if item exists?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I am tuning hyperparameters for a neural network via gridsearch on google colab. I got a \"transport endpoint is not connected\" error after my code executed for 3 4 hours. I found out that this is because google colab doesn't want people to use the platform for a long time period(not quite sure though).\nHowever, funnily, after the exception was thrown when I reopened the browser, the cell was still running. I am not sure what happens to the process once this exception is thrown.\nThank you","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":4543,"Q_Id":55034469,"Users Score":0,"Answer":"In google colab, you can use their GPU service for upto 12 hours, after that it will halt your execution, if you ran it for 3-4 hours, it will just stop displaying Data continuously on your browser window(if left Idle), and refreshing the window will restore that connection.\nIn case you ran it for 34 hours, then it will definitely be terminated(Hyphens matter), this is apparently done to discourage people from mining cryptocurrency on their platform, in case you have to run your training for more than 12 hours, all you need to do is enable checkpoints on your google drive, and then you can restart the training, once a session is terminated, if you are good enough with the requests library in python, you can automate it.","Q_Score":0,"Tags":"python,neural-network,jupyter-notebook,jupyter,google-colaboratory","A_Id":55154500,"CreationDate":"2019-03-07T00:56:00.000","Title":"Google colab Transport endpoint is not connected","Data Science and Machine Learning":1,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"on Twilio's tutorial it sets action parameter to \/handleDialCallStatus but I have no clue what happens when it redirects to the url. How can I handle the status of calls .How can I redirect to another url when the call has completed","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":620,"Q_Id":55048107,"Users Score":1,"Answer":"Twilio evangelist here.\nWhen the <Dial> call ends, the action URL tells Twilio where to send a GET or POST request. A DialCallStatus is passed to the action URL according to one of the following scenarios:\n\nNobody picks up, DialCallStatus=no-answer\nThe line is busy, DialCallStatus=busy\nWhen calling a conference and the call is connected, DialCallStatus=answered\nSomeone answered the call and was connected to the caller, DialCallStatus=connected\nAn invalid phone number was provided, DialCallStatus=failed\nCall canceled via the REST API before it was answered, DialCallStatus=canceled\n\nHow do you handle these scenarios? In the action attribute URL of the Dial verb.\n<Dial timeout='50' action='your_url'>\nThe web app hosted at this action URL can then look at the DialCallStatus and send a response to Twilio telling it what to do next.\nYou can replace your_url with another URL (absolute or relative) to redirect there, and Twilio will continue the initial call after the dialed party hangs up. No TwiML verbs included after that <Dial> will be reachable, so if you want to take more actions on that initial call, you need to respond to Twilio's request with TwiML instructions on how to handle the call.\nAny TwiML verbs included after this <Dial> will be unreachable, as your response to Twilio takes full control of the initial call. If you want to take more actions on that initial call, you must respond to Twilio's request with TwiML instructions on how to handle the call.\nHope this helps.","Q_Score":1,"Tags":"python,twilio,twilio-api","A_Id":55072306,"CreationDate":"2019-03-07T16:03:00.000","Title":"How to handle dial call status with twilio-python","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"Pycharm\nSSH->Remote docker\nWe are using a remote interpreter for python on Pycharm which works great on an SSH connection. We are in a phase to convert our main work on docker container. It is important for us to keep to the development process on the remote servers and not on the local computer. But it is also important to be able to do it over docker container, and not just ssh as this saves a lot of time and effort when starting a new development server.\nRemote docker -> Securely remote docker\nWhat we are seeking is a way to be able to make a remote docker connection within Pycharm securely. It seems that when generating the Tls certificate, we need to bind it to the host IP's. This IP\/host bounding prevents us to quickly start new servers for development as this force to generate a certificate per IP.\nMy question, is it possible to make a secure connection for docker engine from Pycharm without bound the remote docker to its host IP?\nEdit:\nPossible option so far\nWild card certificate:\nAs Jan Garaj suggestions, use a wild card certificate. Then connect each new server to a new subdomain. The wild card will be the same for each of them.\nPros: This suppose to do the trick\nCons: It requires to set up a new subdomain for each server\nSSH tunnle\nSet the docker socket to allow connection from localhost. Then each developer can set up an ssh tunnel from his computer to the remote machine. In Pycharm setup python interpeter to docker socket via localhost with the same port as the tunnel.","AnswerCount":1,"Available Count":1,"Score":0.1973753202,"is_accepted":false,"ViewCount":376,"Q_Id":55049331,"Users Score":1,"Answer":"Use (= buy or create\/install\/use own CA + generate) wildchar TLS certificate, which will cover all current servers and also any new servers.","Q_Score":3,"Tags":"python,python-3.x,docker,pycharm","A_Id":55050534,"CreationDate":"2019-03-07T17:07:00.000","Title":"How to setup a secure connection on remote docker on remote server with pycharm","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":1,"Web Development":0},{"Question":"I'm writing a scraper for the dark web. One step involves creating a torrc file in \/etc\/tor\/, which requires root access. To do this, I just run the python file with sudo (i.e. 'sudo python filename.py').\nHowever, I encountered an error with selenium:\nRunning Firefox as root in a regular user's session is not supported\nI googled the error and found solutions on how to bypass that. I would rather not run it as root if possible. \nHow can I run the first part of the code as root, but not the second part?","AnswerCount":2,"Available Count":2,"Score":0.0,"is_accepted":false,"ViewCount":106,"Q_Id":55108034,"Users Score":0,"Answer":"UNIX\/Linux does not blithely turn root privilege on and off quite so easily. You need to isolate the root portions in a separate script and run only those parts under root privilege. This is also basic system security: grant only the needed privilege for any function.","Q_Score":0,"Tags":"python,linux,selenium,sudo","A_Id":55108243,"CreationDate":"2019-03-11T18:17:00.000","Title":"How do I control what parts of a python script run as root","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I'm writing a scraper for the dark web. One step involves creating a torrc file in \/etc\/tor\/, which requires root access. To do this, I just run the python file with sudo (i.e. 'sudo python filename.py').\nHowever, I encountered an error with selenium:\nRunning Firefox as root in a regular user's session is not supported\nI googled the error and found solutions on how to bypass that. I would rather not run it as root if possible. \nHow can I run the first part of the code as root, but not the second part?","AnswerCount":2,"Available Count":2,"Score":0.0,"is_accepted":false,"ViewCount":106,"Q_Id":55108034,"Users Score":0,"Answer":"I\u2019m not familiar with Tor, but you can try to relax the rights on the tor file.\nRunning something similar to chmod +x \/etc\/tor\/your_file that will allow Firefox to use the file even if not run as a privileged user.","Q_Score":0,"Tags":"python,linux,selenium,sudo","A_Id":55108245,"CreationDate":"2019-03-11T18:17:00.000","Title":"How do I control what parts of a python script run as root","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I have a requirement in which I need to automate the start and stop of the AWS EC2 Instances (within Autoscaling group) daily. This is mainly to prevent cost. I have built a Python script to start and stop EC2 instances but it's not working properly as EC2 instances are within an Autoscaling group.\nDoes anybody know any solution for this?","AnswerCount":2,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":409,"Q_Id":55130327,"Users Score":0,"Answer":"What you need to do is automate the auto scaling parameters, for desired instances, min instances and max instances. Ideally, you want to change the desired instance amount. This will cause the auto scaler to terminate excessive instances, to meet the desired instance amount.","Q_Score":0,"Tags":"python,amazon-web-services,autoscaling","A_Id":55130351,"CreationDate":"2019-03-12T20:41:00.000","Title":"Start and stop AWS EC2 instance","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I've been looking into the hyperledger indy framework and I wanted to start to build an app to get started but I noticed that there's the sdk that uses Libindy but there's also the Libvcx that is on top of Libindy but I don't know which one to use since they both seem to do the same.","AnswerCount":2,"Available Count":2,"Score":0.0996679946,"is_accepted":false,"ViewCount":918,"Q_Id":55133748,"Users Score":1,"Answer":"The indy-sdk repository is the Indy software that enables building components (called agents) that can interact with an Indy ledger and with each other.\nIn 2019, at a \"Connect-a-thon\" in Utah, USA, developers from a variety of organizations gathered to demonstrate interoperability across a set of independently developed agent implementations. At that time, a further idea developed that led to the creation of Hyperledger Aries. What if we had agents that could use DIDs and verifiable credentials from multiple ecosystems? Aries is a toolkit designed for initiatives and solutions focused on creating, transmitting, storing and using verifiable digital credentials. At its core are protocols enabling connectivity between agents using secure messaging to exchange information.\nLibvcx is a c-callable library built on top of libindy that provides a high-level credential exchange protocol. It simplifies creation of agent applications and provides better agent-2-agent interoperability for Hyperledger Indy infrastructure.\nYou need LibVCX if you want to be interoperably exchange credentials with other apps and agents, in others words if you want to be comply with Aries protocol.\nIn this case LibVCX Agency can be used with mediator agency which enables asynchronous communication between 2 parties.","Q_Score":4,"Tags":"python,hyperledger-indy","A_Id":63579718,"CreationDate":"2019-03-13T03:08:00.000","Title":"What's the difference between hyperledger indy-sdk and Libvcx?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I've been looking into the hyperledger indy framework and I wanted to start to build an app to get started but I noticed that there's the sdk that uses Libindy but there's also the Libvcx that is on top of Libindy but I don't know which one to use since they both seem to do the same.","AnswerCount":2,"Available Count":2,"Score":1.2,"is_accepted":true,"ViewCount":918,"Q_Id":55133748,"Users Score":7,"Answer":"As you've said, LibVCX is built on top of LibIndy. \nLibIndy\nProvides low level API to work with credentials and proofs. It provides operations to create create credential requests, credentials, proofs. It also exposes operations for communication with Hyperldger Indy ledger. \nWhat Libindy doesn't handle is the credential exchange. If you write backend which issues credential and a mobile app which can request and receive credentials using Libindy, you'll have to come up with some communication protocol to do so. Is it gonna be HTTP? ZMQ? How are you going to format messages? This is what LibVCX does for you. You will also have to come up with solution how will you securely deliver messages and credentials from server to client when the client is offline.\nLibVCX\nLibVCX is one of several implementations of Hyperledger Aries specification. LibVCX is built on top of LibIndy and provides consumer with OOP-style API to manage connections, credentials, proofs, etc. It's written in Rust and has API Wrappers available for Python, Javascript, Java, iOS.\nLibVCX was designed with asynchronicity in mind. LibVCX assumes existence of so called \"Agency\" between the 2 parties communicating - a proxy which implements certain Indy communication protocol, receives and forwards messages. Therefore your backend server can now issue and send a credential to someone whom it has talked to days ago. The credential will be securely stored in the agency and the receiver can check whether there's any new messages\/credentials addressed for him at the agency.\nYou can think of agency as a sort of mail server. The message is stored there and the client can pull its messages\/credentials and decrypt them locally.\nWhat to use?\nIf you want to leverage tech in IndySDK perhaps for a specific use case and don't care about Aries, you can use vanilla libindy.\nIf you want to be interoperably exchange credentials with other apps and agents, you should comply with Aries protocol. LibVCX is one of the ways to achieve that.","Q_Score":4,"Tags":"python,hyperledger-indy","A_Id":55600304,"CreationDate":"2019-03-13T03:08:00.000","Title":"What's the difference between hyperledger indy-sdk and Libvcx?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I am using the Scrapy (Scrapy==1.6.0) library with Python3. I am wondering, where in the code does Scrapy actually do the HTML request? I want to set a breakpoint there so I can see exactly what headers \/ cookies \/ urls \/ and user agent is actually being passed.\nAlso, where exactly is the response received as well? Right now my spider is failing to find any pages, so I imagine I'm getting either a blank HTML document or a 403 error, however I have no idea where to look to confirm this.\nCan anyone familiar with the scrapy library point me to exactly where in code I can check these parameters?","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":41,"Q_Id":55148001,"Users Score":0,"Answer":"I believe you can check out scrapy\/core\/engine.py method _download.\nThough I'd suggest you make use of scrapy shell. It will let you execute particular request, inspect response, open response in browser to see what was received by Scrapy. Also with a bit more of tuning you can import your spider in your shell and call a particular method of your spider and put a breakpoint there.\nIf your spider fails to find any pages then the problem is likely to be with your spider, not the framework.","Q_Score":0,"Tags":"python,web-scraping,scrapy","A_Id":55148667,"CreationDate":"2019-03-13T17:31:00.000","Title":"Where does Scrapy actually do the html request?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I'm using Selenium to locate elements in a page. Is there any way to combine two methods together?\nExample:\nMethod 1: driver.find_elements_by_partial_link_text('html')\nMethod 2: driver.find_elements_by_class_name('iUh30')\nI will ideally like a method that finds elements that has both the partial link text and class name specified.","AnswerCount":2,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":469,"Q_Id":55159886,"Users Score":1,"Answer":"You can use xpath to combine both selectors:\ndriver.find_elements_by_xpath(\"\/\/*[@class='iUh30'][text()[contains(.,'html')]]\")\nThe \/\/* looks for any element with any tag. Might be <a>, might be <div>, <input>, anything. You can just change it to the desired tag.\nThe above find the element by exact class name. You can also use [contains(@class, 'partial_class')] to find elements by partial class.\nThe [text()[contains(.,'html')]] looks for elements which partial text is \"html\"","Q_Score":0,"Tags":"python-3.x,selenium-webdriver","A_Id":55160119,"CreationDate":"2019-03-14T10:14:00.000","Title":"Selenium locating elements by more than one method","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I am new at this part of web developing and was trying to figure out a way of creating a web app with the basic specifications as the example bellow:\n\nA user1 opens a page with a textbox (something where he can add text or so), and it will be modified as it decides to do it.\n\nIf the user1 has problems he can invite other user2 to help with the typing.\n\n\nThe user2 (when logged to the Channel\/Socket) will be able to modify that field and the modifications made will be show to the user1 in real time and vice versa.\n\n\nOr another example is a room on CodeAcademy:\n\nImagine that I am learning a new coding language, however, at middle of it I jeopardize it and had to ask for help.\n\nSo I go forward and ask help to another user. This user access the page through a WebSocket (or something related to that).\n\n\nThe user helps me changing my code and adding some comments at it in real time, and I also will be able to ask questions through it (real time communication)\n\n\nMy questions is: will I be able to developed certain app using Django Channels 2 and multiplexing? or better move to use NodeJS or something related to that?\nObs: I do have more experience working with python\/django, so it will more productive for me right know if could find a way working with this combo.","AnswerCount":1,"Available Count":1,"Score":0.1973753202,"is_accepted":false,"ViewCount":238,"Q_Id":55170669,"Users Score":1,"Answer":"This is definitely possible. They will be lots of possibilities, but I would recommend the following.\n\nHave a page with code on. The page has some websocket JS code that can connect to a Channels Consumer.\nThe JS does 2 simple things. When code is updated code on the screen, send a message to the Consumer, with the new text (you can optimize this later). When the socket receives a message, then replace the code on screen with the new code.\nIn your consumer, add your consumer to a channel group when connecting (the group will contain all of the consumers that are accessing the page)\nWhen a message is received, use group_send to send it to all the other consumers\nWhen your consumer callback function gets called, then send a message to your websocket","Q_Score":0,"Tags":"python,django,websocket,multiplexing","A_Id":55243910,"CreationDate":"2019-03-14T19:33:00.000","Title":"How does multiplexing in Django sockets work?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I am looking to keep the browser window open even after test execution.\nI would like to keep it open indefinitely.\nAs of now , as a work around I am just using sleep() to keep the window from closing.\nAny help would be much appreciated. Thank you !","AnswerCount":3,"Available Count":1,"Score":0.2605204458,"is_accepted":false,"ViewCount":4086,"Q_Id":55186282,"Users Score":4,"Answer":"Simple - do not call Close Browser at the end.","Q_Score":3,"Tags":"python,selenium,selenium-webdriver,robotframework","A_Id":55197099,"CreationDate":"2019-03-15T15:49:00.000","Title":"Is there a way to keep the browser window open after the test execution in RobotFramework?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"It is visible on the Audience Overview page as a pie chart of Percent New Visitors vs % Returning Visitors","AnswerCount":2,"Available Count":2,"Score":0.1973753202,"is_accepted":false,"ViewCount":728,"Q_Id":55201171,"Users Score":2,"Answer":"There is no metric 'Percent New Visitors' and 'Percent Returned Visitors'. But you can use ga:userType and calculate %.\nAnd also metric ga:percentNewSessions exists - the percentage of sessions by users who had never visited the property before. May be it will be helpful.","Q_Score":0,"Tags":"python,google-analytics,google-analytics-api,google-analytics-firebase","A_Id":55202208,"CreationDate":"2019-03-16T20:25:00.000","Title":"How to find the % returning visitors through the Google Analytics API","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"It is visible on the Audience Overview page as a pie chart of Percent New Visitors vs % Returning Visitors","AnswerCount":2,"Available Count":2,"Score":1.2,"is_accepted":true,"ViewCount":728,"Q_Id":55201171,"Users Score":0,"Answer":"Clarification: The pie chart actually shows % of Sessions by new and returning visitors, not % returning users. As zborovskaya stated, there is a ga:percentNewSessions which can be used (100 - %new = %returning session).","Q_Score":0,"Tags":"python,google-analytics,google-analytics-api,google-analytics-firebase","A_Id":55218898,"CreationDate":"2019-03-16T20:25:00.000","Title":"How to find the % returning visitors through the Google Analytics API","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"Is there any rationale behind the choice to use distinct methods for sending packets at L2 and L3 in Scapy? Could Scapy not just check if the packet being sent is L2 or higher?","AnswerCount":1,"Available Count":1,"Score":0.1973753202,"is_accepted":false,"ViewCount":93,"Q_Id":55205365,"Users Score":1,"Answer":"If Scapy were to detect whether the passed packet was L2 or L3, it would mean it has to hardcode a list of layers that are considered \u201cLayer 3\u201d and \u201cLayer 2\u201d.\nIf you make a custom layer 3, it wouldn\u2019t know in what category it falls, thus it leaves you the choice.\nAlso that\u2019s historical, dates back from 2008, you can\u2019t break it :-)","Q_Score":0,"Tags":"python,scapy","A_Id":55205923,"CreationDate":"2019-03-17T08:52:00.000","Title":"Why uses Scapy different methods to distinguish L2 and L3 packets?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I am trying to grab an ip address from a string and facing an issue.please help.\ninet addr:11.11.11.11 Bcast:11.11.11.111 Mask:111.111.11.1.\nThis is the string I have and I need ip address next to addr:\nI have tried the following code and failed to do in python: \nip = re.findall(r'(?:\\d{1,3}\\.)+(?:\\d{1,3})', line) and get index 0 item.\nResult : This is actually giving me nothing in return","AnswerCount":2,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":250,"Q_Id":55228136,"Users Score":0,"Answer":"Your REGEX could be more specific, I think you could use something like :\naddr:(?<ip>\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}\\.\\d{1,3})\nIn python:\nmatch = re.match(r'addr:(?<ip>\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}\\.\\d{1,3})', line)\nYou can then access the ip group by calling match.group('ip').","Q_Score":2,"Tags":"python,regex,python-3.x","A_Id":55228210,"CreationDate":"2019-03-18T18:46:00.000","Title":"How to grab first ip address from a string","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":1,"System Administration and DevOps":0,"Web Development":0},{"Question":"I need to find the latest git issue number for a repository using python for all users rather than an authenticated user.\nIs there any way i could do this? either through the API or any library?\nI looked at the github docs and from my understanding it is not possible to list issues for all users but only authenticated users.\nI looked at pygithub where all i could find was how to create an issue through the library","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":249,"Q_Id":55236480,"Users Score":0,"Answer":"@Kevin Vizalil\nYou can use the Github API's to get the list of issues or single issue\nplease check https:\/\/developer.github.com\/v3\/issues\/#list-issues\nedit:\ne.g. https:\/\/api.github.com\/repos\/vmg\/redcarpet\/issues?sort=created&direction=desc","Q_Score":0,"Tags":"python,git,github,request,github-api","A_Id":55236763,"CreationDate":"2019-03-19T08:30:00.000","Title":"How to find the latest git issue number on github using python?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I am working on a web automation project. I need to be able to pull pages, assess data, and be able to interact with the page (e.g. login, enter values, and post to the site.) As a derivative of the logins, I think I will need something that will allow me to remain logged in given a credential (e.g. store the credential or cookies.)\nI've already used UrlLib & Requests libraries to pull files and the pages themselves.\nI am trying to decide on the best Python library for the task.\nAny suggestions would be highly appreciated.\nthank you!","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":1516,"Q_Id":55285021,"Users Score":1,"Answer":"@n1c9\n\nIf you can reliably recreate the HTTP requests being used to authenticate logins and speed is important, urllib\/requests for making those HTTP requests and beautifulsoup for parsing the HTML responses would be best. Otherwise, Selenium is where you'll have the most luck. Let me know if you want more details. \n\nLooks like Selenium is the right answer.","Q_Score":1,"Tags":"python,beautifulsoup,urllib3,python-requests-html","A_Id":55291345,"CreationDate":"2019-03-21T16:24:00.000","Title":"beautiful soup vs selenium vs urllib","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I have a contanarized flask app with external db, that logs users on other site using selenium. Everything work perfectly in localhost. I want to deploy this app using containers and found selenium container with google chrome within could make the job. And my question is: how to execute scripts\/methods from flask container in selenium container? I tried to find some helpful info, but I didn't find anything. \nShould I make an API call from selenium container to flask container? Is it the way or maybe something different?","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":115,"Q_Id":55299697,"Users Score":1,"Answer":"As far as i understood, you are trying to take your local implementation, which runs on your pc and put it into two different docker containers. Then you want to make a call from the selenium container to your container containing the flask script which connects to your database.\nIn this case, you can think of your containers like two different computers. You can tell docker do create an internal network between these two containers and send the request via API call, like you suggested. But you are not limited to this approach, you can use any technique, that works for two computers to exchange commands.","Q_Score":0,"Tags":"python,selenium,docker,flask,google-chrome-headless","A_Id":55302692,"CreationDate":"2019-03-22T12:32:00.000","Title":"How to execute script from container within another container?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"In Windows I need to send an NBNS name query packet (which uses UDP protocol), and I need to send 255 packets and get an answer for each. With Scapy it takes a year, so I wanted to know if there is a way to speed it up or maybe use sockets instead?","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":138,"Q_Id":55309306,"Users Score":0,"Answer":"You could use two threads, one that is sending packets using a conf.L3socket() socket from scapy and another thread that is receiving packets using sniff().","Q_Score":1,"Tags":"python-2.7,sockets,networking,scapy","A_Id":72091366,"CreationDate":"2019-03-23T00:07:00.000","Title":"Scapy send packets speed up in Windows","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":1,"Web Development":0},{"Question":"I've got an AWS Lambda, written in python and behind API Gateway, which makes a network request out to a third party.\nShortly after that request, a separate request will be made by the third party to the URL of my choosing - I need to get a hold of the body of that request and return it in the response from my Lambda.\nIf perhaps I have the third party send to a second Lambda, how can I hold the first Lambda open, waiting for an event from the second Lambda?","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":373,"Q_Id":55325063,"Users Score":0,"Answer":"(In the hope that someone offers a better idea...)\nWhat I'm currently intending to do is stand up a redis (Elasticache) cluster. Lambda A will send out a request with an X-Request-ID and then setup a redis pubsub().subscribe(X-Request-ID). Lambda B will receive the response and do a redis pubsub().publish(X-Request-ID, response). Lambda A will then return the response or, if not received in time, timeout.\nInelegant but I think it works.","Q_Score":0,"Tags":"python,amazon-web-services,asynchronous,aws-lambda","A_Id":56402127,"CreationDate":"2019-03-24T14:54:00.000","Title":"AWS Lambda (in python) waiting for asynchronous event","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I am working on a task where I need to web scrape boxofficemojo site.\nI coded everything and it is working perfectly fine in my local machine.\nThere are around 19000 urls that I need to scrape. As it is obviously a time consuming process, I don't want to run it on my local machine. Instead I want to run it on an aws ec2 instance.\nThe ec2 instance is Ubuntu 18.04. I have verified python versions and the libraries used in script are present or not and everything.\nHowever, if I try\nrequests.get('http:\/\/www.boxofficemojo.com') ,\nit is giving me 503 response. If I print the response text, it is saying We are in process of updating site right now. But the same thing is working in my local machine.\nWhy am I getting this wierd behaviour in ec2 instance.\nI tried testing internet connection inside the ec2 instance by issuing ping command . It's working fine.","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":101,"Q_Id":55328681,"Users Score":4,"Answer":"There are public sites, and public api's that specifically block calls from ec2 instances (and probably other cloud providers). \nIt's not impossible that some of the sites you are trying to scrape, simply blacklist ec2 instances ip ranges to cut down on the 'bots' that are eating up their resources ... I have come across this several times, for several sites.\nThe NBA stats api is one example I am familiar with, but I have come across others as well - the sites you are scraping may be some of them as well.","Q_Score":0,"Tags":"python,amazon-web-services,amazon-ec2,web-scraping,python-requests","A_Id":55328905,"CreationDate":"2019-03-24T21:19:00.000","Title":"Why am I getting different http responses from different locations?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I am working on a Project in Raspberry Pi Zero, with apache as web server. The website is in PHP and based on user inputs, it updates a XML file in the server.\nThere is also a python program program running parallely with the web server. This python program contantsly reads the XML and grabs the values from XML, stores them locally and checks for changes in them and if there is any changes it performs some UART communications with external devices, sometimes based on these external communication from the devices, python also updates the XML.\nPython reads the XML every 2 seconds, and the problem is sometimes, when the python is doing the read operation, if the user prodives input and if PHP inserts the new value to the same XML, python crashes. The client wants to reduce the 2 second delay to .1 second, which means Python will be reading fastly and any changes from PHP will crash it.\nIs there a way to get somekind of file lock between python and PHP so that, when Python is reading or writing PHP waits and if PHP is writing Python waits. Priority goes to Python above PHP.","AnswerCount":2,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":65,"Q_Id":55328712,"Users Score":0,"Answer":"Would be better to have an api call first which would suggest if the data is being changed currently and when was the last data changed.\nThis way you can avoid the crash which is happening due to sharing of the resource","Q_Score":0,"Tags":"php,python,xml,file-handling,file-locking","A_Id":55329011,"CreationDate":"2019-03-24T21:23:00.000","Title":"Handling a common XML file between PHP and Python","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I'm trying to use BeautifulSoup method inside my module method. If I import as from BeautifulSoup import BeautifulSoup that is working in while running python through CMD\n How can I import BeautifulSoup3 library that I installed in python 2.7 into my OpenERP module.","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":36,"Q_Id":55341833,"Users Score":0,"Answer":"Just copy and paste the BeautifullSoup.py file from python27\\Lib\\site-packages\\BeautifullSoup.py into the folder path openERP7>Server>server and it will do the trick.","Q_Score":1,"Tags":"python-2.7,beautifulsoup,openerp-7","A_Id":55350713,"CreationDate":"2019-03-25T16:00:00.000","Title":"How to import BeautifulSoup into a python method OpenERP 7 module","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I\u2019m looking into web scraping \/crawling possibilities and have been reading up on the Scrapy program. I was wondering if anyone knows if it\u2019s possible to input instructions into the script so that once it\u2019s visited the url it can then choose pre-selected dates from a calendar on the website. ?\nEnd result is for this to be used for price comparisons on sites such as Trivago. I\u2019m hoping I can get the program to select certain criteria such as dates once on the website like a human would.\nThanks,\nAlex","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":66,"Q_Id":55361023,"Users Score":2,"Answer":"In theory for a website like Trivago you can use the URL to set the dates you want to query but you will need to research user agents and proxies because otherwise your IP will get blacklisted really fast.","Q_Score":1,"Tags":"python,scrapy,web-crawler,data-extraction","A_Id":55368161,"CreationDate":"2019-03-26T15:37:00.000","Title":"Scrapy and possibilities available","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I'm have bandwidth data which identifies protocol usage by tonnage and hour. Based on the protocols, you can tell when something is just connect vs actually being used (1000 bits compared to million or billions of bits) in that hour for that specific protocol. The problem is When looking at each protocol, they are all heavily right skewed. Where 80% of the records are the just connected or what I'm calling \"noise. \nThe task I have is to separate out this noise and focus on only when the protocol is actually being used. My classmates are all just doing this manually and removing at a low threshold. I was hoping there was a way to automate this and using statistics instead of just picking a threshold that \"looks good.\" We have something like 30 different protocols each with a different amount of bits which would represent \"noise\" i.e. a download prototypical might have 1000 bits where a messaging app might have 75 bits when they are connected but not in full use. Similarly they will have different means and gaps between i.e. download mean is 215,000,000 and messaging is 5,000,000. There isn't any set pattern between them.\nAlso this \"noise\" has many connections but only accounts for 1-3% of the total bandwidth being used, this is why we are tasked with identify actual usage vs passive usage.\nI don't want any actual code, as I'd like to practice with the implementation and solution building myself. But the logic, process, or name of a statistical method would be very helpful.","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":36,"Q_Id":55363520,"Users Score":0,"Answer":"Do you have labeled examples, and do you have other data besides the bandwidth? One way to do this would be to train some kind of ML classifier if you have a decent amount of data where you know it's either in use or not in use. If you have enough data you also might be able to do this unsupervised. For a start a simple Naive Bayes classifier works well for binary solutions. As you may be away, NB was the original bases for spam detection (is it spam or not). So your case of is it noise or not should also work, but you will get more robust results if you have other data in addition to the bandwidth to train on. Also, I am wondering if there isn't a way to improve the title of your post so that it communicates your question more quickly.","Q_Score":0,"Tags":"python,r,statistics","A_Id":55462001,"CreationDate":"2019-03-26T17:57:00.000","Title":"Determining \"noise\" in bandwidth data","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I was working on a decentralized python chat room. I was just wondering, if you don't open up a port, you can't receive messages. But, if you open up a port, you are vulnerable to getting hacked or something. So, is there a way to communicate with Python between a server and a client without opening up ports?","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":39,"Q_Id":55399309,"Users Score":0,"Answer":"At least one of the client or the server should have an open port (typically, the server). As soon as a TCP connection is established (typically, by the client), a random (= chosen by the operating system) port will be used by the client to be reachable by the server.","Q_Score":0,"Tags":"python,sockets","A_Id":55399380,"CreationDate":"2019-03-28T13:52:00.000","Title":"Is there a way to communicate between server and client in Python without opening up ports?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I am trying to crawl the football data from the whoscored.com, the website has incapsula web oriented security which is not letting me crawl. Initially, I tried to give user_agent and changed the header then it worked but that's only for the first page. As I need to crawl some other parts of the website it keeps blocking me to request the website. It's getting exhausting now since the blocking time has been increasing.\nIs there anybody who could suggest something to bypass the incapsula security mechanisms.\nI need data for study purposes.\nI have gone through all the old the previous question asked about this topic. but it does not help.\nTools. anaconda, language python, and library beautiful soup\nThanks.","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":343,"Q_Id":55404628,"Users Score":0,"Answer":"if you mimic browser headers and appropiate time between requests, it will probably work\nlook at your request headers and that of your browser","Q_Score":0,"Tags":"python,data-science","A_Id":55663189,"CreationDate":"2019-03-28T18:32:00.000","Title":"By passing incapsula security- python","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I just started studying flask and built a toy website to see how well I am doing. I have a flask website built in python 3.6 and I have tested it on my windows computer and everything goes very well. Now I want to host the website on an ubuntu ec2 instance. But first, I am testing if everything runs well on my ec2 instance and am stuck at trying to access port 5000 on my ec2 instance My app is currently serving on port 127.0.0.1:5000 of my linux server. I have tried to connect to my.ec2.public.ip:5000 and my.ec2.private.ip:5000 with no success. Could someone help me? Thanks.","AnswerCount":2,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":287,"Q_Id":55426401,"Users Score":0,"Answer":"By default a new AWS instance would not allow port 5000 to be accessed, so you will need to modify the security group to allow access on that port. You do this thru the AWS console.","Q_Score":0,"Tags":"amazon-ec2,python-3.6,flask-sqlalchemy,ubuntu-18.04","A_Id":55426431,"CreationDate":"2019-03-29T22:51:00.000","Title":"How do I access 127:.0.0.1:5000 on Ubuntu Server 16.04 LTS (HVM), SSD Volume Type","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":1,"Web Development":1},{"Question":"We are trying to get the owned games of a lot of users but our problem is that after a while the API call limit (100.000 a day) kicks in and we stop getting results. \nWe use 'IPlayerService\/GetOwnedGames\/v0001\/?key=APIKEY&steamid=STEAMID' in our call and it works for the first entries. \nThere are several other queries like the GetPlayerSummaries query which take multiple Steam IDs, but according to the documentation, this one only takes one. \nIs there any other way to combine\/ merge our queries? We are using Python and the urllib.request library to create the request.","AnswerCount":2,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":807,"Q_Id":55505061,"Users Score":0,"Answer":"Depending on the payload of the requests you have the following possibilities:\n\nif each request brings only the newest updates, you could serialize the steam ID's when you get the response that you've hit the daily limit\nif you have the ability to control via the request payload what data you receive, you could go for a multithreaded \/ multiprocessing approach that consume the request queries and the steam ID's from a couple of shared resources","Q_Score":0,"Tags":"python,urllib,steam-web-api","A_Id":55505148,"CreationDate":"2019-04-03T22:08:00.000","Title":"Steam Web API GetOwnedGames multiple SteamIDs","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I'm trying to import xml.dom.minidom in order to read and edit an xml file but am getting an error.\nimport xml.dom.minidom\nI get the error message:\nModuleNotFoundError: No module named 'xml.dom'; 'xml' is not a package","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":253,"Q_Id":55513629,"Users Score":2,"Answer":"This may be due to a file name issue (such as xml.py). There may be a conflict with the file name when compiling. Rename it to solve the issue.","Q_Score":0,"Tags":"python,xml","A_Id":55513753,"CreationDate":"2019-04-04T10:27:00.000","Title":"How can I import xml.dom.mindom","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I've been getting a lot of spam emails from various colleges. I wrote a simple Python program to go through my emails, and find all the ones whose sender ends in .edu.\nThen, to delete it, I though I was supposed to add the \\\\Deleted flag to those emails using: Gmail.store(id, \"+FLAGS\", \"\\\\Deleted\"). This did not work, and some of them didn't even disappear from the inbox. \nAfter more research, I found that instead I had to use Gmail.store(id, \"+X-GM-LABELS\", \"\\\\Trash\"). \nSo I updated the program to use that, but now it doesn't see any of the emails that I previously added the \\\\Deleted flag to.\nIs there any way I can reset all those emails and then trash them afterward?","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":371,"Q_Id":55525410,"Users Score":1,"Answer":"They should be in your All Mail folder. Use the WebUI and search for them and trash them, or select the \"[Gmail]\\All Mail\" folder (watch out for localization, this can change name for non-English users).","Q_Score":0,"Tags":"python,email,gmail,imaplib","A_Id":55544626,"CreationDate":"2019-04-04T21:47:00.000","Title":"Python imaplib recover emails with \\\\Deleted flag and move to trash","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I recent upgraded web2py and starting using username=True, the form returned via auth\/profile no longer contains the user email address.\nHow can a user change email address under the standard api?","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":93,"Q_Id":55547054,"Users Score":1,"Answer":"With or without username=True, the email address is not editable via the current Auth API (this was changed about a year ago, presumably for security reasons). For now, you'll have to implement your own email change functionality. For extra security, you might want to require password verification, and maybe send a verification email to the new address (and possibly a notification to the old address upon completion of the change).","Q_Score":1,"Tags":"python,web2py","A_Id":55550260,"CreationDate":"2019-04-06T08:00:00.000","Title":"How to change email address of user in web2py using standard auth api with username=True","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"When I try to build chromium on Windows10, a python error occurs,\nit says \"ImportError: No module named win32file\". How to savle this?","AnswerCount":1,"Available Count":1,"Score":1.0,"is_accepted":false,"ViewCount":13868,"Q_Id":55551188,"Users Score":20,"Answer":"My solution is: python -m pip install pywin32.\nThen you will see module win32file in the path of C:\/python27\/Lib\/site-packages\/win32file.pyd","Q_Score":11,"Tags":"python-2.7","A_Id":55551200,"CreationDate":"2019-04-06T16:40:00.000","Title":"python ImportError: No module named win32file","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":1,"System Administration and DevOps":0,"Web Development":0},{"Question":"I got a bit lost in the boto3 API details and struggle to find an example to access an S3 bucket using python. I need to use an existing pem file rather than the typical access and secret key. Works fine using an ftp client but I need to get it running also with python.\nAnyone that can point me in the right direction (or suggest alternatives using python)","AnswerCount":2,"Available Count":1,"Score":0.2913126125,"is_accepted":false,"ViewCount":1571,"Q_Id":55552553,"Users Score":3,"Answer":"This is not possible.\nThe types of authentication used on AWS are:\n\nUsername and password associated with an IAM User, used to login to the web management console.\nAccess Key and Secret Key associated with an IAM User, used to make API calls to AWS services\nPrivate Key (PPK\/PEM) used to login to Linux instances.\n\nPrivate Keys are used to login to an operating system and are unrelated to AWS. They are a standard means of accessing Linux systems and identify users defined on the computer itself rather than on AWS.\nAPI calls to AWS require the Access Key and Secret Key and have no relationship to PPK\/PEM keys.","Q_Score":2,"Tags":"python,amazon-web-services,amazon-s3","A_Id":55554101,"CreationDate":"2019-04-06T19:10:00.000","Title":"How do I connect to an Amazon S3 bucket with a pem file using python (boto3)","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I often encounter elements which I cannot right click to inspect their xpath or css. \nI want to ask what other ways exist to click on those elements ?","AnswerCount":4,"Available Count":3,"Score":0.0,"is_accepted":false,"ViewCount":1061,"Q_Id":55556644,"Users Score":0,"Answer":"Control + Shift + C or F12 will open the dev tools for you, you can then click on the cursor mode on your browser.","Q_Score":1,"Tags":"python,selenium,google-chrome,xpath,google-chrome-devtools","A_Id":55558250,"CreationDate":"2019-04-07T07:32:00.000","Title":"How to interact with elements those cannot be inspected through css\/xpath within google-chrome-devtools","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I often encounter elements which I cannot right click to inspect their xpath or css. \nI want to ask what other ways exist to click on those elements ?","AnswerCount":4,"Available Count":3,"Score":0.049958375,"is_accepted":false,"ViewCount":1061,"Q_Id":55556644,"Users Score":1,"Answer":"You can use Ctrl + Shift + C, it will open the devtools with selecting an element to inspect enabled. Just move the mouse cursor to the element and click, it will scroll the html view in devtools to the correct place.\nAlternatively, you can press F12 and toggle the selecting an element to inspect button (top left corner of the devtools).","Q_Score":1,"Tags":"python,selenium,google-chrome,xpath,google-chrome-devtools","A_Id":55558180,"CreationDate":"2019-04-07T07:32:00.000","Title":"How to interact with elements those cannot be inspected through css\/xpath within google-chrome-devtools","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I often encounter elements which I cannot right click to inspect their xpath or css. \nI want to ask what other ways exist to click on those elements ?","AnswerCount":4,"Available Count":3,"Score":0.0,"is_accepted":false,"ViewCount":1061,"Q_Id":55556644,"Users Score":0,"Answer":"If you want to get elements locator but the right click doesn't work then try the following. \nFirst, Open the dev tools window by pressing Ctrl+Shift+I. \nIf that doesn't work then first open the dev tool then load the page. \nAfter opening the dev tools click on \"select element\" tool, the printer icon at the left of the dev tool. You can directly get this tool by pressing Ctrl+Shift+C.\nThen hover on the element that you want to get the locator. The element will be selected in the DOM in the elements tab. Right click on the elements in the DOM. \nThen from the context menu go to copy -> copy selector for CSS selector or copy XPath for XPath. \nNow you have got the locator of that element in the clipboard.","Q_Score":1,"Tags":"python,selenium,google-chrome,xpath,google-chrome-devtools","A_Id":55556724,"CreationDate":"2019-04-07T07:32:00.000","Title":"How to interact with elements those cannot be inspected through css\/xpath within google-chrome-devtools","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"using pip I installed new version of HtmlTestRunner in python 3.6, but while I run python file through command prompt its throws error. \n\nTraceback (most recent call last):\n File \"seleniumUnitTest.py\", line 3, in \n import HtmlTestRunner\n ModuleNotFoundError: No module named 'HtmlTestRunner'","AnswerCount":1,"Available Count":1,"Score":0.3799489623,"is_accepted":false,"ViewCount":1410,"Q_Id":55565407,"Users Score":2,"Answer":"Using python program.py instead of python3 program.py fixed my problem in Windows 10.","Q_Score":2,"Tags":"python,python-3.x,selenium,selenium-webdriver","A_Id":65629114,"CreationDate":"2019-04-08T02:11:00.000","Title":"Already installed Html test runner but it shows error \"ModuleNotFoundError: No module named 'HtmlTestRunner' \"","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I am new to Machine Learning. I am trying to build a classifier that classifies the text as having a url or not having a url. The data is not labelled. I just have textual data. I don't know how to proceed with it. Any help or examples is appreciated.","AnswerCount":2,"Available Count":1,"Score":0.0996679946,"is_accepted":false,"ViewCount":1580,"Q_Id":55576378,"Users Score":1,"Answer":"Since it's text, you can use bag of words technique to create vectors.\n\nYou can use cosine similarity to cluster the common type text.\nThen use classifier, which would depend on number of clusters.\nThis way you have a labeled training set. \n\nIf you have two cluster, binary classifier like logistic regression would work. \nIf you have multiple classes, you need to train model based on multinomial logistic regression\nor train multiple logistic models using One vs Rest technique.\n\nLastly, you can test your model using k-fold cross validation.","Q_Score":0,"Tags":"python,machine-learning,classification","A_Id":55576706,"CreationDate":"2019-04-08T15:02:00.000","Title":"How to classify unlabelled data?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"We have created website online exam portal in nodejs and mongodb hosted on amazon aws. Problem is the descriptive answer evaluation code is in python. So is there any way we can combine both. Run python code in nodejs. We have used express.js for nodejs and mongoose framework for mongodb.","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":37,"Q_Id":55597443,"Users Score":0,"Answer":"You can use lambda for your python, then hit it from nodejs app, and return the response for your nodejs app.","Q_Score":0,"Tags":"python,node.js,mongodb","A_Id":55597496,"CreationDate":"2019-04-09T16:36:00.000","Title":"Is there any way to use python code in nodejs project","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":1,"System Administration and DevOps":0,"Web Development":1},{"Question":"I hope you are all having an amazing day. So I am working on a project using Python. The script's job is to automate actions and tasks on a social media platform via http requests. As of now, one instance of this script access one user account. Now, I want to create a website where I can let users register, enter their credentials to the social media platform and run an instance of this script to perform the automation tasks. I've thought about creating a new process of this script every time a new user has register, but this doesn't seem efficient. Also though about using threads, but also does not seem reasonable. Especially if there are 10,000 users registering. What is the best way to do this? How can I scale? Thank you guys so much in advance.","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":292,"Q_Id":55602143,"Users Score":1,"Answer":"What is the nature of the tasks that you're running?\nAre the tasks simply jobs that run at a scheduled time of day, or every X minutes? For this, you could have your Web application register cronjobs or similar, and each cronjob can spawn an instance of your script, which I assume is short-running, to carry out a the automated task one user at a time. If the exact timing of the script doesn't matter then you could scatter the running of these scripts throughout the day, on seperate machines if need be.\nThe above approach probably won't scale well to 10,000 users, and you will need something more robust, especially if the script is something that needs to run continuously (e.g. you are polling some data from Facebook and need to react to its changes). If it's a lot of communication per user, then you could consider using a producer-consumer model, where a bunch of producer scripts (which run continously) issue work requests into a global queue that a bunch of consumer scripts poll and carry out. You could also load balance such consumers and producers across multiple machines.\nOf course, you would definitely want to squeeze out some parallelism from the extra cores of your machines by carrying out this work on multiple threads or processes. You could do this quite easily in Python using the multiprocessing module.","Q_Score":0,"Tags":"python,web-services,automation,httprequest,scale","A_Id":55603470,"CreationDate":"2019-04-09T22:21:00.000","Title":"How to serve a continuously running python script to multiple users (Social Media Bot)","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I have updated and installed Microsoft Power BI Desktop Report Server (64-bit).\nPrior to this I have already installed python 3.6 in machine. I am trying run python script on Power BI. but I don't see any option.\nI know this can be enabled by selecting \"Python Script\" in Preview Feature.\nBut the problem is I cannot see \"Preview Feature\" available in Options.\nCan anyone help me how I can execute python script there.","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":775,"Q_Id":55613905,"Users Score":0,"Answer":"The Preview Feature is not available with Report server version. We need to have Power BI Desktop Cloud. Also version should be August 2018 or above.","Q_Score":0,"Tags":"python,powerbi,visualization,powerbi-desktop,preview-feature","A_Id":55757798,"CreationDate":"2019-04-10T13:32:00.000","Title":"Preview feature not available in Power BI destop Report server Jan 2019 version","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I am using getstream.io to create feeds. The user can follow feeds and add reaction like and comments. If a user adds a comment on feed and another wants to reply on the comment then how I can achieve this and also retrieve all reply on the comment.","AnswerCount":2,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":583,"Q_Id":55644656,"Users Score":0,"Answer":"you can add the child reaction by using reaction_id","Q_Score":1,"Tags":"python,getstream-io","A_Id":55813711,"CreationDate":"2019-04-12T04:46:00.000","Title":"How to add reply(child comments) to comments on feed in getstream.io python","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"Good afternoon everyone,\nFor a school project I will make a RC car using a c8051 microcontroller and to send the uart data to it I'm using a ESP32 so that I can display a webpage so that the user choose the direction of the car. I've spent a lot of time on micropython doc's page and tutorial for TCP sockets and I see in every one of them that to check if the webpage was requested they use something like:\nIf(request==6):\nAnd I can't figure out why 6, what that represents??\nI appreciate any help given.","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":217,"Q_Id":55690355,"Users Score":0,"Answer":"The answer found in the comment section of the link given\n\"In the while loop, after receiving a request, we need to check if the request contains the \u2018\/?led=on\u2019 or \u2018\/?led=on\u2019 expressions. For that, we can apply the find() method on the request variable. the find() method returns the lowest index of the substring we are looking for.\nBecause the substrings we are looking for are always on index 6, we can add an if statement to detect the content of the request. If the led_on variable is equal to 6, we know we\u2019ve received a request on the \/?led=on URL and we turn the LED on.\nIf the led_off variable is equal to 6, we\u2019ve received a request on the \/?led=off URL and we turn the LED off.\"","Q_Score":0,"Tags":"microcontroller,esp32,micropython","A_Id":55703438,"CreationDate":"2019-04-15T13:32:00.000","Title":"Esp32 micropython webserver TCP socket to check http get request","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"GRPC server does queue the requests and serve them based on the maxWorker configuration which is passed when the sever starts up. How do I print the metric- number of items in the queue .? Essentially, I would like to keep track of the numbers of requests in waiting state.","AnswerCount":2,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":2149,"Q_Id":55699452,"Users Score":0,"Answer":"You can pass your own executor to serverBuilder.executor(). Note that you are then responsible for shutting down the executor service after the server has terminated.","Q_Score":3,"Tags":"grpc,grpc-java,grpc-python","A_Id":55712069,"CreationDate":"2019-04-16T01:49:00.000","Title":"keep track of the request queue in grpc server","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":1,"Web Development":0},{"Question":"I am looking for a way to use web drivers (ChromeDriver, IEDriver, GeckoDriver etc., all together) with my native python app such that, the app will figure out the browser, and choose the driver accordingly, and will do some actions (like click an element or get data). I want to do the task in python without using selenium","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":618,"Q_Id":55707406,"Users Score":1,"Answer":"It would theoretically be possible to use the driver executables without Selenium. All WebDriver implementations operate using the same mechanism. That mechanism is starting an HTTP server running locally, and listening on a well-known set of end points (URLs) for HTTP requests containing well-defined JSON bodies.\nIt\u2019s fully possible to even start a WebDriver implementation like IEDriverServer.exe, geckodriver, or chromedriver and automate the browser even using a tool like cURL, so using a Python HTTP client library and JSON parser is certainly in the realm of the possible. However, doing so requires a fairly thorough understanding of the protocol used in communicating with the driver, and gaining that understanding is distinctly non-trivial. In fact, use of that protocol without needing to know the details of it is one of the very reasons for Selenium\u2019s existence.\nWhile what you say you want to do is possible, I would by no means call it recommended. Attempting to go down that road seems like a lot of effort for a very marginal benefit, when you consider you need to worry about lifetime of the executable process you spawn, proper formatting of the HTTP request bodies, and handling all of the potential responses from the remote end. You\u2019d be reinventing a whole lot of things that Selenium already does for you. Your question doesn\u2019t show any indication of why you don\u2019t want to use Selenium, so it\u2019s difficult to provide any further guidance as to alternatives or mitigations to the things you find objectionable about it.","Q_Score":1,"Tags":"python-3.x,automation,webdriver,selenium-chromedriver","A_Id":55767473,"CreationDate":"2019-04-16T11:45:00.000","Title":"How can we use ChromeDriver, IEDriver, GeckoDriver without selenium in python?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"Uing BAsh console i installed Steem Module for Python3.6\nand everything seems right but when i try to run my python script in Pythonanywhere it says there is problem connecting with api.steemit.com\nbut the script works fine in my PC. \ncode \n import steem\n from steem import Steem\n s=Steem()\n s.get_account_history(your_ac, index_from=-1, limit=12)\nlog is here. Is there any way to solve this issue?\nWARNING:root:Retry in 1s -- MaxRetryError: HTTPSConnectionPool(host='api.steemit.com', port=443): Max retries exceeded with url: \/ (Caused by NewC\nonnectionError(': Failed to establish a new connection: [Errno 111] Connectio\nn refused',))\nWARNING:urllib3.connectionpool:Retrying (Retry(total=19, connect=None, read=None, redirect=0, status=None)) after connection broken by 'NewConnect\nionError(': Failed to establish a new connection: [Errno 111] Connection refu\nsed',)': \/","AnswerCount":1,"Available Count":1,"Score":0.537049567,"is_accepted":false,"ViewCount":159,"Q_Id":55779701,"Users Score":3,"Answer":"External connections for free accounts on PythonAnywhere are passed through a proxy. The error you're getting looks like the library you're using is not using the proxy. Check the docs for the library to see how to configure it to use a proxy. If it does not support it, contact the authors to see if they can add support for proxies.","Q_Score":1,"Tags":"https,pythonanywhere,connection-refused","A_Id":55797086,"CreationDate":"2019-04-21T03:46:00.000","Title":"Connection Refused in pythonAnwhere","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I am trying to extract audio track information (specifically language of the audio) from a live stream that i will be playing with libVLC . Is it possible to do this in javascript or python without writing new code for a wrapper?","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":74,"Q_Id":55780457,"Users Score":0,"Answer":"Not sure about javascript, but the Python wrapper will let you do this.","Q_Score":0,"Tags":"javascript,python,node.js,libvlc","A_Id":55788853,"CreationDate":"2019-04-21T06:29:00.000","Title":"Is it possible to get audio track data from libVLC by using javascript\/python","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I want to access files stored in my s3 buckets running python script in ec2 machine . Boto3 python packages facilitates this but is there some other way files stored in the s3 bucket could be accessed simply providing url of s3 bucket rather than importing boto3 in python program ?","AnswerCount":2,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":1484,"Q_Id":55783565,"Users Score":0,"Answer":"AWS CLI and shell scripts instead of writing a python application and installing boto3 is what I recently did. I worried about python version being installed and didn't want to install boto3, we were using a variant of an Amazon Linux which all will have AWS CLI and will also have installed jq command tool is a great way to get around installing boto3. It can be supplement with python as well. I decided to go with shell scripting, because my program was relatively simple.","Q_Score":0,"Tags":"python,amazon-web-services,amazon-s3,amazon-ec2","A_Id":55786037,"CreationDate":"2019-04-21T14:21:00.000","Title":"How to access files stored in s3 bucket from python program without use of boto3?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I have roughly 80tb of images hosted in an S3 bucket which I need to send to an API for image classification. Once the images are classified, the API will forward the results to another endpoint.\nCurrently, I am thinking of using boto to interact with S3 and perhaps Apache airflow to download these images in batches and forward them to the classification API, which will forward the results of the classification to a web app for display.\nIn the future I want to automatically send any new image added to the S3 bucket to the API for classification. To achieve this I am hoping to use AWS lambda and S3 notifications to trigger this function.\nWould this be the best practice for such a solution?\nThank you.","AnswerCount":1,"Available Count":1,"Score":0.1973753202,"is_accepted":false,"ViewCount":159,"Q_Id":55795614,"Users Score":1,"Answer":"For your future scenarios, yes, that approach would be sensible:\n\nConfigure Amazon S3 Events to trigger an AWS Lambda function when a new object is created\nThe Lambda function can download the object (to \/tmp\/) and call the remote API\nMake sure the Lambda function deletes the temporary file before exiting since the Lambda container might be reused and there is a 500MB storage limit\n\nPlease note that the Lambda function will trigger on a single object, rather than in batches.","Q_Score":0,"Tags":"python,amazon-web-services,amazon-s3,aws-lambda,airflow","A_Id":55800992,"CreationDate":"2019-04-22T14:01:00.000","Title":"Large Scale Processing of S3 Images","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I have an application which handles websocket and http requests for some basic operations and consuming push data over sockets. Nothing is very computation intensive. Some file tailing, occasional file read \/ write is all that it has to do with heavy processing currently. I want to deploy this to Linux. I have no static files to handle \nCan a tornado application handle 50-100 websocket and http clients without needing ngnix ? I don't want to use another server for this. How many clients can it handle on its own ?\nEverywhere I search I get ngnix, and I don't want to involve it","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":132,"Q_Id":55801051,"Users Score":1,"Answer":"Yes, Tornado can easily handle 50-100 websocket and http clients without needing Ngnix. You only need Nginx as a reverse proxy if you're running multiple Tornado processes on separate ports. \nIf you're running a single process or multiple process on a single port, you don't need Nginx.\nI've seen benchmarks which show that with a single Tornado process, you can serve around 5,000 connections per second if your response message size is around 100 KB; and over 20,000 requests per second for 1 KB response size. But this also depends on your CPU speed.\nI think it's safe to assume with an average CPU and around 1 GB RAM, you can easily serve around a 2,000-3,000 requests per second.","Q_Score":0,"Tags":"python,server,tornado,production-environment","A_Id":55805718,"CreationDate":"2019-04-22T20:57:00.000","Title":"Can tornado support 50 -100 websocket clients using its default http server without involving ngnix","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":1,"Web Development":0},{"Question":"I am trying to use Google Cloud Profiler in a python script running locally. It seems it is having problems to connect with a metadata server:\n\nWARNING:googlecloudprofiler.client:Failed to fetch instance\/zone from GCE metadata server: HTTPConnectionPool(host='metadata', port=80): Max retries exceeded with url: \/computeMetadata\/v1\/instance\/zone (Caused by NewConnectionError(': Failed to establish a new connection: [Errno -2] Name or service not known',))\nWARNING:googlecloudprofiler.client:Failed to fetch instance\/name from\n GCE metadata server: HTTPConnectionPool(host='metadata', port=80): Max\n retries exceeded with url: \/computeMetadata\/v1\/instance\/name (Caused\n by NewConnectionError(': Failed to establish a new connection: [Errno -2] Name\n or service not known',))\n\nSince the app seems to be running correctly and the profiler is collecting data successfully, is it OK if I just ignore the warnings or will I likely encounter some problems in the future?","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":588,"Q_Id":55813180,"Users Score":2,"Answer":"If you're running locally (and haven't, for example, manually zone in the config), these warnings are expected, so ignoring them is definitely okay.\n(Disclosure: I work at Google on Stackdriver Profiler)","Q_Score":0,"Tags":"python,google-cloud-platform,google-cloud-profiler","A_Id":56082596,"CreationDate":"2019-04-23T14:09:00.000","Title":"google-cloud-profiler metadata server WARNING","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":1,"Web Development":0},{"Question":"I'm trying to run a Selenium Script (Using PHP) using a Webserver.\nI'm working on Kali and to simulate the Webserver I use Xampp.\nI tried to run the selenium script on Xampp by the following steps:\n-Download the Php Webdriver Bindings, put them in the folder 'htdpcs' of xampp and edit the 'example.php' file following the settings of my own device.\n-Download and execute the Selenium Server Standalone, on port :4444.\nIn the end, I download the geckodriver and I execute the file, but I got the this error:\nOSError: [Errno 98] Address already in use\nHow to fix it in order to run the php-selenium script?","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":269,"Q_Id":55816936,"Users Score":0,"Answer":"The Selenium Server will fork a geckodriver as soon as it needs one to start a new browser session. You should not start a geckodriver yourself. If you want to use its Webdriver API yourself you can start it with the --webdriver-port argument.","Q_Score":0,"Tags":"python,selenium,selenium-webdriver","A_Id":55820106,"CreationDate":"2019-04-23T17:59:00.000","Title":"Selenium Servers and Geckodriver don't run at same port","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I have a problem with my project. It involves downloading data from various websites.\nFor now it's tens of pages, but in the future it will be thousands of pages depending on the country. Each page has its own script.\nTo support these scripts, I created a main script that calls each subsequent script on a separate thread using the multi-threading library. \nThe script on the input has a path to the file with data already downloaded to the database, and on the output it gives the second file only with new data. At the moment, this solution was used only for 4 pages. It works as I assume. But I wonder what happens when these websites will be a few hundred or a few thousand?\nI think I could create a separate main script depending on the country, but as a result, it would give at least several hundred pages (scripts) per country, so according to my logic, several hundred threads run at one time.\nDoes it have a chance to operate on such a number of websites? I opted for multithreading due to the large number of web content download operations.","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":66,"Q_Id":55831248,"Users Score":0,"Answer":"Ok, sounds smart :) Now I have 2 scripts with 'ThreadPool' method. Works fine. But to execute these scripts I have 'main_script' which until today I have separate threads for each script. So in main_script I can use too 'ThreadPool' method? And then how looks like a cost of CPU and generally perfomance? Because in future I want have e.g. 100 scripts in main_script (e.g. max_workers = 5) and every script have too ThreadPool (e.g. max_workers=5), so in one time I have a 5x5 = 25 threads?","Q_Score":0,"Tags":"python,multithreading,web-scraping","A_Id":56000171,"CreationDate":"2019-04-24T13:29:00.000","Title":"How handle multithreading on big-scale web scraping project?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":1,"System Administration and DevOps":0,"Web Development":1},{"Question":"I'm learning a bit about the automation of web applications using python \nI came across this module: \nfrom selenium.webdriver.common.keys import Keys\nWhat is the meant by keys and use of keys?","AnswerCount":2,"Available Count":1,"Score":0.2913126125,"is_accepted":false,"ViewCount":2663,"Q_Id":55877964,"Users Score":3,"Answer":"The Keys are referring to keyboard keys. It's used to ID the different Keys on the keyboard. Ex: Keys.RETURN means your ENTER key in your keyboard.\nThese keys can then be used by selenium for keyboard emulation or other stuff.","Q_Score":0,"Tags":"python","A_Id":55878045,"CreationDate":"2019-04-27T06:37:00.000","Title":"What is the use of from selenium.webdriver.common.keys import Keys","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I receive selenium.common.exceptions.WebDriverException: Message: Service \/usr\/bin\/google-chrome unexpectedly exited. Status code was : 0\nFor background, this is a Linux system and I am typing all the information in the terminal. I looked at many questions. A lot of them recommending uninstalling and reinstalling Chrome. I did that several times. I typed in whereis Google Chrome and found the location. I did not get a .exe file though so I used \"\/usr\/bin\/google-chrome\". Linux doesn't appear to create a .exe file. I am bringing this up because I am not sure if this contributed to my error.\nThis is after I typed in \nmy_path = \"\/usr\/bin\/google-chrome\"\nbrowser = webdriver.Chrome(executable_path=my_path)\nI get many lines of responses on the terminal referencing files in my python3.6 library. Before the main error of it saying it exited unexpectedly, I get \n\"file \"home\/ganesh\/.local\/lib\/python3.6\/site-packages\/selenium\/webdriver\/chrome\/webdriver.py, line 73 in init\"\n\"file \"home\/ganesh\/.local\/lib\/python3.6\/site-packages\/selenium\/webdriver\/chrome\/webdriver.py, line 98 in start\"\nThe thing is that my terminal successfully opens the Chrome browser. However, I get the webdriver exception message I had.\nIn addition, the code ,\nbrowser = webdriver.Chrome(executable_path=my_path)\nclearly didn't fully work because later in the program when I type in \nbrowser.(something else), it doesn't work and says \"name 'browser' is not define\"\nI am hoping for the webdriver exception error to be resolved and for me to successfully be able to call browser in my code later on\nThis question is not a duplicate as marked by someone here. The question that he referred to as answering my question does not answer my question - in that version Chrome exited. In mine, Chrome did not exit, it stayed open. In addition, the previous question has solutions in Windows and Mac, but not for Linux which is my operating system.\nIt is my first week using Linux.","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":87,"Q_Id":55885675,"Users Score":1,"Answer":"Welcome to SO. If the chromedriver file is located in \/usr\/bin\/google-chrome folder then your my_path should be my_path = \"\/usr\/bin\/google-chrome\/chromedriver\"","Q_Score":0,"Tags":"python,linux,python-3.x,google-chrome,selenium-webdriver","A_Id":55885757,"CreationDate":"2019-04-27T23:19:00.000","Title":"Chrome opens succesfully from terminal but I get webdriver common exception message","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":1,"Web Development":0},{"Question":"I made python script using Selenium webdriver. I use that script to login onto system of my faculty and looking on posts. Script refresh page every 60s to see if there is any new post, if there is new post I would recieve Telegram message on my phone. There is no problem with that, problem is that I have to run that script on my laptop 24h to get notifications, which is not possible since I carry it around. \nQuestion is, how can I run that script 24h? Is there any better solution to monitor that page and send messages if there is new post? \nI tried pythonanywhere but I don't have too much experience in that field so I didn't manage to make it work since always some module is missing...","AnswerCount":3,"Available Count":2,"Score":0.0665680765,"is_accepted":false,"ViewCount":905,"Q_Id":55889195,"Users Score":1,"Answer":"I'd just use a Windows virtual machine from AWS\/Microsoft Azure\/Google\/etc. This may be a bit overdoing it in your situation, but you could have one VM connected to another VM that'd be running your script, if it's something that requires an always-on user interface and can't be run in a Linux headless browser. \nInstalling something like AppRobotic personal macro edition on any of the above cloud services would work great. The pro version that's on AWS Marketplace would also work great, but it'd be overdoing it in your use case.","Q_Score":1,"Tags":"python,selenium,webautomation","A_Id":55889276,"CreationDate":"2019-04-28T10:09:00.000","Title":"How to run python script with selenium all the time but not on my PC","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I made python script using Selenium webdriver. I use that script to login onto system of my faculty and looking on posts. Script refresh page every 60s to see if there is any new post, if there is new post I would recieve Telegram message on my phone. There is no problem with that, problem is that I have to run that script on my laptop 24h to get notifications, which is not possible since I carry it around. \nQuestion is, how can I run that script 24h? Is there any better solution to monitor that page and send messages if there is new post? \nI tried pythonanywhere but I don't have too much experience in that field so I didn't manage to make it work since always some module is missing...","AnswerCount":3,"Available Count":2,"Score":0.0665680765,"is_accepted":false,"ViewCount":905,"Q_Id":55889195,"Users Score":1,"Answer":"Welcome !\nThe best way for you would be to use a server so you don't have to run it locally on your computer.\nYou can use an online VPS on which you install your software or you may even try to run it locally on something like a Raspberry Pi.\nIn both case, you will have to deal with linux commands.\nGood luck","Q_Score":1,"Tags":"python,selenium,webautomation","A_Id":55889236,"CreationDate":"2019-04-28T10:09:00.000","Title":"How to run python script with selenium all the time but not on my PC","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I have a list of approx. 52 websites which lead to about approx. 150 webpages that i require scraping on. Based on my ignorance and lack of research i started building crawlers per webpage which is starting to become to difficult to complete and maintain. \nBased on my analysis thus far I already know what information i want to scrape per webpage and it is clear that these websites have their own structure. On the plus side i noticed that each website has some commonalities in their web structure among their webpages.\nMy million dollar question, is there a single technique or single web crawler that i can use to scrape these sites? I already know the information that I want, these sites are rarely updated in terms of their web structure and most of these sites have documents that need to be downloaded. \nAlternatively, is there a better solution to use that will reduce the amount of web crawlers that I need to build? additionally, these web crawlers will only be used to download the new information of the websites that i am aiming them at.","AnswerCount":2,"Available Count":2,"Score":0.0,"is_accepted":false,"ViewCount":76,"Q_Id":55900736,"Users Score":0,"Answer":"[\u2026] i started building crawlers per webpage which is starting to become to difficult to complete and maintain [\u2026] it is clear that these websites have their own structure. [\u2026] these sites are rarely updated in terms of their web structure [\u2026]\n\nIf websites have different structures, having separate spiders makes sense, and should make maintenance easier in the long term.\nYou say completing new spiders (I assume you mean developing them, not crawling or something else) is becoming difficult, however if they are similar to an existing spider, you can simply copy-and-paste the most similar existing spider, and make only the necessary changes.\nMaintenance should be easiest with separate spiders for different websites. If a single website changes, you can fix the spider for that website. If you have a spider for multiple websites, and only one of them changes, you need to make sure that your changes for the modified website do not break the rest of the websites, which can be a nightmare.\nAlso, since you say website structures do not change often, maintenance should not be that hard in general.\nIf you notice you are repeating a lot of code, you might be able to extract some shared code into a spider middleware, a downloader middleware, an extension, an item loader, or even a base spider class shared by two or more spiders. But I would not try to use a single Spider subclass to scrape multiple different websites that are likely to evolve separately.","Q_Score":0,"Tags":"python,scrapy,web-crawler","A_Id":55919394,"CreationDate":"2019-04-29T09:44:00.000","Title":"Using a Single Web Crawler to Scrape Multiple websites in a predefined format with attachments?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I have a list of approx. 52 websites which lead to about approx. 150 webpages that i require scraping on. Based on my ignorance and lack of research i started building crawlers per webpage which is starting to become to difficult to complete and maintain. \nBased on my analysis thus far I already know what information i want to scrape per webpage and it is clear that these websites have their own structure. On the plus side i noticed that each website has some commonalities in their web structure among their webpages.\nMy million dollar question, is there a single technique or single web crawler that i can use to scrape these sites? I already know the information that I want, these sites are rarely updated in terms of their web structure and most of these sites have documents that need to be downloaded. \nAlternatively, is there a better solution to use that will reduce the amount of web crawlers that I need to build? additionally, these web crawlers will only be used to download the new information of the websites that i am aiming them at.","AnswerCount":2,"Available Count":2,"Score":0.0,"is_accepted":false,"ViewCount":76,"Q_Id":55900736,"Users Score":0,"Answer":"I suggest you crawl specific tags such as body, h1,h2,h3,h4,h5, h6,p and... for each links. You can gather all p tags and append them into a specific link. It can be used for each tags you want to crawl them. Also, you can append related links of tags to your database.","Q_Score":0,"Tags":"python,scrapy,web-crawler","A_Id":55920004,"CreationDate":"2019-04-29T09:44:00.000","Title":"Using a Single Web Crawler to Scrape Multiple websites in a predefined format with attachments?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I'm trying to write a simple Python application to manage muted words for Twitter. The interface in the browser and the application are cumbersome to use.\nLooking through the API documentation, it seems that it is possible to create and destroy muted users but not words. Am I missing something, or is this simply not possible?\nI have been trying the python-twitter library but the functionality is missing there too. I realise this is probably an API limitation as opposed to the library.","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":404,"Q_Id":55916527,"Users Score":3,"Answer":"No, this is not possible. The API only has methods for muting users, not words.","Q_Score":4,"Tags":"python,twitter","A_Id":55918277,"CreationDate":"2019-04-30T08:11:00.000","Title":"Is there a way to create and destroy muted words using the Twitter API","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I'm trying to run a python program in the online IDE SourceLair. I've written a line of code that simply prints hello, but I am embarrassed to say I can't figure out how to RUN the program. \nI have the console, web server, and terminal available on the IDE already pulled up. I just don't know how to start the program. I've tried it on Mac OSX and Chrome OS, and neither work. \nI don't know if anyone has experience with this IDE, but I can hope. Thanks!!","AnswerCount":2,"Available Count":1,"Score":0.0996679946,"is_accepted":false,"ViewCount":63,"Q_Id":55929577,"Users Score":1,"Answer":"Can I ask you why you are using SourceLair?\nWell I just figured it out in about 2 mins....its the same as using any other editor for python.\nAll you have to do is to run it in the terminal. python (nameoffile).py","Q_Score":0,"Tags":"python,ide","A_Id":55929861,"CreationDate":"2019-04-30T22:47:00.000","Title":"How to run a python program using sourcelair?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":1,"System Administration and DevOps":0,"Web Development":0},{"Question":"Strugging on this problem for a while so finally asking for some help from the experts.\nLanguage: python \nThe problem\/setup:\nI have many clients, client[n], client[n] .. etc\nI have many servers, server[n], server[n] .. etc\nEach server can plugin to 5 external ws connections. At any time I may need to open [x] ws connections; maybe 2, maybe 32, the total ws connections i need, thus servers needed, is dynamic... \nEach client maybe connecting 1 ws connection from server[1], 1 ws connection from server[2] .. .etc \nHow I imagine the flow working \n\nNew client[1] is loaded, needing 2 ws feeds\nNew client[1] broadcasts [xpub\/xsub ?] message to all servers saying, 'hey, I need these 2 ws connections, who has them?' \nServer[1] with the ws connections reply to client[1] (and only that client) - 'I got what youre looking for, talk to me' \nclient[1] engages in req\/reply communication with server[1] so that client[1] can utilize server[1]'s ws connection to make queries against it, eg, 'hey, server[1] with access to ws[1], can you request [x]' .. server[1] replies to client[1] 'heres the reply from the ws request you made' \n\ntldr\n\nclients will be having multiple req\/rep with many servers\nservers will be dealing with many clients \nclient need to broadcast\/find appropriate clients to be messaging with","AnswerCount":2,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":375,"Q_Id":55934405,"Users Score":0,"Answer":"I'll focus on the discovery problem. How do clients know which servers are available and which ws connections each one has?\nOne approach is to add a third type of node, call it broker. There is a single broker, and all clients and servers know how to reach it. (Eg, all clients and servers are configured with the broker's IP or hostname.)\nWhen a server starts it registers itself with the broker: \"I have ws feeds x,y,z and accept requests on 1.2.3.5:1234\". The broker tracks this state, maybe in a hash table.\nWhen a client needs ws feed y, it first contacts the broker: \"Which server has ws feed y?\" If the broker knows who has feed y, it gives the client the server's IP and port. The client can then contact the server directly. (If multiple servers can access feed y, the broker could return a list of servers instead of a single one.)\nIf servers run for a \"long\" time, clients can cache the \"server X has feed y\" information and only talk to the broker when they need to access a new feed.\nWith this design, clients use the broker to find servers of interest. Servers don't have to know anything about clients at all. And the \"real\" traffic (clients accessing feeds via servers) is still done directly between clients and servers - no broker involved.\nHTH. And for the record I am definitely not an expert.","Q_Score":0,"Tags":"python,sockets,message-queue,zeromq,pyzmq","A_Id":55968720,"CreationDate":"2019-05-01T09:49:00.000","Title":"python zmq many client to many server discovery message patterns","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I have a question about modbus tcp request. There are two options in python for modbus tcp library. Auto_open and Auto_close, these are keep tcp connection open and open\/close tcp connection for each request. Which one should I use? Which one is beneficial for Modbus Tcp devices? What is your suggestions? Thank you.","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":95,"Q_Id":55934968,"Users Score":0,"Answer":"it is better to use Auto_close method and close port after each data transmission\nthis help you to connect more than one mocbus\/tcp devices on same port(502).and the connection should be closed when you haven't data for transmitting","Q_Score":0,"Tags":"python,networking,tcp,modbus-tcp","A_Id":56304206,"CreationDate":"2019-05-01T10:43:00.000","Title":"About Modbus TCP request","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I'm using Python+Selenium to scrape data from a site which lists companies' info.\nFor each company I need 2 data points - email and url.\nThe problem is - for some companies email is not indicated and if I separately get a list of urls and emails I won't be able to fit the pairs (list of emails will be shorter than list of url and I won't know which of the emails is missing).\nSo I thought maybe there is a way to get root elements of each of the companies' blocks (say, it is div with class \"provider\") and then search inside each of them for email and url.\nIs it possible and if yes - how?","AnswerCount":3,"Available Count":1,"Score":0.0665680765,"is_accepted":false,"ViewCount":306,"Q_Id":55943694,"Users Score":1,"Answer":"Ok, I found the solution.\nFirst you collect all the blocks with fields you need to get. Example:\nproviders = browser.find_elements_by_class_name('provider-row')\nAnd then you use find_elements_by_xpath() method with locator starting with \".\/\/\" which means search inside a specific element. Example:\nproviders[0].find_elements_by_xpath(\".\/\/li[@class='website-link website-link-a']\/a[@class='sl-ext']\")","Q_Score":1,"Tags":"python,selenium","A_Id":55944216,"CreationDate":"2019-05-01T22:54:00.000","Title":"Selenium+Python. How to locate several elements within a specific element?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I'm new to kubernetes and I'm setting up some test environments to do some experiments with in minikube. So I have two pods running python, connected trough a service (I verified that they are selected trough the same service). \nI want to open a socket connection between the two. The container ip is equal to the IP specified in the service. On Pod1 I create the socket and connect to localhost and the port belonging to the container. \nI'm not sure whether I should actually connect to localhost or the name of the service. Then on the other pod (Pod2) I connect to the name of the service (if I understand correctly the service should exposes the IP with the name that corresponds to the service name). \nPod2 refuses to connect to Pod1 in the aforementioned configuration. \nIf I let Pod1 create a socket with as IP address the service name I get the \"Cannot assign requested address\" on socket creation.\nSo I think I'm just choosing the IP address wrongly.\nThanks in advance","AnswerCount":3,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":1269,"Q_Id":55977340,"Users Score":0,"Answer":"In Kubernetes, it is difficult to have always the IP address of a given POD. A pretty solution consists in making your application binding to 0.0.0.0 (this means you application will listen on all local interfaces including the Pod primary interface).\nThen you can expose you application running in that Pod using a Kubernetes service.","Q_Score":0,"Tags":"python,docker,sockets,kubernetes","A_Id":67442922,"CreationDate":"2019-05-03T21:32:00.000","Title":"What IP to bind to python socket in Kubernetes?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":1,"Web Development":0},{"Question":"I'm working on a piece of code that will be able to hit multiple web APIs (hardware that has APIs telling about the machine status) not blocking when one is waiting on the other to wait for the response, once the response of one arrives it'll be emitted on a websocket. One requirement is not to kill the APIs so hit them let's say once per 5 seconds as long as the main process is running.\nThe important part I'm struggling with is how to even approach it. \nWhat I did to this point is that the main process is spawning separate threads for different APIs and that thread is hitting the API emitting the response to the websocket time.sleep(5) and repeat. The main process is responsible to start new \"workers\" and kill ones that are not needed anymore also to restart ones that should be working but are not due to i.e. an exception.\nI have no idea if multi-threading is the way to go here - let's say I aim to \"crawl\" through 300 APIs.\nIs spawning long lived workers the right way to achieve this? Should those be processes instead? Should I maybe have the main process coordinate executing short-living threads that will do the API call and die and do that every 5 seconds per API (that seems way worse to maintain)? If the last option, then how to handle cases where a response takes more than 5 seconds to arrive?\nSome people are now talking about Python's asyncio like it's the golden solution for all issues but I don't understand how it could fit into my problem.\nCan someone guide me to the right direction?","AnswerCount":1,"Available Count":1,"Score":0.3799489623,"is_accepted":false,"ViewCount":471,"Q_Id":55994649,"Users Score":2,"Answer":"Let me rephrase this and tell me whether I'm right:\n\nI want to visit ~300 APIs frequently such that each is hit approximately every 5 seconds. How do I approach this and what worker\/process management should I use?\n\nThere's basically two different approaches:\n\nSpawn a thread for each API that is currently being watched (i.e. touched frequently) -- only feasible if at any given time only a subset of your total number of possible APIs is being watched.\nSet up a worker pool where all workers consume the same queue and have a management process fill the queue according to the time restrictions -- probably better when you always want to watch all possible APIs.\n\nEdit after your first comment:\nYou know the number of APIs you want to watch, so the queue's length should never grow larger than that number. Also, you can scan the queue in your main process frequently and check whether an API address you want to add already is in there and don't append it another time.\nTo avoid hitting APIs too frequent, you can add target timestamps along with the API address to the queue (e.g. as a tuple) and have the worker wait until that time is reached before firing the query to that API. This will slow down the consumption of your entire queue but will maintain a minimum delay between to hits of the same API. If you choose to do so, you just have to make sure that (a) the API requests always respond in a reasonable time, and (b) that all API addresses are added in a round-robin manner to the queue.","Q_Score":0,"Tags":"python,multithreading,multiprocessing,python-asyncio","A_Id":55995044,"CreationDate":"2019-05-05T17:38:00.000","Title":"Spawning workers in Python","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":1,"System Administration and DevOps":0,"Web Development":0},{"Question":"I'm looking for a way to download surveys that are still open on Qualtrics so that I can create a report on how many surveys are completed and how many are still in progress. I was able to follow their API documentation to download the completed surveys to a csv file but I couldn't find a way to do the same for the In Progress surveys. Thanks in advance for your help.","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":156,"Q_Id":55997128,"Users Score":0,"Answer":"Not through the API. You can do it manually through the Qualtrics interface.\nIf you need to use the API and the survey is invite only, an alternative would be to download the distribution history for all the distributions. That will tell you the status of each invitee.","Q_Score":0,"Tags":"python,qualtrics","A_Id":56006150,"CreationDate":"2019-05-05T22:56:00.000","Title":"Is there a way to download \"Responses In Progress\" survey from Qualtrics?","Data Science and Machine Learning":1,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I have to execute a .py file (python file) saved on my local machine (say @C:\\User\\abc\\dosomething.py) from postman from its pre-request section.\nEssentially I need to call the .py file from javascript code but from postman only.\nHow do I do that ?","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":2506,"Q_Id":56002752,"Users Score":0,"Answer":"Unfortunately this is not possible. The Pre-request and test-scripts are executed in a sandbox environment.","Q_Score":1,"Tags":"python-2.7,postman","A_Id":56004157,"CreationDate":"2019-05-06T09:52:00.000","Title":"How to execute python script from postman?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I have a SOAP url , while running the url through browser I am getting a wsdl response.But when I am trying to call a method in the response using the required parameter list, and it is showing \"ARERR [149] A user name must be supplied in the control record\".I tried using PHP as well as python but I am getting the same error.\nI searched this error and got the information like this : \"The name field of the ARControlStruct parameter is empty. Supply the name of an AR System user in this field.\".But nowhere I saw how to supply the user name parameter.","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":1235,"Q_Id":56016625,"Users Score":0,"Answer":"I got the solution for this problem.Following are the steps I followed to solve the issue (I have used \"zeep\" a 3rd party module to solve this):\n\nRun the following command to understand WSDL:\n\npython -mzeep wsdl_url\n\nSearch for string \"Service:\". Below that we can see our operation name\nFor my operation I found following entry:\n\nMyOperation(parameters..., _soapheaders={parameters: ns0:AuthenticationInfo})\nwhich clearly communicates that, I have to pass parameters and an auth param using kwargs \"_soapheaders\"\nWith that I came to know that I have to pass my authentication element as _soapheaders argument to MyOperation function.\n\nCreated Auth Element:\n\nauth_ele = client.get_element('ns0:AuthenticationInfo')\nauth = auth_ele(userName='me', password='mypwd')\n\nPassed the auth to my Operation:\n\ncleint.service.MyOperation('parameters..', _soapheaders=[auth])","Q_Score":0,"Tags":"php,python,web-services,soap,wsdl","A_Id":56127550,"CreationDate":"2019-05-07T06:21:00.000","Title":"Getting ARERR 149 A user name must be supplied in the control record","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I am running a Python Script on the Raspberry Pi in order to get measured data out of a Smart Plug. In my script I need to write the IP-Address of the Smart Plug so that I can retrieve the data it was measured. The problem is that I need to be able to take the Smart Plug to different places without having to hard code its new local IP-Address every time. \nI have the MAC Address so I am hoping there is an \"easy\" way to add a couple lines of code and retrieve the local IP-Address from the MAC (?) in the Python Script. Thanks!","AnswerCount":3,"Available Count":1,"Score":-0.1325487884,"is_accepted":false,"ViewCount":7632,"Q_Id":56022026,"Users Score":-2,"Answer":"The local ip address is not based on the MAC address. The router uses DHCP to give the devises an ip address. So there is no way to tell the router which IP he must give you other than changing the settings. \nI would rather try to broadcast the ip and on the raspberry listen on the broadcast channel for the message you are looking for.","Q_Score":2,"Tags":"python,ip-address,hostname,mac-address","A_Id":56022228,"CreationDate":"2019-05-07T11:53:00.000","Title":"Get local IP Address from a known MAC Address in Python?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I Have a python script that monitors a website, and I want it to send me a notification when some particular change happens to the website.\nMy question is how can I make that Python script runs for ever in some place else (Not my machine, because I want it to send me a notification even when my machine is off)?\nI have thought about RDP, but I wanted to have your opinions also.\n(PS: FREE Service if it's possible, otherwise the lowest cost)\nThank you!","AnswerCount":1,"Available Count":1,"Score":0.1973753202,"is_accepted":false,"ViewCount":724,"Q_Id":56056794,"Users Score":1,"Answer":"I would suggest you to setup AWS EC2 instance with whatever OS you want. \nFor beginner, you can get 750 hours of usage for free where you can run your script on.","Q_Score":0,"Tags":"python,web-services,monitoring,network-monitoring","A_Id":56057837,"CreationDate":"2019-05-09T09:52:00.000","Title":"How to make a python script run forever online?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I'm trying to make a program that can be able to make a search request on most websites such as YouTube, ESPN, My university course timetable etc... \nI have looked online for various solutions but many of them point to simply adding your search query at the end of the url you are \"getting\", but that doesn't seem to work with all websites some of them don't update their URL's when you manually make a search, while many others might give each and every URL a unique 'id'. Would it be possible to scrape a search bar from any website and then specifying a search query and entering it? Is there a function for that?","AnswerCount":2,"Available Count":2,"Score":0.0,"is_accepted":false,"ViewCount":557,"Q_Id":56075283,"Users Score":0,"Answer":"It's possible to use a text-based web browser and automate the search with a script. Then you can download the site you get from this browser and scrape it with BeautifulSoup or something else.","Q_Score":0,"Tags":"python,beautifulsoup,request,python-requests","A_Id":56075364,"CreationDate":"2019-05-10T10:02:00.000","Title":"Is there any way to scrape a search box using BeautifulSoup\/requests and then search and refresh?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I'm trying to make a program that can be able to make a search request on most websites such as YouTube, ESPN, My university course timetable etc... \nI have looked online for various solutions but many of them point to simply adding your search query at the end of the url you are \"getting\", but that doesn't seem to work with all websites some of them don't update their URL's when you manually make a search, while many others might give each and every URL a unique 'id'. Would it be possible to scrape a search bar from any website and then specifying a search query and entering it? Is there a function for that?","AnswerCount":2,"Available Count":2,"Score":1.2,"is_accepted":true,"ViewCount":557,"Q_Id":56075283,"Users Score":1,"Answer":"You need to Use Selenium Instance to do that. You can not achieve it using BeautifulSoup or requests.","Q_Score":0,"Tags":"python,beautifulsoup,request,python-requests","A_Id":56075496,"CreationDate":"2019-05-10T10:02:00.000","Title":"Is there any way to scrape a search box using BeautifulSoup\/requests and then search and refresh?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I'm currently working on a project that involves crawling data from various (about 50 websites). There is one website which has multiple pages we need to scrape, but this website doesn't allow multiple session. (the website is authenticated). \nIs there a way to pause the one spider until the other one is finished?\nI've been researching this for the past day. I found some ways you can pause, but it seems these are only working for the whole CrawlerProcess.","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":56,"Q_Id":56114106,"Users Score":0,"Answer":"The solution was actually fairly easy. Each spider has a unique identifying code. When setting the CrawlerProcess up, the program checks if the unique code is the same as the one that needs pausing, and if so, it passes the spider instance to the spider that needs to run first, which will then pause it with self.custom_settings['XX_SPIDER'].crawler.pause() and when done, in the closed() function, will unpause it with self.custom_settings['XX_SPIDER'].crawler.unpause()","Q_Score":0,"Tags":"python,scrapy","A_Id":56518384,"CreationDate":"2019-05-13T14:07:00.000","Title":"Pausing individual spiders in a CrawlerProcess","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I have two computers with internet connection. They both have public IPs and they are NATed. What I want is to send a variable from PC A to PC B and close the connection. \nI have thought of two approaches for this:\n1) Using sockets. PC B will have listen to a connection from PC A. Then, when the variable will be sent, the connection will be closed. The problem is that, the sockets will not communicate, because I have to forward the traffic from my public IP to PC B.\n2) An out of the box idea, is to have the variable broadcasted online somewhere. I mean making a public IP hold the variable in HTML and then the PC would GET the IP from and get the variable. The problem is, how do I make that variable accessible over the internet?\nAny ideas would be much appreciated.","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":233,"Q_Id":56129037,"Users Score":2,"Answer":"Figured a solution out. I make a dummy server using flask and I hosted it at pythonanywhere.com for free. The variables are posted to the server from PC A and then, PC B uses the GET method to get them locally.","Q_Score":1,"Tags":"python,python-3.x,sockets,networking","A_Id":56148950,"CreationDate":"2019-05-14T11:09:00.000","Title":"Send variable between PCs over the internet using python","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I built a website some time ago with Flask. Now all of a sudden when I try to navigate there I get the following:\nNET::ERR_CERT_COMMON_NAME_INVALID\nYour connection is not private\nAttackers might be trying to steal your information from www.mysite.org (for example, passwords, messages, or credit cards). Learn more\nDoes anyone know what's going on?","AnswerCount":3,"Available Count":1,"Score":0.1325487884,"is_accepted":false,"ViewCount":5243,"Q_Id":56138678,"Users Score":2,"Answer":"The error means: The host name you use in the web browser does not match the one used in the certificate.\nIf your server has multiple DNS entries you need to include all of into the certificate to be able to use them with https. If you access the server using it's IP address like https:\/\/10.1.2.3 then the IP address also have to present in the certificate (of course this only makes sense if you have a static IP address that never changes).","Q_Score":3,"Tags":"python,flask,error-handling","A_Id":68351111,"CreationDate":"2019-05-14T21:13:00.000","Title":"NET::ERR_CERT_COMMON_NAME_INVALID - Error Message","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"Is there any python library that allows me to have python server that communicates with python client on a non web port and where arguments as well as results can be passed in native python data type and transformed by library into requisite types needed for transfer protocol","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":25,"Q_Id":56140650,"Users Score":0,"Answer":"You may be looking for Tornado TCP, The data structure can be passed by Json.","Q_Score":0,"Tags":"python,server","A_Id":56140720,"CreationDate":"2019-05-15T01:29:00.000","Title":"python library to create python server and client that can also handle data transfer over ports in native python types","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I have a problem. Last year I developed a Telegram Gateway that uses a socket and the Telethon library (installed through pip).\nThe problem is that this project is installed in an other pc and uses an old version of Telethon, foundamental to use thread (with the new Telethon version I can't use the thread with socket etc., it is changed). \nI need to install the same Telethon version in order to use the same gateway.\nIn the pc, if I run the command:\npip show telethon\nit shows the following message:\nYou are using pip version 8.1.1, however version 19.1.1 is available. You should consider upgrading via etc......\nThen, I need to install the same version of telethon (8.1.1).\nBut if I try to install it in an other pc, through this command:\npip3 install telethon==8.1.1\nit prints the following red error message:\nNo matching distribution found for telethon==8.1.1\nWhy?\nI really need to use the same version of telethon, in order to run the old gateway.\nThank you very much!!!","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":653,"Q_Id":56194596,"Users Score":1,"Answer":"That line is a message printed by pip telling you to consider upgrading pip itself. The version number you are looking for is printed below it:\n\n$ pip3 show telethon\nYou are using pip version 8.1.1, however version 19.0.3 is available.\nYou should consider upgrading via the 'pip install --upgrade pip' command.\nName: Telethon\nVersion: 1.6.2\nSummary: Full-featured Telegram client library for Python 3\nHome-page: https:\/\/github.com\/LonamiWebs\/Telethon\nAuthor: Lonami Exo\nAuthor-email: totufals@hotmail.com\nLicense: MIT\nLocation: \/usr\/local\/lib\/python3.7\/site-packages\nRequires: rsa, pyaes\nRequired-by:","Q_Score":0,"Tags":"python,pip,telethon","A_Id":56194768,"CreationDate":"2019-05-17T23:11:00.000","Title":"Old version of Telethon","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":1,"System Administration and DevOps":0,"Web Development":0},{"Question":"My Telegram bot code was working fine for weeks and I didn't changed anything today suddenly I got [SSL: CERTIFICATE_VERIFY_FAILED] error and my bot code no longer working in my PC.\nI use Ubuntu 18.04 and I'm usng telepot library.\nWhat is wrong and how to fix it?\nEdit: I'm using getMe method and I don't know where is the certificate and how to renew it and I didn't import requests in my bot code. I'm using telepot API by importing telepot in my code.","AnswerCount":1,"Available Count":1,"Score":0.1973753202,"is_accepted":false,"ViewCount":1148,"Q_Id":56220056,"Users Score":1,"Answer":"Probably your certificate expired, that is why it worked fine earlier. Just renew it and all should be good. If you're using requests under the hood you can just pass verify=False to the post or get method but that is unwise.\nThe renew procedure depends on from where do you get your certificate. If your using letsencrypt for example with certbot. Issuing sudo certbot renew command from shell will suffice.","Q_Score":2,"Tags":"python,python-3.x,ssl,telegram-bot,telepot","A_Id":56220074,"CreationDate":"2019-05-20T11:26:00.000","Title":"\"SSL: CERTIFICATE_VERIFY_FAILED\" error in my telegram bot","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"How do I add an additional CA (certificate authority) to the trust store used by my Python3 AWS Lambda function?","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":6930,"Q_Id":56225178,"Users Score":5,"Answer":"If you only need a single CA, then get your crt file and encode it into a pem using the following command in linux:\n\nopenssl x509 -text -in \"{your CA}.crt\" > cacert.pem\n\nIf you need to add CA's to the default CA bundle, then copy python3.8\/site-packages\/certifi\/cacert.pem to your lambda folder. Then run this command for each crt:\n\nopenssl x509 -text -in \"{your CA}.crt\" >> cacert.pem\n\nAfter creating the pem file, deploy your lambda with the REQUESTS_CA_BUNDLE environment variable set to \/var\/task\/cacert.pem. \n\/var\/task is where AWS Lambda extracts your zipped up code to.","Q_Score":6,"Tags":"python,python-3.x,amazon-web-services,aws-lambda,ssl-certificate","A_Id":59638101,"CreationDate":"2019-05-20T16:51:00.000","Title":"Python AWS Lambda Certificates","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I built a crawler that gets the product information of a list of products entered by the user. Sometimes, the crawler freezes, especially if the list of products is long and if the crawler runs on headless mode.\nThe bug seems random and is not reproducible, which makes me think it is caused by the resource utilization of the website being crawled. \nSince this is a non-reproducible bug, I don't think I can fix it, but is there a way to detect that the crawler has frozen and try again?\nHere is some information about the crawler and the bug:\n\nThe crawler is built using Selenium and Python.\nThe bug occurs with different websites and products.\nThe bug occurs in the \"normal\" mode, but occurs more often in the headless mode.\n\nThanks!\nFelipe","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":239,"Q_Id":56247296,"Users Score":0,"Answer":"If the problem isn't related to the browser, it is because the code is busy on getting data in headless mode. If your code working in the normal mode instead of headless mode, you see only the working part.\nI assume you made a GUI. If it is so, you are trying to access GUI but the same program working on crawling. That's why GUI is freezing.\nYou can solve this by using the Threading library or any other multiprocessing method. This will allow you to run more than one process at the same time. So, you can freely use other functions on the GUI and crawl a website without freezing.","Q_Score":1,"Tags":"python,selenium,web-scraping,web-crawler,headless","A_Id":63178050,"CreationDate":"2019-05-21T22:57:00.000","Title":"Python-Selenium crawler freezes, especially in headless mode (non reproducible bug)","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I am connecting my slave via TCP\/IP, everything looks fine by using the Wireshark software I can validate that the CRC checksum always valid \u201cgood\u201d, but I am wondering how I can corrupt the CRC checksum so I can see like checksum \u201cInvalid\u201d. Any suggestion how can I get this done maybe python code or any other way if possible.\nThank you all \nTariq","AnswerCount":2,"Available Count":1,"Score":0.0996679946,"is_accepted":false,"ViewCount":406,"Q_Id":56268062,"Users Score":1,"Answer":"I think you use a library that computes CRC. You can form Modbus packet without it, if you want simulate bad CRC condition","Q_Score":1,"Tags":"python,tcp,checksum,crc,modbus-tcp","A_Id":56268121,"CreationDate":"2019-05-23T04:26:00.000","Title":"How corrupt checksum over TCP\/IP","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"User queries to dialogflow agent through google assistant using voice commands.I want to add some data in that user query. can we change the request parameters sent to agent? If yes, where? The code for google assistant library is in python.\nI am working on python 3.5 on raspberry pi 3. I tried modifying event.py file located at google\/library\/assistant\/ . But before the event for ON_RECOGNISING_SPEECH_FINISHED I got response from google assistant.","AnswerCount":1,"Available Count":1,"Score":0.1973753202,"is_accepted":false,"ViewCount":66,"Q_Id":56269802,"Users Score":1,"Answer":"No, if you're using the Google Assistant SDK, you interact with the agent as if it was any other Google Assistant surface. There is no way to add additional context that is exclusive to your device.","Q_Score":0,"Tags":"python-3.x,raspberry-pi3,dialogflow-es,google-assistant-sdk","A_Id":56278248,"CreationDate":"2019-05-23T07:08:00.000","Title":"change request format sent to dialogflow agent","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I noticed that the same transaction had a different transaction ID the second time I pulled it. Why is this the case? Is it because pending transactions have different transaction IDs than those same transactions once posted? Does anyone have recommendations for how I can identify unique transactions if the trx IDs are in fact changing?","AnswerCount":2,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":415,"Q_Id":56282490,"Users Score":0,"Answer":"Turns out that the transaction ID often does change. When a transaction is posted (stops pending), the original transaction ID becomes the pending transaction ID, and a new transaction ID is assigned.","Q_Score":4,"Tags":"javascript,python,api,banking,plaid","A_Id":57262467,"CreationDate":"2019-05-23T20:27:00.000","Title":"How to identify Plaid transactions if transaction ID's change","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"With graph = nx.node_link_graph(json.loads(\"json_string\")) it is possible to load a graph that is represented in JSON format.\nNow my problem is that I already have a networkx graph in my program and only want to add JSON formated components dynamically during runtime.\nFor example somewhere the string ' {\"source\": 1, \"target\": 2, \"weight\": 5.5} ' is created and should then result in a new edge between node 1 and 2.\nWhat is the best way to realise this?","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":284,"Q_Id":56294819,"Users Score":0,"Answer":"The simpliest way you can add the new edge to the graph G if you have a dict like this:\nd = {\"source\": 1, \"target\": 2, \"weight\": 5.5} (you can convert it from string to dict with dict(s) or json.loads(s)) is:\nG.add_weighted_edges_from([(d['source'], d['target'], d['weight'])])","Q_Score":0,"Tags":"python,json,networkx","A_Id":56295146,"CreationDate":"2019-05-24T14:37:00.000","Title":"Is there a way to add single nodes\/edges in JSON format to networkx graph?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":1,"System Administration and DevOps":0,"Web Development":0},{"Question":"I am currently building a media website using node js. I would like to be able to control Kodi, which is installed of the server computer, remotely from the website browser.How would I go about doing this? My first idea was \n\nto simply see if I could somehow pipe the entire Kodi GUI into the\nbrowser such that the full program stays on the server\nand just the GUI is piped to the browser, sending commands back to\nthe server;\n\nhowever, I could find little documentation on how to do that.\nSecond, I thought of making a script (eg Python) that would be able to control Kodi and just interface node js with the Python script, but again, \nI could find little documentation on that.\n Any help would be much appreciated. \nThank You!","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":93,"Q_Id":56306073,"Users Score":0,"Answer":"Can't you just go to settings -> services -> control and then the 'remote control via http' settings? I use this to login to my local ip e.g. 192.168.1.150:8080 (you can set the port on this page) from my browser and I can do anything from there","Q_Score":0,"Tags":"python,node.js,kodi","A_Id":57015259,"CreationDate":"2019-05-25T15:12:00.000","Title":"Controlling Kodi from Browser","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I need to return some items from inside a spider before starting to parse requests. Because I need to make sure some parent items exist in database before parsing child items.\nI now yield them from the parse method first thing, and this seem to work fine. But I was wondering if there is a better way to do this?","AnswerCount":1,"Available Count":1,"Score":0.1973753202,"is_accepted":false,"ViewCount":67,"Q_Id":56315488,"Users Score":1,"Answer":"Instead of yielding the items, write them into the database directly on the constructor of the pipeline where you add regular items to the database.","Q_Score":0,"Tags":"python,scrapy","A_Id":56322104,"CreationDate":"2019-05-26T16:42:00.000","Title":"Scrapy returning items within spider before first request\/parse","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"Did anybody use to write services, connected with ITSMChangeManagement of OTRS? Is there any API to this thing? I need smth. like GenericTicketConnector. Thank you.","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":63,"Q_Id":56339794,"Users Score":1,"Answer":"No, there is nothing with ITSMChangeManagement. Only Tickets, FAQ and ConfigItems can be used with the GenericInterface of the Community Edition","Q_Score":0,"Tags":"python,api,otrs,itsm","A_Id":56340530,"CreationDate":"2019-05-28T10:08:00.000","Title":"OTRS ITSMChange API service","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I'm currently trying to establish a connection between a chrome-extension and a NativeMessagingHost. Everything works fine on Windows, but it won't do on Linux (either arch, kali or ubuntu). \nHow it fails: \n\nCan't find manifest for native messaging host my_host.json\n\nMy current state:\n\nhave my host manifest under ~\/.config\/google-chrome-beta\/NativeMessagingHosts\/my_host.json\n\nin there: \"name\":\"my_host.json\" and the path to my python script which handles the messages, also the unique chrome-extension code unter allowed_origins\n\nin manifest.json of my extension given the permission for nativeMessaging\nfurthermore in popup.js: var port = chrome.runtime.connectNative(\"my_host.json\"); and the same name used in sendNativeMessage\n\nWhat I tried so far:\n\ntried with google-chrome-beta and -stable\ndeleted file endings e.g. my_host.json to my_host, or removing the python ending\neasier paths where my python script lays\nalso tried to put my_host into \/Library\/Google\/Chrome\/NativeMessagingHosts which typically should be the Mac path - but hey, may it worked (not..)\n\nI get no error starting the application, copied the message from terminal whilst starting chrome with logging.\nI pretty much went through the example google gave, adjusting paths etc.\nAlso went through differnt posts, but it seems no one has the same problem, or no one tries to do something similar.","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":257,"Q_Id":56361604,"Users Score":0,"Answer":"So I figured out where the problem was: \nI changed the name of my host from my_host.json to com.my.host.json and set \"name\" to com.my.host (where I firstly had the ending .json as well, which probably caused the problem too). Furthermore I changed var port = chrome.runtime.connectNative(\"my_host.json\") in my js-file to [...](\"com.my.host\") where the ending .json was not right too. Everything works now. Thank you for your suggestions!","Q_Score":0,"Tags":"javascript,python,google-chrome-extension","A_Id":56376376,"CreationDate":"2019-05-29T13:18:00.000","Title":"Manifest for native messaging host not found","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":1,"Web Development":0},{"Question":"I am running into a limitation with an API I am working with. At the moment the max rows it can return is 10000. The goal is to access a month worth of data which sometimes can be up 200000 rows. This API allows filtering by start_at_row and row_limit. It also returns row_count in the response. I will need to make multiple requests in order to capture all 200000 rows. I'm stuck on figuring out how to trigger the next request after the first 10000 rows and how to pick at exactly at 10001 and run the next 10000 rows. Is there a common method used for this problem? Any advice is welcome. Let me know which pieces of information from my code I can provide that can be useful.","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":89,"Q_Id":56370374,"Users Score":0,"Answer":"Update, I was able to figure this out on my own after a little trial and error and more research on Google.\nI added start_at_row and row_limit as global variables that equal to 1 and 2 respectively.\nFrom there I added a while loop(start_at_row <= row_limit), I then captured the row_count from the response given by the API and made it equal to role_limit. \nTo break the loop I made and an if statement with start_at_row > row_limit","Q_Score":1,"Tags":"xml,python-3.x,beautifulsoup","A_Id":56403222,"CreationDate":"2019-05-30T01:06:00.000","Title":"How to iterate through smaller chunks of data in API response","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I\u2019m writing a HTTP\/1.1 client that will be used against a variety of servers.\nHow can I decide a reasonable default keep-alive timeout value, as in, how long the client should keep an unused connection open before closing? Any value I think of seems extremely arbitrary.","AnswerCount":1,"Available Count":1,"Score":0.3799489623,"is_accepted":false,"ViewCount":1106,"Q_Id":56373516,"Users Score":2,"Answer":"First note that that with HTTP keep alive both client and server can close an idle connection (i.e. no outstanding response, no unfinished request) at any time. This means especially that the client cannot make the server keep the connection open by enforcing some timeout, all what a client-side timeout does is limit how long the client will try to keep the connection open. The server might close the connection even before this client-side timeout is reached.\nBased on this there is no generic good value for the timeout but there actually does not need to be one. The timeout is essentially used to limit resources, i.e. how much idle connections will be open at the same time. If your specific use case will never visit the same site again anyway then using HTTP keep-alive would just be a waste of resources. If instead you don't know your specific usage pattern you could just place a limit on the number of open connections, i.e. close the longest unused connection if the limit is reached and a new connection is needed. It might make sense to have some upper limit timeout of 10..15 minutes anyway since usually after this time firewalls and NAT routers in between will have abandoned the connection state so the idle connection will no longer work for new requests anyway.\nBut in any case you also need to be sure that you detect if the server closes a connection and then discard this connection from the list of reusable connections. And if you use HTTP keep-alive you also need to be aware that the server might close the connection in the very moment you are trying to send a new request on an existing connection, i.e. you need to retry this request then on a newly created connection.","Q_Score":0,"Tags":"python-3.x,sockets,http,tcp,python-asyncio","A_Id":56374276,"CreationDate":"2019-05-30T07:18:00.000","Title":"HTTP\/1.1 client: How to decide good keep-alive timeout default?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I'm trying to get a list of all the items available for purchase on a certain web page. However, the web page only loads 12 items at a time until the user scrolls down and then 12 more are loaded. Is there a way, in C# or Python, using any open source libraries, to be able to \"see\" all of the items available without physically going to the page and scrolling down?\nUsing Chrome's developer tools, I can only \"see\" the 12 items in the HTML window until I physically scroll down on the web page and it loads more.\nNOTE: I'm relatively new at C#\/Python web scraping, so I appreciate any in depth answers!\nEDIT: If I were to use something like Selenium, would it be possible to load everything programatically? If so, how?","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":628,"Q_Id":56380988,"Users Score":0,"Answer":"The Web browser itself can obviously load more items, and a web browser is just a program, so yes, it's possible to write a program to do it.\nBut you'd essentially be writing a mini web browser, which is a lot of work.\nDoes the website itself offer a way to load more items at once? i.e. some websites offer a link or a dropdown menu in the corner of the page that allows you to increase the maximum number of items shown.","Q_Score":1,"Tags":"c#,python,html,web-scraping","A_Id":56381247,"CreationDate":"2019-05-30T15:14:00.000","Title":"How can I load all items on a web page when using an HTML parser?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"There is a table with a list of users and their email on a webpage. At the top of the page, is a search\/filter input field that allows the user to type an email or username and filter for the result as he\/she is typing. \nThe problem: However, when I use the send_keys() method instead of doing this manually, nothing is filtered in the table view.\nThis is happening on the Safari browser on the iPhone 7 Plus (real device, not simulator). Some other information:\n\niOS version: 12.2\nAppium version: 1.13.0-beta.3\nSelenium version: 2.53.1\nProgramming language: Python 2.7.15\n\nIn addition to send_keys(), i've tried to use set_value(), i've also tried to execute JS and setting the attribute value, and also tried to send key for each character (in a for loop with a delay between each character).\nI'm expecting for example, element.send_keys(\"test1000@test.com) to filter the table view on the web page so that the only user that is displayed has the associated test1000@test.com email as it does when I go through the site manually.\nIn actuality, send_keys() does not do that and nothing in the table view gets filtered.\nAny help or guidance would be appreciated!","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":288,"Q_Id":56382223,"Users Score":0,"Answer":"Explicit wait for table to get populate in DOM\nsendKeys search String and additional key Tab\ntextbox.sendKeys(\"test1000@test.com\"+Keys.TAB)\nExplicit Wait for filter to get applied and table to get refreshed.\nfind elements for the newly populated table with applied filters.","Q_Score":0,"Tags":"selenium,appium,appium-ios,python-appium","A_Id":56395049,"CreationDate":"2019-05-30T16:37:00.000","Title":"Using send_keys() on a search\/filter input field does not filter results in table","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I'm new to web scraping and am trying to learn more. I know some websites load products on the back end before they make it available to the general public. Is there a way I can access that information using an HTML parser or any other library?\nI suspect the website developers use dynamic javascript to alter the information after loading. Or use different tags\/classes to hide the information?","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":42,"Q_Id":56382627,"Users Score":0,"Answer":"I see two questions here: \n1)Can I access information on the webserver that isn't sent to the client page? \nNo. You can only scrape what exists on the page. Anything else would be illegally accessing a non-public server and goes beyond scraping into hacking.\n2) If the site loads asynchronously and\/or dynamically, can I access the content that loads after the main portion of the html?\nYes, using browser automation tools like selenium, you can approximate a user experiencing the site and wait for the full content to load before you scrape it. This is different from simple requests\/beautifulsoup, which only gathers the HTML at the point when you send the request.","Q_Score":0,"Tags":"python,web-scraping,beautifulsoup","A_Id":56382871,"CreationDate":"2019-05-30T17:05:00.000","Title":"If a website loads a product on the back end but does not publish it for the public yet, can I access that information?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I want to redirect mysite.com\/index.php to mysite.com\/clientarea.php using .htaccess does not work for me help out.\nwith simple code.\nI have tried several time and finally the site is not available \n\nTry running Windows Network Diagnostics.\n DNS_PROBE_FINISHED_NXDOMAIN","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":46,"Q_Id":56420275,"Users Score":0,"Answer":"Based on that error, it appears you're redirecting to a different hostname than mysite.com - I assume unintentionally. \nIf you can post your .htaccess code, the solution may be easy to provide.","Q_Score":0,"Tags":"django,python-3.x,.htaccess,whmcs","A_Id":56420311,"CreationDate":"2019-06-03T01:49:00.000","Title":"How Can i redirect url to another url with the .htaccess in cpanel","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"Appium is not finding element with MobileBy Accessibility ID and I'm calling the method from AppiumDriver class that inherits from the SeleniumDriver class. Any ideas? Thanks.","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":151,"Q_Id":56421105,"Users Score":0,"Answer":"It ended up being, I had to use MobileBy.ID not MobileBy.Accessibility_ID and the element was found. The framework code was not an issue. Just using wrong locator type.","Q_Score":0,"Tags":"python,automation,appium","A_Id":56429709,"CreationDate":"2019-06-03T04:21:00.000","Title":"Appium is not finding element when using a method to find it from a python appium class that inherits from a selenium class","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I have two simple lambda functions. Lambda 1 is invoking lambda 2 (both do a simple print for text).\nIf both lambdas are outside of a VPC then the invocation succeeds, however as soon as I set them both in to access a VPC (I need to test within a VPC as the full process will be wtihin a VPC) the invocation times out.\nDo I have to give my lambda access to the internet to be able invoke a second lambda within the same VPC?","AnswerCount":2,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":648,"Q_Id":56425420,"Users Score":1,"Answer":"If your lambda functions are inside a VPC you need to configure your both lambda functions into private subnet not public subnet. That is the AWS recommended way.","Q_Score":0,"Tags":"python,python-3.x,aws-lambda","A_Id":56448070,"CreationDate":"2019-06-03T10:27:00.000","Title":"Unable to invoke second lambda within VPC - Python AWS Lamba Function","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I am trying to extract\/sync data through Pardot API v4 into a local DB. Most APIs were fine, just used the query method with created_after search criteria. But the Visit API does not seem to support neither a generic query of all visit data, nor a created_after search criteria to retrieve new items. \nAs far as I can see I can only query Visits in the context of a Visitor or a Prospect.\nAny ideas why, and how could I implement synchronisation? (sorry, no access to Pardot DB...)\nI have been using pypardot4 python wrapper for convenience but would be happy to use the API natively if it makes any difference.","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":185,"Q_Id":56494946,"Users Score":2,"Answer":"I managed to get a response from Pardot support, and they have confirmed that such response filtering is not available on the Visits API. I asked for a feature request, but hardly any chance to get enough up-votes to be considered :(","Q_Score":1,"Tags":"python-3.x,pardot","A_Id":58770292,"CreationDate":"2019-06-07T13:05:00.000","Title":"Pardot Visit query API - generic query not available","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I have a python script that loops through all of the AWS accounts we have and lists the EC2 instances in each one. \nI want to turn it into an AWS Lambda function. But I can't figure out how to pull the AWS credentials that would allow me to list the servers in all the accounts.\nHow can I achieve this in AWS Lambda?","AnswerCount":2,"Available Count":2,"Score":0.0,"is_accepted":false,"ViewCount":316,"Q_Id":56496230,"Users Score":0,"Answer":"When you create lambda you have so specify a role\nIn IAM you can attach required permission to a lambda role. \nIf you want to use some specific set of credentials in a file, you can utilize AWS Systems Manager to retrieve credentials. \nThough, I would recommend role on lambda","Q_Score":0,"Tags":"python,aws-lambda","A_Id":56496798,"CreationDate":"2019-06-07T14:28:00.000","Title":"Loop through AWS Accounts in Lambda Python Function","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I have a python script that loops through all of the AWS accounts we have and lists the EC2 instances in each one. \nI want to turn it into an AWS Lambda function. But I can't figure out how to pull the AWS credentials that would allow me to list the servers in all the accounts.\nHow can I achieve this in AWS Lambda?","AnswerCount":2,"Available Count":2,"Score":1.2,"is_accepted":true,"ViewCount":316,"Q_Id":56496230,"Users Score":1,"Answer":"Create a role with cross account permissions for ec2:ListInstances\nAttach the role to the lambda function","Q_Score":0,"Tags":"python,aws-lambda","A_Id":56501963,"CreationDate":"2019-06-07T14:28:00.000","Title":"Loop through AWS Accounts in Lambda Python Function","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I'm making an http request in my code and I am curious if making a HEAD request still makes the syn, syn-ack, and ack? I've never heard of HEAD until now.","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":65,"Q_Id":56512169,"Users Score":1,"Answer":"\"Does making a HEAD request still require the 3 way handshake?\"\nIt depends. HEAD request is a concept in HTTP, the application layer; while \"3 way handshake\" is a concept in TCP, the transport layer.\nThus, whether HEAD request requires \"3 way handshake\" depends on whether it is using TCP. For HTTP\/1.1 and HTTP\/2, the answer is yes. For HTTP\/3, the answer is no, as HTTP\/3 is based on UDP.","Q_Score":0,"Tags":"http,python-requests,http-headers,httprequest","A_Id":56519591,"CreationDate":"2019-06-09T05:16:00.000","Title":"Does making a HEAD request still require the 3 way handshake?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I use the requests Module with python At the moment. I use \"r.status_code == 200\" to check if the response is valid. But for my project right now it gaves me false posetives. It would be better for me to check if a response is valid with a Keyword check on the sourcecode or something like this is this possible ?","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":1387,"Q_Id":56517575,"Users Score":0,"Answer":"If I understand your question correctly, its possible for your request to give you back empty or invalid data that is not useful to you. \nWithout knowing what this request return you could look for specific fields in the request, for example if the request returns a json look for specific fields using if json_file.get(\"field_name\", False)","Q_Score":0,"Tags":"python,request","A_Id":56517613,"CreationDate":"2019-06-09T19:03:00.000","Title":"Python requests check if success","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I'm trying to build a network listener in python, which is able to capture the local network traffic from my own computer. \nI've tried with sockets, but i find it very hard to use this method, because i'm not on a Linux-machine. \nSo, is there another method i'm not aware of, now when it's about only monitoring my own traffic? :)","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":159,"Q_Id":56538288,"Users Score":0,"Answer":"What you need is called sniffer, try scapy framework.","Q_Score":0,"Tags":"python-3.x,http,networking","A_Id":56566080,"CreationDate":"2019-06-11T07:08:00.000","Title":"How to make a network listener in python?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"Whenever I run my tests on my computer, they work relatively fine. At the very least, selenium runs without problems.\nAs soon as I run the same tests on a docker container I start running in all kinds of errors that selenium started throwing, such as: \"Element not clickable at point...\", \"Element is not interactable...\", etc.\nNone of these happen when I run the tests on my computer normally.\nI have a Linux Debian 9 computer, docker 1.11, Chrome 72.0, chromedriver 2.41, selenium 3.12. Test are done using py.test and in headless chrome.\nMy Dockerfile is simple, installing all of the packages for python and putting my tests in there and running them. I run a custom-made Dockerfile and don't have the option to use the premade seleniumHQ ones.\nI have first tried running a demo test where I first encountered that problem. I managed to solve it by editing the test code to bypass the exception and trying again. After that succeeded, I tried running a few more complicated tests and kept running into different errors I didn't run into before.\nEvery solution I found was aimed at solving the thrown exception, but I suspect there's a deeper issue at hand and I can't figure out what it is.\nTo reiterate: running tests on my computer (both headless and otherwise) works like a charm, but running the same tests in a docker container fails with several selenium errors thrown. For some reason, the tests don't seem to run properly and selenium keeps throwing errors.\nI've been trying to solve it for a while now and can't seem to figure out what the problem is.\nI'd like to know WHY this problem occurs and how to solve the root cause of it. If it's a known issue and the solution is indeed simply to solve every exception as it comes, I'd like to know that too.","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":617,"Q_Id":56546422,"Users Score":1,"Answer":"As far as I could find, and how I solved this, is to just follow the exceptions and fix them as they come.\nThe general problem areas I encountered were the click event on buttons\/textbox and clearing the text from a textbox.\nThe solution involved:\n\nA call to webdriver.execute_script('arguments[0].click();', element) to replace button click events.\nA call to webdriver.execute_script('arguments[0].focus();', element) to replace textbox click events.\nA call to webdriver.execute_script('arguments[0].value = \"\";', element) to replace textbox clears.\n\nFrom where I stand, these solved most if not all the sudden exceptions and the rest worked as intended.","Q_Score":0,"Tags":"python-3.x,selenium-webdriver,docker-container,web-testing","A_Id":56836785,"CreationDate":"2019-06-11T14:51:00.000","Title":"Running Selenium in container brings sudden selenium errors where there were none before","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"What are the differences among the three choices in their usages and functions? Are there lists for their arguments?","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":34,"Q_Id":56547919,"Users Score":0,"Answer":"ChromeOptions are specific to Chromedriver. (you can pass prefs into the ChromeDriver constructor...) Capabilities are for Selenium WebDriver. (so more global... things like setting paths to driver, browser etc...)","Q_Score":0,"Tags":"python,selenium,webdriver,selenium-chromedriver","A_Id":56552660,"CreationDate":"2019-06-11T16:16:00.000","Title":"What are the differences among ChromeOptions, capabilities and preferences in Chromedriver?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"The ACS url in the Google SSO SAML setup where Google is the Identity Provider has to start with https. Therefore, I've not been able to use a localhost url. Is there a way how I could test Google SSO SAML on a local server? What url (or other details) do I need to enter?","AnswerCount":5,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":5382,"Q_Id":56581853,"Users Score":0,"Answer":"I had same problem , i ran my app on localhost with https using local iis with self signed certificate and it worked just fine.\nThis way its easy to debug saml response from google rather than using remote urls.","Q_Score":5,"Tags":"python,google-cloud-platform,single-sign-on,saml-2.0,google-sso","A_Id":59552333,"CreationDate":"2019-06-13T13:40:00.000","Title":"Test Google SSO SAML on Localhost","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":1,"Web Development":1},{"Question":"I tried doing it by creating a css file style.css in the same folder, copied the source code of the bulma link provided and then linked it to my html doc. But then, it shows no css features at all.","AnswerCount":2,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":1985,"Q_Id":56595054,"Users Score":0,"Answer":"Although I didn't achieve what I was trying to do, I guess the only solution is either copying the exact design provided via Bulma link or ending up writing the whole css code all by own which I don't really prefer until essentially needed. So, I'd rather stick to the pre-defined design provided by Bulma.","Q_Score":1,"Tags":"html,css,django,python-3.x,bulma","A_Id":56610545,"CreationDate":"2019-06-14T09:15:00.000","Title":"Is there any way in which I can edit \"https:\/\/cdnjs.cloudflare.com\/ajax\/libs\/bulma\/0.7.5\/css\/bulma.css\" file?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I am calling a web API in Python, getting data back in XML, converting the response into a dictionary using xmltodict, but for several elements, sometimes I get a dictionary (single element) and sometimes I get a list (multiple elements) in response.\nI first started to use \"if isinstance(..., dict):\" - that could solve my problem but is not so elegant and requires quite some lines of code in my case.\nI then found out \"force_list\" which I think is exactly what I need, but I need to apply it to several elements and I can't find the right syntax - I'm not even sure if that's possible.\nThe code I am trying to make work:\nresponse = xmltodict.parse(xml, force_list=({'Child'},{'Brother'}))\nWith force_list={'Child'} only, the code works as expected.\nWith the above code, I do not get any error message but when checking the result with the \"type\" function, I still have dictionaries where I would expect to get lists.\nI tried other syntax and got error messages.","AnswerCount":1,"Available Count":1,"Score":0.3799489623,"is_accepted":false,"ViewCount":1419,"Q_Id":56613717,"Users Score":2,"Answer":"I think I found the right syntax (it seems to be working as I expected):\nresponse = xmltodict.parse(xml, force_list=('Child','Brother'))\nJust posting in case anyone would look for the same answer in the future.","Q_Score":2,"Tags":"python,xmltodict","A_Id":56636203,"CreationDate":"2019-06-15T19:53:00.000","Title":"Can I \"force_list\" more than one parameter in xmltodict?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I\u2019m trying to send a large parquet file to RDS Postgres using Lambda. When I try to test the lambda function. I\u2019m facing the below error:\nCalling the invoke API action failed with this message: Network Error\nI tried sending with limited rows, I haven\u2019t faced any issue but when I tried to send the whole file which is of 300 mb, then I\u2019m getting the above error.\nCan someone help me with this?","AnswerCount":4,"Available Count":4,"Score":0.0,"is_accepted":false,"ViewCount":7702,"Q_Id":56652783,"Users Score":0,"Answer":"I had this problem when the Lambda function was inside a VPC and couldn't mount my EFS volume via File System Config. This was with a totally empty new function, no code yet.\nNo solution yet but that might get you started down the correct path.","Q_Score":10,"Tags":"python","A_Id":72070877,"CreationDate":"2019-06-18T15:46:00.000","Title":"Calling the invoke API action failed with this message: Network Error","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I\u2019m trying to send a large parquet file to RDS Postgres using Lambda. When I try to test the lambda function. I\u2019m facing the below error:\nCalling the invoke API action failed with this message: Network Error\nI tried sending with limited rows, I haven\u2019t faced any issue but when I tried to send the whole file which is of 300 mb, then I\u2019m getting the above error.\nCan someone help me with this?","AnswerCount":4,"Available Count":4,"Score":0.0,"is_accepted":false,"ViewCount":7702,"Q_Id":56652783,"Users Score":0,"Answer":"Update your browser to latest version. Some browsers with additional built-in features may block necessary script on web page.","Q_Score":10,"Tags":"python","A_Id":63502397,"CreationDate":"2019-06-18T15:46:00.000","Title":"Calling the invoke API action failed with this message: Network Error","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I\u2019m trying to send a large parquet file to RDS Postgres using Lambda. When I try to test the lambda function. I\u2019m facing the below error:\nCalling the invoke API action failed with this message: Network Error\nI tried sending with limited rows, I haven\u2019t faced any issue but when I tried to send the whole file which is of 300 mb, then I\u2019m getting the above error.\nCan someone help me with this?","AnswerCount":4,"Available Count":4,"Score":1.0,"is_accepted":false,"ViewCount":7702,"Q_Id":56652783,"Users Score":9,"Answer":"I got the same error, which was weird because it was working fine for very small files. I tried with a larger file (only 5mb), and got the error message.\nSolution: check the memory allocation for the lambda function. I was using the standard minimum (128mb). Changed it to 1gb, it worked fine. The error message is horrible.","Q_Score":10,"Tags":"python","A_Id":59071261,"CreationDate":"2019-06-18T15:46:00.000","Title":"Calling the invoke API action failed with this message: Network Error","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I\u2019m trying to send a large parquet file to RDS Postgres using Lambda. When I try to test the lambda function. I\u2019m facing the below error:\nCalling the invoke API action failed with this message: Network Error\nI tried sending with limited rows, I haven\u2019t faced any issue but when I tried to send the whole file which is of 300 mb, then I\u2019m getting the above error.\nCan someone help me with this?","AnswerCount":4,"Available Count":4,"Score":0.0,"is_accepted":false,"ViewCount":7702,"Q_Id":56652783,"Users Score":0,"Answer":"I get this error every time I set up a Lambda, and then it disappears over the space of a day. Not sure why though.","Q_Score":10,"Tags":"python","A_Id":58645065,"CreationDate":"2019-06-18T15:46:00.000","Title":"Calling the invoke API action failed with this message: Network Error","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I am given a network packet whose last 64 bytes (128 hex characters) are the RSA-512 digital signature of the SHA-256 hash of the packet. I take a truncated version of this packet (everything except the last 64 bytes) and calculate the hash myself, which is working fine, however I need a way to get back the hash that generated the signature in the first place\nI have tried to do this in Python and have run into problems because I don't have the RSA private key, only the public key and the Digital Signature. What I need is a way to take the public key and signature and get the SHA-256 hash back from that to compare it to the hash I've generated. Is there a way to do this? Any crypto libraries would be fine. I am using hashlib to generate the hash","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":902,"Q_Id":56676828,"Users Score":0,"Answer":"The original hash was signed with the private key. To get the original hash, you need to decrypt the signature with the public key, not with the private key.","Q_Score":1,"Tags":"python,rsa,digital-signature","A_Id":56679386,"CreationDate":"2019-06-19T22:44:00.000","Title":"How to get the original hashed input to an RSA digital signature?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I want to be able to search for any user using facebook API v3.3 in python 3. \nI have written a function that can only return my details and that's fine, but now I want to search for any user and I am not succeeding so far, it seems as if in V3.3 I can only search for places and not users\n\nThe following function search and return a place, how can I modify it so that I can able to search for any Facebook users?\n\ndef search_friend():\n graph = facebook.GraphAPI(token)\n find_user = graph.search(q='Durban north beach',type='place')\n print(json.dumps(find_user, indent=4))","AnswerCount":1,"Available Count":1,"Score":0.1973753202,"is_accepted":false,"ViewCount":116,"Q_Id":56687213,"Users Score":1,"Answer":"You can not search for users any more, that part of the search functionality has been removed a while ago.\nPlus you would not be able to get any user info in the first place, unless the user in question logged in to your app first, and granted it permission to access at least their basic profile info.","Q_Score":0,"Tags":"python-3.x,facebook-graph-api","A_Id":56687408,"CreationDate":"2019-06-20T13:36:00.000","Title":"how can i search for facebook users ,using facebook API(V3.3) in python 3","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I automated the Test Suite using Robot Framework-Selenium base on PYTHON 27. This same Test Suite needs to execute on client side. \nBut Company do not want to share code base with client. \nIs there anyway that I can create binary file of Robot Framework and share the same binary file with client instead of sharing a actually code ?\nlike in Java we create .jar file.\nKindly help me I am looking for solution from last three days.","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":343,"Q_Id":56709077,"Users Score":0,"Answer":"I don't think there's anything you can do. Robot doesn't have the ability to parse binary files, so any solution would have to include converting your binary files to text files on the client machine.","Q_Score":1,"Tags":"python,robotframework,executable,robotium","A_Id":56709784,"CreationDate":"2019-06-21T19:00:00.000","Title":"How to hide robot framework code base and share only binary file to client?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"So i have this homework, i need to do a email client that sends emails, notifies when a new email arrives,etc.\nOne of my problems is verifying a domains reputation if the user writes an url in the body or subject of the email, if the domain can be danger i shouldn't send the email. I tried mywot.com api, but i can't get a key to try coding.\nI searched for other apis like domaintools, whoisxml, urlvoid but they have a ton of documentation, and i just get lost reading all of it, also they services are limited to free users.\nIs there another api that i can try? o there's another way to valid user urls?\nThanks for your answers.","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":276,"Q_Id":56711961,"Users Score":0,"Answer":"most of them are free until a certain number of queries are hit (like 1000 month). \nFor a fully free API you'll probably need to implement it yourself, since indexing domain lists takes a lot of effort.\nCheck the free intervals of senderbase and virustotal","Q_Score":0,"Tags":"python,json,xml,url,dns","A_Id":61699316,"CreationDate":"2019-06-22T01:39:00.000","Title":"How to see Domains reputation with python?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I am running some crawls in order to test if the results I get deviate. For this effort I created two test suites, the first one I created with the requests and BeautifulSoup library, the other one is based on selenium. I would like to find out, if pages detect both bots in the same way. \nBut I am still unsure if I am right, by assuming that requests and BeautifulSoup are independent from Selenium. \nI hope its not a dump question, but I haven't find any proper answer yet (maybe because of the wrong keywords). However, any help would be appreciated. \nThanks in advance\nI checked the requests documentation. I wrote a mail to the developer, without any answer. And of course I checked on google. I found something about scrapy vs selenium but well... are requests and BeautyfulSoup related to scrapy?","AnswerCount":2,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":2861,"Q_Id":56728377,"Users Score":1,"Answer":"The python requests module does not use Selenium, neither does BeautifulSoup. Both will run independent of a web browser. Both are pure python implementations.","Q_Score":1,"Tags":"python,selenium,python-requests","A_Id":56728393,"CreationDate":"2019-06-23T23:19:00.000","Title":"Does requests rely on selenium?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I wrote a web scraper in Python, which works very well on my Laptop at home. After deploying it to AWS EC2 the performance of the scraper deteriorates. Now I am confused about the performance of EC2 instances (even of the micro and small instances, see further details below).\nScraper in python:\nGenerally, inner loop of the scrapes does the following:\n(1) the scraper looks up urls on a site (20 per site, one site = one \"site_break\"). In a second step it (2) gets the soruce code of each url, in a third step it (3) extracts the necessary information into an dataframe and in the fourth step it (4) saves the dataframe as pkl.\nAfter all loops it opens and merges the dataframs and saves it as csv.\nThe crucial (most time consuming parts) are:\n(2) download of source codes (I\/O limited by download speed): the program fills the RAM with the source code\n(3) processing of the sources codes (CPU 100%)\nTo use the RAM fully and stick together similar processes, every loop consists of site_break = 100, i.e. 100 sites * 20 urls\/site = 2000 urls. This fills the RAM of my PC to 96% (see below). Since I have to wait for server responses in step 1 and step 2, I implemented threading with maxWorkers=15 (alternatively 20-35 with similar results). This implementation cuts the run time by 80%. I am sure I could get some other .% by implementing asyncio. Nevertheless, I want to start with the lean MVP. In the processor consuming step 3 I didn't implement multiprocessing (yet), because my goal was an cost efficient\/free implemenation on t2.micro (with just one processor).\nSpecification:\nHome PC: Intel Core i7-6500 CPU, 2.59 Ghz (2 Cores, 4 logical Processors), RAM 8.00 GiB, 64-bit, x64, 50Mbit\/s Download-rate (effectively up to 45 Mbit\/s), Python 3.7.3, conda env\nEC2 t2.micro: vCPUs = 1, RAM 1.0 GiB, Network Performance \"low to moderate\" (research in forums tell my this could be something above 50 Mbit), Ubuntu 18.04, Python 3.7.3, conda env\nEC2 t3a.small: vCPUs = 2, RAM 2.0 GiB, Network Performance \"low to moderate\" but another AWS site tells me: \"up to 5 Gbit\/s\", Ubuntu 18.04, Python 3.7.3, conda env\nSince the RAM of the t2.micro is just 1 GiB, I lowered the site_break from 100 to 25. Afterwards, the RAM still got full, so I decreased it in further steps from 25 to 15, 12, 10 and finally 5. For 12, 10 and especially for 5 it works pretty well:\nI needed 5:30min for on loop with site_break = 100 on my PC. t2.micro need 8-10sec for site_break = 5, which leads to 3:00min for analogous 100 sites, which satisfied me in the first moment.\nUnfortunately, the following issue appears: After 20-30 loops the performance plumments. The time for on loop increases rapidly from 8sec to over 2min. My first assumption was it to be the low RAM, during the small loops it doesn't seem run full. After stopping and cleaning the RAM, the performance drops after the second or third loop. If I start it a few hours later the first case (with drop after 20-30 loops) repeats.\nBecause I firstly thought it has to do with the RAM, i launched a second instance on t3a.small with more CPU, RAM and \"up to 5 Gbit\/a\" network performance. I sliced to looks to site_break = 25 and startet the script. I is still running with a constant speed of 1:39-1:55min per loop (which is half as fast as t2.micro in its best phase (10 sec for 5 => 50 sec for 25).\nParallely, I started the script from my home PC with site_break = 25 and it is constantly faster with 1:15-1:30min per loop. (Stopping the time manuall results in 10-15sec slower for downloading and 10-15 sec slower for processing).\nThis all confuses me.\nNow my questions:\n\nWhy does the t2.micro detetoriate after several loops and why does the performance vary so wildly?\nWhy is the t3a.small 50% slower than the t2.micro? I would assume that the \"bigger\" machine would be faster in any regard.\n\nThis lets me stuck:\n\nDon't want to use my home PC for regularly (daily scraping), since the connection aborts at 4am for a tiny period of time and leads to hanging up of the script). Moreover, I don't want the script run manually and the PC all the time and block my private internet stream.\n\nt2.micro: Is useless, because the performance after the deterioration is not acceptable.\n\nt3a.small: performance is 10-20% lower than private PC. I would expect it to be better somehow? This lets my doubt to scrape over an EC2. Moreover, I can't understand the lower performance in comparison to t2.micro at the beginning.","AnswerCount":2,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":830,"Q_Id":56734108,"Users Score":1,"Answer":"Why does the t2.micro detetoriate after several loops and why does the performance vary so wildly?\n\nIf your RAM is not getting full, then this is most likely because Amazon is limiting the resources your instance is consuming whether that is CPU or I\/O. Amazon will give you more compute and throughput for a while (to accommodate any short-term spikes) but you should not mistake that for baseline performance.\n\nWhy is the t3a.small 50% slower than the t2.micro? I would assume that the \"bigger\" machine would be faster in any regard.\n\nT3 instances are designed for applications with moderate CPU usage that experience temporary spikes in use. With t3 you are either paying a premium to be able to afford larger and more frequent spikes, or, you are getting less baseline performance (for the same price) to be able to afford larger and more frequent spikes. This does not match the web-scraping profile where you want constant CPU and I\/O.","Q_Score":1,"Tags":"python,performance,amazon-web-services,amazon-ec2,web-scraping","A_Id":64767158,"CreationDate":"2019-06-24T09:57:00.000","Title":"Web Scraping Performance issues EC2 vs Home PC","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I found upload_from_file and upload_from_filename, but is there a function or method to upload an entire folder to Cloud Storage via Python?","AnswerCount":4,"Available Count":2,"Score":0.0,"is_accepted":false,"ViewCount":2959,"Q_Id":56759262,"Users Score":0,"Answer":"I don't think directly in the Python API, no, but there is in the commandline tool gsutil. You could do a system call from the python script to call out to the gsutil tool as long as you're authenticated on commandline in the shell you're calling the Python from.\nThe command would look something like:\ngsutil -m cp -r <foldername> gs:\/\/<bucketname>","Q_Score":3,"Tags":"python,directory,google-cloud-platform,upload","A_Id":56762660,"CreationDate":"2019-06-25T17:31:00.000","Title":"Upload a folder to Google Cloud Storage with Python?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I found upload_from_file and upload_from_filename, but is there a function or method to upload an entire folder to Cloud Storage via Python?","AnswerCount":4,"Available Count":2,"Score":0.0,"is_accepted":false,"ViewCount":2959,"Q_Id":56759262,"Users Score":0,"Answer":"Google Cloud Storage doesn't really have the concept of \"directories\", just binary blobs (that might have key names that sort of look like directories if you name them that way). So your current method in Python is appropriate.","Q_Score":3,"Tags":"python,directory,google-cloud-platform,upload","A_Id":56780840,"CreationDate":"2019-06-25T17:31:00.000","Title":"Upload a folder to Google Cloud Storage with Python?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"So when I run my python selenium script through Jenkins, how should I write the driver = webdriver.Chrome()\nHow should I put the chrome webdriver EXE in jenkins?\nWhere should I put it?","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":44,"Q_Id":56784344,"Users Score":0,"Answer":"If you have added your repository path in jenkins during job configuration, Jenkins will create a virtual copy of your workspace. So, as long as the webdriver file is somewhere in your project folder structure and as long as you are using relative path to reference it in your code, there shouldn't be any issues with respect to driver in invocation.\nYou question also depends on several params like:\n1. Whether you are using Maven to run the test\n2. Whether you are running tests on Jenkins locally or on a remote machine using Selenium Grid Architecture.","Q_Score":0,"Tags":"python,selenium,jenkins,webdriver","A_Id":56784544,"CreationDate":"2019-06-27T05:15:00.000","Title":"So when I run my python selenium script through jenkins, how should I write the 'driver = webdriver.Chrome()'?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"Can anyone help me to grasp this paragraph :\n\u2022 JSON and XML have good support for Unicode character strings (i.e., humanreadable text), but they don\u2019t support binary strings (sequences of bytes without\na character encoding). Binary strings are a useful feature, so people get around\nthis limitation by encoding the binary data as text using Base64. The schema is\nthen used to indicate that the value should be interpreted as Base64-encoded.\nThis works, but it\u2019s somewhat hacky and increases the data size by 33%\nIf i understand well, for example i use a REST API on my equipement to get some information with python.\nFor each response, it's in JSON and it's in unicode format.\nSo OK\nBut i don't really grasp the story of binary string.\nIs it the fact that each character is not in UTF8 format ?\nWhy my equipement don't response in JSON encoded in byte and not in unicode ?","AnswerCount":2,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":677,"Q_Id":56790769,"Users Score":0,"Answer":"JSON and XML are text formats. They're designed to be readable by people, displayed and manipulated with typical text editors like vi, emacs, or edlin, using whatever character encoding scheme is convenient for the platform. All the character encoding schemes in use, however, have some data patterns that are not allowed to represent characters -- for instance, they may have a pattern that indicates \"end-of-string\".\nIf you want to include completely arbitrary data in JSON or XML, you need a way to encode that data in valid text strings and indicate the text is not the actual data you want to use. Base64 is one way of doing that encoding, and it is commonly used for this purpose. The disadvantages are the overhead of doing the encoding and decoding when writing or reading the data, the encoded text string is slightly larger than the original arbitrary binary, and that you have to remember to do the encoding and decoding.","Q_Score":0,"Tags":"python,json","A_Id":56791501,"CreationDate":"2019-06-27T12:15:00.000","Title":"JSON, response in unicode or binary","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":1,"System Administration and DevOps":0,"Web Development":0},{"Question":"Problem\nI have a Lambda function in Account 1, which retrieves EC2 instance status.\nSimilarly, I want to retrieve EC2 instance status in other 4 accounts.\nWhat I did\nI Created trust relationship with the other 4 account by updating the IAM role.\nQuestion:\nWill my python code (residing in my lambda function in account 1) is enough to retrieve ec2 instance status from the other 4 accounts? or should I do something more ?\nPlease suggest!\nThanks in advance.","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":22,"Q_Id":56809483,"Users Score":1,"Answer":"Each AWS Account is separate. You cannot access details of one AWS Account from another AWS Account. However, you can temporarily assume an IAM Role from another account to gain access.\nInstead, you will need to:\n\nCreate an IAM Role for the central Lambda function (Lambda-Role) and grant it permission to call AssumeRole\nCreate an IAM Role in each account that you wish to access, grant it permission to call DescribeInstances and configure it to trust Lambda-Role\nThe Lambda function can then loop through each account and:\n\n\nCall AssumeRole for that account, which will return temporary credentials\nUse those credentials to call DescribeInstances","Q_Score":0,"Tags":"python-3.x,amazon-web-services,aws-lambda","A_Id":56813988,"CreationDate":"2019-06-28T15:23:00.000","Title":"Is my python code enough to retrieve ec2 instance status from other accounts?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I want to create a program that runs automatically every time I download a file from browser.\nFor example, when I download an image file from chrome, the program runs automatically and perform tasks. Is it possible?","AnswerCount":2,"Available Count":1,"Score":0.1973753202,"is_accepted":false,"ViewCount":2023,"Q_Id":56815065,"Users Score":2,"Answer":"I think you need some kind of \"listening\" script running on background which will monitor files in download directory","Q_Score":2,"Tags":"python","A_Id":56815135,"CreationDate":"2019-06-29T04:26:00.000","Title":"How to make python program run automatically every time I download a file?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"My setup:\nI am using python3 and the socket module to setup a socket using ipv4 and tcp, listening and receiving data via an open port. I have a simple server and client where the client sends a request and the server responses. \nMy problem:\nI did some research and many people suggest that an open port is like an open door so how can I lock it? My goal is to secure my server and my client and not the data that is transmitted (which means the data shouldn't be altered but it does not matter if somebody reads it). I just want to make sure that neither the server nor the client receives wrong data or can be hacked in any way. If both the server and the client are normal computers with build-in firewalls are those sufficient?\nQuestions:\n\nHow can I make sure that the data I transmit can't be altered?\nIs the firewall (normal firewall that every computer has built-in) of the server sufficient when listening, receiving and sending data via an open port? If not what can I do to make sure the server can't be hacked in any way (obviously not entirely but as good as possible)?\nSame as question 2. just for a client (which as far as I am concerned does use an open port or at least not like the server)\n\nPS: If possible using python.","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":412,"Q_Id":56816388,"Users Score":0,"Answer":"A Basic level of security for the server side would be to send a random key along with the data for verification of trusted client. If the list of clients that are going to send data are known you can just whitelist the IP addresses which accept data only from a specific list of IP addresses.","Q_Score":1,"Tags":"python,sockets,security,tcp,server","A_Id":56816527,"CreationDate":"2019-06-29T09:15:00.000","Title":"How to secure a python socket listening, sending and receiving on an open port?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I am creating a Discord bot that needs to check all messages to see if a certain string is in an embed message created by any other Discord bot. I know I can use message.content to get a string of the message a user has sent but how can I do something similar with bot embeds in Python?","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":467,"Q_Id":56822722,"Users Score":0,"Answer":"Use message.embeds instead to get the embed string content","Q_Score":1,"Tags":"python,bots,embed,message,discord","A_Id":56823113,"CreationDate":"2019-06-30T05:53:00.000","Title":"Using a Discord bot, how do I get a string of an embed message from another bot","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I am trying to interact (i.e. clicking buttons within app, taking keyboard input and typing for me, etc.) with the Cisco AnyConnect Client, specifically. \nHowever, I would like to know if there is way that is viable with Python 3 to interact with any other Mac applications.","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":58,"Q_Id":56839710,"Users Score":0,"Answer":"I was able to do this using PyAutoGUI like @Mark Setchell stated","Q_Score":0,"Tags":"python-3.x,macos,user-interface","A_Id":57814669,"CreationDate":"2019-07-01T17:09:00.000","Title":"Is there a way to interact with Mac Applications with Python 3?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I am looking to use a public API running on a distant server from within my company. For security reasons, I am supposed to redirect all the traffic via the company's PROXY. Does anyone know how to do this in Python?","AnswerCount":2,"Available Count":2,"Score":0.0996679946,"is_accepted":false,"ViewCount":312,"Q_Id":56841342,"Users Score":1,"Answer":"Set the HTTP_PROXY environment variable before starting your python script\ne.g. export HTTP_PROXY=http:\/\/proxy.host.com:8080","Q_Score":0,"Tags":"python,proxy,api","A_Id":56841343,"CreationDate":"2019-07-01T15:01:00.000","Title":"Configure proxy with python","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I am looking to use a public API running on a distant server from within my company. For security reasons, I am supposed to redirect all the traffic via the company's PROXY. Does anyone know how to do this in Python?","AnswerCount":2,"Available Count":2,"Score":0.1973753202,"is_accepted":false,"ViewCount":312,"Q_Id":56841342,"Users Score":2,"Answer":"Directly in python you can do :\nos.environ[\"HTTP_PROXY\"] = http:\/\/proxy.host.com:8080.\nOr as it has been mentioned before launching by @hardillb on a terminal :\nexport HTTP_PROXY=http:\/\/proxy.host.com:8080","Q_Score":0,"Tags":"python,proxy,api","A_Id":56841409,"CreationDate":"2019-07-01T15:01:00.000","Title":"Configure proxy with python","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I am having issues accessing my Binance account information via API in Python. It always gives the APIError Exception but I am able to ping the exchange and get candlestick data successfully. I read through the API documentation and made sure that the API key is valid and I don't think I am missing anything.\nbinance_client = BinanceClient(api_key=api_key, api_secret=api_secret)\n print(binance_client.get_account(recvWindow=60000000))","AnswerCount":2,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":4680,"Q_Id":56843717,"Users Score":0,"Answer":"maximum value for recvWindow is 60000.","Q_Score":1,"Tags":"python,binance","A_Id":65251985,"CreationDate":"2019-07-01T23:49:00.000","Title":"binance.exceptions.BinanceAPIException: APIError(code=-1022): Signature for this request is not valid","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I am working with python flask's requests module. I have installed requests module using :\npip install requests\nAnd verified that requests module exists when I run :\npip list\nBut when I run my python application , I receive import Error for requests module.\nI noticed that pip is installing the module in C:\\Users\\xx\\Documents\\Projects\\Python\\Python3REST\\lib\\site-packages\\ folder BUT the interpreter is looking for the module in C:\\Users\\xx\\Documents\\Projects\\Python\\Python3REST\\lib\\site-packages\\flask\\ folder.\nI have tried running the command :\npip install --install-option=\"Path to install in\" requests\nBut threw some other error. \nThe import error I am getting states :\nImportError: cannot import name 'requests' from 'flask'\n(C:\\Users\\xx\\Documents\\Projects\\Python\\Python3REST\\lib\\site-packages\\flask\\__init__.py)\nI am working in virtualenv in Windows 10. \nI appreciate any help I can get.\nThank you","AnswerCount":2,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":5384,"Q_Id":56861605,"Users Score":1,"Answer":"what if you add that folder to your path? using sys.path.extend?","Q_Score":2,"Tags":"python,pip,virtualenv","A_Id":56861629,"CreationDate":"2019-07-03T00:52:00.000","Title":"pip list shows installed module but still getting import error","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":1,"System Administration and DevOps":0,"Web Development":1},{"Question":"I have a rather complex Flask project structure set up that uses blueprints to separate different \"modules\" from one another, and then different routes in each module. I now want to add socket functionalities to this project using the flask-socketio library. My goal is to assign each socket to it's own thread, managed by an object. The problem I am facing now is that I'm unsure how to properly separate the sockets from one another.\n(Please note that I am very deep in experimentation phase right now, I don't have a final concept yet and therefore no conclusive code snippets to show)\nFrom my understanding each socket has a unique socket-ID, which is stored in flask.request.sid while the context is within a socketio-event. That is great, because I can use this to match events to the open socket, meaning I know which \"disconnect\"-event belongs to which \"connect\"-event, since both have the same socket-ID if they are part of the same socket connection.\nNow the problems start though. I am attempting to find a way that based on which route the socket is called from the socket has different event handlers. Meaning the route ...\/admin\/a manages different events than ...\/admin\/b, because both routes lead to different logical parts of my web application. On an even larger scale, I only have one global socketio-object shared over all blueprints of my application. If I add an event handler to react to socket feedback of blueprint 1, I absolutely don't want blueprint 2 to be able to also trigger it with it's own sockets. \nWhat are the recommended ways of handling socket separation? Is that even something that's getting used in practise or do I have a fundamentally wrong understanding of how socket connections should be used in web applications?","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":441,"Q_Id":56867887,"Users Score":2,"Answer":"If you have different logical groups of socket events that you want to be independent of each other, you can use a different namespace for each. In Socket.IO one or more namespaces can be multiplexed on the same physical transport.\nIf you prefer something simpler, just avoid conflicts in the name that you assign to your events.","Q_Score":1,"Tags":"python,flask,socket.io,flask-socketio","A_Id":56878579,"CreationDate":"2019-07-03T10:18:00.000","Title":"How to separate sockets for logically separated Flask routes?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I just wonder how apache server can know the domain you come from you can see that in Vhost configuration","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":19,"Q_Id":56875383,"Users Score":0,"Answer":"By a reverse DNS lookup of the IP; socket.gethostbyaddr(). \nResults vary; many IPs from consumer ISPs won't resolve to anything interesting, because of NAT and just not maintaining a generally informative reverse zone.","Q_Score":0,"Tags":"python","A_Id":56875761,"CreationDate":"2019-07-03T17:35:00.000","Title":"How can I find domain that has been used from a client to reach my server in python socket?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I am getting only upto 10000 members when using telethon how to get more than 10000 \nI tried to run multiple times to check whether it is returning random 10000 members but still most of them are same only few changed that also not crossing two digits\nExpected greater than 10000\nbut actual is 10000","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":131,"Q_Id":56884669,"Users Score":0,"Answer":"there is no simple way. you can play with queries like 'a*', 'b*' and so on","Q_Score":0,"Tags":"python,telegram,telethon","A_Id":56884748,"CreationDate":"2019-07-04T09:21:00.000","Title":"how to get the memebrs of a telegram group greater than 10000","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I'm trying to run a python script in Google Cloud which will download 50GB of data once a day to a storage bucket. That download might take longer than the timeout limit on the Google Cloud Functions which is set to 9 minutes.\nThe request to invoke the python function is triggered by HTTP.\nIs there a way around this problem ? I don't need to run a HTTP Restful service as this is called once a day from an external source. (Can't be scheduled) .\nThe whole premise is do download the big chuck of data directly to the cloud. \nThanks for any suggestions.","AnswerCount":2,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":1242,"Q_Id":56888434,"Users Score":2,"Answer":"9 minutes is a hard limit for Cloud Functions that can't be exceeded. If you can't split up your work into smaller units, one for each function invocation, consider using a different product. Cloud Run limits to 15 minutes, and Compute Engine has no limit that would apply to you.","Q_Score":4,"Tags":"python,google-cloud-functions,long-running-processes","A_Id":56888658,"CreationDate":"2019-07-04T12:55:00.000","Title":"Long running python process with Google Cloud Functions","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"The title clearly says it, \nHow do I go about dragging the message author into the voice channel the bot is currently in?\nSay my bot is in a voice channel alone. A command is called to play sound to the author only. \nBut the author isn't in a voice channel, so I can't use the move_to(*) method, hence the word drag.\nI scrounged the API reference for connections, but I can't seem to find any.\nIs it even possible to drag users into a voice channel?","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":179,"Q_Id":56924550,"Users Score":2,"Answer":"Of course you cannot forcibly connect a client to a voice channel - this would directly violate user privacy. Your idea is akin to 'dragging people into phone calls' unexpectedly.","Q_Score":0,"Tags":"python,discord,discord.py,python-3.7,discord.py-rewrite","A_Id":57067105,"CreationDate":"2019-07-07T17:35:00.000","Title":"discord.py rewrite - dragging message author to a voice channel","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I have a list of about 100 HTML webpages (all have different structures, such as divs, anchors, classes, etc.) and I am trying to scrape the title of each page (where the title is under a certain div and class). To do this, I was using get requests and Beautifulsoup, however, this takes way to long (10 minutes every time I want to do it)!\nI used a timer to see what is taking the most time: it's the get requests. Apparently Python (3.7) executes the code one after another, and since each get request takes about 5-6 seconds, it's taking approximately 500-600 seconds to complete the ~100 requests. \nI've searched for ways to make these requests work faster and came across many different solutions. However, a common theme seemed to be that making my requests asynchronous (so all requests start at the same time) will solve the problem (by making it faster).\nThere were many possible solutions for doing this that I read online including: multithreading, using grequest, using Scrapy, parsing lxml, etc. However, I'm new to programming and am not skilled enough to learn and experiment with each way (in fact, I tried following the answers to similar questions on SO, but wasn't successful), so I am unsure what is the best route for me to take. \nI don't need anything fancy; all I want to do is extract the titles from the HTML documents as text and then print them out. I don't need to download any CSS files, images, media, etc. Also, I'm hoping to keep the code as simple\/bare as possible. How can I do this as fast as possible in Python? I'd appreciate it if someone could suggest the best path to take (i.e. using Scrapy), and give a short explanation of what I must do using that tool to get the results I'm hoping for. You don't have to write out the whole code for me. Thanks!","AnswerCount":3,"Available Count":1,"Score":0.0665680765,"is_accepted":false,"ViewCount":154,"Q_Id":56928294,"Users Score":1,"Answer":"One of the idea which I can suggest is taking all the urls in Csv and keep few headings like path,title div,body div, image div as per your requirement and keep adding the particular div(div class=\u201dtitle\u201d).\nEx:\n PATH TITLE DIV IMAGE DIV BODY DIV \nSimilarly, you can give all links in one csv file nd read it through python script so that all data is pulled.","Q_Score":1,"Tags":"python,html,parsing,web-scraping,scrapy","A_Id":56928682,"CreationDate":"2019-07-08T04:05:00.000","Title":"How to scrape many HTML documents quickly using Python?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"The above mentioned error occurred while trying to run a cloud function in Google Cloud Platform. The error occurred in its main.py written in python in the line \"storage_client=storage.Client()\" \nI have also checked the github repository for google-cloud-python\/storage\/google\/cloud\/storage\/_http.py line 33 where it is showing error but I have done nothing with those variables anywhere, I reckon\nAny help will be appreciated","AnswerCount":1,"Available Count":1,"Score":0.3799489623,"is_accepted":false,"ViewCount":369,"Q_Id":56932945,"Users Score":2,"Answer":"I just experienced this same issue-- \nShort answer-- upgrade the google-cloud-core package: e.g. in my case I had \ngoogle-cloud-core==0.29.1\nUpgrading to version 1.0.2 solved my issue:\npip3 install --upgrade google-cloud-core==1.0.2\nFor me, this arose from installing all my python packages from a requirements.txt file, which had explicit versions. Sometime later, I must have upgraded and the packages did not stay aligned.","Q_Score":1,"Tags":"google-cloud-platform,google-cloud-functions,google-cloud-storage,google-api-python-client","A_Id":57914005,"CreationDate":"2019-07-08T10:24:00.000","Title":"How to fix error \"File \"\/...\/google\/cloud\/storage\/_http.py\", line 33, in __init__ TypeError: __init__() takes 2 positional arguments but 3 were given\"","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"Hello I am developing a web scraper and I am using in a particular website, this website has a lot of URLs, maybe more than 1.000.000, and for scraping and getting the information I have the following architecture.\nOne set to store the visited sites and another set to store the non-visited sites.\nFor scraping the website I am using multithreading with a limit of 2000 threads.\nThis architecture has a problem with a memory size and can never finish because the program exceeds the memory with the URLs\nBefore putting a URL in the set of non-visited, I check first if this site is in visited, if the site was visited then I will never store in the non-visited sites.\nFor doing this I am using python, I think that maybe a better approach would be storing all sites in a database, but I fear that this can be slow\nI can fix part of the problem by storing the set of visited URLs in a database like SQLite, but the problem is that the set of the non-visited URL is too big and exceeds all memory\nAny idea about how to improve this, with another tool, language, architecture, etc...?\nThanks","AnswerCount":2,"Available Count":2,"Score":1.2,"is_accepted":true,"ViewCount":108,"Q_Id":56938689,"Users Score":1,"Answer":"At first, i never crawled pages using Python. My preferd language is c#. But python should be good, or better.\nOk, the first thing your detected is quiet important. Just operating on your memory will NOT work. Implementing a way to work on your harddrive is important. If you just want to work on memory, think about the size of the page.\nIn my opinion, you already got the best(or a good) architecture for webscraping\/crawling. You need some kind of list, which represents the urls you already visited and another list in which you could store the new urls your found. Just two lists is the simplest way you could go. Cause that means, you are not implementing some kind of strategy in crawling. If you are not looking for something like that, ok. But think about it, because that could optimize the usage of memory. Therefor you should look for something like deep and wide crawl. Or recursive crawl. Representing each branch as a own list, or a dimension of an array.\nFurther, what is the problem with storing your not visited urls in a database too? Cause you only need on each thread. If your problem with putting it in db is the fact, that it could need some time swiping through it, then you should think about using multiple tables for each part of the page.\nThat means, you could use one table for each substring in url:\nwwww.example.com\/\nwwww.example.com\/contact\/\nwwww.example.com\/download\/\nwwww.example.com\/content\/\nwwww.example.com\/support\/\nwwww.example.com\/news\/\nSo if your url is:\"wwww.example.com\/download\/sweetcats\/\", then you should put it in the table for wwww.example.com\/download\/.\nWhen you have a set of urls, then you have to look at first for the correct table. Afterwards you can swipe through the table.\nAnd at the end, i have just one question. Why are you not using a library or a framework which already supports these features? I think there should be something available for python.","Q_Score":0,"Tags":"python,performance,web-scraping,architecture","A_Id":56953619,"CreationDate":"2019-07-08T16:03:00.000","Title":"What is the best approach to scrape a big website?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"Hello I am developing a web scraper and I am using in a particular website, this website has a lot of URLs, maybe more than 1.000.000, and for scraping and getting the information I have the following architecture.\nOne set to store the visited sites and another set to store the non-visited sites.\nFor scraping the website I am using multithreading with a limit of 2000 threads.\nThis architecture has a problem with a memory size and can never finish because the program exceeds the memory with the URLs\nBefore putting a URL in the set of non-visited, I check first if this site is in visited, if the site was visited then I will never store in the non-visited sites.\nFor doing this I am using python, I think that maybe a better approach would be storing all sites in a database, but I fear that this can be slow\nI can fix part of the problem by storing the set of visited URLs in a database like SQLite, but the problem is that the set of the non-visited URL is too big and exceeds all memory\nAny idea about how to improve this, with another tool, language, architecture, etc...?\nThanks","AnswerCount":2,"Available Count":2,"Score":0.0996679946,"is_accepted":false,"ViewCount":108,"Q_Id":56938689,"Users Score":1,"Answer":"2000 threads is too many. Even 1 may be too many. Your scraper will probably be thought of as a DOS (Denial Of Service) attach and your IP address will be blocked.\nEven if you are allowed in, 2000 is too many threads. You will bottleneck somewhere, and that chokepoint will probably lead to going slower than you could if you had some sane threading. Suggest trying 10. One way to look at it -- Each thread will flip-flop between fetching a URL (network intensive) and processing it (cpu intensive). So, 2 times the number of CPUs is another likely limit.\nYou need a database under the covers. This will let you top and restart the process. More importantly, it will let you fix bugs and release a new crawler without necessarily throwing away all the scraped info.\nThe database will not be the slow part. The main steps:\n\nPick a page to go for (and lock it in the database to avoid redundancy).\nFetch the page (this is perhaps the slowest part)\nParse the page (or this could be the slowest)\nStore the results in the database\nRepeat until no further pages -- which may be never, since the pages will be changing out from under you.\n\n(I did this many years ago. I had a tiny 0.5GB machine. I quit after about a million analyzed pages. There were still about a million pages waiting to be scanned. And, yes, I was accused of a DOS attack.)","Q_Score":0,"Tags":"python,performance,web-scraping,architecture","A_Id":57539538,"CreationDate":"2019-07-08T16:03:00.000","Title":"What is the best approach to scrape a big website?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I need to scrape a web page, but the problem is when i click on the link on website, it works fine, but when i go through the link manually by typing url in browser, it gives Access Denied error, so may be they are validating referrer on their end, Can you please tell me how can i sort this issue out using selenium in python ? \nor any idea that can solve this issue? i am unable to scrape the page because its giving Access Denied error.\nPS. i am working with python3\nWaiting for help.\nThanks","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":322,"Q_Id":56940886,"Users Score":0,"Answer":"I solved myself by using seleniumwire ;) selenium doesn't support headers, but seleniumwire supports, so that solved my issue.\nThanks","Q_Score":0,"Tags":"python","A_Id":56940887,"CreationDate":"2019-07-08T17:12:00.000","Title":"How to set Referrer in driver selenium python?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I was trying to using python requests and mechanize to gather information from a website. This process needs me to post some information then get the results from that website. I automate this process using for loop in Python. However, after ~500 queries, I was told that I am blocked due to high query rate. It takes about 1 sec to do each query. I was using some software online where they query multiple data without problems. Could anyone help me how to avoid this issue? Thanks!\nNo idea how to solve this.\n--- I am looping this process (by auto changing case number) and export data to csv....\nAfter some queries, I was told that my IP was blocked.","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":46,"Q_Id":56943620,"Users Score":0,"Answer":"Optimum randomized delay time between requests. \nRandomized real user-agents for\neach request.\nEnabling cookies.\nUsing a working proxy pool and\nselecting a random proxy for each request.","Q_Score":0,"Tags":"python,web-scraping,python-requests,export-to-csv","A_Id":56954734,"CreationDate":"2019-07-08T23:22:00.000","Title":"While query data (web scraping) from a website with Python, how to avoid being blocked by the server?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I want to build REST API which will communicate with MYSQL DB The application will have some heavy processing after I fetch data from DB and return. Node.js being single threaded might have some issues i fell.\nI want to know if I should go with node.js or python is there any other technology I should be using ?","AnswerCount":1,"Available Count":1,"Score":0.1973753202,"is_accepted":false,"ViewCount":294,"Q_Id":56945173,"Users Score":1,"Answer":"With my exp, python work with MySQL better than NodeJS in multi thread. But i think you should try some other solution if it's really heavy process, like using Spark for data processing.","Q_Score":0,"Tags":"python,node.js,rest,api,backend","A_Id":56945205,"CreationDate":"2019-07-09T04:03:00.000","Title":"Node.js, python for API","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":1,"System Administration and DevOps":0,"Web Development":0},{"Question":"I am currently developing a proprietary PDF parser that can read multiple types of documents with various types of data. Before starting, I was thinking about if reading PowerPoint slides was possible. My employer uses presentation guidelines that requires imagery and background designs - is it possible to build a parser that can read the data from these PowerPoint PDFs without the slide decor getting in the way? \nSo the workflow would basically be this:\n\nAt the end of a project, the project report is delivered in the form of a presentation. \nThe presentation would be converted to PDF.\nThe PDF would be submitted to my application.\nThe application would read the slides and create a data-focused report for quick review.\n\nThe goal of the application is to cut down on the amount of reading that needs to be done by a significant amount as some of these presentation reports can be many pages long with not enough time in the day.","AnswerCount":2,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":201,"Q_Id":56975372,"Users Score":0,"Answer":"A PowerPoint PDF isn't a type of PDF.\nThere isn't going to be anything natively in the PDF that identifies elements on the page as being 'slide' graphics the originated from a PowerPoint file for example.\nYou could try building an algorithm that makes decision about content to drop from the created PDF but that would be tricky and seems like the wrong approach to me.\nA better approach would be to \"Export\" the PPT to text first, e.g. in Microsoft PowerPoint Export it to a RTF file so you get all of the text out and use that directly or then convert that to PDF.","Q_Score":0,"Tags":"python,parsing,pdf,pdf-scraping","A_Id":56994393,"CreationDate":"2019-07-10T16:55:00.000","Title":"Is it possible for a PDF data parser to read PowerPoint PDFs?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":1,"System Administration and DevOps":0,"Web Development":1},{"Question":"By trying to find an optimization to my server on python, I have stumbled on a concept called select. By trying to find any code possible to use, no matter where I looked, Windows compatibility with this subject is hard to find.\nAny ideas how to program a TCP server with select on windows? I know about the idea of unblocking the sockets to maintain the compatibility with it. Any suggestions will be welcomed.","AnswerCount":1,"Available Count":1,"Score":0.1973753202,"is_accepted":false,"ViewCount":98,"Q_Id":56979085,"Users Score":1,"Answer":"Using select() under Windows is 99% the same as it is under other OS's, with some minor variations. The minor variations (at least the ones I know about) are:\n\nUnder Windows, select() only works for real network sockets. In particular, don't bother trying to select() on stdin under Windows, as it won't work.\nUnder Windows, if you attempt a non-blocking TCP connection and the TCP connection fails asynchronously, you will get a notification of that failure via the third (\"exception\") fd_set only. (Under other OS's you will get notified that the failed-to-connect TCP-socket is ready-for-read\/write also)\nUnder Windows, select() will fail if you don't pass in at least one valid socket to it (so you can't use select([], [], [], timeoutInSeconds) as an alternative to time.sleep() like you can under some other OS's)\n\nOther than that select() for Windows is like select() for any other OS. (If your real question about how to use select() in general, you can find information about that using a web search)","Q_Score":0,"Tags":"python,windows,sockets,select","A_Id":56979188,"CreationDate":"2019-07-10T21:47:00.000","Title":"TCP Socket on Server Side Using Python with select on Windows","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I am currently using python to write some appium test. Because I am behind a corporate firewall my traffic needs to go via a proxy. \nI have set my http_proxy and https_proxy variables, but it seems like this is not being picked up by python during execution.\nI tried the exact same test using javascript and node and the proxy get picked up and everything works so I am sure the problem is python not following the proxy settings.\nHow can I make sure python is using correct proxy settings?\nI am using python 2.7 on macos mojave\nThanks!","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":154,"Q_Id":56987653,"Users Score":0,"Answer":"So I figured out that appium currently does not support options to provide a proxy when making a remote connection. As temporary solution I modified the remote_connection module of selenium that appium inherits forcing it to use a proxy url for the connection.\nMy python knowledge is not that good but I think it shoudnt take much effort for someone to make a module that wraps\/override the appium webdriver remote connection to include a proxy option.","Q_Score":0,"Tags":"python","A_Id":57199327,"CreationDate":"2019-07-11T10:56:00.000","Title":"python ignores environmet proxy settings","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I wanted to get the waiting time of vehicles in SUMO and work it into the TraCI interface. For example I want to receive the getwaitingtime() of each vehicle Id within a certain area of the network.(meaning they are stopped or waiting in a queue). Then I want to add the total waiting time of Vehicles based on lane or each direction. After the total Time is added I want to assign this value to lets say X. and Use the value of X to perform some mathematical calculations and give decision to change the traffic light.\ngetwaitingtime(). VehID().","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":607,"Q_Id":56998904,"Users Score":0,"Answer":"When the vehicle is stopped it doesn't accumulate waiting time. Rather you can do this check using isStopped command. This will return True for each simulation step the vehicle is stopped or is in a stopped state.\nAs for the accumulation of waiting time, the waiting time counter is set to 0 each time the vehicle's speed is greater than 0.1 ms. So getWaitingTime might not give you an accurate measure of the total waiting time for a single vehicle. Use getAccumulatedWaitingTime to get the accumulated waiting time for the pre-defined or user-defined waiting time memory. This accumulated waiting time can be tested against the simulation time steps (aggregate) and then you can know for sure if the vehicle has been in the queue for a long time or not.","Q_Score":0,"Tags":"python,sumo,traffic-simulation","A_Id":57004916,"CreationDate":"2019-07-12T00:38:00.000","Title":"SUMO TraCi : How to assign the getwaitingtime() of a VehId and add that total Waiting time per lane?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I have tried automating Facebook login using Selenium WebDriver. Can I do the same automatic login on Android applications instead of web pages? And if yes, what dependency do I need? Or do I need some sort of API?","AnswerCount":2,"Available Count":2,"Score":0.0,"is_accepted":false,"ViewCount":82,"Q_Id":57042809,"Users Score":0,"Answer":"the answer is no selenium dosen't supp mobile app but you can use Appium a good free open source tool.","Q_Score":0,"Tags":"python,python-3.x,python-2.7,selenium,selenium-webdriver","A_Id":57042967,"CreationDate":"2019-07-15T15:23:00.000","Title":"Selenium automatic login can be used on web pages; can it also be used on Android apps?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I have tried automating Facebook login using Selenium WebDriver. Can I do the same automatic login on Android applications instead of web pages? And if yes, what dependency do I need? Or do I need some sort of API?","AnswerCount":2,"Available Count":2,"Score":0.0,"is_accepted":false,"ViewCount":82,"Q_Id":57042809,"Users Score":0,"Answer":"One option how I do it is to run the android app in an emulator such as Android Studio on Windows, and then use a Windows automation tool like AppRobotic Personal to simulate user actions, allowing me to log in on the android app.","Q_Score":0,"Tags":"python,python-3.x,python-2.7,selenium,selenium-webdriver","A_Id":57043197,"CreationDate":"2019-07-15T15:23:00.000","Title":"Selenium automatic login can be used on web pages; can it also be used on Android apps?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I'm using rpyc server to get data using selenium when a connection to a client is established, the problem is that the url I'm trying to access occasionally prompts a reCaptcha to fill in order to access the data needed.\nI don't really need to find a way to automate a completion, what I do want is to find a way to stream the browser from the server to the client if it encounters a reCaptcha, in a manner that allows the user to interact with the browser, and fill the reCaptcha manually himself, and from there to let the server go on with the rest of his code.\nSomething similar to Teamviewer's functionality, to implement in my setup.\nI actually couldn't find any direction to follow on that subject yet, and couldn't figure out a method to try myself.","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":70,"Q_Id":57053912,"Users Score":0,"Answer":"If you work with Selenium, then you have the opportunity to programmatically wait for elements or to detect elements. You could just have your program wait for the ReCaptcha element and make an output to the user in the console that he should solve the ReCaptcha. In the background your program already waits for the elements that appear when the ReCaptcha is solved. Once the user has solved the ReCaptcha, the program will automatically resume.","Q_Score":0,"Tags":"python,recaptcha,rpyc","A_Id":57054034,"CreationDate":"2019-07-16T09:15:00.000","Title":"Is it possible to let the client interact with recaptcha on the server side?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I made a chatbot in AWS Lex and deployed it to facebook. However, I want the chat to display a typing animation before the bot replies. Right now the bot just replies instantly. I want there to be a typing animation to make the bot been more human like. \nIs there some setting in FB Developer I can turn on? I can't seem to find it anywhere. All I see are things for the API call but I am not using any REST calls in my chatbot.","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":25,"Q_Id":57096570,"Users Score":0,"Answer":"What if you wait 1 second before sending the message?","Q_Score":0,"Tags":"python,amazon-web-services","A_Id":57096707,"CreationDate":"2019-07-18T14:12:00.000","Title":"AWS Lex and Facebook typing animation","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I'm trying to download files from a website using python requests module and beautifulsoup4 but the problem is that you have to wait for 5 seconds before the download button appears.\nI tried using requests.get('URL') to get the page and then parse it with beautifulsoup4 to get the download link but the problem is that you have to wait 5 seconds (if you were to open it with an actual browser) in order for the button to appear so when I pass the URL to requests.get() the initial response object doesn't have the button element I searched a lot on google but couldn't find any results that helped me.\nIs there a way to \"refresh\" the response object? or \"wait\"? that is to update it's contents after five seconds as if it were opened with a browser?\nI don't think this is possible with the requests module. What should I do?\nI'm running Windows10 64x \nI'm new so sorry if the formatting is bad. :(","AnswerCount":2,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":1237,"Q_Id":57108768,"Users Score":0,"Answer":"HTTP is stateless, every new request goes as a different request to the earlier one. We typically imeplement states in cookies, browser stoarges and so on. Being a plain HTTP client, there is no way for requests to refresh a request, and the next request will be a compleletly new request.\nWhat you're looking for is some client that understands JavaScript and can handle page update automatically. I suggest you to look at selenium which can do browser automation.","Q_Score":1,"Tags":"python,html,python-requests","A_Id":57108853,"CreationDate":"2019-07-19T08:46:00.000","Title":"Is there a way to \"refresh\" a request?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I wrote a Python script which scrapes a website and sends emails if a certain condition is met. It repeats itself every day in a loop.\nI converted the Python file to an EXE and it runs as an application on my computer. But I don't think this is the best solution to my needs since my computer isn't always on and connected to the internet.\nIs there a specific website I can host my Python code on which will allow it to always run?\nMore generally, I am trying to get the bigger picture of how this works. What do you actually have to do to have a Python script running on the cloud? Do you just upload it? What steps do you have to undertake?\nThanks in advance!","AnswerCount":3,"Available Count":2,"Score":0.1973753202,"is_accepted":false,"ViewCount":1140,"Q_Id":57126697,"Users Score":3,"Answer":"well i think one of the best option is pythonanywhere.com there you can upload your python script(script.py) and then run it and then finish.\ni did this with my telegram bot","Q_Score":1,"Tags":"python,cloud,hosting","A_Id":63171389,"CreationDate":"2019-07-20T16:40:00.000","Title":"How to host a Python script on the cloud?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I wrote a Python script which scrapes a website and sends emails if a certain condition is met. It repeats itself every day in a loop.\nI converted the Python file to an EXE and it runs as an application on my computer. But I don't think this is the best solution to my needs since my computer isn't always on and connected to the internet.\nIs there a specific website I can host my Python code on which will allow it to always run?\nMore generally, I am trying to get the bigger picture of how this works. What do you actually have to do to have a Python script running on the cloud? Do you just upload it? What steps do you have to undertake?\nThanks in advance!","AnswerCount":3,"Available Count":2,"Score":0.1325487884,"is_accepted":false,"ViewCount":1140,"Q_Id":57126697,"Users Score":2,"Answer":"You can deploy your application using AWS Beanstalk. It will provide you with the whole python environment along with server configuration likely to be changed according to your needs. Its a PAAS offering from AWS cloud.","Q_Score":1,"Tags":"python,cloud,hosting","A_Id":57130698,"CreationDate":"2019-07-20T16:40:00.000","Title":"How to host a Python script on the cloud?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"Some modules (for example lxml) return ModuleNotFoundError when I run code with them using python3 command. But when I use python3.6 command everything works well. Why it is so?\nP.S.\npython3 --version returns Python 3.7.2","AnswerCount":1,"Available Count":1,"Score":0.3799489623,"is_accepted":false,"ViewCount":47,"Q_Id":57180069,"Users Score":2,"Answer":"This is due to your environment variables pointing python to 3.7.2 you probably have python 3.6 installed too but your python is pointing to 3.7.2.\nYou can change it under the computer environment variables and changing to path of your python to your 3.6","Q_Score":0,"Tags":"python,python-3.x,python-3.6","A_Id":57180136,"CreationDate":"2019-07-24T09:51:00.000","Title":"ModuleNotFoundError when i use python3","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":1,"System Administration and DevOps":0,"Web Development":0},{"Question":"I have been successfully using the Python Tweepy library to download data (tweets) from Twitter. I had substantial help with my code using Tweepy. For the next stage of my research, I need to access the Premium Search API, which I cannot do using Tweepy. Twitter recommends using TwitterAPI for premium search, available from @geduldig on GitHub. The problem is I'm new to Python and it would be a steep learning curve for me to learn TwitterAPI. Am I able to use TwitterAPI just to access the premium search API, but use Tweepy for other tasks (implement search query, etc)?","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":69,"Q_Id":57186757,"Users Score":0,"Answer":"I do not know specifics about the libraries you asked, but of course, you could use both the libraries in your program. \nYou need to spend a little bit more time grokking the TwitterAPI library. I do not think that amounts to a steep learning curve.","Q_Score":0,"Tags":"python,github,twitter,tweepy,twitterapi-python","A_Id":57186922,"CreationDate":"2019-07-24T15:45:00.000","Title":"Python Novice: where to start with Python and TwitterAPI","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I have a python code in PyCharm in which i am using\nimport requests\nbut the terminal is showing me the following error:\n(venv) ahmad@Ahmad:~\/Desktop\/Spiders$ python test.py\nTraceback (most recent call last):\n File \"test.py\", line 1, in \n import requests\nModuleNotFoundError: No module named 'requests'\nBut I have installed pip and requests as well.","AnswerCount":4,"Available Count":3,"Score":0.0,"is_accepted":false,"ViewCount":19489,"Q_Id":57212573,"Users Score":0,"Answer":"You can just found the icon of Search in right highest corner and search for 'import modules'. Then click on \"Project: [YOUR_PROJECT]\" choose 'Python Interpreter' and then click on 'plus' button. Thus you can install 'requests' module manually.\n(Some sites say that you can call Search by pressing Alt + Enter, but it doesn`t work in my case)","Q_Score":4,"Tags":"python,pip,python-requests","A_Id":68539185,"CreationDate":"2019-07-26T02:58:00.000","Title":"ModuleNotFoundError: No module named 'requests' in PyCharm","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":1,"System Administration and DevOps":0,"Web Development":0},{"Question":"I have a python code in PyCharm in which i am using\nimport requests\nbut the terminal is showing me the following error:\n(venv) ahmad@Ahmad:~\/Desktop\/Spiders$ python test.py\nTraceback (most recent call last):\n File \"test.py\", line 1, in \n import requests\nModuleNotFoundError: No module named 'requests'\nBut I have installed pip and requests as well.","AnswerCount":4,"Available Count":3,"Score":0.0,"is_accepted":false,"ViewCount":19489,"Q_Id":57212573,"Users Score":0,"Answer":"I had to go to the CLi (command line) and run \">pip install requests\" - this after trying unsuccessfully from the Pycharm 'Setting\/Interpreter' gui menu.","Q_Score":4,"Tags":"python,pip,python-requests","A_Id":69285809,"CreationDate":"2019-07-26T02:58:00.000","Title":"ModuleNotFoundError: No module named 'requests' in PyCharm","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":1,"System Administration and DevOps":0,"Web Development":0},{"Question":"I have a python code in PyCharm in which i am using\nimport requests\nbut the terminal is showing me the following error:\n(venv) ahmad@Ahmad:~\/Desktop\/Spiders$ python test.py\nTraceback (most recent call last):\n File \"test.py\", line 1, in \n import requests\nModuleNotFoundError: No module named 'requests'\nBut I have installed pip and requests as well.","AnswerCount":4,"Available Count":3,"Score":1.0,"is_accepted":false,"ViewCount":19489,"Q_Id":57212573,"Users Score":8,"Answer":"you can use this : \njust go to file > sitting > project : name > project interpreter > add > search and install","Q_Score":4,"Tags":"python,pip,python-requests","A_Id":57226478,"CreationDate":"2019-07-26T02:58:00.000","Title":"ModuleNotFoundError: No module named 'requests' in PyCharm","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":1,"System Administration and DevOps":0,"Web Development":0},{"Question":"I have a list of url and many of them are invalid. When I use scrapy to crawl, the engine will automatically filter those urls with 404 status code, but some urls' status code aren't 404 and will be crawled so when I open it, it says something like there's nothing here or the domain has been changed, etc. Can someone let me know how to filter these types of invalid urls?","AnswerCount":3,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":1284,"Q_Id":57216139,"Users Score":0,"Answer":"In your callback (e.g. parse) implement checks that detect those cases of 200 responses that are not valid, and exit the callback right away (return) when you detect one of those requests.","Q_Score":0,"Tags":"python,scrapy,web-crawler","A_Id":57308027,"CreationDate":"2019-07-26T08:32:00.000","Title":"How to check if a url is valid in Scrapy?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I tried to connect it with SVN in aws ec2. It is showing Network timeout. After that i tried scp to transfer again it is showing the same error connection refused. Then i tried AWS S3 ec2 copy command to copy the file from local machine to S3 bucket but again it is showing the same error.","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":13,"Q_Id":57218711,"Users Score":0,"Answer":"The approach which I use to move code or files from local server to ec2 instance is git.\nIt maintains my file changes with proper commit messages as well as a proper set of revisions or iterations I have done until now.","Q_Score":0,"Tags":"python-3.x","A_Id":57218757,"CreationDate":"2019-07-26T11:01:00.000","Title":"AWS to local machine file sharing","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"My gethostbyaddr() is giving a different host name from what I need. The host name that I need does not even show up in the alias list.\nSo, I was confused with the weird hostnames that I get when i try gethostbyaddr for different websites. \nSo, I tried getting the ip of amazon using gethostbyname. Then I used the resulting IP in gethostbyaddr() but I did not get the hostname of amazon. I read the official documentation and it states that the alias list returned contains the alternative host names, but I still do not get www.amazon.com\nSo This is what I tried doing.\nsocket.gethostbyname('www.amazon.com')\nAnd my result was: '13.35.134.162'\nThen I input this IP:\nsocket.gethostbyaddr('13.35.134.162')\nBut my result is:\n('server-13-35-134-162.del54.r.cloudfront.net', [], ['13.35.134.162'])\nCan someone explain why 'www.amazon.com' is not displayed and what is this hostname that I get?","AnswerCount":3,"Available Count":2,"Score":0.0665680765,"is_accepted":false,"ViewCount":343,"Q_Id":57230607,"Users Score":1,"Answer":"A website name is not equal to hostname. There is no 1:1 relationship in general. One computer can serve many websites. OTOH a busy website is served by many computers (so called load balancing). CDNs (content delivery networks) use some BGP-4 tricks (BGP-4 = an important routing protocol) to connect you to a server geographically near you - they run several \"clones\" of a webiste in different locations.\nWhat are your needs? If you want to be sure you are connected to the right website, rely on HTTPS certificates.","Q_Score":0,"Tags":"python,python-3.x,sockets","A_Id":57230747,"CreationDate":"2019-07-27T08:47:00.000","Title":"gethostbyaddr gives a different host name","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"My gethostbyaddr() is giving a different host name from what I need. The host name that I need does not even show up in the alias list.\nSo, I was confused with the weird hostnames that I get when i try gethostbyaddr for different websites. \nSo, I tried getting the ip of amazon using gethostbyname. Then I used the resulting IP in gethostbyaddr() but I did not get the hostname of amazon. I read the official documentation and it states that the alias list returned contains the alternative host names, but I still do not get www.amazon.com\nSo This is what I tried doing.\nsocket.gethostbyname('www.amazon.com')\nAnd my result was: '13.35.134.162'\nThen I input this IP:\nsocket.gethostbyaddr('13.35.134.162')\nBut my result is:\n('server-13-35-134-162.del54.r.cloudfront.net', [], ['13.35.134.162'])\nCan someone explain why 'www.amazon.com' is not displayed and what is this hostname that I get?","AnswerCount":3,"Available Count":2,"Score":0.0665680765,"is_accepted":false,"ViewCount":343,"Q_Id":57230607,"Users Score":1,"Answer":"This has nothing to do with python, but rather how DNS works. A single IP address can host many web sites and thus have many host names. As a result, the name to IP lookup is not always reversible.","Q_Score":0,"Tags":"python,python-3.x,sockets","A_Id":57230754,"CreationDate":"2019-07-27T08:47:00.000","Title":"gethostbyaddr gives a different host name","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I need to parse a page, keeping HTML and JS the same as in my own browser. Site must think, that I am logged using the same browser, I need to \"press\" some buttons using JS and find some elements.\nWhen using requests library or selenium.webdriver.Firefox(), site think I am from a new browser. But I think selenium must help.","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":112,"Q_Id":57232605,"Users Score":0,"Answer":"Requests cannot process JavaScript, nor can it parse HTML and CSS to create a DOM. Requests is just a very nice abstraction around making HTTP requests to any server, but websites\/browsers aren't the only things that use HTTP.\nWhat you're looking for is a JavaScript engine along with an HTML and CSS parser so that it can create an actual DOM for the site and allow you to interact with it. Without these things, there'd be no way to tell what the DOM of the page would be, and so you wouldn't be able to click buttons on it and have the resulting JavaScript do what it should. \nSo what you're looking for is a web browser. There's just no way around it. Anything that does those things, is, by definition, a web browser. \nTo clarify from one of your comments, just because something has a GUI, that doesn't mean it isn't automatic. In fact, that's exactly what Selenium is for (i.e. automating the interactions with the GUI that is the web page). It's not meant to emulate user behavior exactly 1:1, and it's actually an abstraction around the WebDriver protocol, which is meant for writing automated tests. However, it does allow you to interact with the webpage in a way that approximates how a user would interact with it.\nYou may not want to see the GUI of the browser, but luckily, Chrome and Firefox have \"headless\" modes, and Selenium can control headless instances of those browsers. This would have the browser GUI be hidden while Selenium controls it, which sounds like what you're looking for.","Q_Score":0,"Tags":"python,selenium,parsing,web,browser","A_Id":57240490,"CreationDate":"2019-07-27T13:30:00.000","Title":"Navigate webpage as if from my browser (Python, selenium)","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I built a python app that ran happily in its docker container until I added some http calls.. Using the library 'requests.' I did some research and alpine linux 3.7 has a library called 'py-requests'.. Only I can't seem to install this on my own system, to change my code to use it, and errors are thrown just leaving 'import requests' in my code. Py-pip doesn't seem to work either.","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":1115,"Q_Id":57261543,"Users Score":-1,"Answer":"I couldn't use pip install because I was blocked by a corporate firewall. I added --proxy mycompanyproxy.com:80 --trusted-host pypi.org --trusted-host files.pythonhosted.org to my dockerfile's pip install command and was able to get what I needed. Thanks for commenting, and sorry for the vagueness, I am a noob :)","Q_Score":0,"Tags":"python,docker,alpine","A_Id":57317442,"CreationDate":"2019-07-29T21:18:00.000","Title":"How to use py-requests in alpine linux docker?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":1,"Web Development":0},{"Question":"I am trying to implement very simple authentication for updating values on my server. Here's the situation:\nI have a door sensor hooked up to a raspberry pi. Every time the sensor is triggered ('Opened' or 'Closed'), I send a POST request out to my Digital Ocean droplet at 'api.xxxxxx.com' which points to a restify server. The POST request body contains the sensor state, a time-stamp, and an API key. The RESTify server also has a file called 'constants.js' that contains the same API key. If the API key sent from the RPi is the same as the one in the constants file on my droplet, it allows values to update (latest state\/time). If not, it just sends back an error message.\nThe API key is a password sent through SHA3-256.\nIs this scheme okay for what I'm doing? The only thing I could think of is if someone found the endpoint, they might be able to spam requests to it, but nothing else. The API key (on my local raspberry pi and on the droplet) are kept in different files and excluded from git, so viewing git files would not reveal anything.\nI don't expect anyone to have access to my droplet or raspberry pi either, so if I set up SSH correctly I don't see how it (the API key in the files) could be leaked either.\nEDIT: Forgot to say that I'm using Python on the Raspberry Pi to send out POSTs. The droplet is running a RESTify server (JS).","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":48,"Q_Id":57262893,"Users Score":1,"Answer":"Well, you are vulnerable to network snooping. If anyone can snoop either of the network links, then they can steal the API key and are free to use your service with it. \nHTTPS on both links would prevent that. HTTPS could also prevent any sort of DNS hijack that could trick the Pi into sending the APIKey to a false host (thus stealing it that way).\nOther than that, your API key is your secret that controls access so as long as it is secured in storage at both ends and secured in transit and it's sufficiently hard to guess, you are OK.","Q_Score":0,"Tags":"javascript,python,node.js,authentication,restify","A_Id":57263175,"CreationDate":"2019-07-30T00:41:00.000","Title":"Very simple authentication on my own server, is this okay?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"Trying to figure out how to make python play mp3s whenever a tag's text changes on an Online Fantasy Draft Board (ClickyDraft).\nI know how to scrape elements from a website with python & beautiful soup, and how to play mp3s. But how do you think can I have it detect when a certain element changes so it can play the appropriate mp3?\nI was thinking of having the program scrape the site every 0.5seconds to detect the changes,\nbut I read that that could cause problems? Is there any way of doing this?","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":40,"Q_Id":57291451,"Users Score":0,"Answer":"The only way is too scrape the site on a regular basis. 0.5s is too fast. I don't know how time sensitive this project is. But scraping every 1\/5\/10 minute is good enough. If you need it quicker, just get a proxy (plenty of free ones out there) and you can scrape the site more often.\nJust try respecting the site, Don't consume too much of the sites ressources by requesting every 0.5 seconds","Q_Score":1,"Tags":"python,html,beautifulsoup,mp3","A_Id":57292691,"CreationDate":"2019-07-31T13:05:00.000","Title":"Is it possible to write a Python web scraper that plays an mp3 whenever an element's text changes?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I have 2 web elements with the same XPath and name but under the different navigation links. Both links point to the same page but with a different title. How should I access each of the links uniquely?\nI am a beginner to python selenium programming","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":23,"Q_Id":57300618,"Users Score":0,"Answer":"You have the answer in the question itself. You must try identifying the elements using title with combination of other attributes if they have unique titles.\nPS: Please(always) add the code you have tried and the relevant(in this case HTML) required files\/snippet which can help us understand and investigate better. Questions with enough information get faster replies :)","Q_Score":0,"Tags":"python-3.x,selenium-chromedriver","A_Id":57302283,"CreationDate":"2019-08-01T01:05:00.000","Title":"Web element with same name and xpath under two elements in the navigation pane","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I was learning how to play music using selenium so I wrote a program which would be used as a module to play music. Unfortunately I exited the python shell without exiting the headless browser and now the song is continuously playing.\nCould someone tell me how I can find the current headless browser and exit it?","AnswerCount":2,"Available Count":2,"Score":0.0996679946,"is_accepted":false,"ViewCount":84,"Q_Id":57305632,"Users Score":1,"Answer":"You need to include in your script to stop the music before closing the session of your headless browser.","Q_Score":0,"Tags":"python-3.x,selenium-webdriver","A_Id":57306386,"CreationDate":"2019-08-01T09:16:00.000","Title":"Stop music from playing in headless browser","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I was learning how to play music using selenium so I wrote a program which would be used as a module to play music. Unfortunately I exited the python shell without exiting the headless browser and now the song is continuously playing.\nCould someone tell me how I can find the current headless browser and exit it?","AnswerCount":2,"Available Count":2,"Score":0.0996679946,"is_accepted":false,"ViewCount":84,"Q_Id":57305632,"Users Score":1,"Answer":"If you are on a Linux box, You can easily find the process Id with ps aux| grep chrome command and Kill it. If you are on Windows kill the process via Task Manager","Q_Score":0,"Tags":"python-3.x,selenium-webdriver","A_Id":57308855,"CreationDate":"2019-08-01T09:16:00.000","Title":"Stop music from playing in headless browser","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I am trying to add a package to PyPi so I can install it with Pip. I am trying to add it using twine upload dist\/*. \nThis causes me to get multiple SSL errors such as raise SSLError(e, request=request) requests.exceptions.SSLError: HTTPSConnectionPool(host='upload.pypi.org', port=443): Max retries exceeded with url: \/legacy\/ (Caused by SSLError(SSLError(\"bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')])\"))).\nI am using a school laptop and I presume that this is something my administrator has done however I can install stuff with pip by using pip3 install --trusted-host pypi.org --trusted-h\\ost files.pythonhosted.org.\n I was wondering if there was another to add my package to pip?","AnswerCount":2,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":1790,"Q_Id":57354747,"Users Score":5,"Answer":"My guess is your school has something in place where they are replacing the original cert with their own, you could maybe get around it using --cert and referencing the path for your schools cert, but I think an easier workaround is to copy the files to a non school computer and upload from there.","Q_Score":5,"Tags":"python,ssl,pypi,twine","A_Id":57393456,"CreationDate":"2019-08-05T08:24:00.000","Title":"When adding package to PyPi SSL error occurs","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":1,"System Administration and DevOps":0,"Web Development":0},{"Question":"All,\nI have googled this to death but I can't figure out how to parse this type of JSON packet.\n{\n\"recipients\":[\n\"\\\"Name 1\\\" \",\n\"\\\"Name 2\\\" \",\n\"\\\"Name 3\\\" \"\n]\n}\nAny help would be appreciated.","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":12,"Q_Id":57364095,"Users Score":0,"Answer":"very simple, as my code:\nimport json\nj = json.loads('{\"recipients\": [\"Name 1\", \"Name 2\",\"Name 3\"] }')\nprint (j['recipients'])\n=> ['Name 1', 'Name 2', 'Name 3']","Q_Score":0,"Tags":"json,python-2.7","A_Id":57364203,"CreationDate":"2019-08-05T18:19:00.000","Title":"Assistance with parsing Json packet","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":1,"System Administration and DevOps":0,"Web Development":0},{"Question":"I am in a situation where I have two endpoints I can ask for a value, and one may be faster than the other. The calls to the endpoints are blocking. I want to wait for one to complete and take that result without waiting for the other to complete.\nMy solution was to issue the requests in separate threads and have those threads set a flag to true when they complete. In the main thread, I continuously check the flags (I know it is a busy wait, but that is not my primary concern right now) and when one completes it takes that value and returns it as the result. \nThe issue I have is that I never clean up the other thread. I can't find any way to do it without using .join(), which would just block and defeat the purpose of this whole thing. So, how can I clean up that other, slower thread that is blocking without joining it from the main thread?","AnswerCount":2,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":1503,"Q_Id":57377390,"Users Score":0,"Answer":"One bad and dirty solution is to implement a methode for the threads which close the socket which is blocking. Now you have to catch the exception in the main thread.","Q_Score":2,"Tags":"python,python-3.x,multithreading,asynchronous","A_Id":57377511,"CreationDate":"2019-08-06T13:39:00.000","Title":"Clean up a thread without .join() and without blocking the main thread","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":1,"System Administration and DevOps":0,"Web Development":0},{"Question":"I am doing a small project involving web scraping basic html pages hosted on an internal server we use at work. Upon entry to these pages a popup . . . which looks exactly like a windows.prompt() asks for username and a password with a message like \"Enter credentials for Proxy: \". Is there a way to automatically inject and submit these values on the prompt box using JavaScript or even python?\nYes I do have access but every time I go to a different page it will re-prompt me and I am trying to do this for a very large amount of pages.\nI have already tried inspecting the page but there does not seem to be any element for a popup.","AnswerCount":1,"Available Count":1,"Score":0.1973753202,"is_accepted":false,"ViewCount":86,"Q_Id":57401627,"Users Score":1,"Answer":"I suspect you may be getting the login prompt due to the web server's setup for the page and not due to anything included in the content of the page in question. You may need to adjust your app's code to handle the login behavior (setting cookies in your request header for example).","Q_Score":0,"Tags":"javascript,python,html","A_Id":57402523,"CreationDate":"2019-08-07T20:06:00.000","Title":"How to enter in values for a window.prompt() using javascript?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I have a large tar.gz archive file having nxml files and total size is around 5gb.\nMy aim is to extract files from it but, I do not have to extract all of them. I have to extract all those files whose name is greater than a threshold value.\nFor example:\nLet us consider 1000 is our threshold value. So\npath\/to\/file\/900.nxml will not be extracted but\npath\/to\/file\/1100.nxml will be extracted.\nSo my requirement is to make a conditional extraction of files from the archive.\nThanks","AnswerCount":2,"Available Count":1,"Score":0.0996679946,"is_accepted":false,"ViewCount":123,"Q_Id":57406452,"Users Score":1,"Answer":"You can also use --wildcards option of tar.\nFor example in the case when your threshold is 1000 you can use tar -xf tar.gz --wildcards path\/to\/files\/????*.nxml. The ? will match one character and using * will match any number of character. This pattern will look for any file name with 4 or more characters.\nHope this helps.","Q_Score":1,"Tags":"python,bash,gzip,python-2.x,tar","A_Id":57424232,"CreationDate":"2019-08-08T06:31:00.000","Title":"Conditional extraction of files from an Archive file","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":1,"Web Development":0},{"Question":"I am very new to python and cannot seem to figure out how to accomplish this task. I want to connect to a website and extract the certificate information such as issuer and expiration dates.\nI have looked all over, tried all kinds of steps but because I am new I am getting lost in the socket, wrapper etc.\nTo make matters worse, I am in a proxy environment and it seems to really complicate things.\nDoes anyone know how I could connect and extract the information while behind the proxy?","AnswerCount":2,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":1079,"Q_Id":57432064,"Users Score":0,"Answer":"Python SSL lib don't deal with proxies.","Q_Score":0,"Tags":"python,ssl,python-requests,pyopenssl,m2crypto","A_Id":71436836,"CreationDate":"2019-08-09T14:23:00.000","Title":"Trying to extract Certificate information in Python","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I have a program that imports the selenium package. It runs fine inside pyCharm but would not run on command prompt, saying \"No module named selenium\" (Windows 7). Is there an easy way to get all the setup in pyCharm and make them available in the command prompt?","AnswerCount":2,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":29,"Q_Id":57433767,"Users Score":0,"Answer":"This is because you have imported the packages under a virtual environment(venv) . Therefore the package imports are valid only for the virtual environment. So in order to work it globally you have to install it via command prompt without activating a venv.","Q_Score":0,"Tags":"python,python-venv","A_Id":57433864,"CreationDate":"2019-08-09T16:08:00.000","Title":"Making Packages Available Outside PyCharm (Python3)","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":1,"System Administration and DevOps":0,"Web Development":0},{"Question":"Does anyone know of a solution for checking if a tweet has replies or not, without checking the reply_count field of the JSON response? \nI'm building a crawler and already have a method for scraping a timeline for tweets as well as replies to tweets. In order to increase efficiency I want to find out if a tweet has any replies at all before calling my reply method. I have a standard developer account with Twitter so I do not have access to reply_count.","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":123,"Q_Id":57436251,"Users Score":0,"Answer":"Looking for this as well. Only way I found was scraping the page (which is against the Terms of Service)\nreply-count-aria-${tweet.id_str}.*?(\\d+) replies","Q_Score":0,"Tags":"twitter,tweepy,twitterapi-python","A_Id":61215841,"CreationDate":"2019-08-09T19:44:00.000","Title":"Is there a workaround for non-premium Twitter developers for getting reply_count?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I have a conversation where the bot send out multiple replies.\nFor example:\nUser: \u201cHi\u201d\nBot: \u201cHello\u201d\nBot: \u201cHow can I help you?\u201d \nHere the bot sends out multiple messages. In interactive mode this is working fine. However, after I setup the REST API (following this How to serve chatbot from server?), the bot does not send out multiple messages for obvious reasons. How can I get this working?","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":583,"Q_Id":57478153,"Users Score":0,"Answer":"you can use \\n\\n inside a text response to split it into two messages..\nUnfortunately it does not work with button response messages.\nHere You can use custom actions with as many dispatcher.utter_message() events as you want..","Q_Score":2,"Tags":"python,rasa-nlu,rasa-core,rasa","A_Id":71426079,"CreationDate":"2019-08-13T12:53:00.000","Title":"How to make the bot send out multiple messages in sequence over REST API? [Rasa]","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I am receiving a SyntaxError: invalid syntax when using Import requests using python 3.7\nI have re-installed requests. \nImport requests\n... \n File \"\", line 10\n Import requests\n ^\nSyntaxError: invalid syntax\nI did not expect any problem. Using Automate The Boring Stuff with Python by Al Sweigart page 237","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":484,"Q_Id":57482476,"Users Score":0,"Answer":"I believe your issue is that you're using Import with a capital 'I' instead of import with a lowercase 'i'.","Q_Score":0,"Tags":"python-3.x,python-requests,installation,python-import","A_Id":57482527,"CreationDate":"2019-08-13T17:14:00.000","Title":"Syntax error on import requests with python 3.7","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":1,"System Administration and DevOps":0,"Web Development":0},{"Question":"I want to create a webserver on linux in python so that I can use it to get csv files from the linux machine onto a windows machine. I am quite unfamiliar with networking terminology so I would greatly appreciate it if the answer is a little detailed. I dont want to create a website, just the webserver to get the csv file requested.","AnswerCount":2,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":755,"Q_Id":57485604,"Users Score":0,"Answer":"You can create an simple file server, using flask, django or any other python web-server frameworks. There are just some small tips:\n\nYou should send csv files through HTTP POST request to web-server. I think multi-part\/form-data is appropriate for this. \nAn small\/simple Database with track files received and their locations on disk.\nYou can also have something to translate csv files to database records on demand. After that you will be able to do very useful things on them.\n\nFor any more questions, ask different questions to get appropriate\/expert answers. (welcome to SO)","Q_Score":0,"Tags":"python,webserver","A_Id":57485935,"CreationDate":"2019-08-13T21:35:00.000","Title":"How to make a python webserver that gives a requested csv file","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":1,"Web Development":0},{"Question":"I have made a pretty big spider that basically extracts the data from an amazon product page.\nThe problem is that sometimes, no data comes back when I extract. After that happens I check the URL that was processed and, following the xpath with a chrome tool, the data is in fact there.\nI know that what me and the Chrome tool sees is not the same as what the spider processes so, is it there any way to actually see the source code the spider is trying to extract from? and will the XPath I make with the chrome tool's help be trustworthy?","AnswerCount":2,"Available Count":1,"Score":0.0996679946,"is_accepted":false,"ViewCount":497,"Q_Id":57486244,"Users Score":1,"Answer":"Check the view-source with (Ctrl-U in Chrome). Chrome tools will not always line up with the html source. Probably due to the JavaScript on the page.","Q_Score":0,"Tags":"python-3.x,web-scraping,scrapy","A_Id":57501420,"CreationDate":"2019-08-13T22:56:00.000","Title":"Checking source code in a scrapy response","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I want to send multiple, delayed responses from the webhook written in python, once an intent is been triggered by the user. First response I want immediately after triggering the intent and another response I want after some processing to be performed on top of the user utterance.\nFor example:\n\nUser : I want my account balance.\nBOT : Please tell your account number for details.\nUser : my account number is 218497234.\nBOT : Hold-on a bit we are fetching your details.\nBOT : Welcome John, your account balance is $70000.\n\nIn the above example, this is a bank-bot, which replies to user queries. Currently fetching-up account balance for a user supplying account number. The last two responses from the BOT are from the webhook when say \"account_balance_check\" intent is been triggered. First response is immediate, asking the user to be patient and wait for the account details , while the second response is after fetching the account details from a db using account number.\nAnother way could be to trigger response from the bot, without utterance from the user. In the above case, is there anyway, bot itself can send response to user after telling him to wait? Please note that after telling user to wait, we don't want user utterance to trigger second response.","AnswerCount":1,"Available Count":1,"Score":0.1973753202,"is_accepted":false,"ViewCount":331,"Q_Id":57522331,"Users Score":1,"Answer":"Unfortunately, you cannot do that in Dialogflow because once you send the response then your agent will not be able to send the response without any user input or event call.\nHere is an answer if your process does not take a long time. Once you get the user request, send them the waiting message with the \"OK\" suggestion. Once the user clicks on the suggestion, you can show the response. Also, process the request with some API and save your data in a common file that you can access through both API and agent and then show the response to the user from the file.","Q_Score":2,"Tags":"python,dialogflow-es,dialogflow-es-fulfillment","A_Id":57523379,"CreationDate":"2019-08-16T09:40:00.000","Title":"How to send multiple delayed responses from Python Webhook once an intent is been triggered?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I'm developing a discord bot using discord.py (rewrite branch) for my servers and I need to invite the bot to multiple servers and use it simultaneously. \nMy question is: \nDo I need to set up a new thread for every server or does the bot queue events and handle them one by one? if it does queue them, should I just use that or use separate threads?\nSorry if this is a noobish question but I'm fairly new to discord.py and I don't really understand how it works just yet.\nThanks for reading","AnswerCount":2,"Available Count":1,"Score":-0.0996679946,"is_accepted":false,"ViewCount":8081,"Q_Id":57550170,"Users Score":-1,"Answer":"Multiprocesses, threads or queues can all be used to approach this issue each with their respective advantages and disadvantages . Personally I would use threads as the events that need to take place on each server are independent of each other mostly.","Q_Score":3,"Tags":"python,discord.py-rewrite","A_Id":57550391,"CreationDate":"2019-08-19T03:32:00.000","Title":"How does a discord bot handle events from multiple servers","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"We are using selenium python to automate web application. To launch firefox browser, we need to download geckodriver and place it in \/usr\/bin. but, we found that linux version geckodriver is not compatible with Solaris os. whenever I am running selenium python to run code on solaris v5.11 , we got an error like \"Bad System call(core dumped)\"\nsolaris 11.4\npython 2.7.14\nselenium 3.141.0\ngeckodriver 0.24.0\nplease help to resolve the issue","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":78,"Q_Id":57570736,"Users Score":1,"Answer":"Solaris & Linux use very different system calls, and binaries must be compiled specifically for each one - you cannot copy them across from one system to the other - so you will need to either compile geckodriver yourself or find a version already compiled for Solaris, not Linux.","Q_Score":0,"Tags":"python,selenium,solaris,selenium-firefoxdriver","A_Id":57576724,"CreationDate":"2019-08-20T09:37:00.000","Title":"Geckodriver for Solaris OS","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"Scrapy gets 302 redirect to another link. In the link 'https:\/\/xxxxxx.queue-it.net?c.....com' Scrapy does not add the '\/'. It should be'https:\/\/xxxxxx.queue-it.net\/?c.....com'. \nI have tried adding '\/' in middleware.py. Under downloaderMiddleware function. But, it does not work. \nScrapy crawls when I manually add the redirect link with '\/'. However, it is not very dynamic.","AnswerCount":1,"Available Count":1,"Score":0.1973753202,"is_accepted":false,"ViewCount":74,"Q_Id":57602863,"Users Score":1,"Answer":"Set 'dont_redirect': True in the specific request or disable redirect globally by setting REDIRECT_ENABLED setting to False.","Q_Score":0,"Tags":"python,web-scraping,scrapy","A_Id":57604799,"CreationDate":"2019-08-22T06:00:00.000","Title":"Scrapy get's redirected to follow 302 and it does not crawl the site","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I am using paramiko to create a SFTP server. I have succeeded in uploading and downloading files to and from server on client request.But, I need to send a file from server to client whenever I need without client request. So, instead of breaking my head on making server send a file to client I want to make both machines act as both server and client in different ports so that when I need to send a file from machine A to B I can just Upload it to the SFTP server running on that port. Is this hypothesis possible?","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":450,"Q_Id":57615144,"Users Score":0,"Answer":"You already know that you cannot send a file from an server to a client:\nCan I send a file from SFTP Server to the Client without any request from it?\n(The question on Server Fault has been deleted)\n\nTo answer your port question:\nYou do not care about client's port. It is automatically assigned to any available port, without you ever needing to know its value. In general, that's true for any TCP\/IP connection, not only SFTP.\nSo you can just run SFTP server on both machines on the standard port 22. And use your client code on the other machine to connect to it.","Q_Score":0,"Tags":"python,windows,sftp,paramiko","A_Id":57616068,"CreationDate":"2019-08-22T18:23:00.000","Title":"Is it possible to run both SFTP server and client in a same machine on different ports?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"My setup is flask-socketio with a flask-restful webserver.\nEventlet is installed, so in production mode, eventlet webserver is used.\nI understand flask-socketio and eventlet webserver themselves are event-loop based.\nDoes flask-socketio and eventlet webserver runs on the same eventloop (same thread) or in two different threads?","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":692,"Q_Id":57619141,"Users Score":1,"Answer":"I think you are confusing the terminology.\nThe event loop is the task scheduler. This is provided by eventlet, and a single event loop is used for the whole application, including the Flask and the Flask-SocketIO parts.\nEach time a request arrives to the eventlet web server, it will allocate a new task for it. So basically each request (be it Flask or Flask-SocketIO, HTTP or WebSocket) will get its own task. Tasks are constantly being created and destroyed as requests are handled.\nWhen you use eventlet, tasks are not threads, they are greenlets, that is why I avoided calling them threads above and used the more generic \"task\" term. They behave like threads in many ways, but they are not.","Q_Score":0,"Tags":"python,flask,flask-restful,flask-socketio,eventlet","A_Id":57630398,"CreationDate":"2019-08-23T02:28:00.000","Title":"Flask-SocketIO with eventlet: Web and Websockets Thread","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I am trying to detect the drive failure in Datanode in a Hadoop Cluster. Cloudera Manager API don't have any specific API for that. CM API are only related to Name node or restart services. Are there any suggestions here? Thanks a lot!","AnswerCount":1,"Available Count":1,"Score":0.1973753202,"is_accepted":false,"ViewCount":520,"Q_Id":57621257,"Users Score":1,"Answer":"If you have access to NameNode UI, the JMX page will give you this information. If you hit the JMX page directly it'll be a JSON formatted page, which can be parsed easily.\nWe use HortonWorks primarily, haven't touched Cloudera in a long time, but I assume that can be made available somehow.","Q_Score":0,"Tags":"python-3.x,hadoop,hadoop-yarn,cloudera,cloudera-manager","A_Id":57633508,"CreationDate":"2019-08-23T06:58:00.000","Title":"How to detect in Hadoop cluster if any Datanode drive (Storage) failed","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":1,"Web Development":0},{"Question":"Seeking a bit of guidance on a general approach as to how one would automate the retrieval of data from a My Google Map. While I could easily export any given layer to KML\/KMZ, I'm looking for a way to do this within a larger script, that will automate the process. Preferably, where I wouldn't even have to log in to the map itself to complete the data pull.\nSo, what do you think the best approach is? Two possible options I'm considering are 1) using selenium\/beautiful soup to simulate page-clicks on Google Maps and export the KMZ or 2) making use of Python Google Maps API. Though, I'm not sure if this API makes it possible to download Google Maps layer via a script. \nTo be clear, the data is already in the map - I'm just looking for a way to export it. It could either be a KMZ export, or better yet, GeoJSON.\nAny thoughts or advice welcome! Thank you in advance.","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":280,"Q_Id":57664000,"Users Score":0,"Answer":"I used my browser\u2019s inspection feature to figure out what was going on under the hood with the website I was interested in grabbing data from, which led me to this solution. \nI use Selenium to login and navigate said website, then transfer my cookies to Python\u2019s Requests package. I have Requests send a specific query to the server whose response is in the form of JSON. I was able to figure out what query to send and what form the response would be through the inspection feature previously stated. Once I have the response in JSON I use Python\u2019s JSON package to convert into a Python dict to use however I need. \nSounds like you might not necessarily need Selenium but it does sound like the Requests package would be useful to your use case. I think your first step is figuring out what form the server response is when you interact with the website naturally to get what you want. \nHopefully this helps to some degree!","Q_Score":2,"Tags":"python,selenium,google-maps,geospatial,kmz","A_Id":57664626,"CreationDate":"2019-08-26T19:41:00.000","Title":"How to automate pulling data (KMZ? JSON?) from My Google Maps","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I am an extreme beginner with Python and its libraries and installation in general. I want to make an extremely simple google search web scraping tool. I was told to use Requests and BeautifulSoup. I have installed python3 on my Mac by using brew install python3 and I am wondering how to get those two libraries\nI googled around and many results said that by doing brew install python3 it will automatically install pip so I can use something like pip install requests but it says pip: command not found.\nby running python3 --version it says Python 3.7.4","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":165,"Q_Id":57665963,"Users Score":0,"Answer":"Since you're running with Python3, not Python (which usually refers to 2.7), you should try using pip3.\npip on the other hand, is the package installer for Python, not Python3.","Q_Score":0,"Tags":"python-3.x,pip","A_Id":57666013,"CreationDate":"2019-08-26T23:20:00.000","Title":"How to install stuff like Requests and BeautifulSoup to use in Python?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":1,"Web Development":1},{"Question":"This is probably a really simple question, but I can't seem to find an answer online.\nI'm using a Google Cloud Function to generate a CSV file and store the file in a Google Storage bucket. I've got the code working on my local machine using a json service account.\nI'm wanting to push this code to a cloud function, however, I can't use the json service account file in the cloud environment - so how do I authenticate to my storage account in the cloud function?","AnswerCount":1,"Available Count":1,"Score":0.6640367703,"is_accepted":false,"ViewCount":89,"Q_Id":57720636,"Users Score":4,"Answer":"You don't need the json service account file in the cloud environment.\nIf the GCS bucket and GCF are in the same project, you can just directly access it.\nOtherwise, add your GCF default service account(Note: it's App Engine default service account ) to your GCS project's IAM and grant relative GSC permission.","Q_Score":0,"Tags":"python-3.x,google-cloud-platform,google-cloud-functions,google-cloud-storage","A_Id":57721212,"CreationDate":"2019-08-30T04:47:00.000","Title":"Authenticating Google Cloud Storage SDK in Cloud Functions","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I have written a script in selenium python which is basically opening up a website and clicking on links in it and doing this thing multiple times..\nPurpose of the software was to increase traffic on the website but after script was made it has observed that is not posting real traffic on website while website is just taking it as a test and ignoring it.\nNow I am wondering whether it is basically possible with selenium or not? \nI have searched around and I suppose it is possible but don't know how. Do anyone know about this? Or is there any specific piece of code for this?","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":1011,"Q_Id":57729315,"Users Score":0,"Answer":"It does create traffic, the problem is websites sometimes defends from bots and can guess if the income connection is a bot or not, maybe you should put some time.wait(seconds) between actions to deceive the website control and make it thinks you are a person","Q_Score":1,"Tags":"python,selenium,automation,web-traffic","A_Id":57729437,"CreationDate":"2019-08-30T15:12:00.000","Title":"Can selenium post real traffic on a website?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I need to download a file from onedrive or google drive using wget or a faster method!\nI've tried lots of code i could find on the internet but have not had any luck.","AnswerCount":1,"Available Count":1,"Score":0.1973753202,"is_accepted":false,"ViewCount":153,"Q_Id":57733624,"Users Score":1,"Answer":"This has worked for me to download a non-protected-link file:\n\nGo to your OneDrive\nClick on the share link and copy. It will look like: https:\/\/...\/EvZJeK2tIMOs54OA?<other-stuff>\nAppend download=1 after ? looking like https:\/\/...\/EvZJeK2tIMOs54OA?download=1\n\nNow you can use it with wget like:\nwget https:\/\/...\/EvZJeK2tIMOs54OA?download=1 -O <output-file-name\\>.<extension\\>\nNote: The -O (capital O) is to define an output name otherwise it will have EvZJeK2tIMOs54OA?download=1 as the file name.","Q_Score":0,"Tags":"python","A_Id":71328453,"CreationDate":"2019-08-30T21:57:00.000","Title":"How to download file from one drive\/google drive","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"How to set the source IP address for UDP multicast packages to something else than the interface IP?\nI am trying to write a small router that selectively routes UDP SSDP packages from one network to another. The plan is to do it in python, although I am flexible on that.\nIt seems fairly easy to route SSDP NOTIFY messages: I receive them on one interface and decide which interface to re-broadcast them on. However the protocol for M-SEARCH messages require that the source IP is set to the original source of the message as any service that chooses to respond will respond with a unicast message to the source IP and port.\nExample (heavily simplified):\n\nNetwork A: 192.168.10.0\/24\nNetwork B: 192.168.11.0\/24\nMy router application runs on a multihomed computer on 192.168.10.2 and 192.168.11.2.\n\nA client on network A with IP 192.168.10.10 sends an M-SEARCH message:\n\nSrc IP\/Port: 192.168.10.10 port 40000\nDst IP\/Port: 239.255.255.250 port 1900\n\nMy \"router application\" on 192.168.10.2 receives the packet and would like to rebroadcast it on network B. However I cannot find any method in the socket API that allows me to set the source IP address. Only to pick the source interface.\nThus the rebroadcasted packet now looks like this:\n\nSrc IP\/Port: 192.168.11.2 port xxxxx\nDst IP\/Port: 239.255.255.250 port 1900\n\nAnd now the receiving service is unable to unicast back to the client as the original IP and port are lost.","AnswerCount":1,"Available Count":1,"Score":0.537049567,"is_accepted":false,"ViewCount":403,"Q_Id":57744381,"Users Score":3,"Answer":"How to set the source IP address for UDP multicast packages to something else than the interface IP?\n\nThe only way I know of is to use a RAW socket and construct the IP headers manually. Note that use of RAW sockets is typically restricted to admin users on most platforms.\n\nHowever I cannot find any method in the socket API that allows me to set the source IP address. \n\nBecause there isn't one.\n\nAnd now the receiving service is unable to unicast back to the client as the original IP and port are lost.\n\nCan't you just have your router remember the original source when it receives the M-SEARCH request, and when it receives the unicast reply then forward it to the original requester? That is how most routers usually work.","Q_Score":0,"Tags":"python,sockets,multicast","A_Id":57752433,"CreationDate":"2019-09-01T08:26:00.000","Title":"Setting source address when sending UDP multicast messages","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":1,"Web Development":0},{"Question":"I am trying to send Confidential emails via gmail, although I am not sure it is supported. Via the gmail API, the User.messages fields do not seem to indicate whether an email is confidential or not. I retrieved a confidential and regular email via the v1 gmail api get method and the messages are identical. Is there a way send a confidential gmail email, either with the official api or other service?","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":319,"Q_Id":57782111,"Users Score":1,"Answer":"To my knowlage you can't send messages with \"Confidential mode\" through the Gmail API. This is not currently supported.","Q_Score":0,"Tags":"python,google-api,gmail-api,google-api-python-client","A_Id":57782321,"CreationDate":"2019-09-04T05:41:00.000","Title":"Send Confidential email through gmail api","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I need help to find the bottleneck with my scrapy\/python based scraper.\nWe are scraping products from Amazon (Italy at the moment) but we are struggling with overall requests throughput.\nWe are using backconnect rotating proxies: StormProxies (50 threads plan) + Proxyrotator (100 threads) + TOR but even 250+ available threads we can scrape only 1\/2 URLs per second...\nWe are running it on OVH dedicated server, 8 core x 16GB ram, redis celery and docker as additional tools\nI am an IT technician, the software is developed by my indian dev guy, if you need additional info or code just ask!\nThanks in advance","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":604,"Q_Id":57795692,"Users Score":0,"Answer":"Amazon, like other big websites, is using IA to detect the incoming requests. It means that if you are requesting a lot of traffic to the same \"logical\" order on the same website, they will detect you and try to ban you (even if you are using other proxies).\nTry to separate and do it on different servers. Instead of using only one big server, use several small servers.\nMy recommendations:\n\nUse proxies that are reliable (different providers)\nUse random time to request every URL (just to confuse the detection algorithm and to avoid being identified as the same user using different IP)\nUse different agents too (rotating)\nBe nice to the site you are going to scrape. See robots.txt file to know more\nIf you are scraping a list of elements in Amazon (for instance, the top products sell in books category), don't scrape it sequentially. Pick up randomly to scrape that list of products in order to be unpredictable\nUse a good structure to scrape (parallelizing). I suggest use:\n\n\nStore the URLs to scrape in Redis\nUse differents ScrapyD servers to run horizontal scraping\nUse monitoring system to review logs, to review errors and manage the servers\n\n\nI hope it helps!","Q_Score":0,"Tags":"python,web-scraping,scrapy,amazon","A_Id":59702840,"CreationDate":"2019-09-04T21:11:00.000","Title":"Scrapy extremely slow: probably bottleneck","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"We are using protobufs to model our networking software. There are many instances, like priorities, where 0 is a valid value. But, when we transport, the fields with 0 values are suppressed. \nIs there a way to change this behavior? That is, differentiate a filed with a valid value of 0, from a field which has not been set, which can be suppressed?\nOur client is gRPC-Java and server is gRPC-Python.\nThank you for your time.","AnswerCount":1,"Available Count":1,"Score":0.1973753202,"is_accepted":false,"ViewCount":106,"Q_Id":57796407,"Users Score":1,"Answer":"You can use Protobuf version 2, which can distinguish whether the field has been set. However, gRPC recommend to use Protobuf version 3.\nAn alternative is to set the field to an invalid value, e.g. -1, if the field is NOT set.","Q_Score":1,"Tags":"protocol-buffers,grpc,grpc-java,grpc-python","A_Id":57798038,"CreationDate":"2019-09-04T22:41:00.000","Title":"Protobuf fields where a value of 0 is valid","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I have an url of tweet and want to get texts of tweet from the url in Python. But I have no idea. I searched about Tweepy, But I think It's for search, upload tweets not to get texts from url.","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":183,"Q_Id":57824627,"Users Score":0,"Answer":"You could probably use Beautiful Soup to grab the info you need via cURL. They have decent documentation. Does Twitter have an API you could use?","Q_Score":0,"Tags":"python,twitter,tweets","A_Id":57824762,"CreationDate":"2019-09-06T15:13:00.000","Title":"How to get texts from an url of tweets in Python?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I have some kind of single page application which composes XHR requests on-the-fly. It is used to implement pagination for a list of links I want to click on using selenium.\nThe page only provides a Goto next page link. When clicking the next page link a javascript function creates a XHR request and updates the page content.\nNow when I click on one of the links in the list I get redirected to a new page (again through javascript with obfuscated request generation). Though this is exactly the behaviour I want, when going back to the previous page I have to start over from the beginning (i.e. starting at page 0 and click through to page n)\nThere are a few solutions which came to my mind:\n\nblock the second XHR request when clicking on the links in the list, store it and replay it later. This way I can skim through the pages but keep my links for replay later\nSomehow 'inject' the first XHR request which does the pagination in order to save myself from clicking through all the pages again \n\nI was also trying out some simple proxies but https is causing troubles for me and was wondering if there is any simple solution I might have missed.","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":150,"Q_Id":57861058,"Users Score":0,"Answer":"browsermobproxy integrates easily and will allow you to capture all the requests made. It should also allow you to block certain calls from returning.\nIt does sound like you are scraping a site, so it might be worth parsing the data the XHR calls make and mimicking them.","Q_Score":0,"Tags":"python,selenium","A_Id":57914026,"CreationDate":"2019-09-09T20:53:00.000","Title":"Logging and potentially blocking XHR Requests by javascript using selenium","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"What's going to happen?\nI got a big amount of Scrapy crawlers, written in Python 2.7. I need to convert them all to support Python 3.\nThis transition must be done completely in 1 go. I can only start using the Python 3 crawlers once they're all ready for deployment. I cannot deploy 1 crawler at a time.\nWhat have I done\/what do I have?\nI currently have a remote branch, which is the master branch. Lets call that Remote-A. That is the branch that holds all the crawlers, and they get executed daily. This branch must remain functional.\nFor that remote branch, I have the local folder\/branch, where I fix bugs and create new ones. Lets call that Local-A. From the master, I push and pull from that.\nNow, as \"all operations must remain active\", I need a separate remote branch for Python 3 crawlers. That branch can be called Remote-B. I've created that manually, so the whole repository has two branches now: Master(Remote-A), and the one for Python 3 crawlers(Remote-B), which is a copy of the Master branch.\nI've also created a folder manually, and downloaded a zip from the Python 3 branch. That folder is called Local-B. My idea is to either delete all the (2.7) crawlers from Python 3 branch, or just start replacing them one by one.\nTo sum it up. I got Local A connected to Remote A. I also need Local B connected to Remote B. These two connections should not get mixed.\nMind you, I'm not very comfortable with GIT, and I'm the only responsible for the transition project, so I want everything to go smooth as silk. I know that it's easy to cause a lot of damage in GIT.\nSo, my workflow requires me to keep the running crawlers operational on daily basis, and work on upgrading the old ones to Python 3. How do I make the switches between A- and B-sides easy, without causing a havoc?","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":107,"Q_Id":57869506,"Users Score":2,"Answer":"If I understand your question, I am afraid you are a bit confused about the concepts of branch and remote. A remote would be github, and also a local mirror of your git repo.\nYou can have as many mirrors as you like. All would contain all the branches.\nIn your case, and also if I understand your question, I would do as follows:\n\nhave a deploy branch (which what you seem to name as Remote-A - a name that is confusing to me): this branch should always be correct\nhave a python2 development branch (suggested name: py2dev), in case you need to perform modifications in the python2 deployed code before deploying the python3 code\nhave a python3 migration branch (suggested name: py3) where you would be migrating the python2 code\n\nWhile the python3 migration is not ready, you would develop on the py2dev branch in your computer. In case you want to publish some change, you would publish these changes in your remote (i.e. github), and then pull these changes from that remote (github) in your deployed repo.\nWhen the python3 migration is ready, you would push your changes again to a remote, and then fetch them, and do a git checkout py3 in the deployed server. If things go wrong, you can do a git checkout deploy and you would be safe again.\nThere are many git workflows. This is supposed to be a simple one.","Q_Score":0,"Tags":"python,git,github,scrapy","A_Id":57869932,"CreationDate":"2019-09-10T11:06:00.000","Title":"How do I switch between remote github branches and local branches\/folders with ease?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I have an API (in python) which has to alter files inside an EC2 instance that is already running. I'm searching on boto3 documentation, but could only find functions to start new EC2 instances, not to connect to an already existing one.\nI am currently thinking of replicating the APIs functions to alter the files in a script inside the EC2 instance, and having the API simply start that script on the EC2 instance by accessing it using some sort of SSH library. \nWould that be the correct approach, or is there some boto3 function (or in some of the other Amazon\/AWS libraries) that allows me to start a script inside existing instances?","AnswerCount":3,"Available Count":1,"Score":0.1973753202,"is_accepted":false,"ViewCount":2456,"Q_Id":57896987,"Users Score":3,"Answer":"Unless you have a specific service running on that machine which allows you to modify mentioned files. I would make an attempt to log onto EC2 instance as to any other machine via network. \nYou can access EC2 machine via ssh with use of paramiko or pexpect libraries.","Q_Score":1,"Tags":"python,amazon-web-services,amazon-ec2,boto3","A_Id":57897084,"CreationDate":"2019-09-11T21:24:00.000","Title":"Running Python Script in an existing EC2 instance on AWS","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"According to performance it is more than obvious that web scraping with BautifulSoup is much faster than using a webdriver with Selenium. However I don't know any other way to get content from a dynamic web page. I thought the difference comes from the time needed for the browser to load elements but it is definitely more than that. Once the browser loads the page(5 seconds) all I had to do is to extract some <tr> tags from a table. It took about 3-4 minutes to extract 1016 records which is extremely slow in my opinion. I came to a conclusion that webdriver methods for finding elements such as find_elements_by_name are slow. Is find_elements_by.. from webdriver much slower than the find method in BeautifulSoup? And would it be faster if I get the whole html from the webdriver browser and then parse it with lxml and use the BeautifulSoup?","AnswerCount":4,"Available Count":1,"Score":0.049958375,"is_accepted":false,"ViewCount":1214,"Q_Id":57938852,"Users Score":1,"Answer":"Look into 2 options:\n1) sometimes these dynamic pages do actually have the data within <script> tags in a valid json format. You can use requests to get the html, beautifulsoup will get the <script> tag, then you can use json,loads() to parse.\n2) go directly to the source. Look at the dev tools and search the XHR to see if you can go directly to the url\/API and that generates the data and return the data that way (most likely again in json format). In my opinion, this is by far the better\/faster option if available.\nIf you can provide the url, I can check to see if either of these options apply to your situation.","Q_Score":0,"Tags":"python,performance,selenium,web-scraping,beautifulsoup","A_Id":57942694,"CreationDate":"2019-09-14T20:02:00.000","Title":"Web Scraping with Python Selenium performance","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"Have some code, it works with raw code but once i turned it into a executable with pyinstaller i get the requests.packages.urllib3.exceptions.SSLError: [Error 2] no such file or directory\nit always works with raw code but not with pyinstaller\ntried using requests from both python 2 and python3, tried on multiple PCs.\ni expect it to execute the code when using a .exe with pyinstaller","AnswerCount":3,"Available Count":2,"Score":0.0,"is_accepted":false,"ViewCount":981,"Q_Id":57942006,"Users Score":0,"Answer":"Post some code, without it I am going blindly. But i can guest that you have somewhere in your code relative(like .\/file), while everything is in your original directory(raw code) it works, but after compiling,programm can't find it","Q_Score":1,"Tags":"python,python-requests,pyinstaller,instagram-api","A_Id":57942058,"CreationDate":"2019-09-15T07:22:00.000","Title":"requests.packages.urllib3.exceptions.SSLError: [Error 2] no such file or directory","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":1,"System Administration and DevOps":0,"Web Development":0},{"Question":"Have some code, it works with raw code but once i turned it into a executable with pyinstaller i get the requests.packages.urllib3.exceptions.SSLError: [Error 2] no such file or directory\nit always works with raw code but not with pyinstaller\ntried using requests from both python 2 and python3, tried on multiple PCs.\ni expect it to execute the code when using a .exe with pyinstaller","AnswerCount":3,"Available Count":2,"Score":0.0,"is_accepted":false,"ViewCount":981,"Q_Id":57942006,"Users Score":0,"Answer":"I found the solution\nUsing auto-py-to-exe and pyinstaller to convert to exe.\nClick in \"on file\" button use \"one directory\" option.\nThen you have to go to your python's directory and find the directory \"requests\".\nCopy this directory (\"requests\") to your app directory created with auto-py-to-exe.\nIt's done.","Q_Score":1,"Tags":"python,python-requests,pyinstaller,instagram-api","A_Id":71662789,"CreationDate":"2019-09-15T07:22:00.000","Title":"requests.packages.urllib3.exceptions.SSLError: [Error 2] no such file or directory","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":1,"System Administration and DevOps":0,"Web Development":0},{"Question":"I have created a chatbot using RASA to work with free text and it is working fine. As per my new requirement i need to build button based chatbot which should follow flowchart kind of structure. I don't know how to do that what i thought is to convert the flowchart into graph data structure using networkx but i am not sure whether it has that capability. I did search but most of the examples are using dialogue or chat fuel. Can i do it using networkx.\nPlease help.","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":614,"Q_Id":57961205,"Users Score":1,"Answer":"Sure, you can.\nYou just need each button to point to another intent. The payload of each button should point have the \/intent_value as its payload and this will cause the NLU to skip evaluation and simply predict the intent. Then you can just bind a trigger to the intent or use the utter_ method.\nHope that helps.","Q_Score":0,"Tags":"python,networkx,flowchart,rasa","A_Id":58013231,"CreationDate":"2019-09-16T16:45:00.000","Title":"How to create button based chatbot","Data Science and Machine Learning":1,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"So I want to be able to input a username into my Python program and then it checks how many followers that account has (if it's not private) and displays the number. I'm sure it has something to do with APIs.","AnswerCount":2,"Available Count":1,"Score":0.0996679946,"is_accepted":false,"ViewCount":379,"Q_Id":57964704,"Users Score":1,"Answer":"first idea that came to my mind is to scrape it with beautifulsoup.","Q_Score":0,"Tags":"python,api,instagram","A_Id":57964909,"CreationDate":"2019-09-16T21:35:00.000","Title":"How can I check how many followers someone has on Instagram with Python?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I'm trying to setup a python script that uses the requests library to get data from a website. The script works without issues running in a virtual environment on my windows 10 pc or on a azure vm. \nHowever, when I try to create a docker container using the python:3.6-slim image I get DH_KEY_TOO_SMALL errors. Testing the website on ssllabs.com revealed that it supports weak DH key exchange parameters. What could be causing this error and how can I fix it?","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":2231,"Q_Id":58011032,"Users Score":4,"Answer":"I've managed to fix the issue. The problem was caused by openssl versions. Both my windows 10 pc and ubuntu 18.04 vm run an older version that had no problem connecting to the website. The python docker images contain a newer version of openssl that refused to connect.","Q_Score":4,"Tags":"python,docker,openssl,python-requests","A_Id":58012193,"CreationDate":"2019-09-19T12:26:00.000","Title":"Docker python requests results in DH KEY TOO SMALL error","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":1,"Web Development":0},{"Question":"I am running a python application which is running inside a docker container using Beanstalk in a private subnet, and I want to get the Private\/Local IP of the EC2 instance. Is it possible to get the Local IP address without using curl http:\/\/169.254.169.254\/latest\/meta-data\/local-ipv4 inside a docker container.\nAlthough I tried docker run --net=host <image_name> but still its not accessible.","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":404,"Q_Id":58064056,"Users Score":0,"Answer":"You can get the local ip of a Linux instance with this command:\nhostname -I | awk '{print $1}'\nFor EB, use .ebexentions and write a bash script in a script_name.config to run: export HOST_IP=$(hostname -I | awk '{print $1}')","Q_Score":1,"Tags":"python-3.x,amazon-web-services,docker,amazon-ec2,amazon-elastic-beanstalk","A_Id":58065828,"CreationDate":"2019-09-23T13:54:00.000","Title":"How to get EC2 instance private IP in a private subnet?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":1,"Web Development":0},{"Question":"I want to Automate Web App testing using Selenium and PyTest and I am new to both. So here is the structure of my project folder: \nParent Folder\nLocator.py\nSection 1(Folder) -> test_sectionone.py\nLocator.py contains all my Xpaths. Now when I try to import Locator.py by typing import Locator and then fetching the xpath using Locator.xpathElement1 in the test_sectionone.py, pytest fails with the following error: \nHint: make sure your test modules\/packages have valid Python names.\nTraceback:\ntest_mailboxProvisioning.py:5: in \n import Locator\nE ModuleNotFoundError: No module named 'Locator'\nIs there something I am missing? Is there a naming convention because of which I am facing the issue?","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":67,"Q_Id":58069050,"Users Score":0,"Answer":"Convert Parentfolder into a python package by including an empty file __init__.py\nConvert sub folder \"Section1\" into a python package by including an empty file __init__.py\nIn test_sectionone.py now either import ParentFolder.Locator (using absolute path) or from ..Locator import function using relative path.\nBoth should work.","Q_Score":0,"Tags":"python,selenium,pytest","A_Id":58069402,"CreationDate":"2019-09-23T19:32:00.000","Title":"Unable to import .py file in Pytest","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I have a list of all wikipedia articles and I want to scrape a body for research purposes. My script is working fine, but at the current speed this will take 40 days straight.\nMy questions is:\nCan I run this script, lets say 10 times parallel in different terminal windows. If I just set it like this:\nScript 1:\nstart point: 0\nend point: len(list)\/10\nScript 2:\nstart start point: len(list)\/10\nend point: len(list)\/(10*2)\n...\nScript 10\nand so on.\nThis could leave me with 4 days, which is reasonable imo.\nDoes my approach work? Is there a better approach?\nThanks.","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":86,"Q_Id":58104906,"Users Score":0,"Answer":"Possible yes, ideal no. Why do you think its running so slowly? Also are you using the wiki api or are you scraping the site? There's factors that affect either of the two so knowing what your actually doing will help us give a better answer.","Q_Score":0,"Tags":"python,web-scraping,wikipedia,wikipedia-api","A_Id":58105252,"CreationDate":"2019-09-25T18:55:00.000","Title":"Can you run python wikipedia api multiple times at the same time?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":1,"System Administration and DevOps":0,"Web Development":0},{"Question":"In teradataml does create_context support ldap? What other authentication mechanisms are supported? \nHow can user specify ldap server while creating context?","AnswerCount":2,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":389,"Q_Id":58104992,"Users Score":0,"Answer":"How can user specify ldap server while creating context?\n\nYou do not specify the LDAP server in the connection parameters. That is not possible and is not necessary.\nLDAP authentication only works if your Teradata Database system has been configured on the server side to offer LDAP authentication. The LDAP server that is used is the LDAP server that the Teradata Database system is configured to use.","Q_Score":0,"Tags":"python,teradata","A_Id":58240835,"CreationDate":"2019-09-25T19:01:00.000","Title":"teradataml: what are the various authentication mechanisms supported by context manager in teradataml?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I'm having a problem while tring to read data from a website using urllib in Python 3 that causes my program to stand still (it does not give me any error message, it looks like it is stucked on a loop). The weird part is that this same script works with any other websites that I try, except this one.\nI've been doing some research and read people talking about that \"HTTP Error 403: Forbidden\" traceback that can be fixed using a custom User-Agent to open the url as a browser, but this does not fix my problem.\nI'm almost 100% sure that the problem is some type of restriction from the PC because I'm using a PC from work and both the PC and the Network has some restrictions here, but the weird thing is that the website opens when I'm using Chrome.\nThis code was posted by Kardi Teknomo on another topic and is an interesting way to show my problem.\nimport urllib.request\n\nurl = \"http:\/\/py4e-data.dr-chuck.net\/known_by_Fikret.html\"\n\npage = urllib.request.Request(url,headers={'User-Agent': 'Chrome\/76.0.3809.132'})\ninfile = urllib.request.urlopen(page).read()\ndata = infile.decode()\n\nprint(data)\n\nA simpler code that tries to do the same and fails in the same way, is this one:\nimport urllib.request, urllib.parse, urllib.error\n\nFhandle = urllib.request.urlopen(\"http:\/\/py4e-data.dr-chuck.net\/known_by_Fikret.html\")\nfor lines in Fhandle:\n lines = lines.decode().strip()\n print(lines)\n\nI'm trying to do something more complex, but for the sake of fixing the problem, all I'm trying to do with this code is to read an HTML page and print its content, but the program always get stucked right after I press enter if this I use the URL from theses examples.","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":902,"Q_Id":58120737,"Users Score":0,"Answer":"I have had the same issue while running code with urllib.request.urlopen(link, context=ctx).read() in IDE PyCharm . The code runs perfectly in console. In order to run this code in IDE, once you put url via url = input('Enter url: ') move the cursor to the beginning of the line and then press 'Enter'.","Q_Score":0,"Tags":"python,urllib","A_Id":61984020,"CreationDate":"2019-09-26T16:04:00.000","Title":"Website opens in Chrome but not while using urllib in Python","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I want to extract all Twitter feeds of a specific language only. (to extract Sinhala language tweets) How can I do that using python?","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":57,"Q_Id":58143644,"Users Score":2,"Answer":"regex is useful.\nSinhala language use codes U+0D80\uff5eU+0DFF in utf-8. First, try extracting tweets that contain Sinhala characters with regex.","Q_Score":1,"Tags":"python,tweepy,tweets","A_Id":58143708,"CreationDate":"2019-09-28T05:30:00.000","Title":"How to extract all Twitter feeds of sinhala language language only using python?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"Problem:\n1) Need a python library which accepts path of file instead of opening the file(as it can be huge) to make a PUT request.\n2) Also want that library to automatically do the redirect if exists(Requests library does that)\nThe shell command for this is curl -T\nI've tried requests, it does redirect but needs to open file to send the file. Same is the case for pyCurl. The only alternative left is running a shell command in the python code.","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":283,"Q_Id":58213185,"Users Score":0,"Answer":"In pycurl, both READDATA and READFUNCTION should stream the data from the open file. As pointed out in the comments, opening the file is required to do anything with its contents.\nIn general you can verify that your program is streaming data rather than reading the file in memory by monitoring its memory usage while it is executing and providing it with a large input file (e.g. > 100 MB).","Q_Score":0,"Tags":"python,api,curl,python-requests,pycurl","A_Id":58369834,"CreationDate":"2019-10-03T06:06:00.000","Title":"How to upload file by PUT on python without opening the file","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I want to make something like this: I will have list of email and password of a certain website. I want to check that email and password work for that website. I want it to go put that email and password in email and password box of certain website. If that logged in then it will say Logged in else not logged in.\nCan I built it with Python module called Scrapy?","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":80,"Q_Id":58222912,"Users Score":0,"Answer":"Yes, you can.\nWhether Scrapy is the best choice for your use case depends on the details you have not shared.","Q_Score":0,"Tags":"python-3.x,scrapy","A_Id":58410314,"CreationDate":"2019-10-03T15:59:00.000","Title":"Can Python Scrapy module help us to make an account checker?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"My problem is that I want to be able to upload a PHP file to my pythonanywhere server and then get its public url and use it to get notifications from a webhook to that url or specifically to the file responsible for processing those notifications. But I can't find a way to get this url and being honest is the first time I find myself using this server. Thank you and excuse my ignorance.","AnswerCount":1,"Available Count":1,"Score":0.1973753202,"is_accepted":false,"ViewCount":65,"Q_Id":58236668,"Users Score":1,"Answer":"PythonAnywhere does not support PHP.","Q_Score":0,"Tags":"pythonanywhere","A_Id":58268294,"CreationDate":"2019-10-04T12:42:00.000","Title":"How to get the public url of a file?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"Trying to get Firefox Portable v46 to function in headless mode. (only portable version that works)\nChrome portable can do this but I would like to know if firefox portable can.\nI must include capabilities_argument[\"marionette\"] = False otherwise an exception is thrown:\nselenium.common.exceptions.SessionNotCreatedException: Message: Unable to find a matching set of capabilities\nIn normal Firefox (not portable + recent version) ... disabling marionette also causes ops.add_argument(\"--headless\") to fail ... this is why I believe disabling marionette is the issue.\nos.environ['MOZ_HEADLESS'] = '1' ... Does not work either.\nbrowser.set_window_position(-3000, 0) ... works, but only after being visible\nI am also running multiple instances of firefox, hiding from task manager might not be an option.\nIs there a way I can import marionette or get it to work somehow?","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":217,"Q_Id":58257620,"Users Score":0,"Answer":"I have discovered that (in windows), once Firefox (most recent gecko-driver compatible version) is installed, the folder in which firefox.exe is kept, is portable.\nThe folder can be transferred to any windows machine even if it has never had Firefox installed. And the folder containing Firefox.exe will run Firefox successfully\nThis is the same with Chrome","Q_Score":0,"Tags":"python-3.x,selenium,geckodriver,selenium-firefoxdriver","A_Id":58266615,"CreationDate":"2019-10-06T13:19:00.000","Title":"python - selenium firefox portable headless fails","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I have an application that uses two servers that need to communicate with one another. One is a python server that implements BaseHTTPRequestHandler's do_METHODs. The other is node js's development server\/production server that hosts a React JS application.\nThe node js application uses a POST request to get a message from an input which the python server extracts that message from that POST request and then calls Amazon's Lex using Boto3 to get a response.\nWhenever I run node js development server (on localhost:3000) and the python server (on localhost:3001), the python server is able to recognize that a POST request has been made and goes ahead and gets a response. However, when I run node js production server (on localhost:5000) and the python server (on localhost:3001), the node js server is unable to communicate with the python server. Why is this?","AnswerCount":2,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":243,"Q_Id":58262399,"Users Score":0,"Answer":"After much research, I felt like having two local applications running on different ports is a little impractical or rather more complicated than it should be. \nAs such, I have migrated my project to use Django instead of using React JS and integrate it with python. This way Django's back-end can handle the POST requests I make and update the DOM as needed. \nAll this is for a chatbot application that is suppose take a user's question, call Amazon's Lex for a intent and entity which then spaCy, a ML framework, searches our database for an answer. The answer is then returned and the DOM is then updated without refreshing the page.","Q_Score":1,"Tags":"python,node.js,reactjs","A_Id":58384477,"CreationDate":"2019-10-07T00:51:00.000","Title":"Communicate between a python server and node server","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I am trying to use the paramiko and boto3 libraries in Python to transfer files from a S3 bucket to an Unix server. Currently I am able to achieve this by bringing the file to local by using the download method. Then i will use the ftp client to upload the file to Unix server. Is there a simple way to do this directly from s3 to unix server without bringing the file to local?","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":416,"Q_Id":58265368,"Users Score":0,"Answer":"There is hardly a simple API what will allow you to transfer a file between two remote S3 and FTP servers.\nBut of course, you can either:\n\ninitiate the download from S3 on the Unix server, or\nsetup a server on AWS that will push\/upload the file to the Unix server.","Q_Score":0,"Tags":"python,amazon-s3,sftp,boto3,paramiko","A_Id":58268330,"CreationDate":"2019-10-07T07:46:00.000","Title":"Transfer files from AWS S3 to Unix Server using SFTP","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I am scraping elements from a web page & I can see the element being visible (numeric value) on the web page in a grayed out box ,but when tried to inspect the element I cant find it between the tags. I assumed the URL might be any webservice endpoint & tried to GET from postman but it returned mere HTML code not a JSON response.\nIn general, we can get values between the tags by finding the element & getting innerText attribute in selenium that too failed as there is no text in between the tags.\nI cannot post any URL or responses due to security compliance issues in my org. Please advise any other way I can work around.","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":173,"Q_Id":58274532,"Users Score":0,"Answer":"Got my answer , tried running JS code \"document.querySelector('input[name=assets_tot_mfr]').value;\" and ran it through python execute_script. Thanks","Q_Score":0,"Tags":"json,python-3.x,web-scraping","A_Id":58274633,"CreationDate":"2019-10-07T17:35:00.000","Title":"Value present on browser but not available in inspect element","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I am having hard time to install a python lib called python3-saml\nTo narrow down the problem I created a very simple application on ibm-cloud and I can deploy it without any problem, but when I add as a requirement the lib python3-saml \nI got an exception saying:\npkgconfig.pkgconfig.PackageNotFoundError: xmlsec1 not found\nThe above was a deployment on ibm-cloud, but I did try to install the same python lib locally and I got the same error message, locally I can see that I have the xmlsec1 installed.\nAny help on how to successfully deploy it on ibm-cloud using python3-saml?\nThanks in advance","AnswerCount":2,"Available Count":1,"Score":0.1973753202,"is_accepted":false,"ViewCount":448,"Q_Id":58288228,"Users Score":2,"Answer":"I had a similar issue and I had to install the \"xmlsec1-devel\" on my CentOS system before installing the python package.","Q_Score":1,"Tags":"python,cloud,ibm-cloud,xmlsec1","A_Id":59429632,"CreationDate":"2019-10-08T14:18:00.000","Title":"xmlsec1 not found on ibm-cloud deployment","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":1,"Web Development":0},{"Question":"Need to sudo su - diffuser after connection via SSHClient function.\nI cannot login with ssh directly to the required user id in order to run commands owned by that required user. When at the local cli of the target ssh system, I simply run sudo su - diffuser, and then run the required commands.\nIs there anyway to simulate this using the Paramiko SSHClient commands?","AnswerCount":1,"Available Count":1,"Score":0.1973753202,"is_accepted":false,"ViewCount":67,"Q_Id":58307003,"Users Score":1,"Answer":"I was able to resolve this using the SSHClient's invoke_shell method and associated methods (x.send, x.recv, etc.)","Q_Score":0,"Tags":"python","A_Id":58310387,"CreationDate":"2019-10-09T14:57:00.000","Title":"paramiko SSHClient sudo to new user","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I want to scrape a stocks website and get the prices using selenium. I can't use normal HTML requests as the HTML is dynamic. \nI am using the headless selenium webdriver to get the data, but it takes around 30 seconds for each request. \nIs there a faster way to get the dynamic HTML?","AnswerCount":3,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":793,"Q_Id":58315190,"Users Score":3,"Answer":"No, you are stuck with Selenium's wait-time with rendering\nDynamic HTML requires a full browser. There is not much negotiating with that. If your pages are separate and distinct, i.e. you are scraping stocks.com\/oilandgas as well as stocks.com\/agriculture, there is a possible way to speed things up. \nThe one option you might have is to create a separate thread for each Selenium Webdriver instance and have both web pages scraped at the same time by two different Selenium Webdrivers. \nThe caveat to that is that it will only speed things up if the bottleneck (what is causing the slowness) is the rendering of the website.\nIf it is the internet speed, the processing power of your computer, or the server speed of the website, this would not improve things. \nActually, Daniel Farrell below suggests that it would improve the networking speed. You may want to give this a shot.","Q_Score":0,"Tags":"python,selenium,web-scraping,dynamic-html","A_Id":58315244,"CreationDate":"2019-10-10T03:48:00.000","Title":"Fastest way to get dynamically updated HTML content from website?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I am trying to import some data from web, where the data is being updated every 1 second. \nHowever, the problem is that when I try to soup this data, all data are not imported. When the data is not being updated, my code works well. But when the data is being updated(from 8 a.m. to 12 a.m.), some of those are not imported properly. \nIt's a table with 500 rows and 12 columns, where each column is stored in a separate variable. It is expected that the variables have the same length. This is true when data are not updated, but when I run the code from 8 a.m. to 12 a.m.(when data are being updated) the variables do not have the same length. Any advice or recommendation would be greatly appreciated.\nI am using selenium to open the web page and execute some commands to read data with beautiful-soup.","AnswerCount":1,"Available Count":1,"Score":0.537049567,"is_accepted":false,"ViewCount":148,"Q_Id":58338256,"Users Score":3,"Answer":"Since there is no code and I don't know the website, here are some general thoughts:\n\nAs Maxime suggested - Is there an API you could use?\nCan you ditch selenium? Using something like requests might be faster and will help you get the timings right.\nIs the stuff you want to scrape already loaded when you try to save it?\nIs your internet connection keeping up with the amount of requests you sent?","Q_Score":0,"Tags":"python,python-3.x,selenium-webdriver,beautifulsoup","A_Id":58338671,"CreationDate":"2019-10-11T09:28:00.000","Title":"what is the best way to import high frequency updating data in python?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"For example in requests library we have get method which is defined inside api.py file\nbut i can directly call it with requests.get .\nas per my understanding it should be called like this requests.api.get\nHow python modules handles this.","AnswerCount":2,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":44,"Q_Id":58340553,"Users Score":1,"Answer":"Your intuition is correct that the name should be requests.api.get. In fact, if you look at requests.get.__module__, you will find that it is requests.api. Similarly, requests.get.__globals__ is a reference to requests.api.__dict__.\nMany libraries will export functions defined elsewhere through the top level namespace. This is usually done with an import like from .api import get in the __init__.py file of requests.","Q_Score":2,"Tags":"python,python-3.x","A_Id":58341056,"CreationDate":"2019-10-11T11:41:00.000","Title":"How functions are mapped to correct filename in python packages?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":1,"System Administration and DevOps":0,"Web Development":0},{"Question":"I'm writing a server extension for jupyter lab and i can use ServerConnection.makeRequest() from @jupyterlab\/services to send POST or GET to my custom URL in typescript.\nNow i want to make some request from notebook to this URL by using library requests of python but i always get 403 error.\nIs there any equivalent of ServerConnection.makeRequest() in jupyter lab python library to send request to server ?","AnswerCount":1,"Available Count":1,"Score":0.1973753202,"is_accepted":false,"ViewCount":108,"Q_Id":58342869,"Users Score":1,"Answer":"Your request from within a notebook is most likely forbidden due to the xsrf check by the jupyter server. When you start jupyter server pass the parameter --NotebookApp.disable_check_xsrf=True in the command line to disable it. Or you need to handle passing the xsrf token. Note that disabling this check in an external-facing (production) system is not recommended.\nYou may also have to pass or suppress token. The token can be suppressed by passing --NotebookApp.token='' when you start the server.","Q_Score":0,"Tags":"python,jupyter,jupyter-lab","A_Id":58343883,"CreationDate":"2019-10-11T13:59:00.000","Title":"Equivalent of ServerConnection.makeRequest() on python side","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I have this html code:\n<button class=\"_2ic5v\"><span aria-label=\"Like\" class=\"glyphsSpriteComment_like u-__7\"><\/span><\/button>\nI am trying to locate all the elements that meet this class with phyton, and selenium webdriver library:\nlikeBtn = driver.find_elements_by_class_name('_2ic5v')\nbut when I print \nlikeBtn\nit prints \n[]\nI want to locate all of the buttons that much this div\/span class, or aria-label\nhow do I do that successfully? Thanks in advance\nupdate - when I do copy Xpath from page the print stays the same","AnswerCount":4,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":544,"Q_Id":58382323,"Users Score":0,"Answer":"Is it button class name dynamic or static? \nHow if you try choose By.CssSelector?\nYou can find element by copy selector in element","Q_Score":1,"Tags":"python,selenium,selenium-webdriver","A_Id":58391703,"CreationDate":"2019-10-14T18:35:00.000","Title":"how do I locate the btn by class name?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I have two separate AWS Virtual Machines set up within a region (different availability zones) both are connected via WebSocket (in Python) to a different load balancer (Cloudfront) of the same host server (also hosted with AWS) and receive frequent small WebSocket payloads - every 5ms.\nNB: I do not own the host server I am merely on the receiving end.\nBoth machines are receiving the same updates and I would like to measure on which machine the updates\/payloads\/packets are arriving first\nIn essence I would like to figure out which load balancer is \"closer\" to the host and so has the least latency overhead in transmitting the signal since my application is highly latency sensitive.\nI have tried using the system clock to get timestamps of the data arrival however it is not guaranteed that the two instances have their time synced to an appropriate accuracy.","AnswerCount":1,"Available Count":1,"Score":0.1973753202,"is_accepted":false,"ViewCount":38,"Q_Id":58431786,"Users Score":1,"Answer":"follow this.\n\n\nSend a request to the load balancer with the body of the request containing the timestamp when it was sent to the server. You can\n easily do this using the DateTime api of your fav language.\nAfter that packet arrives to your backend server residing on your instance (can be a simple node server or a rails server), you can get\n that request, and compare it to the current timestamp. \nYou can do this on both the servers and can easily compare which was faster.","Q_Score":0,"Tags":"python,amazon-web-services,networking,time,websocket","A_Id":58431962,"CreationDate":"2019-10-17T11:40:00.000","Title":"Determining which packet\/update arrived first between two machines","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":1,"Web Development":1},{"Question":"I am writing test code in Appium of an Andriod Project in python . The problem is that I am not able to access two button in two different Activity having same Id . I have tried to access the second button in this way.But none of them works. How to resolve the issue?\ndriver.find_element_by_id(\"com.myapp.testApp:id\/login[1]\").click(), driver.find_element_by_class_name(\"android.widget.Button\").click()\ndriver.find_element_by_xpath(\"(\/\/button[@id='login'])[1]\").click()\ndriver.find_element_by_xpath(\"\/\/android.widget.Button[@text='Change Password']\").click()","AnswerCount":5,"Available Count":2,"Score":0.0,"is_accepted":false,"ViewCount":2487,"Q_Id":58433478,"Users Score":0,"Answer":"If buttons texts are different, try by accessibility_id\ndriver.find_element_by_accessibility_id(\"Login\").click()\ndriver.find_element_by_accessibility_id(\"Change Password\").click()\nOR\nIf you are familiar with uiautomator,\ndriver.find_element_by_android_uiautomator('new UiSelector().resourceId(\"com.myapp.testApp:id\/login[1]\")')\ndriver.find_element_by_android_uiautomator('new UiSelector().resourceId(\"com.myapp.testApp:id\/login[2]\")')","Q_Score":0,"Tags":"appium,appium-android,appium-ios,python-appium,appium-desktop","A_Id":60395628,"CreationDate":"2019-10-17T13:09:00.000","Title":"How to find multiple element in appium having same id in Python?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I am writing test code in Appium of an Andriod Project in python . The problem is that I am not able to access two button in two different Activity having same Id . I have tried to access the second button in this way.But none of them works. How to resolve the issue?\ndriver.find_element_by_id(\"com.myapp.testApp:id\/login[1]\").click(), driver.find_element_by_class_name(\"android.widget.Button\").click()\ndriver.find_element_by_xpath(\"(\/\/button[@id='login'])[1]\").click()\ndriver.find_element_by_xpath(\"\/\/android.widget.Button[@text='Change Password']\").click()","AnswerCount":5,"Available Count":2,"Score":0.0,"is_accepted":false,"ViewCount":2487,"Q_Id":58433478,"Users Score":0,"Answer":"You can do link this, It will give you proper Output:\nInput is:\n\nd1 = driver.find_elements_by_id(\"com.application.zomato:id\/price\")\nitemtotalprice = d1[0].get_attribute(\"text\")\n\nHere, is my output:\n\nMandarin Oak\n\u20b9200\n\u20b9166.75","Q_Score":0,"Tags":"appium,appium-android,appium-ios,python-appium,appium-desktop","A_Id":69285758,"CreationDate":"2019-10-17T13:09:00.000","Title":"How to find multiple element in appium having same id in Python?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I am trying to grab a bunch numbers that are presented in a table on a web page that I\u2019ve accessed using python and Selenium running headless on a Raspberry Pi. The numbers are not in the page source, rather they are deeply embedded in complex html served by several URLs called by the main page (the numbers update every few seconds). I know I could parse the html to get the numbers I want, but the numbers are already sitting on the front page in perfect format all in one place. I can select and copy the numbers when I view the web page in Chrome on my PC. \nHow can I use python and get Selenium webdriver to get me those numbers? Can Selenium simply provide all the visible text on a page? How? (I've tried driver.page_source but the text returned does not contain the numbers). Or is there a way to essentially copy text and numbers from a table visible on the screen using python and Selenium? (I\u2019ve looked into xdotool but didn\u2019t find enough documentation to help). I\u2019m just learning Selenium so any suggestions will be much appreciated!","AnswerCount":2,"Available Count":1,"Score":0.1973753202,"is_accepted":false,"ViewCount":2512,"Q_Id":58443994,"Users Score":2,"Answer":"Well, I figured out the answer to my question. It's embarrassingly easy. This line gets just what I need - all the text that is visible on the web page: \npage_text = driver.find_element_by_tag_name('body').text","Q_Score":0,"Tags":"python,selenium,selenium-webdriver","A_Id":58453255,"CreationDate":"2019-10-18T04:33:00.000","Title":"How to get visible text from a webpage using Selenium & python?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I need to write some messages in discord with my bot, but I don't know how to do it. It seems that discord.py can't send messages autonomously.\nDoes anyone know how to do it?","AnswerCount":3,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":289,"Q_Id":58458098,"Users Score":0,"Answer":"I solved putting a while loop inside the function on_message.\nSo I need to send only a message and then my bot can write as many messages as he wants","Q_Score":0,"Tags":"python,python-3.x,discord.py","A_Id":58462241,"CreationDate":"2019-10-18T20:56:00.000","Title":"How to write in discord with discord.py without receiving a message?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"Code:\n<input type=\"file\" accept=\"image\/png, image\/jpeg\" style=\"display: none;\">\nWhat I am trying to do:\nupload = driver.find_elements_by_xpath(\"\/\/input[@type = 'file']\")\nupload.send_keys('\/Users\/username\/Desktop\/images\/filename.jpg')\nPS : This code is not in any iframe and element can be found (I checked its attribute to be sure)\nNothing happens, I have written the same logic to upload files in other test cases and it worked.\nThanks for your help!","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":96,"Q_Id":58458960,"Users Score":0,"Answer":"I was able to figure out the issue. Below the input field accepts \"jpeg and png\" and I was trying to upload \"jpg\". Once I changed the input to \"jpeg or png\" it worked.\nCode:\n<input type=\"file\" accept=\"image\/png, image\/jpeg\" style=\"display: none;\">\nWhat I was doing :\nupload.send_keys('\/Users\/username\/Desktop\/images\/filename.jpg')\nSolution : Change the file extension to jpeg or png.\nupload.send_keys('\/Users\/username\/Desktop\/images\/filename.jpeg')","Q_Score":0,"Tags":"python,selenium,file-upload","A_Id":58509726,"CreationDate":"2019-10-18T22:40:00.000","Title":"Python -> Selenium : Not able to upload the file","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I have a free trial of Chegg and would like to save the solutions to a few chapters for my math textbook. \nWhat I would like to do is open the link for each solution, and save that webpage as a PDF using Python 3. I was able to figure out the code to save the site as a PDF using some libraries including Requests. However, I'm having a hard time figuring out how to log in.\nI watched (too) many videos and read tutorials, but I can't figure out what my payload should include when sending a request to Chegg. I've easily been able to log in to some other sites, but Chegg doesn't have typical stuff that others do, such as \"username\" and \"password\" fields in the code (when you look at the Network tab in Chrome Developer Tools). Does anyone know what I'm missing? How can I can send a request to log in to Chegg specifically, without using any \"complex\" things like Selenium (I'm a beginner)?","AnswerCount":1,"Available Count":1,"Score":0.1973753202,"is_accepted":false,"ViewCount":805,"Q_Id":58467933,"Users Score":1,"Answer":"Why don't you copy authenticated cookies from your browser?\n\u2013 t.m.adamOct 20 '19 at 10:09\n\n\nEither copy the cookies as a string and pass it in the headers\nparameter, .get(url, headers={'Cookie':cookies}) or create a\ndictionary and pass it in the cookies parameter, .get(url, cookies=cookies). Web sites keep track of users with cookies, so if\nyour request has authenticeded cookies it should have the same\nprivileges as your user account. Of course, you could try to reproduce\nthe login process with requests, but I think the method I proposed is\neasier.\n\u2013 t.m.adam Oct 21 '19 at 0:27\n\nThought I would post this so that people that find this question won't miss the answer in the comments!","Q_Score":1,"Tags":"python,authentication,python-requests","A_Id":67375284,"CreationDate":"2019-10-19T20:57:00.000","Title":"How to log into Chegg with Python?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I'm using boto3 to download files from an s3 bucket & I need to support canceling an active file transfer in my client UI - but I can't find how to do it.\nThere is a progress callback that I can use for transfer status, but I can not cancel the transfer from there.\nI did find that boto3's s3transfer.TransferManager object has a .shutdown() member, but it is buggy (.shutdown() passes the wrong params to ._shutdown() a few lines below it) & crashes.\nIs there another way to safely cancel an active file_download?","AnswerCount":2,"Available Count":1,"Score":0.0996679946,"is_accepted":false,"ViewCount":478,"Q_Id":58469915,"Users Score":1,"Answer":"Can you kill the process associated with the file? \nkill $(ps -ef | grep 'process-name' | awk '{print $2}')","Q_Score":7,"Tags":"python,amazon-s3,boto3","A_Id":58807904,"CreationDate":"2019-10-20T03:58:00.000","Title":"How can I cancel an active boto3 s3 file_download?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I need to retrieve a url from shortcuts on Windows and Linux using python. \nThe shortcuts have the same name but different links.\nHow do I actually retrieve the url from the shortcuts?","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":91,"Q_Id":58505338,"Users Score":0,"Answer":"Desktop shortcuts in most desktop environments are just some machine readable files. I guess there is no way around parsing those files and retrieving the information you need that way.\nNote: The format of those files might differ substantially from desktop environment to desktop environment.","Q_Score":1,"Tags":"python-3.x,linux,windows","A_Id":58505526,"CreationDate":"2019-10-22T13:23:00.000","Title":"How do I retrieve the url a desktop shortcut is pointing to?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"Currently needing help doing the following: \nTake the highlighted data from the POST Request(s) Summary See Image and use python\/ pycharm to load them into .csv file as they come in. \nEach of these post requests will be a different link and I just need them to be automatically sent to the .csv file using python\/ pycharm.\nI am using TWILIO to send the post requests to NGROK.\nTHANK YOU SOOO MUCH","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":102,"Q_Id":58512681,"Users Score":0,"Answer":"First of all Ngrok cant be used to gather data, its a tunneling tool that exposes your local server to the internet. To achieve what you need you need to evaluate the POST request and parse the so that you can fetch the fields you need. Post the code you have to get further assistance.","Q_Score":0,"Tags":"python-3.x,automation,pycharm,ngrok","A_Id":58516764,"CreationDate":"2019-10-22T21:42:00.000","Title":"How do I use NGROK to gather post data and send it automatically to a .csv file","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I'm working on a Python script that connects to the Twitter API to pull in some tweets into an array, then pushes this to a mysql database. It's a pretty basic script, but I'd like to set it up to run weekly.\nI'd like to know the best way to deploy it so that it can automatically run weekly, so that I don't have to manually run it every week.","AnswerCount":2,"Available Count":2,"Score":1.2,"is_accepted":true,"ViewCount":46,"Q_Id":58512952,"Users Score":0,"Answer":"This depends on platform where you intend to run your python code. As martin says, this is not a python question and more of scheduling related question.","Q_Score":0,"Tags":"python,cloud","A_Id":58513538,"CreationDate":"2019-10-22T22:14:00.000","Title":"How to run Python code periodically that connects to the internet","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I'm working on a Python script that connects to the Twitter API to pull in some tweets into an array, then pushes this to a mysql database. It's a pretty basic script, but I'd like to set it up to run weekly.\nI'd like to know the best way to deploy it so that it can automatically run weekly, so that I don't have to manually run it every week.","AnswerCount":2,"Available Count":2,"Score":0.0,"is_accepted":false,"ViewCount":46,"Q_Id":58512952,"Users Score":0,"Answer":"You can create a batch file that can activate python and run your script and then, use task scheduler to schedule youe batch file execution weekly","Q_Score":0,"Tags":"python,cloud","A_Id":58513008,"CreationDate":"2019-10-22T22:14:00.000","Title":"How to run Python code periodically that connects to the internet","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I have 2 servers and a client. There is 2 way communication between the server and the client. Would I need multiple sockets on the client to communicate with the servers? I used only one socket and a few of the packets from the servers are missing. How many sockets would I need to communicate with the server?","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":37,"Q_Id":58530182,"Users Score":1,"Answer":"With UDP you almost always only need a single socket; you can call sendto() and recvfrom() on it to send and receive UDP packets from anywhere.\nAs for missing UDP packets, that is a fact of life with UDP; UDP packets can and sometimes will get dropped at any step of the path from sender and receiver. You'll need to design your app to tolerate that, or alternatively come up with a mechanism by which the receiver can detect that a packet was lost and request a resend (or otherwise somehow handle that situations).","Q_Score":0,"Tags":"python,sockets,udp","A_Id":58530257,"CreationDate":"2019-10-23T20:00:00.000","Title":"Socket programming in UDP","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I am in this kind of situation: I have a file on a server and I use a script to make a REST API call. This call is gonna open the file, read it and search inside the value I sent; in the end, the file is being closed (usual with open as synthax).\nNow, inside the file there is a timestamp which I use to say to the script: if value of time.now - timestamp is over 30 minutes, then update the file. Updating consist in download some files, merge them and finally create my new main file.\nMy question is: what happens if, considering that more than 30 minutes passed, I do 2 (or more) calls at the same time ?\nI premise that for now no controls are implemented.\nWhat I expect is that the \"first\" call will lock the file to update it, but the \"second\" ? I will have any kind of error ?\nThanks","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":26,"Q_Id":58537276,"Users Score":0,"Answer":"No error but ultimately the file will be overwritten by the last call. There will be no lock. You'll have to implement that yourself.","Q_Score":0,"Tags":"python,rest,api,with-statement","A_Id":58537323,"CreationDate":"2019-10-24T08:30:00.000","Title":"API call - What happen to file?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":1,"System Administration and DevOps":0,"Web Development":0},{"Question":"I came across the following example of creating an Internet Checksum:\n\nTake the example IP header 45 00 00 54 41 e0 40 00 40 01 00 00 0a 00 00 04 0a 00 00 05:\n\nAdding the fields together yields the two\u2019s complement sum 01 1b 3e.\nThen, to convert it to one\u2019s complement, the carry-over bits are added to the first 16-bits: 1b 3e + 01 = 1b 3f.\nFinally, the one\u2019s complement of the sum is taken, resulting to the checksum value e4c0.\n\n\nI was wondering how the IP header is added together to get 01 1b 3e?","AnswerCount":2,"Available Count":1,"Score":0.0996679946,"is_accepted":false,"ViewCount":324,"Q_Id":58567845,"Users Score":1,"Answer":"The IP header is added together with carry in hexadecimal numbers of 4 digits.\ni.e. the first 3 numbers that are added are 0x4500 + 0x0054 + 0x41e0 +...","Q_Score":0,"Tags":"python,networking,hex","A_Id":58567886,"CreationDate":"2019-10-26T03:37:00.000","Title":"Internet checksum -- Adding hex numbers together for checksum","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"Is it possible to create as many threads to use 100% of CPU and is it really efficient? I'm planning to create a crawler in Python and, in order to make the program efficient, I want to create as many threads as possible, where each thread will be downloading one website. I tried looking up for some information online; unfortunately I didn't find much.","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":81,"Q_Id":58573612,"Users Score":1,"Answer":"You are confusing your terminology, but that is ok. A very high level overview would help.\nConcurrency can consist of IO bound (reading and writing from disk, http requests, etc) and CPU bound work (running a machine learning optimization function on a big set of data).\nWith IO bound work, which is what you are referring to I am assuming, in fact your CPU is not working very hard but rather waiting around for data to come back.\nContrast that with multi-processing where you can use multiple core of your machine to do more intense CPU bound work.\nThat said multi-threading could help you. I would advise to use the asyncio and aiohttp modules for Python. These will help you make sure whilst you are waiting for some response to be returned, the software can continue with other requests.\nI use asyncio, aiohttp and bs4 when I need to do some web-scraping.","Q_Score":0,"Tags":"python,multithreading","A_Id":58573791,"CreationDate":"2019-10-26T18:36:00.000","Title":"Multithreaded crawler in Python","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":1,"System Administration and DevOps":0,"Web Development":0},{"Question":"I am getting following error \"line 30, in \n from exceptions import PendingDeprecationWarning\nModuleNotFoundError: No module named 'exceptions'\"\nI have imported docx module, yet I am getting this error\nimport docx\ndoc = docx.Document()\ndoc.add_paragraph(input(''))","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":56,"Q_Id":58578944,"Users Score":0,"Answer":"It seems you installed docx which is not maintained anymore and you should instead install the python-docx package.","Q_Score":0,"Tags":"python,python-docx","A_Id":58579078,"CreationDate":"2019-10-27T11:27:00.000","Title":"python, docx, what would have been error in my code while using docx module","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":1,"System Administration and DevOps":0,"Web Development":0},{"Question":"I am trying to write a script that will tell me the number of views a given YouTube video has. I know that I can grab all of the existing text on a page but I am curious if there is a way to get it to only output the amount of views the video has, and how I might go about implementing that. Thanks!","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":30,"Q_Id":58600774,"Users Score":1,"Answer":"requests library will only get you the page's content. You'll need HTML parse like BeautifulSoup to parse and get whatever information you need from the content.","Q_Score":0,"Tags":"python","A_Id":58600811,"CreationDate":"2019-10-29T02:37:00.000","Title":"Is there a way to search for specific text on a website using the requests library?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I have an S3 prefix. I want to find out, if it's a file or a whole folder. How do I do this with boto3?","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":858,"Q_Id":58610499,"Users Score":1,"Answer":"As was stated in a comment, there is no notion of directories in S3. You can find out if the object is a file by attempting to retrieve it, but in order to determine if it is a \"directory\", you would have to list all the objects in the bucket and determine if the provided value is a prefix to any of them. This may not be feasible for you depending on how many items exist (or can exist) in the bucket.","Q_Score":1,"Tags":"python,amazon-s3,boto","A_Id":58610620,"CreationDate":"2019-10-29T15:25:00.000","Title":"How to check, if S3 prefix corresponds to a file or folder?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I have this code related with the download, unzip, search and upload to a AWS S3. \nI have a function to download, another to unzip, another to make the searches and another function to upload the found files; to finally call all this functions into one function. \nThe problem it's that at some point (specifically in the last function) the execution gets an error. \nThe code has 1684 lines of code, and could take even 4 hours to execute. \nIf an error is found in a function, the try\/catches guarantee the final return. \nI've tried to call every function sequentially and they work. \nIf trying to call all the functions, except the last one it still works. \nIf trying to call the last function (the upload to S3), it works. \nI believe it could be related with the RAM \nTrust me, it's huge","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":37,"Q_Id":58613422,"Users Score":1,"Answer":"Basically, my problem was solved through the creation of a \".sh\" format script that calls all the functions sequentially.","Q_Score":0,"Tags":"python,python-3.x,amazon-web-services,amazon-s3","A_Id":58718308,"CreationDate":"2019-10-29T18:28:00.000","Title":"Two functional functions when called together creates an error","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I have a single page dashboard with 2 different slicers, one for country and the other for year I would like to export to PDF for all possible combinations of the slicer selection. Is that programmatically possible? Using R or Python or any other method. Need guidance.","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":393,"Q_Id":58627728,"Users Score":0,"Answer":"This is not directly possible in the Power BI Service, or currently supported by the Power BI REST API, it would have to be manual process to do this.","Q_Score":0,"Tags":"python,r,powerbi,dax,dashboard","A_Id":58651730,"CreationDate":"2019-10-30T14:34:00.000","Title":"Export to PDF Power BI Dashboard for all slicer options","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I am trying to copy files from 2 different bucket which are in two different aws account using same access key.\nSo it provide an error saying 403 forbidden.So what i want to do is, I want to check whether the access key that i am using to copy the file has permission to those bucket before i copy the file using boto3. Is there are anyway to do this?","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":579,"Q_Id":58638363,"Users Score":1,"Answer":"If you are copying objects between Amazon S3 buckets that belong to different AWS accounts, then you will need to use a single set of credentials that have:\n\nGetObject permission on the source bucket\nPutObject permission on the destination bucket\n\nAlso, the CopyObject command should be sent to the destination bucket to avoid problems with object ownership.\nTherefore, I would recommend:\n\nUse credentials from the destination account (dest-IAM-user)\nAdd a bucket policy to the source bucket that permits GetObject access by dest-IAM-user","Q_Score":3,"Tags":"python,amazon-s3,boto3","A_Id":58639883,"CreationDate":"2019-10-31T07:09:00.000","Title":"How to check whether s3 access key has access to a specific bucket or not in aws using boto3","Data Science and Machine Learning":0,"Database and SQL":1,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I need to query from the BBG API the nearest quote to 14:00 o'clock for a number of FX currency pairs. I read the developers guide and I can see that reference data request provides you with the latest quote available for a currency however if I run the request at 14.15 it will give me the nearest quote to that time not 14.00. Historical and intraday data output too many values as I need only the latest quote to a given time. \nWould you be able to advise me if there is a type of request which will give me what I am looking for.","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":293,"Q_Id":58641891,"Users Score":0,"Answer":"Further to previous suggestions, you can start subscription to \/\/blp\/mktdata service before 14:00 for each instrument to receive stream of real-time ticks. Cache the last tick, when hitting 14:00 mark the cache as pre-14:00, then mark the first tick after as post:14, select the nearest to 14:00 from both.","Q_Score":1,"Tags":"python,api,quote,bloomberg","A_Id":58652497,"CreationDate":"2019-10-31T10:58:00.000","Title":"Bloomberg API how to get only the latest quote to a given time specified by the user in Python?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"Every UDP server example I can find uses a while True loop to listen for incoming data. I'm attempting to use a single UDP socket server as part a kivy window that's also doing other things. As soon as I implement the server's while True loop everything locks up, as I guess I would expect it to do.\nHow do I listen on a UDP port and also have the rest of the program continue functioning?\nI've tried moving the UDP server handling to another (udp_server.py) file and then importing the function, but since I'm importing the while loop nothing changes.\nI've also tried assigning the received data to a variable inside udp_server.py and then just importing that variable, with udp_server.py already running separately, but even that is locking up my main program.\nI'm 99.99% sure it's just some basic thing that I should already know, but I'm new to Python. Thanks in advance for any help.","AnswerCount":2,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":479,"Q_Id":58664927,"Users Score":0,"Answer":"Can you share your python files?\nSo that i can understand how did you do it.\nI have a if __name__ == '__main__': loop, which listens gpio's.\nAnd also i would like to get commands form udp port.\nThanks\nHW is rPi Zero","Q_Score":3,"Tags":"python,sockets,while-loop,server,udp","A_Id":64665624,"CreationDate":"2019-11-01T19:34:00.000","Title":"How do I keep a UDP server listening in Python 3 without the WHILE loop locking up the program?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":1,"System Administration and DevOps":0,"Web Development":0},{"Question":"I am looking to connect to a car wirelessly using socketCAN protocol on MacOS using the module python-can on python3. I don't know how to install the socketCAN protocol on MacOS. Pls help.","AnswerCount":2,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":1978,"Q_Id":58700886,"Users Score":0,"Answer":"SocketCAN is implemented only for the Linux kernel. So it is not available on other operating systems. But as long as your CAN adapter is supported by python-can, you don't need SocketCAN.","Q_Score":2,"Tags":"python,python-3.x,can-bus,bus,python-can","A_Id":58708908,"CreationDate":"2019-11-04T20:54:00.000","Title":"Is it possible to use socketCAN protocol on MacOS","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"One of data sources I extract data from provides access through REST API in form of JSON responses. That's great, because I get data already structured, i.e., less pain with scraping and parsing unstructured HTML documents. \nHowever, they constrain HTTP traffic with rate limiting: requests per minutes\/hour\/month\/IP\/user email.\nWhen I was scraping HTML documents with Scrapy I could easily configure number of requests per second, delays between subsequent requests, number of threads, etc. I will call it \"load strategy\". The way it works in Scrapy under the hood, is that I generate a number of HTTP requests that Scrapy puts into the queue, and process requests from the queue with respect to the given \"load strategy\".\nIs there something like that for REST APIs? \nTo give some context, I'm using Python REST client generated from data source Swagger definitions. The client uses urlib3 under the hood. The client provides a way to execute requests in an asynchronous way and a way to configure a thread pool but it looks like I would need to play a bit around to configure it. I'm looking for out-of-the-box solution.","AnswerCount":1,"Available Count":1,"Score":0.1973753202,"is_accepted":false,"ViewCount":116,"Q_Id":58717607,"Users Score":1,"Answer":"With a generated client you will be able to make requests to the corresponding REST API. However, you'll need to build your own code\/logic for inserting delays between requests and request queuing. Much of the convenience that Scrapy is providing for you will need to be implemented by you. Or you'll need to find tools\/package that will provide this functionality for you.","Q_Score":0,"Tags":"python","A_Id":58717737,"CreationDate":"2019-11-05T18:43:00.000","Title":"Scheduling \"scraping\" of data from REST APIs","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I have a telegram bot , i have a code in python the publish links to different channels.\nIs there any way to know who press the link , i tried with get_updates , but it only shows messages.\nThanks in advance","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":157,"Q_Id":58728252,"Users Score":0,"Answer":"Text and InlineKeyboardButton link clicks are not tracked by Telegram. You can try using third-party services (url-shorteners) to track your links.\nNOTE: this feature is not declared by Telegram. In my experience, Telegram did not send Update about the same actions.","Q_Score":1,"Tags":"python-3.x,api,telegram","A_Id":58731036,"CreationDate":"2019-11-06T10:42:00.000","Title":"Telegram API get updates in a channel","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"For a project in school i have to compare two ontologies whose are in RDF formats.\nI can code in python but i'm not against any kind of language.\nMy main question is : Is it possible to compare classes of different RDF files?","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":60,"Q_Id":58762441,"Users Score":0,"Answer":"Pedantic note on \"RDF format\", RDF has a number of representations \nincluding ntriples, quads, turtle, rdfxml, notation3 but the\nrepresentation does not matter as they all represent a graph which is \nnone of them.\nShort answer to the question is something like\n yes! of course you can ... you will just probably be wrong.\nBut that is a little misleading, for the formal answer search on \"graph isomorphism\". \nFor a useful way to start thinking about it consider the\ndifference between two prose documents, which is easy, \nbecause each is an ordered list of items (words). \nEven when they are not identical you can say which parts \nare the same, and which parts are close or really divergent. \nWhen the documents are ontologies they are not an ordered list of items, \nthey are unordered lists of unordered lists of items where the items are allowed different names\/representations in each document. \nSo we punt on the fact it is (nigh) impossible as the ontologies grow larger and find heuristics that allow us to say something useful in the meanwhile. \n\"Semantic Similarity\" the art of judging if two graph fragments\nrepresent the same concept (are these classes the same?).\nIt can use whatever you can afford to bring to bear on the problem.","Q_Score":0,"Tags":"python,linux,rdf,rdfs","A_Id":59146291,"CreationDate":"2019-11-08T07:45:00.000","Title":"How to compare classes within two differents RDF","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":1,"System Administration and DevOps":0,"Web Development":0},{"Question":"Sorry if this is a silly question, I'm still a beginner in Android and couldn't find an answer to my question. I'm making an Android application that sends an input to a server and executes a Python script on that server to process the given input and generates an output. I was successful in sending data from Android client to Python server using SSH. \nI can also use SSH to retrieve the output back to the Android client. However, the Python script takes some time to generate the output, and I can't seem to find a way for the Android client to wait for the Python script to finish generating the output. I was able to do this on internet connection with using Firebase database and have the Python script upload the output into Firebase database and have the Android client listen for changes in database. But I'm looking for a way to do this locally without internet (i.e. Firebase).\nSo is there a way to make my Android application wait for a message from Python to know it has finished with generating the output so it can retrieve it back using SSH or any other way?","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":63,"Q_Id":58776295,"Users Score":0,"Answer":"Was a protocol what was send a code what was suppost to say if that 2 files are the same. but i do not remember the name. For u will be more easy to send the storage information's in bits (to the device) and after to start that important transfer. when the device will lost the connection will need to compare the real storage memory with the first information's send it and if has not match to ask the user to reconnect to internet or something like that. Maybe is helping until some one will give u the answer.","Q_Score":0,"Tags":"java,python,android","A_Id":58776390,"CreationDate":"2019-11-09T04:01:00.000","Title":"How to make Android app wait for a message from Python on a local server without internet?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"i have a question for you\nI'm using the udp socket server using baserequesthandler on python\nI want to protect the server against spoofing - source address changes.\nDoes client_address is the actual ip address of established to server ?\nIf not, how do I get the actual address?","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":76,"Q_Id":58790204,"Users Score":0,"Answer":"Authenticate the packets so that you know that every message in session X from source address Y is from the same client.\nBy establishing a shared session key which is then used along with a sequence number to produce a hash of the packet keyed by the (sequence, session_key) pair. Which is then included in every packet. This can be done in both directions protecting both the client and server.\nWhen you receive a packet you use its source address and the session number to look up the session, then you compute HMAC((sequence, session_key), packet) and check if the MAC field in the message matches. If it doesn't discard the message.\nThis might not be a correct protocol but it is close enough to demonstrate the principle.","Q_Score":0,"Tags":"python,socketserver","A_Id":58790759,"CreationDate":"2019-11-10T15:41:00.000","Title":"python baserequesthandler client address is real established ip?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"If I close a server socket while in a s.recv_into() loop from a client, I keep getting 0 from recv_into() infinitely. \nAs I understand, the only way to know a socket is closed was not through the Python socket mechanism itself, but using select \/ poll \/ etc.\nWhat's the rational of not returning -1 when the socket is closed by the other side? Why can't I properly differentiate \"no data currently\" from \"no one there anymore\"?\nEDIT:\nPerhaps the correct question is - why is no exception thrown when using recv(), if the other side terminated the connection?","AnswerCount":2,"Available Count":1,"Score":0.0996679946,"is_accepted":false,"ViewCount":588,"Q_Id":58790806,"Users Score":1,"Answer":"If I close a server socket while in a s.recv_into() loop from a client, I keep getting 0 from recv_into() infinitely. \n\nYes, as expected. A closed connection is indicated by 0, not -1.\n\nAs I understand, the only way to know a socket is closed was not through the Python socket mechanism itself, but using select \/ poll \/ etc.\n\nNo, that's not true. Simply calling recv in a non-blocking mode will do the job without involving any selector mechanism. In a non-blocking mode the \"no data yet\" state will be indicated by an appropriate exception (note that you have to inspect that exception, it's not the only case when recv throws). \nHowever this doesn't mean that the peer is still alive (the default TCP timeout is 15 minutes AFAIK). This cannot be reliably detected. The best you can do is to implement some pinging protocol over the wire.\n\nWhat's the rational of not returning -1 when the socket is closed by the other side? \n\nFor syscalls (such as recv) the -1 return code is (typically) reserved for errors. However Python translates these into exceptions (and so AFAIK you will never get -1 in Python). \nAnyway 0 is not an error. It's an information that the other side closed the connection. Perhaps this is a completely valid situation, there's no way for Python (or any other language) to know that.","Q_Score":0,"Tags":"python,sockets,recv","A_Id":58790848,"CreationDate":"2019-11-10T16:45:00.000","Title":"Why won't Python's socket.recv() return -1","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"It is not disabled by default.\nI have written a spider which consumes almost 2 GB of data per hour. Now I want to save my data consumption, images are of no use for me, so want to make sure they not being fetched.\nGiven that this is a P0 scenario, it should be a simple flag in settings.py but surprisingly from docs I couldn't find any. I found a lot of details about ImagesPipeline, enabling those pipelines, their storage etc, but no flag for people not interested in images. Let me know if I am missing anything.","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":492,"Q_Id":58801190,"Users Score":1,"Answer":"Scrapy does not download images unless you explicitly tell it to do it.\nYou can check in the run time logs the URLs that Scrapy downloads. If a image URL does not appear in the logs, it is not being downloaded, even if a webpage that contains images is downloaded.\nWhen you open a downloaded page in a web browser, images are downloaded on the fly by the web browser. They do not come from the downloaded webpage, they are not (usually) embedded in the webpage, the webpage indicates where in the Internet they are, and the web browser downloads them to display them, but Scrapy does not.\nThe only exception would be that images are actually embedded in the HTML code, as base64. This is uncommon, and probably not your case. And when that happens, there is no way you can prevent their download, you cannot download a webpage excluding part of its content.","Q_Score":0,"Tags":"python,python-3.x,web-scraping,scrapy","A_Id":58814419,"CreationDate":"2019-11-11T12:17:00.000","Title":"What's the best way to disable image download in scrapy?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"Can anyone please help me how to open a URL in Chrome browser in Icognito mode, as the URL that i'm trying to open in chrome is giving 404 error but is opening successfully in Icognito.\nPlease guide me what Robot command should be given to open in Icognito mode.\nThank you.","AnswerCount":3,"Available Count":1,"Score":0.0665680765,"is_accepted":false,"ViewCount":2692,"Q_Id":58801529,"Users Score":1,"Answer":"This should work:\nOpen Browser | ${url} | Chrome | options=add_argument(\"--incognito\")","Q_Score":0,"Tags":"python-2.7,browser,robotframework","A_Id":67016997,"CreationDate":"2019-11-11T12:38:00.000","Title":"How can we open a URL in Chrome Icognito mode using Robot Framework","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I want to click on the link from email and download the file in my laptop by using python script. Any one has any idea about this?\nI could see most of the information's which i am seeing in the internet is download the file in linux itself.","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":103,"Q_Id":58812527,"Users Score":0,"Answer":"The Instruction to do that:\nWriting a code to do the following:\n\nlogin to your email.\nparse your inbox and locate the specific email.\nparse the html source and locate the url for the download link.\ndownload it by connecting directly to it.","Q_Score":0,"Tags":"python","A_Id":58812603,"CreationDate":"2019-11-12T05:24:00.000","Title":"How to download the file by clicking the link in email by using the python script","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I'm searching for Python Script where I can make a hTTP Server, that allows to download the files of a specific folder (Linux)","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":67,"Q_Id":58824419,"Users Score":4,"Answer":"You can do it with the command:\npython -m http.server 8180","Q_Score":2,"Tags":"python","A_Id":58824428,"CreationDate":"2019-11-12T18:13:00.000","Title":"How to make a Python HTTP server where I can download files of a directory","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I can use selenium with pycharm but not in gnome-terminal.\nModuleNotFoundError:No module named 'selenium' \nis shown when I try to execute(in gnome-terminal) the python file which include the sentence \"from selenium import webdriver\".\nWhat is the difference between 2 environment?.","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":65,"Q_Id":58830285,"Users Score":0,"Answer":"Uninstall selenium in PyCharm and reinstall selenium in gnome-terminal and reinstall selenium in PyCharm.","Q_Score":0,"Tags":"python,selenium,pycharm,gnome-terminal","A_Id":58835319,"CreationDate":"2019-11-13T04:19:00.000","Title":"I can use selenium only in pycharm but not in gnome-terminal.It only works in pycharm","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I am trying to automate downloading of certain csv files from a website.\nThis is how I manually do it:\n\nI log in to the website.\nClick on the button export as csv.\nThe file gets downloaded. \n\nThe problem is the button does not have any link to it so I was not able to automate it using wget or requests.","AnswerCount":2,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":434,"Q_Id":58836664,"Users Score":0,"Answer":"You can use selenium in python. There is an option to click using \"link text\" or \"partial link text\". It is quite easy and efficient.\ndriver.findElement(By.linkText(\"click here\")).click()\nIt kind of looks like this.","Q_Score":0,"Tags":"python-3.x,python-requests,wget","A_Id":61072914,"CreationDate":"2019-11-13T11:59:00.000","Title":"Automate downloading of certain csv files from a website","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"There is a server that contains a json dataset that I need \nI can manually use chrome to login\nto the url and use chrome developer tool to read the request header for said json data\nI determined that the minimum required headers that should be sent to the json endpoint are ['cookie', 'x-xsrf-token', 'user-agent']\nI don't know how I can get these values so that I can automate fetching this data. I would like to use request module to get the data\nI tried using selenium, to navigate to the webpage that exposes these header values, but cannot get said headers values (not sure if selenium supports this)\nIs there a way for me to use request module to inch towards getting these header values...by following the request header \"bread crumbs\" so to speak?\nIs there an alternative module that excels at this?\nTo note, I have used selenium to get the required datapoints successfully, but selenium is resource heavy and prone to crash; \nBy using the request module with header values greatly simplifies the workflow and makes my script reliable","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":430,"Q_Id":58847818,"Users Score":0,"Answer":"Based on pguardiario's comment\nSessions cookies and csrf-token are provided by the host when a request is made against the Origin url. These values are needed to make subsequent requests against the endpoint with the JSON payload. By using request.session() against the Origin url, and then updating the header when using request.get(url, header). I was able to access the json data","Q_Score":0,"Tags":"python,selenium,web-scraping,python-requests","A_Id":58896850,"CreationDate":"2019-11-14T00:50:00.000","Title":"get json data from host that requires headers={'user-agent', 'cookie', x-xsrf-token'}","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"Is there a way to make your Selenium script undetectable in Python using geckodriver?\nI'm using Selenium for scraping. Are there any protections we need to use so websites can't detect Selenium?","AnswerCount":4,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":21277,"Q_Id":58873022,"Users Score":0,"Answer":"It may sound simple, but if you look how the website detects selenium (or bots) is by tracking the movements, so if you can make your program slightly towards like a human is browsing the website you can get less captcha, such as add cursor\/page scroll movements in between your operations, and other actions which mimics the browsing. So between two operations try to add some other actions, Add some delay etc. This will make your bot slower and could get undetected.\nThanks","Q_Score":20,"Tags":"python,selenium,firefox,geckodriver,selenium-firefoxdriver","A_Id":71659726,"CreationDate":"2019-11-15T08:29:00.000","Title":"How can I make a Selenium script undetectable using GeckoDriver and Firefox through Python?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I want a reverse proxy (using mitmproxy\/mitmdump) for an insecure site. I want the proxy to run on a non-standard port (not 443) and to be accessible only via https. This is the closest I've gotten:\nmitmdump -p 2112 --mode reverse:http:\/\/localhost:41781 --set block_global=false --certs full.pem\nThis works when I access it via https: https:\/\/localhost:2112. The problem is, it is also accessible via http: http:\/\/localhost:2112.\nHow do I disable that and make it only accessible via https?","AnswerCount":2,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":2329,"Q_Id":58892960,"Users Score":0,"Answer":"(Obligatory: Use Apache\/nginx\/... if you need a production-grade reverse proxy)\nMitmproxy currently always supports both HTTP and HTTPS to simplify setups (we had lots of bug reports because of misconfigurations). If you want to disable HTTP, you could write an addon that detects flow.request.scheme == \"http\" and then either invokes flow.kill() or sets flow.response = mitmproxy.http.HTTPResponse.make(<redirect data here>). Lastly, you can also make sure to set HSTS headers in the response hook in mitmproxy.","Q_Score":1,"Tags":"python,ssl,reverse-proxy,mitmproxy","A_Id":58916214,"CreationDate":"2019-11-16T16:45:00.000","Title":"mitmproxy reverse proxy - want SSL only on non-standard port","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I'm using azureml-sdk behind a corporate's proxy and therefore forced to manually add the proxy's certificate to the list of trusted hosts. It works fine by manually append the server certificate to the file \"cert_ca.pem\" of the package certifi. \nAs mentioned in the documentation of python-requests, this should also be possible by setting the environment variable \"REQUESTS_CA_BUNDLE\" to a specific file. However, the environment variable seems to be ignored by azureml-sdk. Is this the correct behavior? Or how is it possible to specify an own \"pem\" file?","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":243,"Q_Id":58892964,"Users Score":0,"Answer":"The correct environment variable is REQUESTS_CA_BUNDLE - note the plural.","Q_Score":2,"Tags":"python,azure,ssl,azure-machine-learning-service,certifi","A_Id":58896952,"CreationDate":"2019-11-16T16:45:00.000","Title":"azureml-sdk ignores the REQUESTS_CA_BUNDLE environment variable","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"okay guys been out of coding for a while and im trying to get back in because i love it\ncurrently im writing a python script and im having trouble finding the correct way to click button ive tried several different times and this is where im stuck at. \nAdd to Bag\nwhen copying css selector: \n#dwfrm_product_addtocart_d0rmouqnbpev > div > div.pdp-button-block > button.pdp-add-to-bag.add-to-cart.pid-19CE-ac0eff0691331d3dc991483b06\nwhen copying xpath: \n\/\/*[@id=\"dwfrm_product_addtocart_d0byuazkjuyr\"]\/div\/div[5]\/button[1]\ncode i have in script:\npython_button = driver.find_element_by_xpath(\"\/\/*[@id='dwfrm_product_addtocart_d0byuazkjuyr']\/div\/div[5]\/button[1]\")[0] \n python_button.click()\n\nive tried different things and currently having a hard time","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":49,"Q_Id":58900666,"Users Score":0,"Answer":"Few points in the line used:\nfind_element_by_xpath(\"\/\/*[@id='dwfrm_product_addtocart_d0byuazkjuyr']\/div\/div[5]\/button[1]\")[0]\n\nfind_element_by_xpath returns a WebElement and not a list of WebElements. So you can remove the [0] as it picks the 1st element matching the xpath.\nOr you can use find_elements_by_xpath if you want to match a list and pick one element using index.\nThe last part of id looks randomly generated, so better use contains instead of going for an exact match.\n\nYou can verify in console if the locator used is correct(using $$ for css and $x for xpath). If it is correct, then see if any other element is overlapping the element that you're trying to click and try clicking on that.\nHope this helps!\nGood luck","Q_Score":0,"Tags":"python,html,css,xpath","A_Id":58909747,"CreationDate":"2019-11-17T12:56:00.000","Title":"find css selector to add to python script using selenium","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I am unable to download a .py file from an email. I get the error \"file not supported.\" The file was saved from a Jupyter-notebook script.\nI have Python 3.6.6 and Jupyter downloaded on my Windows 10 laptop and tried to access the file through Chrome and through my computer's email app, but this didn't resolve the problem.\nAny ideas on how to make the file compatible with my computer?\nEDIT: I had to have the .ipynb file sent rather than the .py file.","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":3687,"Q_Id":58920830,"Users Score":0,"Answer":"Generally email providers block any thing which can possibly execute on clients machine.\nBest option to share will be transfer via email will be .py.txt \nor any cloud drives.","Q_Score":0,"Tags":"python,jupyter-notebook","A_Id":60748514,"CreationDate":"2019-11-18T18:32:00.000","Title":"How to allow my computer to download .py files from an email","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":1,"System Administration and DevOps":0,"Web Development":0},{"Question":"I want to collect the list of favorite(like) in each tweet for a specific celebrity so that I can get the top three users who like his\/her posts the most in a period. However, I can't find any information using Tweepy(Twitter API) to gather it. Is it the only possible to use Selenium to collect the favorite list for each tweet?\nThanks","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":36,"Q_Id":58925031,"Users Score":0,"Answer":"Twitter's API does not provide this information. Also note that using Selenium or other web scraping methods is against the Twitter Terms of Service.","Q_Score":0,"Tags":"python-3.x,twitter,tweepy","A_Id":58937988,"CreationDate":"2019-11-19T00:42:00.000","Title":"How to find lists of favorite for each tweet?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I installed openstackclient:4.0.0 on Python 2.7.5 and getting the below error\nException raised: cannot import name redirect_stdout\nAny thoughts\nThanks","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":179,"Q_Id":58940943,"Users Score":0,"Answer":"Well, a complete exception trace would have been more helpful.\n\nThe exception which you are getting is due to missing contextlib which\n is part of python3.\n Openstackclient is using contextlib in network related operations\n currently, however not using redirect_stdout.\n\nLooks like you are using a lib which is using redirect_stdout from contextlib. It doesn't seems to be openstackclient.","Q_Score":0,"Tags":"python-2.7,openstack","A_Id":58966906,"CreationDate":"2019-11-19T18:48:00.000","Title":"openstackclient: Exception raised: cannot import name redirect_stdout","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I am trying to intercept data sent from the server to a client , where the script is json and server is nginx (https). Taking for example I am also a client and I am penetration testing the server for vulnerabilities.\nI have tried inspecting elements from chrome and everything seems fine (the data is only sent to the other client), I have checked the network files and found nothing, I also tried python import request method, and I have recieved the HTTP 4xx error.\nMy question is here, what way is there to be able to intercept the data without complete access to server?","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":247,"Q_Id":58949731,"Users Score":0,"Answer":"Have you tried to look for it on the 'Network' or 'Console' tab specifically at the element inspector?\nPlease note that JSon is not tecnically a script, it is just a object notation, a way of sending information from one place to another, wheareas scripts are intended for interpretation and execution (Bash, JavaScript and Python are examples)\nIf that data is not being desplayed there, you will probably need some kind of network sniffer (Like wireshark) and filter information comming form that server to your computer. It might be encrypted though, and undreadable for humans.\nHope I helped.","Q_Score":0,"Tags":"python,json,google-chrome","A_Id":58950496,"CreationDate":"2019-11-20T08:25:00.000","Title":"How am I able to intercept server client requests using Chrome and Python?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I am currently trying to automate a process using Selenium with python, but I have hit a roadblock with it. The list is part of a list which is under a tree. I have identified the base of the tree with the following xpath \nitem = driver.find_element_by_xpath(\"\/\/*[@id='filter']\/ul\/li[1]\/\/ul\/\/li\")\nitems = item.find_elements_by_tag_name(\"li\")\nI am trying to Loop through the \"items\" section but need and click on anything with an \"input\" tag\nfor k in items:\n WebDriverWait(driver, 10).until(EC.element_to_be_clickable((k.find_element(By.TAG_NAME, \"input\")))).click()\nWhen execute the above I get the following error:\n\"TypeError: find_element() argument after * must be an iterable, not WebElement\"\nFor some reason .click() will not work if I use something like the below.\nk.find_element_by_tag_name(\"input\").click()\nit only works if i use the webdriverwait. I have had to use the web driver wait method anytime i needed to click something on the page.\nMy question is:\nWhat is the syntax to replicate items = item.find_elements_by_tag_name(\"li\")\nfor WebDriverWait(driver, 10).until(EC.element_to_be_clickable((k.find_element(By.TAG_NAME, \"input\")))).click()\ni.e how do I use a base path and append to the using the private methods find_elements(By.TAG_NAME)\nThanks in advance","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":115,"Q_Id":58978646,"Users Score":0,"Answer":"I have managed to find a work around and get Selenium to do what i need. \nI had to call the javascript execution, so instead of trying to get\nWebDriverWait(driver, 10).until(EC.element_to_be_clickable((k.find_element(By.TAG_NAME, \"input\")))).click() to work, i just used\ndriver.execute_script(\"arguments[0].click();\", k.find_element_by_tag_name(\"input\"))\nIts doing exactly what I needed it to do.","Q_Score":0,"Tags":"python,selenium,selenium-webdriver,selenium-chromedriver","A_Id":58995977,"CreationDate":"2019-11-21T15:43:00.000","Title":"Looping through webelements with selenium Python","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I would like my bot to recover the attachment of a message and send it in another channel with an embed, I make this:\nembed.set_image(url=message.attachments[0].url\nbut it only work with images (or gif, but video doesn't work)","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":235,"Q_Id":59007265,"Users Score":0,"Answer":"embed.set_image as the name implies is for settings images. Unfortunately Discord does not let bots send videos inside embeds. Your limited options are to take the url and post it as just the content of the message and let discord embed it.","Q_Score":0,"Tags":"python-3.x,discord,discord.py,discord.py-rewrite","A_Id":59056271,"CreationDate":"2019-11-23T11:46:00.000","Title":"Discord bot recover the attachment of a message and send it in another channel with an embed","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I am using google cloud functions with Python. I want to format all logs with some additional data, e.g. customer id. I achieved this without any problem using Stackdriver Logging library with CloudLoggingHandler. In the same manner, I also like to add this information to uncaught error logs and tracebacks.\nI tried to modify sys.excepthook and sys.stderr but it did not work, probably they are handled exclusively by cloud functions.\nIs there any way I can modify uncaught exceptions or modify handled errors, e.g. by using Stackdriver error reporting? Or do you have any alternative solution for this (without catching all exceptions)?","AnswerCount":2,"Available Count":1,"Score":0.1973753202,"is_accepted":false,"ViewCount":732,"Q_Id":59022947,"Users Score":2,"Answer":"Cloud Functions provides the (currently) highest level abstraction for code execution. The philosophy is that your bring the code that implements your desired logic and Cloud Functions provides the highest level environment for execution. This has pluses and minuses. \nFurthermore, the biggest plus is that you have the very least to concern yourself with in order to get the execution you desired. \nOn the other hand, you have very little in the way of operational control (the vision is that Cloud Functions provides the maximum in operational control).\nAs a consequence, if you want more control over the environment at the cost of you having to do more \"work\", I suggest you Cloud Run. In Cloud Run, you package your application logic as a Docker container and then ask it to take care of all execution of such logic. In your container, you can do anything you want ... including using technology such as Stackdriver logging and defining a CloudLoggingHandler. Cloud Run then takes care of your scaling and execution environment from there.\nTo sum up, the answer then becomes \"No\" you don't have control over error logs in Cloud Functions but you can achieve your desired outcome by leveraging Cloud Run instead.","Q_Score":1,"Tags":"python-3.x,google-cloud-functions,stackdriver,google-cloud-logging,google-cloud-error-reporting","A_Id":59023178,"CreationDate":"2019-11-24T22:12:00.000","Title":"How to modify error logs in google cloud functions?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I have my web application. Now i want to integrate salesforce into my web app so that i can push data from my app to any salesforce org after the authentication(OAuth).\nI found 2 ways:\n1. Connected Apps\n2. via wsdl generation file and use\nI created a connected app from my developer account and i authenticated using consumer key, cusumer secret key(from my connected app) and username of user and secret token of the user account.\nI tried with another free trail account, It's validating and fetching the details and post data also working.\nMy question is, shall i deploy my connected app into app exchange, then only i caan use REST APIs ?\n\ngenerating wsdl and coding around is the better option than the above ?\n\nIs there a option, only one time authentication enough for any number of sessions and use the REST APIs?\nPlease suggest me a best way to proceed.","AnswerCount":1,"Available Count":1,"Score":0.1973753202,"is_accepted":false,"ViewCount":43,"Q_Id":59031743,"Users Score":1,"Answer":"You're mixing up a couple of independent issues here.\n\nIf you're going to authenticate via OAuth, you must have a Connected App. \nA SOAP API login() call requires you to store a username and password, which is undesirable.\nThere is no way to \"permanently\" authenticate, i.e., to get a session id that never expires. Your app must always be ready to get a new access token via the OAuth refresh token it obtains and stores (via, for example, the Web Server OAuth flow), or to reauthenticate via JWT flow.\nConnected Apps are global metadata in most cases. You don't need to deploy a Connected App into a target org in order to authenticate using its Client Id and Secret into that org. The only exception I'm aware of is if you want to use the JWT flow with a certificate and preauthorized Profiles and Permission Sets.\nBased on what you've shared, I don't see any reason for the AppExchange to be involved.","Q_Score":0,"Tags":"python-2.7,salesforce","A_Id":59143337,"CreationDate":"2019-11-25T12:21:00.000","Title":"want to push\/pull data into\/from SalesForce Accounts\/leads from\/to my external web app","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I developed a program to interact between Telegram and other 3rd party Software. It's written in Python and I used the Telethon library. \nEverything works fine, but since it uses my personal configuration including API ID, API hash, phone number and username, I would like to know how to handle all of this if I wanted to distribute the software to other people. \nOf course they can't use my data, so should they login into Telegram development page and get all the info? Or, is there a more user-friendly way to do it?","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":118,"Q_Id":59032433,"Users Score":1,"Answer":"Since the API ID and the API Hash in Telegram are supposed to be distributed with your client all you need to do is prompt the user for their Phone Number. \nYou could do this using a GUI Library (like PySide2 using QInputDialog) or if it is a command line application using input(). Keep in mind that the user will also need a way to enter the code they receive from Telegram and their 2FA Password if set.","Q_Score":0,"Tags":"python,telegram,telethon","A_Id":59032610,"CreationDate":"2019-11-25T12:59:00.000","Title":"Python and Telethon: how to handle sw distribution","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I am trying to make a calculator (with matrix calculation also). I want to make interface in JavaScript and calculation stuff in Python. But I don't know how to send parameters from python to JavaScript and from JavaScript to python.\nEdit: I want to send data via JSON (if possible).","AnswerCount":4,"Available Count":1,"Score":0.0996679946,"is_accepted":false,"ViewCount":1401,"Q_Id":59149354,"Users Score":2,"Answer":"You would have to essentially set both of them up as API's and access them via endpoints. \nFor Javascript, you can use node to set up your API endpoint, and for Python use Flask.","Q_Score":1,"Tags":"javascript,python,json","A_Id":59149367,"CreationDate":"2019-12-03T02:24:00.000","Title":"How to send data from Python script to JavaScript and vice-versa?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":1,"System Administration and DevOps":0,"Web Development":0},{"Question":"I need all video's ID between X to Y dates of specific subject. I used YouTube data API v3 and every thing was OK, until it returned empty item after 1000 result. I searched about this issue and figured out it's API limitation!!! Does anyone have any solution?\nIf it helps I code in python.I did test in php too and same result. It doesn't depend on languages.\nthanks","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":142,"Q_Id":59165991,"Users Score":0,"Answer":"Just split your date range into multiple requests, so each request < result limit.","Q_Score":0,"Tags":"python,youtube-api","A_Id":59167151,"CreationDate":"2019-12-03T21:58:00.000","Title":"How to Cross the 1000 Results Limit of youtube data API?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I have a question so in telegram bot there is a InlineKeyboardMarkup that can have a callback, but its not persistent, does not stick to the botton of the screen, when you write a new message it will go up, like a normal message. And on the other hand there is ReplyKeyboardMarkup that does persist and stay at the botton, but does not have a callback. So is there a way to create a button that stays at the botton but allso accepts a callback that you can catch with callback_query_handler","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":329,"Q_Id":59201308,"Users Score":0,"Answer":"No, there is no way to create a button that you can capture with callback_query_handler in a ReplyKeyboardMarkup.\nWhen you press a KeyboardButton, it sends a message with its text. What you could do is put your bot's commands in the buttons or you can make a message handler that captures non-command messages and execute some code for each text of your buttons.","Q_Score":0,"Tags":"python,telegram","A_Id":59201831,"CreationDate":"2019-12-05T18:32:00.000","Title":"Fixed button for Telegram bot that accepts a callback","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I want packets to be sent immediately (and individually) but even when setting socket.setsockopt(IPPROTO_TCP, TCP_NODELAY, 1) before socket.send(...), packets are still being received by the client (using socket.recv(1024)) in multiples.","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":132,"Q_Id":59203296,"Users Score":2,"Answer":"Disabling the Nagle algorithm only makes sure that writing data does not get delayed at the sender side. It does not prevent accumulation of the payloads of multiple packets on the recipients side. A recv(1024) will try to get up to 1024 bytes from the read buffer and it does not care if these bytes were send within a single or within multiple packets.\nIn general, TCP is a byte stream and has no implicit message semantics. This does not change by invoking some socket options, it still has to be treated as a byte stream. Any message semantics you want to have must be implemented in the application on top of this byte stream.","Q_Score":0,"Tags":"python,sockets,tcp","A_Id":59203624,"CreationDate":"2019-12-05T21:09:00.000","Title":"Nagle not being disabled by TCP_NODELAY","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"i run a python program using beautifulsoup and requests to scrape embedded videos URL , but to download theses videos i need to bypass a ads popups and javascript reload only then the m3u8 files start to appear in the network traffic;\nso i need to simulate the clicks to get to the javascript reload (if there's a method better than selenium, trying to reduce script dependencies) and then when the m3u8files appear i need to get their url.","AnswerCount":2,"Available Count":1,"Score":-0.0996679946,"is_accepted":false,"ViewCount":1293,"Q_Id":59227170,"Users Score":-1,"Answer":"Blockquote\n `i run a python program using beautifulsoup and requests to scrape embedded videos URL , but to download theses videos i need to bypass a ads popups and javascript reload only then the m3u8 files start to appear in the network traffic;\n\nso i need to simulate the clicks to get to the javascript reload (if there's a method better than selenium, trying to reduce script dependencies) and then when the m3u8files appear i need to get their url\n`Can I have your code please?","Q_Score":0,"Tags":"javascript,python-3.x,beautifulsoup,python-requests,m3u8","A_Id":59507440,"CreationDate":"2019-12-07T14:47:00.000","Title":"How to scrape m3u8 from Network-Stream after a js reload (Python)","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I have a python program in which I am trying to do a post sending file contents as a string. I am doing the following:\nheader 'content-type':'multipart\/form-data'\ndata {file1: filedata}\nfiledata is a string I build by reading a file putting the contents in the string\nwhen I make the post call I get a 500 back and looking in the catalina log I see the error \"couldn't determine the boundary from the message\".\nthe thing is when I put this in insomnia it works properly and the catalina log shows the header had ; boundary=X-INSOMNIA-BOUNDARY appended to the content-type I defined\nWhy would this work for insomnia but not when I do it in python using requests? This is my requests call (auth is set to None)\nresponse = requests.post(url, data=data, headers=headers, auth=auth, timeout=REQUEST_TIMEOUT, verify=False)\nheaders = 'content-type':'multipart\/form-data\ndata = {'timepunch': 'datastring'}","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":136,"Q_Id":59250063,"Users Score":0,"Answer":"I found my problem - when sending a file I need the request to be files-data instead of data=data - not sure exactly what happens in the background but this resolved my issue","Q_Score":1,"Tags":"python","A_Id":59251085,"CreationDate":"2019-12-09T13:36:00.000","Title":"Python Post Getting couldn't determine the boundary from the message","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":1,"System Administration and DevOps":0,"Web Development":1},{"Question":"After updating to Spyder 4.0.0 (from Spyer 3.7) I get following error on scripts that worked fine before:\nFile \"C:\\Users\\Usuario\\Anaconda3\\lib\\zipfile.py\", line 1395, in getinfo\n 'There is no item named %r in the archive' % name)\nKeyError: \"There is no item named '[Content_Types].xml' in the archive\"\nWith python 3.7.4 I did not get this error. Now with 3.7.1 I have this issue. \nAny suggestion? where can I find the root of the issue?\nThank you.\nBest regards,","AnswerCount":1,"Available Count":1,"Score":0.1973753202,"is_accepted":false,"ViewCount":149,"Q_Id":59257452,"Users Score":1,"Answer":"Issue solved: the excel file was corrupted","Q_Score":0,"Tags":"python,spyder","A_Id":59272780,"CreationDate":"2019-12-09T22:15:00.000","Title":"Key error after updating Spider and Anaconda","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I'm currently working on a program that uses a reverse geocoder and Google's street view metadata API to generate random coordinates and locate street view locations from these coordinates. However, I only wish to obtain road cameras, not those produced from Google Trekker. Is there a way to find or determine all Trekker locations so I can exclude them from my dataset?","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":58,"Q_Id":59259523,"Users Score":0,"Answer":"You could potentially check if there are any other points within a small radius (3 or so metres) as trekker locations will usually have a smaller distance between each panorama than road panoramas.\nDid some testing myself and found that the average for trekking panoramas is about 4 metres while for road panoramas the average is about 10 metres.\nThis is by no means a guaranteed solution but with tuning it could work with reasonable accuracy.","Q_Score":1,"Tags":"python,python-3.x,reverse-geocoding,google-street-view","A_Id":59344772,"CreationDate":"2019-12-10T02:53:00.000","Title":"Google Maps - How to find all Google Trekker Locations?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I am running some test scripts and while it is running at the background I want to do my other jobs on excel or another web page? But sometimes I got same errors since the selenium can not find the specific field to click? What can be the reason? Is it not working well at the background?","AnswerCount":2,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":209,"Q_Id":59287278,"Users Score":0,"Answer":"To run selenium concurrently while i am working on other projects i just run it in a headless mode, that tends to work most of the time. \nThe thing is that you have to have completed the building of the script before you do this otherwise debugging could be a nightmare if you are new or fairly new to selenium web-driver","Q_Score":0,"Tags":"python,selenium,testing,automation,automated-tests","A_Id":59287821,"CreationDate":"2019-12-11T13:51:00.000","Title":"Can selenium python work good at background while I am doing other processes on main page?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"Is it possible to write TCP SYN scan (half-open scanning) in python without using driver? I want to implement this kind of scanning in python without using scapy as I know it requires installing a driver.","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":195,"Q_Id":59301532,"Users Score":0,"Answer":"If you're on Linux, BSD, OSX, yeah sure, using Raw sockets.\nI assume if you talk about a \"driver\" it means you're on Windows in which case you will be restricted (can't access layer 2), and will require administrative privileges anyways.\nIn any case, you can still use Scapy without any \"driver\" (libpcap-based) installed. Some functions will be disabled","Q_Score":0,"Tags":"python,scapy,nmap,port-scanning","A_Id":59310650,"CreationDate":"2019-12-12T09:35:00.000","Title":"Python-TCP SYN scan without using driver","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":1,"Web Development":0},{"Question":"I have a question regarding SSL verification within the requests library for Python, but I believe it to me more general than that.\nI am currently ignoring certificate verification because the third party API I need to connect to is using a self-signed certificate.\nWhat are the implications for turning SSL verification off in requests? And what are the implications for not verifying SSL certificates in the real-world. Can I gaurantee the data transported is secure\/encrypted?","AnswerCount":2,"Available Count":2,"Score":0.1973753202,"is_accepted":false,"ViewCount":1626,"Q_Id":59303814,"Users Score":2,"Answer":"Can I gaurantee the data transported is secure\/encrypted?\n\nThe data is encrypted (this is TLS confidentiality guarantee) but since you did not authenticate the remote part (if you disable certificate validation or bypass all errors) you could be as well sending the encrypted content to anyone, including an attacker, which of course on his side will read it in plain, as the TLS handshake succeeded if you do not validate the remote party.\nTLS provides multiple features, two major ones being authentication and confidentiality. They are orthogonal (you can have one without the other) but it may not be so useful to not have all of them.\nContrary to natural thinking, authentication is more important than confidentiality because if you have no insurance about who is the remote party, what do you gain by sending it encrypted? Nothing.","Q_Score":3,"Tags":"python-3.x,ssl,python-requests,ssl-certificate,urllib3","A_Id":59308892,"CreationDate":"2019-12-12T11:42:00.000","Title":"What are the implications of ignoring SSL certificate verification?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I have a question regarding SSL verification within the requests library for Python, but I believe it to me more general than that.\nI am currently ignoring certificate verification because the third party API I need to connect to is using a self-signed certificate.\nWhat are the implications for turning SSL verification off in requests? And what are the implications for not verifying SSL certificates in the real-world. Can I gaurantee the data transported is secure\/encrypted?","AnswerCount":2,"Available Count":2,"Score":1.2,"is_accepted":true,"ViewCount":1626,"Q_Id":59303814,"Users Score":3,"Answer":"This is a security sin, as anyone could spoof this certificate and intercept your traffic. You should just add the self-signed certificate to the trusted certificate chain of the machine which is using the API.\nHow you do that depends on the operating system and specific setup, but a quick google will guide you to the right solution.","Q_Score":3,"Tags":"python-3.x,ssl,python-requests,ssl-certificate,urllib3","A_Id":59303990,"CreationDate":"2019-12-12T11:42:00.000","Title":"What are the implications of ignoring SSL certificate verification?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I'm building a telegram bot using pyTelegramBotAPI libraries, I wanted to know if there is a way to know when a user deletes a chat with my bot, so the bot will not send more messages towards that specific id.\nreading the telegram API I found nothing, can you help me?","AnswerCount":2,"Available Count":1,"Score":0.0996679946,"is_accepted":false,"ViewCount":1203,"Q_Id":59311680,"Users Score":1,"Answer":"As I know, there is no way. When you chat with a friend, is there any way to know that he\/she has deleted the chat page of you? NO.\nTelegram bot is completely similar to simple chat in this case up to\n this date. It doesn't have considered in Telegram Bot API up to this\n date.","Q_Score":0,"Tags":"python,telegram-bot","A_Id":59360428,"CreationDate":"2019-12-12T19:40:00.000","Title":"How to delete a Telegram user id when user delete the chat of the bot","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I'm using PyCharm. I try to install Selenium but I have a problem with proxy. I try to add packages manually to my project\/environment but I don't know how.\nI downloaded files with Selenium. Could you tell me how to add this package to Project without using pip?","AnswerCount":2,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":2667,"Q_Id":59320019,"Users Score":0,"Answer":"open pycharm\nclick on settings (if u use mac click on preference )\nclick project\nthen click projecti nterpreter\nclick the + button on the bottom of the window you can see a new window search Selenium package and install","Q_Score":3,"Tags":"python,selenium,pip,pycharm","A_Id":59320231,"CreationDate":"2019-12-13T09:54:00.000","Title":"Add manually packages to PyCharm in Windows","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":1,"System Administration and DevOps":0,"Web Development":0},{"Question":"I'm trying to export all the comments on a post with Facebook Graph.\nI can get all the comments fine along with the tags but I'm having some problems getting the user who actually commented. I own the page and post that I'm doing this too. Is there a specific permission I need? I have manage_pages, pages_show_list, and public_profiles.\nAny help would be great, thanks!","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":25,"Q_Id":59329025,"Users Score":0,"Answer":"It is important that you are using a Page Token. With a simpel App Token, you will not get the from field in the response.","Q_Score":0,"Tags":"python,facebook,facebook-graph-api","A_Id":59335735,"CreationDate":"2019-12-13T19:55:00.000","Title":"How can I View Who a Comment was From on Facebook API","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"My server can run on either Python 3.6 or Python 3.8. The problem seems to be the client.\nIf I run the client on Python 3.6, I get TLSv1.2 working with no error. But if I run the same client code on Python 3.8, I get NO_SHARED_CIPHER on the server and SSLV3_ALERT_HANDSHAKE_FAILURE on the client.\nI suspect it has something to do with server or client certificate.\nUPDATE: I created some new server and client certificates to experiment around with: ed25519, rsa, and secp256k1. From what I've seen, the problem above only shows up when the server certificate is secp256k1 and both the client and server are running Python 3.8 (and therefore with TLSv1.3). The client can use any type of certificate.","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":298,"Q_Id":59333458,"Users Score":0,"Answer":"... NO_SHARED_CIPHER ...\n I suspect it has something to do with server or client certificate.\n\nThis is a problem of cipher selection and cipher selection has almost nothing to do with the certificate, at least if the the certificate type is still the same, i.e. ECC vs. RSA. And I assume that you are still using the same certificate or at least the same certificate type in the working and non-working cases.\nIt is more likely that your client tries to use some RC4 or 3DES based cipher which is today considered obsolete and thus is no longer supported by a server when using a newer version of the software stack (i.e. Python+OpenSSL).","Q_Score":0,"Tags":"python-3.x,sockets,ssl,openssl","A_Id":59333535,"CreationDate":"2019-12-14T08:17:00.000","Title":"Python SSL\/TLS - no shared cipher on Python 3.8 but working on Python 3.6","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"my question is if I use a vpn and try to send a message on a python socket message, not of any protocol, will my socket traffic go through the vpn and my ip be hidden from the message receiver? I found that I can't see the traffic using wireshark because it's not of any protocol. THANKS FOR THE HELP!","AnswerCount":1,"Available Count":1,"Score":0.3799489623,"is_accepted":false,"ViewCount":2273,"Q_Id":59339029,"Users Score":2,"Answer":"To establish a socket connection with a vpn is possible but...\nI tried the same project and got this result:\n\nTo establish a socket connection in the VPN server, you need to connect your device in which you execute your code to the vpn.\n\nType in ipconfig and get the vpn's IP address and bind a socket in Python using the IP.\n\nNow if you execute the project within the same pc the project works.\n\n\nBut if you use a different Wi-Fi connection in another pc it doesn't but why?\nHere comes port forwarding:\nThe vpn allocates a private IP address to your computer. A private IP address can be used to communicate with devices within your LAN (modem). But if you try to connect private IP address from other network, it would fail to communicate with your pc. So, to solve this public IP address is used. Each modem (even vpn servers) have a public IP address. It is like a tool for communication of devices across any network. For example online game servers runs using public IP address. This is the reason Windows firewall pops up when you enter an online game in Windows. But even if you type the public IP the socket doesn't work.\nHere comes port forwarding. Think, many devices are connected to your network or vpn server. The modem or vpn does not know which device should it transmit the message. If you port forward the private IP address and the exact port number. It now finally works!\n\nNOTE: Public IP's are sensitive and hackers can enter any VPN server using public IP address. So most VPN providers do not provide public IP. Only with the main VPN server you can port forward. So you need to contact the VPN service provider and request for a port forwarding to your private IP and finally program you socket and your project may work.\n\n\nNOTE: You need a TCP\/UDP protocol for a socket.","Q_Score":2,"Tags":"python,sockets,vpn","A_Id":64458547,"CreationDate":"2019-12-14T20:50:00.000","Title":"python socket and vpn","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I never did this before so I am sorry if my question is not very clear (I am not sure what the name of each step is). I have a code which is supposed to give me an interactive interface in a browser (similar to jupyter notebook I guess). After I run the code (I am using a remote machine, not my personal computer) one of the messages is: Running on http:\/\/127.0.0.1:5000\/ When I try to enter that in my browser I am getting a This site can\u2019t be reached error. Also the terminal window used to run this code isn't active anymore (it is similar to when I run a jupyter notebook). What should I do to get access to that interactive windows created? Thank you!","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":17,"Q_Id":59341519,"Users Score":0,"Answer":"http:\/\/127.0.0.1 is called loopback ip and is equivalent to localhost.\nThe remote machine is hosting the program on localhost, i.e., you cannot access it from other machine unless you host it on the machine's public ip. \nEasiest way to do that is to host on 0.0.0.0:5000. where 5000 is just a port and can be almost anything. Now that would probably print Running on http:\/\/0.0.0.0:5000\/ but you need to access it with the machine's public ip.","Q_Score":0,"Tags":"python,browser","A_Id":59342467,"CreationDate":"2019-12-15T05:48:00.000","Title":"Get the output of a code on a remote machine in a browser","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"My scripts don't work anymore and I can't figure it out.\nIt is a chrome version problem apparently... But I don't know how to switch to another version (not the latest?) Does exist another way?\nMy terminal indicates : \nTraceback (most recent call last):\nFile \"\/Users\/...\/Documents\/SCRIPTS\/PYTHON\/Scripts\/# -- coding: utf-8 --.py\", line 21, in \n driver = webdriver.Chrome()\nFile \"\/opt\/anaconda3\/lib\/python3.7\/site-packages\/selenium\/webdriver\/chrome\/webdriver.py\", line 81, in init\n desired_capabilities=desired_capabilities)\nFile \"\/opt\/anaconda3\/lib\/python3.7\/site-packages\/selenium\/webdriver\/remote\/webdriver.py\", line 157, in init\n self.start_session(capabilities, browser_profile)\nFile \"\/opt\/anaconda3\/lib\/python3.7\/site-packages\/selenium\/webdriver\/remote\/webdriver.py\", line 252, in start_session\n response = self.execute(Command.NEW_SESSION, parameters)\nFile \"\/opt\/anaconda3\/lib\/python3.7\/site-packages\/selenium\/webdriver\/remote\/webdriver.py\", line 321, in execute\n self.error_handler.check_response(response)\nFile \"\/opt\/anaconda3\/lib\/python3.7\/site-packages\/selenium\/webdriver\/remote\/errorhandler.py\", line 242, in check_response\n raise exception_class(message, screen, stacktrace)\nselenium.common.exceptions.SessionNotCreatedException: Message: session not created: Chrome version must be between 71 and 75\n(Driver info: chromedriver=2.46.628411 (3324f4c8be9ff2f70a05a30ebc72ffb013e1a71e),platform=Mac OS X 10.14.5 x86_64)\nAny idea?","AnswerCount":2,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":363,"Q_Id":59344813,"Users Score":0,"Answer":"This possibly happens, as your Chrome Browser or Chromium may be updated to newer versions automatically. But you still run your selenium scripts using the old version of the chromedriver.\nCheck the current version of your Google chrome or Chromium, then download the chromedriver for that specific version.\nThen your scripts may work fine!","Q_Score":0,"Tags":"python,google-chrome,selenium-webdriver,version","A_Id":63296566,"CreationDate":"2019-12-15T14:24:00.000","Title":"My python scripts using selenium don't work anymore. Chrome driver version problem","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I have been searching all over internet for solution but couldn't find any. I am using captcha solving api to solve captcha in scrapy-splash but when captcha page appears, scrapy sends request to api while trying to scrape next page. This is happening due to concurrency. I have tried CONCURRENT_REQUESTS=1 but same result. I have tried setting wait and pausing engine but in most cases, two or more threads arrive exactly at same line and so while first thread pause engine, 2 or more thread also pauses engine and tries to solve captcha. Can anyone please suggest anything to solve this issue.\nsorry if you didn't understand, Please comment any questions you have, I reply in seconds","AnswerCount":1,"Available Count":1,"Score":-0.1973753202,"is_accepted":false,"ViewCount":88,"Q_Id":59368079,"Users Score":-1,"Answer":"Use a blocking HTTP client library (e.g. Requests) for your API calls.","Q_Score":0,"Tags":"python,scrapy,scrapy-splash","A_Id":59377590,"CreationDate":"2019-12-17T05:27:00.000","Title":"I am having problem in scrapy concurrency with captcha","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I have made a bot with BotFather and got the access token. What I want is to add my bot to a number of groups(I will provide the invite link to it) and it should be able to join the group through an API. So, let's say I have a list of 100 public groups with their invite link, then my bot should be able to join them, this process should be automated. \nI tried searching the telegram API docs, but I wasn't able to find there. Also, the questions on StackOverflow mostly helpful when I want to add the bot to my own group, which I don't want. \nI just want a direction, maybe a function from telethon or any other client. I will then be able to do it by myself.","AnswerCount":2,"Available Count":2,"Score":0.0,"is_accepted":false,"ViewCount":206,"Q_Id":59442068,"Users Score":0,"Answer":"You can't add bot to the group without corresponding admin permission, also what you're trying to do looks like a spam and leads to account restriction(or full deletion).","Q_Score":0,"Tags":"python,python-3.x,telegram,telegram-bot,telethon","A_Id":59442105,"CreationDate":"2019-12-22T06:18:00.000","Title":"How to add a telegram bot to different groups with python?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I have made a bot with BotFather and got the access token. What I want is to add my bot to a number of groups(I will provide the invite link to it) and it should be able to join the group through an API. So, let's say I have a list of 100 public groups with their invite link, then my bot should be able to join them, this process should be automated. \nI tried searching the telegram API docs, but I wasn't able to find there. Also, the questions on StackOverflow mostly helpful when I want to add the bot to my own group, which I don't want. \nI just want a direction, maybe a function from telethon or any other client. I will then be able to do it by myself.","AnswerCount":2,"Available Count":2,"Score":0.0,"is_accepted":false,"ViewCount":206,"Q_Id":59442068,"Users Score":0,"Answer":"Only an admin of a group or channel can add the bot. The bot itself cannot add it anywhere. Even a user have to initiate the interaction with bot first.","Q_Score":0,"Tags":"python,python-3.x,telegram,telegram-bot,telethon","A_Id":59468608,"CreationDate":"2019-12-22T06:18:00.000","Title":"How to add a telegram bot to different groups with python?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I've made a telegram bot, that sends a request to a certain API and returns the answer to the bot.\nwhen I run the app locally, it takes like 2-3 seconds to execute a request and send the answer to the user, but when its webhook to pythonanywhere, it takes 5-10 minutes to execute a request.\nis it really that slow? or something is wrong?","AnswerCount":1,"Available Count":1,"Score":0.1973753202,"is_accepted":false,"ViewCount":80,"Q_Id":59458765,"Users Score":1,"Answer":"That really seems like something's wrong. Instrument your code with some timing information so you can see where it's spending it's time. Then you can start to try to work out why there's such a big difference.","Q_Score":0,"Tags":"python-3.x,telegram-bot,pythonanywhere,python-telegram-bot","A_Id":59481625,"CreationDate":"2019-12-23T16:51:00.000","Title":"pythonanywhere - SUPER SLOW?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I'm trying connect to internal Jira instance use my AWS lambda function. The lambda cannot connect to this web-resource because VPN is required. (all work properly on my local machine under VPN connection).\nPlease any suggestion regarding this issue ?","AnswerCount":2,"Available Count":2,"Score":0.0,"is_accepted":false,"ViewCount":166,"Q_Id":59468906,"Users Score":0,"Answer":"Since there is an AWS Direct Connect connection between an Amazon VPC and your corporate network, the AWS Lambda function can be configured to connect to the VPC.\nThe Lambda function can then connect to the JIRA instance on the corporate network by using the private IP address of the JIRA instance.\nYou might be able to configure the VPC to defer to your corporate DNS server to resolve the DNS Name into the IP address, but I would recommend you start by using the private IP address itself to ensure that connectivity is working.","Q_Score":0,"Tags":"python,aws-lambda,python-jira","A_Id":59478372,"CreationDate":"2019-12-24T12:31:00.000","Title":"How connect to internal Jira instance via AWS lambda","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I'm trying connect to internal Jira instance use my AWS lambda function. The lambda cannot connect to this web-resource because VPN is required. (all work properly on my local machine under VPN connection).\nPlease any suggestion regarding this issue ?","AnswerCount":2,"Available Count":2,"Score":0.0,"is_accepted":false,"ViewCount":166,"Q_Id":59468906,"Users Score":0,"Answer":"It would be best to use SQS messaging.\n\nDefine a message format\nFrom your lambda function, publish a message when you want to query something\nHave an app\/worker running inside your network, which is listening to your SQS queue.\nYour app\/worker receives that message and can query to your Jira instance.\nWhen the result is ready, your app\/worker can publish it on other SQS queue\nYou can receive the message in another lambda function, or If you want to receive the response in a same lambda function, you can poll for messages and wait till you get your desired message.\nIf you have multiple such lambda functions, or many requests going on. You can put a request-id in messages. So, when receiving a message, you can come to know which is your desired message and delete that one only.","Q_Score":0,"Tags":"python,aws-lambda,python-jira","A_Id":61316127,"CreationDate":"2019-12-24T12:31:00.000","Title":"How connect to internal Jira instance via AWS lambda","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"Blueprism gives the possibility to spy elements (like buttons and textboxes) in both web-browsers and windows applications. How can I spy (windows-based only) applications using Python, R, Java, C++, C# or other, anything but not Blueprism, preferrably opensource.\n\nFor web-browsers, I know how to do this, without being an expert. Using Python or R, for example, I can use Selenium or RSelenium, to spy elements of a website using different ways such as CSS selector, xpath, ID, Class Name, Tag, Text etc.\nBut for Applications, I have no clue. BluePrism has mainly two different App spying modes which are WIN32 and Active Accessibility. How can I do this type of spying and interacting with an application outside of Blueprism, preferrably using an opensource language?\n\n(only interested in windows-based apps for now)\nThe aim is of course to create robots able to navigate the apps as a human would do.","AnswerCount":3,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":497,"Q_Id":59470191,"Users Score":0,"Answer":"There is a free version of Blue Prism now :) Also Blue Prism uses win32, active accessibility and UI Automation which is a newer for of the older active accessibility.\nTo do this yourself without looking into Blue Prism you would need to know how to use UIA with C#\/VB.new or C++. There are libraries however given that Blue Prism now has a free version I would recommend using that. Anything specific can be developed withing a code stage within Blue Prism.","Q_Score":1,"Tags":"c#,python,windows,blueprism,rpa","A_Id":59556005,"CreationDate":"2019-12-24T14:27:00.000","Title":"Blueprism-like spying and bot development","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I am trying to run a test case using the command below\npipenv run py.test --cov=src --cov-fail-under=85 -vv test\/unit --debug\nIt complains as below\n<CallInfo when='collect' excinfo=<ExceptionInfo CollectError(\"ImportError while importing test module '\/Users\/omsingh0\/Applications\/aws\/awspipeline\/examples\/ecommerce...in <module>\\n from requests_aws4auth import AWS4Auth\\nE ModuleNotFoundError: No module named 'requests_aws4auth'\") tblen=7>>\nWhen I run pip list I can notice that requests-aws4auth is there\nPackage Version \n\ncertifi 2019.11.28\nchardet 3.0.4\nelasticsearch 7.1.0\nidna 2.8\npip 19.2.3\npipenv 2018.11.26\nrequests 2.22.0\nrequests-aws4auth 0.9\nsetuptools 39.0.1\nurllib3 1.25.7\nvirtualenv 16.7.9\nvirtualenv-clone 0.5.3 \nPlease let me know in case you have any suggestion.","AnswerCount":1,"Available Count":1,"Score":0.1973753202,"is_accepted":false,"ViewCount":1259,"Q_Id":59488208,"Users Score":1,"Answer":"l was able to run the below command and after, it was successful.\n$ pip install requests\n$ sudo pip install requests-aws4auth\nGive it a try","Q_Score":1,"Tags":"python,python-unittest","A_Id":64043149,"CreationDate":"2019-12-26T12:04:00.000","Title":"python unit test - unable to import requests_aws4auth","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I am writing a code to run a pi-based robot. It has several sensors and communicates with external computers over wifi as well. Some of the sensor and computer communication data is done over sockets (to other scripts) to keep things modular and simple (for me). As long as I stay at less than 10 sockets, will I come across any problems? I am mostly wondering if there is anything inherently wrong with this coding strategy.","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":69,"Q_Id":59490736,"Users Score":0,"Answer":"Q : As long as I stay at less than 10 sockets, will I come across any problems?\n\nMaybe yes, maybe not. The number of ZeroMQ Socket-instances does not matter in this, the configuration of the ZeroMQ Context()-instances and thereof of the underlying operating system's resources do matter.\n\nQ : anything inherently wrong with this coding strategy?\n\nNo, unless you do right things in a wrong way. Particularly \"this coding\" remains magically hidden as an unseen, so far, code, the less the MCVE-formulated code-example and the general answer for such a problem thus suffers from a principal undecidability, until an MCVE-code is let to run in the problem-specific eco-system, with known limits for the control-loops and the robot physical properties ( which is left to the kind readers to review ad-hoc ).\nZeroMQ will have Zero-problems in running on the RPi-platform, given due configuration does not suffocate the Context()-instance(s) by not providing adequate resources.","Q_Score":1,"Tags":"python,websocket,zeromq,pi,sensors","A_Id":59505155,"CreationDate":"2019-12-26T16:03:00.000","Title":"is there such a thing as too many python websockets?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I want to use youtube-dl in a python script, not using commands. I installed youtube-dl using pip install youtube-dl and I also tried installing it using pip install --upgrade youtube-dl and also pip3 install youtube-dl. It keeps saying I have it installed so I'm guessing the installation is fine.\nWhen I try to use import youtube_dl it gives me the error: unresolved import 'youtube_dl'Python(unresolved-import)\nPlease help!","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":1091,"Q_Id":59511510,"Users Score":1,"Answer":"This is a VS Code issue and not a youtube-dl issue. What you need to do is \n\nGo into the workspace folder (here workspaceRootFolder) and create a .env file\nIn this empty .env file add the line PYTHONPATH=codeFolder (replace codeFolder with your folder name)\nAdd \"python.envFile\": \"${workspaceFolder}\/.env\" to the settings.json\nRestart VS code","Q_Score":0,"Tags":"python,windows,visual-studio-code,pip,youtube-dl","A_Id":59511558,"CreationDate":"2019-12-28T13:53:00.000","Title":"Use Youtube-dl in python script using import youtube_dl","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":1,"System Administration and DevOps":0,"Web Development":0},{"Question":"I have made a Python program that webscrapes IMDB with Beautifulsoup to make a mySQL database with tables of all the top rated movies in the different categories. So far so good. My problem is that I am doing this from Norway, and many of the movie titles are translated to Norwegian. For example, in the top list of IMDB opened from a Norwegian IP adress, \"The Shawshank Redemption\" is translated to \"Frihetens Regn\". I want all the titles in English. Are there maybe some free VPNs that you can activate from Python and that works with Beautifulsoup? Or do anyone have another solution to this?","AnswerCount":2,"Available Count":1,"Score":0.1973753202,"is_accepted":false,"ViewCount":7622,"Q_Id":59511919,"Users Score":2,"Answer":"I agree that using proxies will work better rather than using a vpn. \nHowever, don't go with a free proxy, if you want results. If it's something you can invest in, get a decent paid provider, otherwise most likely nothing good will come out of this, as you will constantly get blocked.","Q_Score":2,"Tags":"python,web-scraping,beautifulsoup,ip,vpn","A_Id":59613310,"CreationDate":"2019-12-28T14:46:00.000","Title":"How to webscrape with VPN in Python?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"Why do we need socket despite of request library in python?\nIf we wanna socket to connect to other server so what is request library for?","AnswerCount":2,"Available Count":2,"Score":0.2913126125,"is_accepted":false,"ViewCount":43,"Q_Id":59518614,"Users Score":3,"Answer":"Request is a higher level API for handling HTTP requests (which uses socket internally). There are dozens of other network protocols not covered by it. Of course, you could handle HTTP by using socket directly, but unless you have an extremely good reason to do so, you'd just be reinventing the wheel.","Q_Score":1,"Tags":"python,socket.io,request","A_Id":59518647,"CreationDate":"2019-12-29T10:34:00.000","Title":"When do we need socket and when do we need request?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"Why do we need socket despite of request library in python?\nIf we wanna socket to connect to other server so what is request library for?","AnswerCount":2,"Available Count":2,"Score":0.1973753202,"is_accepted":false,"ViewCount":43,"Q_Id":59518614,"Users Score":2,"Answer":"Requests is a Python HTTP library, whereas sockets are used for sending or receiving data on a computer network. HTTP is an application layer protocol that specifies how request and replies from client and server should be made. In socket programming, you make connection by specifying destination IP\/Port and send your data to remote host.","Q_Score":1,"Tags":"python,socket.io,request","A_Id":59518810,"CreationDate":"2019-12-29T10:34:00.000","Title":"When do we need socket and when do we need request?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"how can I open the browser page without using chromedriver, for example, I want data to be saved in the main browser used by the user when executing the program, when data is not saved with the use of chromedriver.","AnswerCount":2,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":67,"Q_Id":59551371,"Users Score":0,"Answer":"Use profile,\nfrom selenium import webdriver\noptions = webdriver.ChromeOptions() \noptions.add_argument(\"user-data-dir=C:\\Path\") #Path to your chrome profile\nw = webdriver.Chrome(executable_path=\"C:\\Users\\chromedriver.exe\", chrome_options=options)","Q_Score":0,"Tags":"python-3.x,selenium","A_Id":59554716,"CreationDate":"2020-01-01T09:37:00.000","Title":"another way to open the browser using the selenium module Python","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I want to upload files to Google Drive, but the < input type = \"file\"> tag does not exist in HTML. File selection window opens How can I select the file I want in this section?","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":75,"Q_Id":59553820,"Users Score":0,"Answer":"Pyautogui like so,\nhover over your position You want to click\nget coordinates\nImport pyautogui\nPyautogui.position()\nUse those coordinates to click\nPyautogui.click(x,y)","Q_Score":1,"Tags":"python,selenium","A_Id":59554690,"CreationDate":"2020-01-01T15:43:00.000","Title":"How to upload selected file with Selenium in Python?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"Has anyone had this problem before? I get the below error message when I try to run a Python script in Spyder IDE while using NVIDIA Nsight Systems.\n\nCommError: Can't make a request to a closed comm","AnswerCount":4,"Available Count":4,"Score":0.049958375,"is_accepted":false,"ViewCount":4854,"Q_Id":59573606,"Users Score":1,"Answer":"Restart the spyder app and run the program. This worked out for me","Q_Score":3,"Tags":"python,anaconda,nsight","A_Id":59979493,"CreationDate":"2020-01-03T05:28:00.000","Title":"Can't make a request to closed comm Python Spyder IDE error","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"Has anyone had this problem before? I get the below error message when I try to run a Python script in Spyder IDE while using NVIDIA Nsight Systems.\n\nCommError: Can't make a request to a closed comm","AnswerCount":4,"Available Count":4,"Score":0.0996679946,"is_accepted":false,"ViewCount":4854,"Q_Id":59573606,"Users Score":2,"Answer":"Does this error occurs when trying to generate plots? If so, I was also in such a situation and this works for me:\n\nPreference > IPython console > Graphic: Check \"Graphic Backend\" is set to Automatic\nView > Panes > Plots: Check that \"Mute inline plotting\" is checked and view your plots using this pane afterward.","Q_Score":3,"Tags":"python,anaconda,nsight","A_Id":59754708,"CreationDate":"2020-01-03T05:28:00.000","Title":"Can't make a request to closed comm Python Spyder IDE error","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"Has anyone had this problem before? I get the below error message when I try to run a Python script in Spyder IDE while using NVIDIA Nsight Systems.\n\nCommError: Can't make a request to a closed comm","AnswerCount":4,"Available Count":4,"Score":0.049958375,"is_accepted":false,"ViewCount":4854,"Q_Id":59573606,"Users Score":1,"Answer":"Yep. Same here with Spyder 4.0. Downgrading to 3.3.6 resolves.","Q_Score":3,"Tags":"python,anaconda,nsight","A_Id":59711341,"CreationDate":"2020-01-03T05:28:00.000","Title":"Can't make a request to closed comm Python Spyder IDE error","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"Has anyone had this problem before? I get the below error message when I try to run a Python script in Spyder IDE while using NVIDIA Nsight Systems.\n\nCommError: Can't make a request to a closed comm","AnswerCount":4,"Available Count":4,"Score":0.1488850336,"is_accepted":false,"ViewCount":4854,"Q_Id":59573606,"Users Score":3,"Answer":"Apparently the restart kernel functionality doesn't work properly. You can stop that kernel and open up a new console, this triggers the init of a new kernel without restart the whole program (Spyder 4)","Q_Score":3,"Tags":"python,anaconda,nsight","A_Id":64241246,"CreationDate":"2020-01-03T05:28:00.000","Title":"Can't make a request to closed comm Python Spyder IDE error","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"This is not a programming related question, but I've used python requests a lot of times. \nThe headers I usually use are:\n\naccept-encoding \norigin \nreferrer \nuser-agent \naccept-language \naccept \ncookie \n\nWhile examining the requests sent to a server when performing a task such as logging in, you notice headers like:\n\nHost\ncontent-length\nTE\n\nIf I decided to add these headers to my requests, would it improve my chance of being recognized as a human since it tells the server information about the \"browser\".","AnswerCount":1,"Available Count":1,"Score":0.3799489623,"is_accepted":false,"ViewCount":44,"Q_Id":59574451,"Users Score":2,"Answer":"Not really.\nHost is a required header in HTTP v.1.1 protocol, so it is sent anyway. The requests module sets this header implicitly. You should not set it manually and in 99 cases of 100 you should never care about this header.\nAlthough Content-Length is an optional header, it is being used almost all the time, because it is the easiest way to tell the remote server where HTTP message actually ends. The most popular alternative is Transfer-Encoding: chunked. Any of these two headers do not disclose any information about the client, so you should not worry about them.\nFinally, by using TE header client tells the remote server the desired Transfer-Encoding format. Again, this header just negotiates data format between client and server, it will hardly improve your chances to be recognized as a human.","Q_Score":2,"Tags":"python,http,web-scraping,python-requests,http-headers","A_Id":59579490,"CreationDate":"2020-01-03T07:02:00.000","Title":"do more headers mean better chance the server thinks you're a human","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I'm using Selenium with Python API and Chrome to do the followings:\n\nCollect the Performance Log;\nClick some <a, target='_blank'> tags to get into other pages;\n\nFor example, I click a href in Page 'A', which commands the browser opens a new window to load another URL 'B'.\nBut when I use driver.get_log('performance') to get the performance log, I can only get the log of Page 'A'. Even though I switch to the window of 'B' as soon as I click the href, some log entries of the page 'B' will be lost.\nSo how can I get the whole performance log of another page without setting the target of <a> to '_top'?","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":180,"Q_Id":59612799,"Users Score":0,"Answer":"I had the same problem and I think it is because the driver does not immediately switch to a new window.\nI switched to page \"B\" and reloaded this page, then uses get_log and it worked.","Q_Score":1,"Tags":"python,selenium,logging","A_Id":66257822,"CreationDate":"2020-01-06T13:14:00.000","Title":"How to get the Performance Log of another tab from Selenium using Python?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"How to restream\\redirect incoming rtmp stream to several other services with python? Is it even possible with python?\nI know that there is a nginx solution, but I need to change links for restream without restarting, so I want to use Python.","AnswerCount":1,"Available Count":1,"Score":0.1973753202,"is_accepted":false,"ViewCount":250,"Q_Id":59648916,"Users Score":1,"Answer":"Yes, or corse it possible in python (or any other language for that matter) but it\u2019s much harder. \nUse nginx. Nginx can call your python code with the \u201cexec\u201d options. The. You can then spin up ffmpeg instances in python to restream wherever you want.","Q_Score":0,"Tags":"python,rtmp","A_Id":59650185,"CreationDate":"2020-01-08T15:22:00.000","Title":"How to restream rtmp with python?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I am trying to clone my device data and stream the device data to another IoThub in an entirely different organization, the organization here I mean a different vendor cloud. \nTrial:\nI have tried routing the messages to service bus topics from my IoThub and then written a container using the service bus SDK which can subscribe to these topics and push the data to Azure IoT hub in a different vendor cloud. \nASK: \nIs there any other best way for sending the device stream to another vendor Azure IoThub other than the above using Azure alone. \nI can directly stream the device data from the data source, which is clearly not the use case.\nUsing IoT-device, not IoT-Edge.","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":62,"Q_Id":59655683,"Users Score":0,"Answer":"What you want to achieve is not possible out of the box. You need a processor in the middle that picks up messages from IoT Hub A - for instance on its built-in Event Hub endpoint. Then the processor would need to impersonate each device for which it got a message from A, and send the message to IoT Hub B, using the device credentials for the respective device.\nYou could for example build this with an Azure Function. But you need to build all this yourself.","Q_Score":0,"Tags":"azure-iot-hub,azure-eventhub,azure-servicebus-queues,azure-servicebus-topics,azure-sdk-python","A_Id":59682644,"CreationDate":"2020-01-09T00:31:00.000","Title":"Streaming device data from IoThub in one organization(Org\/Client A) to IoThub in entirely different organization(Org\/Client B)","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":1,"Web Development":0},{"Question":"I had launched an EC2 with free trial (t2-micro)(WINDOWS SERVER 2019) in ap-south-2(Mumbai) . I am running a discord bot in python script. Running that script on the Mumbai server, I faced no problem.\nBut then due to ping issues i changed my region to us-west-1(N.California). But while running python file on California server (and any other at this point) I am having this problem with certification.\naiohttp.client_exceptions.ClientConnectorCertificateError: Cannot connect to host discordapp.com:443 ssl:True [SSLCertVerificationError: (1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1108)')]\nAny help will be greatly appreciated","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":1498,"Q_Id":59658986,"Users Score":1,"Answer":"So I recently faced the same problem. I found my way around that by terminating all the EC2 instances. Then giving it a rest for about 1-2 hours then making a new instance.\nAnd then by running code in that, I didn't face the problem. I don't know how it fixes the problem but I hope this helps.","Q_Score":3,"Tags":"python-3.x,amazon-web-services,amazon-ec2,discord.py","A_Id":59659016,"CreationDate":"2020-01-09T07:19:00.000","Title":"ClientConnectorCertificateError: Cannot connect to host discordapp.com:443, Certification error on AWS.(ec2)","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I am writing a telegram to the bot. I ran into such a problem. I need the bot to send a message (text) when clicked on which it was copied (as a token from @BotFather)","AnswerCount":2,"Available Count":1,"Score":0.0996679946,"is_accepted":false,"ViewCount":13652,"Q_Id":59713920,"Users Score":1,"Answer":"You can simply by editing the text in following manner\nWrite ``` then write your text then again write that 3 character .\nBingo!!!","Q_Score":4,"Tags":"python,bots,telegram,telegram-bot,py-telegram-bot-api","A_Id":65958439,"CreationDate":"2020-01-13T09:40:00.000","Title":"How to make that when you click on the text it was copied pytelegrambotapi","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"Is there anyway to do a one to one comparison between Pardot and Salesforce. I am aware that these two are synced, but I do not find the connected data very reliable. I am more interested in writing in a code in python by using Salesforce and Pardot API and do one to one comparison. Has anyone come across such situation?","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":49,"Q_Id":59716020,"Users Score":0,"Answer":"Yes, it is possible to write such code. You would need to be conscious of the limits both systems impost (governor limits of Salesforce, Pardot API record limits).\nAnother option for you is to check the Pardot field sync options for the mapping to see what is going on. If these are not set as you like, even the best script would not yield the results you are looking for and the data would just be overwritten.","Q_Score":0,"Tags":"python,api,salesforce,pardot","A_Id":60611151,"CreationDate":"2020-01-13T11:44:00.000","Title":"Comparing data in Salesforce and Pardot","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I want to set socket.TCP_KEEPIDLE of a Pyro4 object for keeping heartbeat. Is there any way to do this? Or workaround is also welcome.\nThanks in advance.","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":58,"Q_Id":59729555,"Users Score":0,"Answer":"If you really want, you can get to the actual socket object in the Proxy via proxy._pyroConnection.sock but then you're using an internal API, so no guarantees that your code keeps working.","Q_Score":0,"Tags":"python,sockets,pyro,pyro4","A_Id":61733366,"CreationDate":"2020-01-14T07:58:00.000","Title":"How to get\/config socket from Pyro Proxy","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I am trying to create a website in Node.js. Though, as I am taking a course on how to use Artificial Intelligence and would like to implement such into my program. Therefore, I was wondering if it was feasible to connect Python Spyder to a Node.js based web application with somewhat ease.","AnswerCount":2,"Available Count":1,"Score":0.0996679946,"is_accepted":false,"ViewCount":1193,"Q_Id":59738972,"Users Score":1,"Answer":"Yes. That is possible. There are a few ways you can do this. You can use the child_process library, as mentioned above. Or, you can have a Python API that takes care of the AI stuff, which your Node app communicates with. \nThe latter example is what I prefer as most my projects run on containers as micro services on Kubernates.","Q_Score":1,"Tags":"python,node.js,artificial-intelligence,spyder","A_Id":59740436,"CreationDate":"2020-01-14T17:32:00.000","Title":"Can I use Node.js for the back end and Python for the AI calculations?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":1,"System Administration and DevOps":0,"Web Development":1},{"Question":"I have a django project and external resource. The external resource has XML data which changes time to time but it still static if nothing happened. I need to collect new data if something happened.\nThe ways I see:\n1. Grab content one time in some period and check if it is different.\n2. Save content as file and check length (worse way).\nUnfortunately, I can't predict when content will change also I can't use WebSockets, Long Polling and WebHooks.\nPlease, describe me the ways that in modern WWW today may solve my problem!","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":18,"Q_Id":59791391,"Users Score":0,"Answer":"If you can add some hook, when external resource updated, you can use it.\nBut if it is impossible - on way to do it is create periodical task for checking external resources (Last-Modified header for example).","Q_Score":0,"Tags":"django,python-3.x,http,https","A_Id":59791479,"CreationDate":"2020-01-17T16:32:00.000","Title":"Whats the best way to add listener to external resource?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I made a python script into an executable file by turning it into a .command file and adding the #!\/usr\/bin\/env python3 on the first line of the file. \nWhen I double click to run the file, the file does run on the terminal, however the installed modules do not seem to be linked to the file. \nimport requests\nModuleNotFoundError: No module named 'requests'\nIs there a way to use external module such as the 'requests' module when running an executable form of a python script?","AnswerCount":2,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":118,"Q_Id":59796078,"Users Score":0,"Answer":"By \".\" you mean \".\/\"?\nHow to you normaly run it?","Q_Score":0,"Tags":"python,python-3.x,python-requests,command,executable","A_Id":59796166,"CreationDate":"2020-01-17T23:29:00.000","Title":"How to use external python modules such as the module 'requests' when running executable python scripts such as .command files?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I am building a python script that is trying to stream a screen capture to my Facebook Page and to be able to retrieve all the comments from the Facebook Live stream real time so that I can do some processing in the middle of the stream.\nThe Facebook App was set up (in development mode) but when I tried to retrieve the comments from my live stream, I am only able to retrieve comments with their name and id (\"from\") that are made as the Facebook Page Admin, not comments that are made by other users. I need the user's id, user's name and their comments.\nI understand that I need to get Facebook App to be live mode in order to retrieve all the comments with their details tagged to it. When I tried to get it, it tells me that I need to get the permission approved. I tried to fill in most of the stuff and try to get the two permission (manage_page for the comments and live video API for the streaming) but I was unable to because I left the platform empty.\nBelow is the message I got:\n\nYou do not have any platforms eligible for review. Please configure a platform on your Settings page.\n\nThe problem is when I tried to choose a platform that was shown in the list, python script does not fall in the list of platform. \nDoes anyone know of a solution or a different way to achieve what I need to retrieve?","AnswerCount":2,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":1082,"Q_Id":59810273,"Users Score":0,"Answer":"In order to get the \"from\" field when retrieving the comments, you need to have manage_pages permission from your Facebook App that is linked to your Facebook Page. You will need to submit an App review for your Facebook App that usually takes 1-3 days to process. If you are lucky, it will probably take about 6-8 hours. \nOnce it is approved, you can request the permission and get your application to go live.\nAlso use the Page Access token in your \"access_token\" field when invoking the API so that it will allow you to pull the \"from\" field, which contains the id and name of the user.","Q_Score":5,"Tags":"python,facebook,facebook-graph-api","A_Id":61137300,"CreationDate":"2020-01-19T13:05:00.000","Title":"Retrieving Facebook Page comments with python script","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I would like to create a tool with Python and the Twitter API to be able to create lists of tweets that match certain criteria like \"contains the word Python\" or \"has at least 2 likes\". Or simple stats like top posters, most liked, etc.\nAll of my search pointed me to the Tweepy project. But for that I need 0Auth tokens. So I applied for a developer account and was denied with the comment \"we are unable to serve your use case\".\nDo I have any alternatives?","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":158,"Q_Id":59811949,"Users Score":0,"Answer":"Well, as a general answer for these situations, you can always use a web-based automation tool, which is basically a library that interacts with the remote feature of the browsers and replicates what would be you \"opening the website, logging in, etc\" and can subsequently parse all data from the rendered elements.\nTry looking at selenium, i've used that library in the past to raw scrap facebook and it worked flawless.\nEdit: Note that this isn't a twitter specific library, you will have to find the html tags in the login website and use them to log in, same for parsing data, etc.","Q_Score":3,"Tags":"python,twitter","A_Id":70450120,"CreationDate":"2020-01-19T16:16:00.000","Title":"using Python and the Twitter API without authentication tokens","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I want python to click on a key on my keyboard for example the arrow down key when a specific word for example google is present somewhere in the browser or in the searh bar. Is it possible with selenium or the os module. Any suggestions?","AnswerCount":2,"Available Count":1,"Score":0.0996679946,"is_accepted":false,"ViewCount":3289,"Q_Id":59815209,"Users Score":1,"Answer":"you could search for the element with an xpath looking for the text you are searching e.g. $x('\/\/*[.=\"Text\"]') and then use sendKey() to press the key","Q_Score":0,"Tags":"python,selenium,automation,keyboard,arrow-keys","A_Id":59817029,"CreationDate":"2020-01-19T23:05:00.000","Title":"How to simulate pressing the arrow down key when a specific element is present in the HTML using Selenium and Python","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I am doing a project where I need to send device parameters to the server. I will be using Rasberry Pi for that and flask framework. \n1. I want to know is there any limitation of HTTPS POST requests per second. Also, I will be using PythonAnywhere for server-side and their SQL database. \nInitially, my objective was to send data over the HTTPS channel when the device is in sleep mode. But when the device (ex: car) wakes up I wanted to upgrade the HTTPS to WebSocket and transmit data in realtime. Later came to know PythonAnywhere doesn't support WebSocket.\nApart from answering the first question, can anyone put some light on the second part? I can just increase the number of HTTPS requests when the device is awake (ex: 1 per 60 min in sleep mode and 6 per 60sec when awake), but it will be unnecessary data consumption over the wake period for transmission of the overhead. It will be a consistent channel during the wake period.","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":569,"Q_Id":59822489,"Users Score":2,"Answer":"PythonAnywhere developer here: from the server side, if you're running on our platform, there's no hard limit on the number of requests you can handle beyond the amount of time your Flask server takes to process each request. In a free account you would have one worker process handling all of the requests, each one in turn, so if it takes (say) 0.2 seconds to handle a request, your theoretical maximum throughput would be five requests a second. A paid \"Hacker\" plan would have two worker processes, and they would both be handling requests, to that would get you up to ten a second. And you could customize a paid plan and get more worker processes to increase that.\nI don't know whether there would be any limits on the RPi side; perhaps someone else will be able to help with that.","Q_Score":1,"Tags":"python-3.x,http,flask,websocket,pythonanywhere","A_Id":59829441,"CreationDate":"2020-01-20T11:38:00.000","Title":"Limit of HTTPS request per seconds","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I want to develop a system for trace and debugging an external device via COM port.\nMain service will be developed using python to receive, analyse & store logs data. \nWe decided to stream log data to web browser with gRPC protocol and draw live charts.\nHighest rate if data is 50K of signals per second and maximum size of every signal is just 10 bytes.\nSystem will be used in local network or same PC so we do not have bandwidth limits.\nWe want to make sure the web-grpc platform can cover this rate per second.\nThanks for your recommendations.","AnswerCount":1,"Available Count":1,"Score":0.1973753202,"is_accepted":false,"ViewCount":93,"Q_Id":59849029,"Users Score":1,"Answer":"The throughput limit is mostly decided by the browser and the protobuf overhead. Since the latter is application specific, you should do a benchmark with real data on your preferred browsers.","Q_Score":0,"Tags":"benchmarking,grpc,rate,grpc-python","A_Id":59872414,"CreationDate":"2020-01-21T20:50:00.000","Title":"web-gRPC Performance Rate per second","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I have a web page with data in different tables. I want to extract a particular table and compare with an excel sheet and see whether there are any differences. Note the web page is in a internal domain. I tried with requests and beautifulsoup but I got 401 error. Could anyone help how I can achieve this?","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":112,"Q_Id":59871748,"Users Score":0,"Answer":"401 is an Unauthorized Error - which suggests your username and password may be getting rejected, or their format not accepted. Review your credentials and the exact format \/ data names expected by the page to ensure you're correctly trying to connect.","Q_Score":1,"Tags":"python,web-scraping,beautifulsoup","A_Id":59872815,"CreationDate":"2020-01-23T04:51:00.000","Title":"Scrape and compare and Web page data","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I am trying to create a multi-thread programming using python selenium.In my program i want to open 50+ different urls simultaneously, so my code is opening each url in different browsers.Whenever i ran my program my cpu utilization is reaching 100%.So sometime some of the urls are not opening.\nSo My question is if i open multiple tabs within single browser is it reduce my cpu utilization ?\nIf yes, please help me to open multiple tabs in single browser using python selenium.","AnswerCount":2,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":1093,"Q_Id":59873157,"Users Score":0,"Answer":"Open Multiple Tabs in one browser will speed down your program, As webdriver will work only in one tab and then will move to the second tab. can't work simultaneously. its same as to reduce the number of URLs.","Q_Score":1,"Tags":"python,multithreading,selenium,cpu-usage","A_Id":59873602,"CreationDate":"2020-01-23T07:05:00.000","Title":"opening multiple chrome browser and multiple tabs in single browser is same?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I'm building a dashboard in Data Studio and want to add a 'button' like feature that basically calls\/runs a python function that downloads some content locally(on the client machine). I have the python function almost ready. \nThe question I have is where do I host this function so that it is callable from Data Studio? \nIdeally, I wanted to create a Cloud function that would host the python function and get triggered when the button is clicked. This would work till this point but, will not download content locally. What options do I have to accomplish this?","AnswerCount":2,"Available Count":2,"Score":0.0,"is_accepted":false,"ViewCount":526,"Q_Id":59884775,"Users Score":0,"Answer":"Currently it is not possible to trigger Cloud Function or something similar from the Data Studio Front end, especially when you are intending to download content locally.\nThe only thing I can think of is, you can create a custom connector which will call the Cloud Function (via URL trigger). Then use the connector to create a data source and attach that data source to a table or chart. That way, every time that page (with the table\/chart) is refreshed, the connector will call the Cloud Function and retrieve the associated data.","Q_Score":1,"Tags":"python,google-cloud-platform,google-cloud-functions,cloud,google-data-studio","A_Id":59889328,"CreationDate":"2020-01-23T18:12:00.000","Title":"How do I call a python function from a dashboard?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I'm building a dashboard in Data Studio and want to add a 'button' like feature that basically calls\/runs a python function that downloads some content locally(on the client machine). I have the python function almost ready. \nThe question I have is where do I host this function so that it is callable from Data Studio? \nIdeally, I wanted to create a Cloud function that would host the python function and get triggered when the button is clicked. This would work till this point but, will not download content locally. What options do I have to accomplish this?","AnswerCount":2,"Available Count":2,"Score":0.0,"is_accepted":false,"ViewCount":526,"Q_Id":59884775,"Users Score":0,"Answer":"A function can be triggered by HTTP, so you can make it publicly available and just drop a link to it in a dashboard.\nJust add a gray square around this link and make it look like a button. You may want to check the box to open the URL in a new tab, so your dashboard is not closed when the link is clicked.","Q_Score":1,"Tags":"python,google-cloud-platform,google-cloud-functions,cloud,google-data-studio","A_Id":67154973,"CreationDate":"2020-01-23T18:12:00.000","Title":"How do I call a python function from a dashboard?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I have set up the entire environment to run mininet using an Ubuntu image on VirtualBox. I also found a JSON graph topology which I run with Python to simulate it in mininet (I use PuTTy for remote access from Windows..)\nI read before that using the command sudo mn --controller=none \ncreates a non-SDN simulation. This is not bad for a start, but is there another way of doing that? And if so, how is it possible to remove a specific node and replace it with an SDN switch to test the network performance?\nMany thanks in advance.\nSet up used:\nWindows 10, Oracle VM Virtual Machine, Ubuntu, Mininet, NetworkX, Python, WinTCP..PuTTy..","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":182,"Q_Id":59917211,"Users Score":0,"Answer":"Yes, it is possible. You start a normal SDN network with the desired topology, and you:\n\nset the forwarding-rules on the 'non' SDN-nodes to live forever (so the controller is never asked for new\/expired rules), and\nleave the rest of the node under the control of the controller.","Q_Score":0,"Tags":"python,networking,hybrid,sdn,mininet","A_Id":61058427,"CreationDate":"2020-01-26T09:53:00.000","Title":"Is there a way to create a traditional network with mininet then replace some of its nodes with SDN?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":1,"Web Development":0},{"Question":"Let suppose I Have a webhook that will request to my python server and I want to allow that request to process without applying any middleware authorization of python.\nI don't want to make more costly process of request\/response by processing all the requests in my custom middleware.","AnswerCount":1,"Available Count":1,"Score":0.1973753202,"is_accepted":false,"ViewCount":43,"Q_Id":59953583,"Users Score":1,"Answer":"You could create a separate settings file for a different web server instance where it would use that settings file. You'd also likely need to create a separate urls.py so that you don't grant access to other views.","Q_Score":0,"Tags":"python,django","A_Id":59953775,"CreationDate":"2020-01-28T16:56:00.000","Title":"how to exclude specific api from applying all the authorization of python middleware","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"So I've been racking my brain trying to implement a system in which computers on a network (where there are always three or more computers on the network) are able to asynchronously communicate with each other by sending each other data.\nSo far, all I've been able to find as far as solutions go is sockets--which, to my knowledge, requires a client and a server script. My first problem is that I'd like to remove any client or server roles since all of the computers on the network are decentralized and running the same script concurrently without a server. Secondly, all of the computers are sending other nodes (chosen at random) sensor data from a specific point in time. If, for example, I have 4 computers on the network and--since they're all running the same script--they decide to send their data to another computer at the same time, wouldn't that cause a wait lock since all of the nodes are trying to communicate with another computer, but those computers are unable to accept the connection because they're also trying to send data?\nI've considered using multithreading to run my begin_sync and wait_sync functions concurrently, but I'm not sure whether or not that would work. Does anyone have any suggestions or ideas for solutions that I could look into?\nThanks for your time!","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":106,"Q_Id":59954817,"Users Score":0,"Answer":"As per NotTheBatman's response, I was able to get this to work using sockets on multiple ports. As far as how I handled being able to wait for sensor data and query other nodes, I simply used multithreading with great success.","Q_Score":0,"Tags":"python,network-programming,gossip","A_Id":60660334,"CreationDate":"2020-01-28T18:20:00.000","Title":"Is decentralized communication between 3+ computers in a network possible in python?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"Recently I've been trying to use RGB Colours, i.e.\n166, 245, 255.\nHowever I'm stuck in here:\ndiscord.Colour()\nwhere I can only input ONE value in the brackets. How can I solve it?","AnswerCount":2,"Available Count":1,"Score":0.1973753202,"is_accepted":false,"ViewCount":1813,"Q_Id":59961455,"Users Score":2,"Answer":"For 0-255 RGB range: discord.Colour.from_rgb(r, g, b)\nFor hexa: discord.Colour(0xff55ff)","Q_Score":1,"Tags":"python,colors,discord.py","A_Id":59962161,"CreationDate":"2020-01-29T06:18:00.000","Title":"How can I use RGB codes in discord.Colour in Python?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I am building a package that uses the Google Analytics API for Python.\nBut, in severous cases when I have multiple dimensions the extraction by day is sampled.\nI know that if I use sampling_level = LARGE will use a sample more accurate.\nBut, somebody knows if has a way to reduce a request that you can extract one day without sampling?\nGrateful","AnswerCount":3,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":840,"Q_Id":59969386,"Users Score":0,"Answer":"setting sampling to LARGE is the only method we have to decide the amount of sampling but as you already know this doesn't prevent it.\nThe only way to reduce the chances of sampling is to request less data. A reduced number of dimensions and metrics as well as a shorter date range are the best ways to ensure that you dont get sampled data","Q_Score":0,"Tags":"python-3.x,google-analytics,google-analytics-api","A_Id":59969860,"CreationDate":"2020-01-29T14:33:00.000","Title":"How to get unsampled data from Google Analytics API in a specific day","Data Science and Machine Learning":1,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"When i load data, these errors show up:\n....\nFile \"\/Users\/Georg\/PycharmProjects\/990-xml-database\/venv\/lib\/python3.6\/site-packages\/urllib3\/contrib\/pyopenssl.py\", line 318, in recv_into\nraise SocketError(str(e)) \nOSError: (54, 'ECONNRESET')\nDuring handling of the above exception, another exception occurred:\n...\nraise SocketError(str(e))\nurllib3.exceptions.ProtocolError: ('Connection aborted.', OSError(\"(54, 'ECONNRESET')\",))\nDuring handling of the above exception, another exception occurred:\n...\nraise ConnectionError(err, request=request)\nrequests.exceptions.ConnectionError: ('Connection aborted.', OSError(\"(54, 'ECONNRESET')\",))\nAny idea what's wrong \/ how I can solve this?","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":912,"Q_Id":59994700,"Users Score":0,"Answer":"I solved it adding import urllib3.contrib.pyopenssl in the code","Q_Score":0,"Tags":"python,download,pycharm,connection","A_Id":59996007,"CreationDate":"2020-01-30T21:21:00.000","Title":"OSError 54 - when loading data","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"new user of mitmproxy here. I've figured out how to edit a request and replay it, and I'm wondering how to undo my edit.\nMore specifically, I go to a request's flow, hit 'e', then '8' to edit the request headers. Then I press 'd' to delete one of the headers. What do I press to undo this change? 'u' doesn't work.","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":57,"Q_Id":60031716,"Users Score":0,"Answer":"It's possible to revoke changes to a flow, but not while editing. In your case, 'e' -> '8' -> 'd' headers, now press 'q' to go back to the flow -> press 'V' to revoke changes to the flow.","Q_Score":0,"Tags":"python,mitmproxy","A_Id":60541411,"CreationDate":"2020-02-02T23:55:00.000","Title":"mitmproxy: shortcut for undoing edit","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I am using python's urllib.parse.urlencode to encode a request body to make a 'get' request for a url. I am running into trouble when encoding a UTC datetime string. \nThe endpoints expects the date to look like: \nbegin=2004-01-01T00:00:00&end=2019-04-21T00:00:00\nbut after using urlencode, the date payload looks like: \nbegin=2020-02-04T17%3A00%3A00&end=2020-02-04T20%3A00%3A00\nThe format of the first half of the UTC datetime looks okay, (YYYY-MM-DDT), but second half is screwed up. It SHOULD look like T00:00:00 but does not. \nI am passing a dictionary of parameters into the urlencode function:\nparams = {\n 'begin':'2020-02-04T17:00:00',\n 'end':'2020-02-04T20:00:00',\n}\nreturn urlencode(params, doseq=True)\n\nHow can I encode the date in the correct format? Please let me know if there is anything else I can provide to help you all. Thank you.","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":3417,"Q_Id":60048889,"Users Score":3,"Answer":"%3A is : when urlencoded.\nHence begin=2020-02-04T17:00:00&end=2020-02-04T20:00:00 when encoded will be begin=2020-02-04T17%3A00%3A00&end=2020-02-04T20%3A00%3A00, as stated in your question.\nThe format is correct and there is nothing wrong with your code.","Q_Score":1,"Tags":"python-3.x,http,datetime,python-requests,urllib","A_Id":60296370,"CreationDate":"2020-02-04T00:15:00.000","Title":"How to urlencode UTC datetime strings?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":1,"System Administration and DevOps":0,"Web Development":0},{"Question":"I am currently working on obtaining data from a nested JSON response called \"Result\"\nNow after reviewing the API documentation, they say that they only return 100 records per request, so which means if we have 425 records I would have to pass the request. get at least 4 times with:\n\n\/example\n\/example?$skip=100 \n\/example?$skip=200\n\/example?$skip=400\n\nAfter that is done it should write the response list in a csv file.I have parsed the response from the get to json.loads, I have converted the dictionary to list and created a for loop that writes whatever is in the \"Result\" dictionary. \nMy question is how can I create that it loops also the request.get and increments the url value to skip 100,200,300,400. Hope this makes sense","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":326,"Q_Id":60062887,"Users Score":0,"Answer":"So after searching and searching the best way that worked for me was.\n\nCreating a for loop with the number of times that it needs to loop over.\ntoSkip = (i+1) * 100\nConcatenate the string with 'url string' + '?$Skip=' + str(toSkip)\nCreate a request with passing the authorization header\nParse it with json.loads\nWrite the result to a csv file or google sheets API","Q_Score":0,"Tags":"python-3.x,api","A_Id":60080377,"CreationDate":"2020-02-04T17:44:00.000","Title":"Python - API multiple responses","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":1,"System Administration and DevOps":0,"Web Development":0},{"Question":"I can't find any example on how to make a convertcurrency request using the paypal API in python, can you give me some examples for this simple request?","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":65,"Q_Id":60066232,"Users Score":0,"Answer":"Is this an existing integration for which you have an Adaptive APP ID? If not, the Adaptive Payments APIs are very old and deprecated, so you would not have permissions to use this, regardless of whether you can find ready-made code samples for Python.","Q_Score":1,"Tags":"python,paypal,request,paypal-adaptive-payments","A_Id":60066533,"CreationDate":"2020-02-04T22:07:00.000","Title":"PayPal Adaptive Payments ConvertCurrency Request (deprecated API) in Python","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"72.0.2 (64-bit) on MacOS 10.14.6 (18G3020)\nRan a python selenium script for automation and after that FF still shows the orange stripes bar and an icon which says \"Browser is under remote control\"\nRebooted machine but no luck.\nIs there way to take out FF in normal mode now?\n(Not able to access saved logins in this mode)\nThanks!","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":181,"Q_Id":60067833,"Users Score":0,"Answer":"This did the trick.\n\nSetup FireFox sync account\nCreate a new empty profile and make it default\nRestart FF with that profile\nLogin to FF sync and get back bookmarks\/saved logins etc.\n\nNow FF is auto-filling passwords again.\nStrange thing is that address bar still shows slanted orange\/light orange stripes and \"Browser is under remote control\" but works like regular browser.","Q_Score":0,"Tags":"python,macos,selenium,firefox","A_Id":60314201,"CreationDate":"2020-02-05T01:28:00.000","Title":"FireFox 72.0.2 on Mac Stuck in \"Remote Control\" mode","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I am trying to automate a situation wherein I need to close internet connection of device and then resume it again. Can anyone help me out?","AnswerCount":2,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":41,"Q_Id":60072604,"Users Score":0,"Answer":"If it operating system is ubuntu :\n1. Launch terminal using 'os' module in python and shut down networking interface\n command : os.system(\"gnome-terminal -e 'sudo ifconfig eth0 down'\")\n2. Do your required task\n3. Again, make network interface up and running\n command : os.system(\"gnome-terminal -e 'sudo ifconfig eth0 up'\")","Q_Score":0,"Tags":"python,selenium","A_Id":60088874,"CreationDate":"2020-02-05T09:22:00.000","Title":"Is there any function in selenium that closes internet connection and resumes after specific amount of time?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I have a multiplayer game written in python and uses TCP, So when I send two packets at the same time they get mixed up example if I send \"Hello there\" and \"man\" the client receives \"hello thereman\".\nWhat should I do to prevent them from getting mixed?","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":279,"Q_Id":60087125,"Users Score":3,"Answer":"That's the way TCP works. It is a byte stream. It is not message-based.\nConsider if you write \"Hello there\" and \"man\" to a file. If you read the file, you see \"hello thereman\". A socket works the same way.\nIf you want to make sense of the byte stream, you need other information. For example, add line feeds to the stream to indicate end of line. For a binary file, include data structures such as \"2-byte length (big-endian) followed by <length> bytes of data\" so you can read the stream and break it into decipherable messages.\nNote that socket methods send() and recv() must have their return values checked. recv(1024) for example can return '' (socket closed) or 1-1024 bytes of data. The size is a maximum to be returned. send() can send less than requested and you'll have to re-send the part that didn't send (or use sendall() in the first place).\nOr, use a framework that does all this for you...","Q_Score":0,"Tags":"python-3.x,sockets","A_Id":60087186,"CreationDate":"2020-02-06T03:05:00.000","Title":"Python TCP packets getting mixed","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I using selennium with python to crawl webpage. But I want to pause and resume everytime when scrolling webpage. Is it possible?","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":89,"Q_Id":60097213,"Users Score":0,"Answer":"You could use an explicit wait or poll for certain elements every time you scroll. I would be interested to know if you are dealing with some lazy loading, where the elements you are looking for are not already in the DOM.","Q_Score":0,"Tags":"python,selenium,selenium-webdriver,web-crawler","A_Id":60100509,"CreationDate":"2020-02-06T14:27:00.000","Title":"Is it possible to pause and resume like event when scrolling webpage using selenium python?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"What exactly does this error mean and how can i fix it, am running server on port 8000 of local host.\nConnectionRefusedError: [WinError 10061] No connection could be made because the target machine actively refused it","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":2412,"Q_Id":60110234,"Users Score":0,"Answer":"Is firewall running on the server? If so, that may be blocking connections. You could disable firewall or add an exception on the server side to allow connections on port 8000.","Q_Score":0,"Tags":"python,django,localhost","A_Id":60110521,"CreationDate":"2020-02-07T09:05:00.000","Title":"ConnectionRefusedError: [WinError 10061][WinError 10061] No connection could be made because the target machine actively refused it","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I have a webmap which is made in python using Folium. I am adding various geojson layers from an underlying database. I would like to do spatial analysis based on the user's location and their position relative to the various map overlays. As part of this I want to display a marker on the map which indicates the user's current position, and which updates regularly as they move around.\nI know how to add markers to the map from within python, using Folium.\nI know how to get a constantly updating latitude \/ longitude of the user using JS \nnavigator.geolocation.watchPosition(showPosition)\nwhich then passes a position variable to the function showPosition.\nI am currently just displaying this as text on the website for now.\nWhat I have not been able to do is to add a marker to the Folium map from inside the webpage, using JS\/Leaflet (as Folium is just a wrapper for Leaflet, i think I should be able to do this). \nThe Folium map object seems to be assigned a new variable name every time the webpage is loaded, and I don't know how to \"get\" the map element and add a marker using the Leaflet syntax \nL.marker([lat, lon]).addTo(name_of_map_variable_which_keeps_changing)\nAlternatively there might be a way to \"send\" the constantly changing lat\/lon variables from the webpage back to the python script so that I can just use folium to add the marker. \nBut I have been unable to figure this out or find the right assistance online and would appreciate any help.","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":1042,"Q_Id":60127471,"Users Score":0,"Answer":"OK, I have figured out a main part of the question - how to add a user location marker to the Folium map. It is actually very simple:\nhttps:\/\/python-visualization.github.io\/folium\/plugins.html#folium.plugins.LocateControl\nI am still unable to pass the user's lat\/lon through to my python script so that I can perform spatial queries using that location. So am looking forward to anyone being able to answer that part. Though I may have to post that as a separate question perhaps...","Q_Score":0,"Tags":"javascript,python,geolocation,leaflet,folium","A_Id":60205173,"CreationDate":"2020-02-08T13:56:00.000","Title":"How do I add a mobile geolocation marker into a Folium webmap that automatically updates position?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I have a Node project that's bundled and added to Github as releases. At the moment, it checks my Github for a new release via the API and lets the user download it. The user must then stop the Node server, unzip the release.zip to the folder and overwrite everything to update the project.\nWhat I'm trying to do is write a Python script that I can execute in Node by spawning a new process. This will then kill the Node server using PM2, and then Python script will then check the Github API, grab the download url, downloads it, unzips the contents to the current folder, deletes the zip and then starts up the Node server again.\nWhat I'm struggling with though is checking the Github API and downloading the latest release file. Can anyone point me in the right direction? I've read that wget shouldn't be used in Python, and instead use urlopen","AnswerCount":2,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":65,"Q_Id":60137429,"Users Score":0,"Answer":"You need to map out your workflow and dataflow better. You can do it in words or pictures. If you can express your problem clearly and completely in words step by step in list format in words, then translate it to pseudocode. Python is great because you can go almost immediately from a good written description, to pseudocode, to a working implementation. Then at least you have something that works, and you can optimize performance, simplify functionality or usability from there. This is the process of translating a problem into a solution. \nWhen asking questions on SO, you need to show your current thinking, what you've already tried, preferably with your code that doesn't yet work, or work the way you need it to work. People can vote you down and give you negative reputation points if you ask a question with just a vague description, a question that is an obvious cry for help with homework (yours is not that), or a muse or a vague question with not even an attempt at a solution, because it does not contribute back to the community in any way. \nDo you have any code or detailed pseudocode steps for checking the GitHub API and checking for the \"latest release\" of file(s) you are trying to update?","Q_Score":0,"Tags":"python,python-3.x","A_Id":60137925,"CreationDate":"2020-02-09T14:05:00.000","Title":"Download latest version from Github, unzip to folder and overwrite contents?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I'm trying to write a short python script which fetches data from Google Places API and exports it as a .csv file. \nUnfortunately I'm stuck right at the beginning. \nI want to use requests package but my machine cannot find it. \nI've installed Python 3.5 on my Mac and when I use pip to install requests it says: \n\nRequirement already satisfied: requests in \/Library\/Frameworks\/Python.framework\/Versions\/3.5\/lib\/python3.5\/site-packages\n\nBut when I run the program I get an import error which says: \n\nImportError: No module named requests\n\nDo you have any ideas how i can fix this issue? \nThanks in advance. \nMuch love.","AnswerCount":1,"Available Count":1,"Score":0.1973753202,"is_accepted":false,"ViewCount":57,"Q_Id":60147769,"Users Score":1,"Answer":"Use python -m pip install requests instead. Your pip installation was probably for a different version.","Q_Score":0,"Tags":"google-api,python-requests","A_Id":60185360,"CreationDate":"2020-02-10T09:52:00.000","Title":"Fetch Google Places Data using Python","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":1,"System Administration and DevOps":0,"Web Development":0},{"Question":"I'm working with responder the python web framework and trying to get remote_addr.\nI looked req.header but there's no data. No client or remote_addr field in req.\nAnd I googled for a while but I didn't get any information.\nHow can I solve this problem?\nThank you.","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":52,"Q_Id":60148527,"Users Score":0,"Answer":"First argument of the handler (declare it req commonly) stores starlette.requests.Request in req._starlette.\nSo we can get remote_addr by looking at req._starlette.client.host.\nAdditionally, we can get remote_addr from WebSocket object by ws.scope[\"client\"][0] (index 1 is the port).","Q_Score":0,"Tags":"python,web,server,frameworks,web-frameworks","A_Id":60149603,"CreationDate":"2020-02-10T10:34:00.000","Title":"How to get remote_addr with responder?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I'm trying to add someone to a specific server and then DM said person with just the discord ID.\nThe way it works is that someone is logging himself in using discord OAuth2 on a website and after he is logged in he should be added to a specific server and then the bot should DM saying something like Welcome to the server! \nHas anyone an idea how to do that?\nThanks for any help","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":91,"Q_Id":60149133,"Users Score":0,"Answer":"It is not possible to leave or join servers with OAuth2. Nor is it possible to DM a user on Discord with a bot unless they share a mutual server.","Q_Score":1,"Tags":"python,oauth-2.0,discord,discord.py","A_Id":60342379,"CreationDate":"2020-02-10T11:09:00.000","Title":"Discord.py: Adding someone to a discord server with just the discord ID","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I am an absolute beginner in AWS: I have created a key and an instance, the python script I want to run in the EC2 environment needs to loop through around 80,000 filings, tokenize the sentences in them, and use these sentences for some unsupervised learning. \nThis might be a duplicate; but I can't find a way to copy these filings to the EC2 environment and run the python script in EC2, I am also not very sure as to how I can use boto3. I am using Mac OS. I am just looking for any way to speed things up. Thank you so so much! I am forever grateful!!!","AnswerCount":2,"Available Count":2,"Score":0.0,"is_accepted":false,"ViewCount":136,"Q_Id":60158883,"Users Score":0,"Answer":"Here's one way that might help:\n\ncreate a simple IAM role that allows S3 access to the bucket holding your files\napply that IAM role to the running EC2 instance (or launch a new instance with the IAM role)\ninstall the awscli on the EC2 instance\nSSH to the instance and sync the S3 files to the EC2 instance using aws s3 sync\nrun your app\n\nI'm assuming you've launched EC2 with enough diskspace to hold the files.","Q_Score":0,"Tags":"python,amazon-web-services,machine-learning,amazon-ec2,boto3","A_Id":60159170,"CreationDate":"2020-02-10T21:41:00.000","Title":"Run Python Script on AWS and transfer 5GB of files to EC2","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":1,"Web Development":0},{"Question":"I am an absolute beginner in AWS: I have created a key and an instance, the python script I want to run in the EC2 environment needs to loop through around 80,000 filings, tokenize the sentences in them, and use these sentences for some unsupervised learning. \nThis might be a duplicate; but I can't find a way to copy these filings to the EC2 environment and run the python script in EC2, I am also not very sure as to how I can use boto3. I am using Mac OS. I am just looking for any way to speed things up. Thank you so so much! I am forever grateful!!!","AnswerCount":2,"Available Count":2,"Score":1.2,"is_accepted":true,"ViewCount":136,"Q_Id":60158883,"Users Score":0,"Answer":"Here's what I tried recently:\n\nCreate the bucket and keep the bucket accessible for public.\nCreate the role and add HTTP option.\nUpload all the files and make sure the files are public accessible.\nGet the HTTP link of the S3 file.\nConnect the instance through putty.\nwget copies the file into EC2\ninstance.\n\nIf your files are in zip format, one time copy enough to move all the files into instance.","Q_Score":0,"Tags":"python,amazon-web-services,machine-learning,amazon-ec2,boto3","A_Id":60161528,"CreationDate":"2020-02-10T21:41:00.000","Title":"Run Python Script on AWS and transfer 5GB of files to EC2","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":1,"Web Development":0},{"Question":"I'm a beginner and I've tried using with ZMQ monitor queue but I'm not getting expected result","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":249,"Q_Id":60169702,"Users Score":0,"Answer":"Q : How to know socket status(connected or not) in python?\n\na ) By exception(s) :thrown in common in situations, when the requested call to API ( API service ) was asked in such a moment, the socket-object was not in a state to execute it( for this, see the native-API documentation for details on reporting failed state via zmq_errno() et al )\nb ) By introspection : ZeroMQ API reports the rc ( Return Code detecting the { PASS | FAIL } result of a native-API call )\n\nPython wrappers ( pyzmq vX.Y ), the more the recent efforts to re-wrap this already wrapper-re-formulated native-API tools, complicate the way, how to learn intrinsic details about the { PASS | FAIL } result of the most recent re-wrapped call to native-API.\nBest read first the original ZeroMQ native API documentation, so as to see what gets when reported via the native zmq_errno() and all the rest about the internal state(s) of the system is to be deducted from FSA-events ( reported via the native-API socket_monitor tool ( having this again re-wrapped by a wrapper-mediated counterparty in pyzmq - so test its suitability for your use-case ) ).","Q_Score":1,"Tags":"python-3.x,zeromq,pyzmq","A_Id":60171966,"CreationDate":"2020-02-11T13:14:00.000","Title":"In ZMQ PUSH\/PULL How to know socket status(connected or not) in python?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I am a beginner, I apologize in advance for any ambiguity in my explanation\nPython: Python 3.7.4\nConsole used: Anaconda Command Prompt\nInfo: I am using a work computer and work network, which has a firewall preventing me from using \"pip\" commands to install anything. I downloaded Selenium from their webpage\nDescription of the problem\nI have a python script that utilizes Selenium to web-scrape a website. Every time I restart my computer and I launch the script, I get ImportError: No module named 'selenium' \nThe temporary solution I have found for this problem is running the setup.py script within the Selenium folder, and then running the install command.\nHowever, this temporary solution is problematic as I wish to start the script at a predetermined time using TASK SCHEDULER. From my futile attempts, when I try to run the original script via TASK SCHEDULER (no .bat file, but the .py file), I get the problem stated in the title, despite having \"installed\" the setup.py previously.\nDoes anyone know what this problem is due to? I have tried changing the location of the Selenium Folder to no avail.\nFYI: The file path where I have saved all Selenium related files is ('X's used for confidentiality reasons):\nF:\\DEFI-TAU-CLI-XXX-FCE-XXX\\XXX_XX_XXXX\\XX XXXXX XXXXXXXX\\Scraping Python\\PYTHON - Code XXXX\\Selenium\\selenium-3.141.0","AnswerCount":3,"Available Count":2,"Score":0.0665680765,"is_accepted":false,"ViewCount":70,"Q_Id":60185831,"Users Score":1,"Answer":"You could update the path environment variable by getting into the advanced system settings > environment variables \nOr you could uninstall the anaconda navigator and while reinstalling check the first box which says Add anaconda to PATH environment variable.","Q_Score":0,"Tags":"python,selenium,web-scraping","A_Id":60187255,"CreationDate":"2020-02-12T10:10:00.000","Title":"ImportError: No module named 'selenium' - need to run setup.py","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I am a beginner, I apologize in advance for any ambiguity in my explanation\nPython: Python 3.7.4\nConsole used: Anaconda Command Prompt\nInfo: I am using a work computer and work network, which has a firewall preventing me from using \"pip\" commands to install anything. I downloaded Selenium from their webpage\nDescription of the problem\nI have a python script that utilizes Selenium to web-scrape a website. Every time I restart my computer and I launch the script, I get ImportError: No module named 'selenium' \nThe temporary solution I have found for this problem is running the setup.py script within the Selenium folder, and then running the install command.\nHowever, this temporary solution is problematic as I wish to start the script at a predetermined time using TASK SCHEDULER. From my futile attempts, when I try to run the original script via TASK SCHEDULER (no .bat file, but the .py file), I get the problem stated in the title, despite having \"installed\" the setup.py previously.\nDoes anyone know what this problem is due to? I have tried changing the location of the Selenium Folder to no avail.\nFYI: The file path where I have saved all Selenium related files is ('X's used for confidentiality reasons):\nF:\\DEFI-TAU-CLI-XXX-FCE-XXX\\XXX_XX_XXXX\\XX XXXXX XXXXXXXX\\Scraping Python\\PYTHON - Code XXXX\\Selenium\\selenium-3.141.0","AnswerCount":3,"Available Count":2,"Score":0.0665680765,"is_accepted":false,"ViewCount":70,"Q_Id":60185831,"Users Score":1,"Answer":"Python is not able to find the selenium files that you have in your F drive.\nUpdate the PATH environment variable to include your selenium file directory.","Q_Score":0,"Tags":"python,selenium,web-scraping","A_Id":60185952,"CreationDate":"2020-02-12T10:10:00.000","Title":"ImportError: No module named 'selenium' - need to run setup.py","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I gone through lots of tutorial but not getting proper solution for this error\n\nTypeError: maximize_window() missing 1 required positional argument: 'self'","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":56,"Q_Id":60186333,"Users Score":0,"Answer":"from selenium import webdriver\ndriver = webdriver.Chrome()\n driver.maximize_window()\nThis works just fine, assuming you have downloaded chromedriver.\nYou can also download geckodriver and work with Firefox, personally i prefer this.","Q_Score":0,"Tags":"python-3.x,selenium-webdriver,python-requests,pycharm,selenium-chromedriver","A_Id":60186679,"CreationDate":"2020-02-12T10:34:00.000","Title":"how to maximize browser window in selenium webdriver 3","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"following docusign documentations I have written code to send and downloaded signed docs from docusign this was done using the OAuth token generator for development provided by docusign, now I am getting application token using api_client.request_jwt_application_token which hits \/oauth\/token end point, I am using the access_token provided by this to create and send envelop but I get the following error\nerrorCode\": \"USER_AUTHENTICATION_FAILED\",\\r\\n \"message\": \"One or both of Username and Password are invalid. invalid_request\nI've tried different solutions but so far nothing worked for me","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":142,"Q_Id":60191700,"Users Score":0,"Answer":"Here are the things that you must ensure:\n\nYou have an Integration key (clientID) that is configured correctly.\nYou have an RSA Private key. that key was copied\/pasted exactly as given to your configuration file. New lines must be preserved\nYour URLs are matching the environment. Meaning you use account-d and demo.docusign.net for the sandbox env endpoints.\nYou need the userId which is a GUid for the user that would be impersonated. That user must consent to the application. You have to ensure you pass userId and not accountId and that it is for the same account that you would be using.\n\nIf you confirm all of this and still get an error - I would consider to use our code example to start. I don't see code in your question, but our code example should be a good way to start.","Q_Score":0,"Tags":"python-3.x,jwt,docusignapi","A_Id":60459358,"CreationDate":"2020-02-12T15:26:00.000","Title":"Docusign JWT application access token fails when creating and sending envelop","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"All the articles about web scraping in Python I came across quoted BeautifulSoup and Selenium as the main tools to use to retrieve HTML and JavaScript content from a webpage.\nHowever, I have not found a way to automatically update the data received from this same webpage without the need to send it a new request.\nIs it impossible to do in Python what you would do with Google Chrome, that is, simply opening a webpage and looking at it being automatically updated by the host itself?","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":259,"Q_Id":60257740,"Users Score":0,"Answer":"Yes, it is possible. For example, you can download elements of the DOM tree in a loop if the content is dynamically modified using JavaScript. You can also call your own JavaScript to perform a specific action. Alternatively, you can even write and upload your own browser plugin. In general, in selenium you can control the browser to a large extent, just like a normal user, without the need to refresh the page again.","Q_Score":0,"Tags":"javascript,python,selenium,beautifulsoup,screen-scraping","A_Id":60258897,"CreationDate":"2020-02-17T07:30:00.000","Title":"Web scraping: update data like in Chrome?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I have set up a small flask webpage but in only runs on localhost while I would like to make it run on my local network, how do I do that?","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":29,"Q_Id":60283309,"Users Score":1,"Answer":"Just my 2 cents on this, I just did some research, there are many suggestions online...\nAdding a parameter to your app.run(), by default it runs on localhost, so change it to app.run(host= '0.0.0.0') to run on your machines IP address.\nFew other things you could do is to use the flask executable to start up your local server, and then you can use flask run --host=0.0.0.0 to change the default IP which is 127.0.0.1 and open it up to non local connections.\nThe thing is you should use the app.run() method which is much better than any other methods.\nHope it helps a little, if not good luck :)","Q_Score":0,"Tags":"python,flask,localhost,wifi,python-3.7","A_Id":60283756,"CreationDate":"2020-02-18T14:37:00.000","Title":"I have set up a small flask webpage but in only runs on localhost while I would like to make it run on my local network python3.7","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I got the following link for a app\nhttps:\/\/www.facebook.com\/dialog\/oauth?client_id=SOME_NUMER_I_HAVE&redirect_uri=https:\/\/someSITE.com\/fblogin&state=SOME_VALID_STATE\nI want to send the name and password, and get the cookie.\nDoes anyone know what the request would look like? \n(Or a library that takes care of this)\nMy setup, I'm trying to crawl a website, i need to login to this website to do so.","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":35,"Q_Id":60290799,"Users Score":0,"Answer":"a bit more info about your setup would be helpful. \nIf you are using Django I would recommend using django-allauth for social authentication.","Q_Score":0,"Tags":"python,urllib3","A_Id":60290895,"CreationDate":"2020-02-18T23:14:00.000","Title":"Login into facebook using oauth in python?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I have a REST API service that I want to call on at scheduled times. Currently, I call the API using a basic cronjob on a server every day at 8:00am.\nI want to scale up and allow my users to schedule a time they would like to receive the notification from my API call. How could I go about doing this? I know I would need to keep a database of user requests and their associated times, however I am not sure if continuing to use cron is the best way about this... (I would prefer not to use third party services in order to keep costs down)\nI am having trouble wrapping my head this, if anybody has any advice that would be much appreciated!","AnswerCount":1,"Available Count":1,"Score":0.1973753202,"is_accepted":false,"ViewCount":42,"Q_Id":60309869,"Users Score":1,"Answer":"If the time frame is going to be something simple, like one-per-day, once-per-week, etc., using the cron.d folder is a fairly trivial and in my opinion appropriate solution. \nThe simplest way would be each user having their own file with a simple one-line cron statement that reflects their selected time. When the user selects their time, part of your service creates the correct file for that user. You can go on from there. \nWhether or not you put them in a database is really a question of your own system design; given a proper file naming scheme, you could feasibly do this without having to keep that requested time in persistent storage.","Q_Score":0,"Tags":"python,rest,api,automation,cron","A_Id":60329310,"CreationDate":"2020-02-19T21:55:00.000","Title":"How could I schedule an API call that users sign up for?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":1,"Web Development":0},{"Question":"This month, I learnt Python. I compiled the code and am able to run it on my laptop. A simple code that asks the user yes\/no responses to questions that helps them traverse through a flowchart and end up with a final choice\/result. \nHow do I turn this into a Python Web Application that can be accessed by others over a URL and not be dependent on my laptop to run it?","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":208,"Q_Id":60354832,"Users Score":1,"Answer":"You need to separate the parts of your app that have its logic from the ones which react to and update window content.\nThen you need to rewrite the window management part for the web. From the scratch. Also other parts if they rely on things that are not available in a browser (usually local filesystem access).\nIt is doable, but not that trivial as web apps have different architecture from desktop apps, there are issues with concurrency, preserving app state, and browser limitations, to name few.\nAnything more detailed requires you to describe what it is that your app does and how.","Q_Score":0,"Tags":"python,saas","A_Id":60354901,"CreationDate":"2020-02-22T17:29:00.000","Title":"How to move my Python application to the web?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I'm using selenium to collect cookies, and here's the question: does cookies store information about selenium?","AnswerCount":2,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":162,"Q_Id":60362601,"Users Score":0,"Answer":"You can check the cookies list by driver.manage().getCookies();\nReturn type is Set <Cookies>.\nHope this will help.","Q_Score":0,"Tags":"python,selenium,selenium-webdriver,cookies,pickle","A_Id":60363564,"CreationDate":"2020-02-23T13:20:00.000","Title":"What information about device does cookie store?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I am new to Python unit testing and I am not sure how i can create a unit test of this function that returns a connection make sure that it always returns something?","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":275,"Q_Id":60415665,"Users Score":0,"Answer":"For unit testing any SQL connection, you can use the command SELECT 1 which will always return 1 if the command executed successfully. You can specify more columns, column names, and types by just specifying the values in the query.\nExample: SELECT 1 as NUMBER, 'hello world' as TEXT","Q_Score":0,"Tags":"python,python-3.x,python-2.7,unit-testing,amazon-redshift","A_Id":60443580,"CreationDate":"2020-02-26T14:03:00.000","Title":"Redshift connection unit testing in Python","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"My Company is using JIRA.\nAnd My task is need to use API get data in Service Desk.\nCurrent we are exporting csv file from JIRA. and we want to improve use api get data instead of export file csv.\nSome Field im csv file is \" Summary, Issue key, Issue id, Issue Type, Status, Priority, Resolution, Assignee, Reporter, Creator, Created, Updated, Last Viewed, Resolved, Comment ..... \"\nI have learned about jira's api ,\nEx : \/rest\/servicedeskapi\/request.\nIt only retrieves requests made by the user or assigned to the user (Ex user is me).\nI need to get all the requests in that project.\nI need to your help with api ( python ) get data in Service Desk Jira.\nThanks All.","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":207,"Q_Id":60427303,"Users Score":0,"Answer":"The API should return all the requests the user making the request has access to. Are you able to view all the requests when you log in through the JIRA UI? If the permissions are set up to restrict your account from viewing all the requests, you won't be able to access those tickets via the API. Make sure your account is a Project Administrator for the Service Desk project you're working in, and if it's not have your account added to that role.","Q_Score":0,"Tags":"python,api,jira","A_Id":60571330,"CreationDate":"2020-02-27T06:19:00.000","Title":"Use API get data in Service Desk JIRA","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I want to draw two graphs are almost same, the only difference is that the second one will have some edges grayed out. And I want nodes in the second graph stay exactly where they are in the first graph.","AnswerCount":1,"Available Count":1,"Score":0.1973753202,"is_accepted":false,"ViewCount":131,"Q_Id":60443607,"Users Score":1,"Answer":"You need to define the optional pos argument.\nIn your case using circular layout pos = nx.circular_layout(G). Then you call the plotting commands like nx.draw(G, pos, other_arguments...).","Q_Score":0,"Tags":"python,networkx","A_Id":60443882,"CreationDate":"2020-02-28T00:18:00.000","Title":"How to fix position of nodes when using draw_circular in networkx?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I have a Machine learning module which uses Google Colab's free GPU for NLP tasks, and I want to make a web app out of it. I've been thinking of using React js for frontend and spring boot for the back end and was wondering whether there is a way to connect the code at Google Colab with the backend.\nWant to know other alternative suggestions to building a web app incorporating the ML module in Colab as well. Any sort of help is appriciated.","AnswerCount":1,"Available Count":1,"Score":0.1973753202,"is_accepted":false,"ViewCount":2835,"Q_Id":60443948,"Users Score":1,"Answer":"It's possible. But there's no good examples of it.\nThere's an example of serving local resources, using http-server. Flask examples used to work too. But not much else.","Q_Score":2,"Tags":"python,api,machine-learning,nlp,google-colaboratory","A_Id":60452454,"CreationDate":"2020-02-28T01:10:00.000","Title":"Is there a way to turn Google Colab code into web services or rest APIs","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I am working on odoo11. I want to redirect to Invoicing menu from Odoo website on button click. How can I achieve it through code?","AnswerCount":2,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":246,"Q_Id":60446222,"Users Score":0,"Answer":"You can use controller to redirect website page to backend but make sure on your controller you use auth=user for security reason so public user can not go to backend.","Q_Score":0,"Tags":"python-3.x,odoo,odoo-11","A_Id":60969021,"CreationDate":"2020-02-28T06:15:00.000","Title":"How to redirect to invoice menu from Odoo website?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I am trying to solve a modified version of the TSP. In my version, multiple visits to a city are allowed, as long as the path is the shortest, and also, only subset of the cities are compulsory to visit, as in, you can go through other cities to visit all the subset cities if path is shorter, but if not, the other cities can be ignored. NetworkX has approx. solution for the traditional TSP using the dwave_networkx.algorithms.tsp.traveling_salesperson, but I have trouble solving this one. A naive approach can be to find all possible combinations of the subset cities and check which has the shortest total path length, but that solution will have a n^2 complexity for trying each combination, plus the complexity for finding shortest path for each two cities. So, what should I use to solve this problem using NetworkX.","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":706,"Q_Id":60473238,"Users Score":0,"Answer":"You can select a path randomly and optimize the way on it. Basically, randomly assign a way between two nodes. Than on nodes, try to find optimal way for n+2 nodes. A --> B --> C if there is a path between shortest then try A--> D --> C---E if there is path between D and E shortest than D --> K --> E then again iterate A--> D --> F --> E simply that sound to me a good idea. I don't have proof now but it can give you the possible shortest path. I hope that will be helpful. Good luck.","Q_Score":0,"Tags":"python,algorithm,graph,networkx,traveling-salesman","A_Id":60486146,"CreationDate":"2020-03-01T08:18:00.000","Title":"Solving a modified Travelling Salesman Problem(TSP) using networkx","Data Science and Machine Learning":1,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"When I run chrome driver from selenium the browser opens in minimized windows. but I want it to open by default as maximized","AnswerCount":2,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":378,"Q_Id":60507053,"Users Score":1,"Answer":"You can either use \ndriver.maximize_window() or \nchrome_options.add_argument(\"--start-maximized\") which will maximize the browser when ever it opens.","Q_Score":0,"Tags":"python","A_Id":60507125,"CreationDate":"2020-03-03T12:15:00.000","Title":"How to maximize chrome browser in default when using selenium in python","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I use Python and Spotipy lib.\nIs there any way to list all songs from spotify playlist? \nMethod playlist_tracks() has limit to print only 100 songs.\nSame question about other methods eg. current_user_saved_tracks() with its 20 limits.\nWhy spotify api have these limits? \nThanks","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":380,"Q_Id":60513415,"Users Score":0,"Answer":"In order to get all songs in a playlist you use the first request to determine the total song count which is part of the spotify response.\nDividing the total count by the max song limit per request (100) and rounding up the result will give you the necessary amount of api calls needed to get all song.\nWith this information in mind you simply loop from 0 to call count and request the next 100 song.\nInside the loop you have to set the offset property of the request to x * 100.\nFurthermore your have to add the requested songs to a temporary array to get the entire list at the end.\nI can only assume that the general limit of 100 songs per request is a technical need at spotify.","Q_Score":0,"Tags":"python,api,spotify,spotipy","A_Id":60513747,"CreationDate":"2020-03-03T18:24:00.000","Title":"spotify api playlist_tracks() limits","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I want to implement a backend server that can read (to perform some action) users gmail every time a new mail is received. I am able to figure out that using gmail API users.watch, my server can be notified every time a new email is received. Now, for fetching new mails from gmail my server needs User credentials (Auth token) that are provided by the user at the time of opting in to be watched. Is there anyway these credentials can be sent to my server along with the push notification (maybe using users.watch API). \nOne method I came across to achieve the same is to store auth and refresh token in a DB, that will be accessible only by my server. But it will be better if the purpose can be achieved without storing credentials in the DB.","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":137,"Q_Id":60525479,"Users Score":2,"Answer":"When the user authenticates your application you are given a refresh token assuming that you requested offline access. You should store this in a secure place associated with the user on your server.\nWhen you get a push notification you should then retrieve the refresh token that you have stored on your server and use that to request a new access token that you can use to access the users data.\nThe push notification system has no way of sending you the authorization nor would it be a very wise idea if it was storing your authorization.","Q_Score":2,"Tags":"gmail-api,google-cloud-pubsub,google-api-python-client,google-cloud-python","A_Id":60528026,"CreationDate":"2020-03-04T11:55:00.000","Title":"How to read users gmail after receiving push notification from gmail users.watch API?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I am having a little tough time importing the xmltodict module into my visual studio code. \nI setup the module in my windows using pip. it should be working on my visual studio as per the guidelines and relevant posts I found here. \nbut for some reasons it isn't working in the visual studio. \nPlease advise on how can I get the xmltodict module installed or imported on visual studio code \nThanks in Advance","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":425,"Q_Id":60557938,"Users Score":0,"Answer":"I had the same issue and it turned out that it wasn't installed in that virtual environment even though that was what I had done. Try:\nvenv\/Scripts\/python.exe -m pip install xmltodict","Q_Score":0,"Tags":"python,windows,visual-studio,xmltodict","A_Id":60617030,"CreationDate":"2020-03-06T05:12:00.000","Title":"import xmltodict module into visual studio code","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":1,"System Administration and DevOps":0,"Web Development":0},{"Question":"I don't know much about that topic so I'm asking for your understanding...\nThis is not question about Bluetooth connection!\nI have WIRELESS headphones and I want to detect the connection between the 2.4GHz receiver and the headphones (Steelseries Arctis 7, if that helps). The receiver is connected via USB to the PC. \nHow do I do it in Python? I tried trackerjacker module but it's on linux only and I'm on Windows 10.\nAlso, gamesense module is not for this stuff I guess so I'm really clueless if it's even possible to this...","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":78,"Q_Id":60580667,"Users Score":0,"Answer":"You can try \"winwifi\" package of python for windows env.","Q_Score":0,"Tags":"python,python-3.x,detection,wireless,headphones","A_Id":60580735,"CreationDate":"2020-03-07T18:04:00.000","Title":"How to detect a WIRELESS (not Bluetooth) connection?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I'm helping out a newly formed startup build a social media following, and I have a csv file of thousands of email addresses of people I need to follow. From looking at the twitter API, I see its possible to follow the accounts if I knew their usernames, but its unclear how to look them up by email. Any ideas?","AnswerCount":2,"Available Count":2,"Score":0.0996679946,"is_accepted":false,"ViewCount":65,"Q_Id":60590628,"Users Score":1,"Answer":"This does not appear to be an option with their API, you can use either user_id or screen name with their GET users\/show or GET users\/lookup options.","Q_Score":0,"Tags":"python,api,twitter,twitter-oauth","A_Id":60590725,"CreationDate":"2020-03-08T18:20:00.000","Title":"How can I use the Twitter API to look up accounts from email addresses?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I'm helping out a newly formed startup build a social media following, and I have a csv file of thousands of email addresses of people I need to follow. From looking at the twitter API, I see its possible to follow the accounts if I knew their usernames, but its unclear how to look them up by email. Any ideas?","AnswerCount":2,"Available Count":2,"Score":0.0,"is_accepted":false,"ViewCount":65,"Q_Id":60590628,"Users Score":0,"Answer":"There is no way to do a lookup based on email address in the Twitter API.","Q_Score":0,"Tags":"python,api,twitter,twitter-oauth","A_Id":60601136,"CreationDate":"2020-03-08T18:20:00.000","Title":"How can I use the Twitter API to look up accounts from email addresses?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I'm trying to run an idle file in python 3.8.2... An error keeps popping up and says \"No module named 'discord'\". I'm trying to make a discord bot, but IDLE can't seem to find a module called discord. It keeps highlighting the import discord statement on line 1.","AnswerCount":1,"Available Count":1,"Score":0.1973753202,"is_accepted":false,"ViewCount":635,"Q_Id":60590764,"Users Score":1,"Answer":"I guess you didn't installed discord module. Please install it using \"pip install discord\"","Q_Score":0,"Tags":"python","A_Id":60590934,"CreationDate":"2020-03-08T18:33:00.000","Title":"Error message in IDLE 3.8.2: No module named 'discord'","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"Plese help me with below question\nsample_list = ['Ironman.mdc.googlesuite.net', 'Hulk.nba.abc.googlekey.net',\n 'Thor.web.gg.hh.googlestream.net', 'Antman.googled.net',\n 'Loki.media.googlesuite.net','Captain.googlekey.net']\n\nI would want everything preceeding 'googlesuite.net', 'googlekey.net','googlestream.net' and 'googled.net' in list1 and corresponding prefixes in another list as:\n\nresult_list1=['Ironman.mdc', 'Hulk.nba.abc', 'Thor.web.gg.hh', 'Antman',\n 'Loki.media', 'Captain']\n\n\nresult_list2=['googlesuite.net', 'googlekey.net', 'googlestream.net', 'googled.net',\n 'googlesuite.net', 'googlekey.net']","AnswerCount":3,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":83,"Q_Id":60595247,"Users Score":0,"Answer":"thank you for the answers, it does help but what if I have list like this:\nsample_list = ['Ironman.mdc.googlesuite.net', 'Hulk.nba.abc.googlekey.net',\n 'Thor.web.gg.hh.googlestream.net', 'Antman.googled.net', 'Loki.media.googlesuite.net','Captain.googlekey.net']\n\nI would want everything preceeding 'googlesuite.net', 'googlekey.net','googlestream.net' and 'googled.net' in list1 and corresponding prefixes in another list as:\nresult_list1=['Ironman.mdc', 'Hulk.nba.abc', 'Thor.web.gg.hh', 'Antman', 'Loki.media', 'Captain']\nresult_list2=['googlesuite.net', 'googlekey.net', 'googlestream.net', 'googled.net',\n 'googlesuite.net', 'googlekey.net']","Q_Score":0,"Tags":"python,string","A_Id":60595969,"CreationDate":"2020-03-09T05:43:00.000","Title":"How to split parts of a string in a list based on predefined part of in the string","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":1,"System Administration and DevOps":0,"Web Development":0},{"Question":"Is there anyway I can list all resources\/components under a sub-net using boto-3?\nI'm unable to find a method which gives me result I'm looking for","AnswerCount":2,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":74,"Q_Id":60635513,"Users Score":1,"Answer":"There is no single method that lists everything, but there are API calls to list each type of component.\nFor example, you could list EC2 instances, ENIs, Load Balancers, etc.\nActually, everything in a subnet will have an ENI associated with it, so you could start there. However, it's often not easy to determine the component that is linked to the ENI.","Q_Score":0,"Tags":"python,amazon-web-services,aws-sdk,boto3","A_Id":60646170,"CreationDate":"2020-03-11T11:58:00.000","Title":"List all components under a subnet AWS","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I am trying to pull data from an external API and dump it on S3 . I was thinking on writing and Airflow Operator rest-to-s3.py which would pull in data from external Rest API .\nMy concerns are :\n\nThis would be a long running task , how do i keep track of failures ?\nIs there a better alternative than writing an operator ?\nIs it advisable to do a task that would probably run for a couple of hours and wait on it ?\n\nI am fairly new to Airflow so it would be helpful.","AnswerCount":1,"Available Count":1,"Score":0.761594156,"is_accepted":false,"ViewCount":2262,"Q_Id":60645797,"Users Score":5,"Answer":"Errors - one of the benefits of using a tool like airflow is error tracking. Any failed task is subject to rerun (based on configuration) will persist its state in task history etc..\nAlso, you can branch based on the task status to decide if you want to report error e.g. to email\nAn operator sounds like a valid option, another option is the built-in PythonOperator and writing a python function.\nLong-running tasks are problematic with any design and tool. You better break it down to small tasks (and maybe parallelize their execution to reduce the run time?) Does the API take long time to respond? Or do you send many calls? maybe split based on the resulting s3 files? i.e. each file is a different DAG\/branch?","Q_Score":0,"Tags":"python,airflow-scheduler,airflow","A_Id":60651388,"CreationDate":"2020-03-11T23:42:00.000","Title":"Airflow Operator to pull data from external Rest API","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":1,"Web Development":0},{"Question":"I am trying to import a Python module (fiasco). I cloned it from GitHub and everything appeared to be working fine. Importing it works, but when I try to type, for example iron = fiasco.Element('iron', [1e4, 1e6, 1e8]*u.K), I get the error module 'fiasco' has no attribute 'Element'. I am using Spyder's iPython console. This also fails if I start iPython from the terminal, but works if I start python3 from the terminal. \nI had done this on two different computers - on one, it worked at first, but started giving me the same error after I restarted the kernel. On the other, it never worked at all. \nIf it helps: after importing, I tried typing fiasco. When I did this on the computer where it originally worked, the output was <module 'fiasco' from '\/Users\/shirman\/fiasco\/fiasco\/__init__.py'>. Now, and on the computer it never worked on, it just says <module 'fiasco' (namespace)>. So maybe this has something to do with paths? \nAddition: sys.path points to \/Users\/shirman, and several paths within \/Users\/shirman\/anaconda3. The fiasco folder is in \/Users\/shirman.","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":101,"Q_Id":60677713,"Users Score":2,"Answer":"You have inadvertently created a namespace package due to your sys.path setting. Namespace packages are directories without an __init__.py in the Python search path and allow loading of submodules or -packages from different paths (e.g. path1\/foo\/a.py and path2\/foo\/b.py can be imported as foo.a and foo.b).\nThe problem is that import fiasco finds \/Users\/shirman\/fiasco first and imports it as a namespace package. If you set sys.path such that \/Users\/shirman\/fiasco comes before \/Users\/shirman, the importer finds the actual package \/Users\/shirman\/fiasco\/fiasco first.\nNamespace packages are a Python 3.3 feature, so either the other machine had a different sys.path setting, a really old Python 3 installation, or you were using Python 2.","Q_Score":1,"Tags":"python","A_Id":60678229,"CreationDate":"2020-03-13T21:56:00.000","Title":"Module imports, but doesn't have any attributes","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":1,"System Administration and DevOps":0,"Web Development":0},{"Question":"I'm trying to write my first Python code in Lambda function that will check whether i'm able to SSH (port 22) in to an EC2 instance.\nI have created an EC2 instance with Security Group 22 CidrIP my public IP\n then, created a Lambda function with python 3.8 as runtime in the same account\nNow, through code i,m trying to SSH into EC2 by passing EC2 Public IP, Username, Key pair\nand execute one command, example: sudo su\nQuestion:\n\nWhere should i place my keypair?\nWhat is the code to SSH in to EC2 from lambda funtion?","AnswerCount":1,"Available Count":1,"Score":1.0,"is_accepted":false,"ViewCount":4179,"Q_Id":60686223,"Users Score":6,"Answer":"The first thing I would say is that you should almost never SSH from Lambda into EC2. There are much better ways to remotely run scripts on EC2, including:\n\nSSM Run Manager\nExpose an API on the EC2 instance and call that API\n\nIf you really want to do this, perhaps for some academic reason, then:\n\nstore the keypair in Secrets Manager and give the Lambda permission to read it\nuse a Python package such as Fabric or Paramiko\n\n[Update: it seems that you're trying to validate that SSH access is blocked]\nThe best way to validate security groups is to use the EC2 API, describe the instance(s), enumerate the security groups and their inbound rules. If you don't trust that approach then you could try to SSH to the instance using the method I proposed above (though you only need to try to connect for the test to be useful, presumably).\nThe problem you're going to have is that the security groups could potentially have been set up to block all SSH access (which is the default, by the way) with the exception of a single 'attacker' IP address which is allowed. Your Lambda SSH connection attempt will fail, because it's not coming from that one 'attacker' IP, yet your Lambda test will report \"I cannot access the web server over SSH, test is successful\". That's an invalid test.","Q_Score":1,"Tags":"python,amazon-web-services,amazon-ec2,ssh,aws-lambda","A_Id":60686978,"CreationDate":"2020-03-14T18:50:00.000","Title":"How to ssh into EC2 instance from lambda function","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I'm looking for a way to simulate an active session on a website using Python. What I mean by that, is I want to create a software, which makes the website think that an actual user with an actual browser has the website open. I've found urllib3 and it's request.urlopen methon, but it seems that this only reads the content provided from url and closes the connection. Thanks for any suggestions","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":182,"Q_Id":60698928,"Users Score":0,"Answer":"You can try simulate browser requests to get necessary cookies for authentication. Google Chrome Dev Tools and requests python lib will do the job.\nSome websites have another way to handle sessions, but I believe the majority is using cookies set through post requests.","Q_Score":0,"Tags":"python,url,connection,simulate","A_Id":60699011,"CreationDate":"2020-03-16T00:07:00.000","Title":"Simulate active session on a website with python","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I am writing a socket program to create a simple server. \nWhen I write ip = socket.gethostbyname(socket.gethostname()) and then I print ip it prints 127.0.1.1\nWhy does this keep happening?\nMy device is connected to a mobile hotspot connection still the ip address remains of a local host.\nI am using ubuntu 19.04 OS","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":1395,"Q_Id":60707849,"Users Score":0,"Answer":"The problem is that a host has multiple interfaces. It is not a problem is you use a true DNS or a carefully handwritten \/etc\/host file because then the system will look there to find the translation. But depending on the configuration, the host name can be bound to all the available interfaces, including the loopback one. And gethostbyname returns the address of the first of those interfaces in its own order.\nTo make sure of that, you should use gethostbyname_ex which returns a list of all the interfaces, and you should find the hostspot connected one, in addition to the loopback one.","Q_Score":0,"Tags":"python,sockets,ubuntu,localhost,ip-address","A_Id":60708187,"CreationDate":"2020-03-16T14:32:00.000","Title":"Why does socket.gethostbyname(socket.gethostname) binds only to local ip address?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I have prepared a draft email in Gmail that I need to send to several people but Gmail does not allow me to. The draft email has text, an image in it, and formatting.\nI'd ideally want to send the draft to my list of contacts one by one just changing who it's addressed to. I can put the list of contacts in one column of excel and the name in another. \nI could also just make the draft start with \"Dear sir\/madam\" and send the same draft to my contacts without any modification.\nIf the body of my email was just text I guess I'd just use SMTP, but with all the formatting, and image in it I don't know what to do.\nWhat is the easiest way to do this? Would using Selenium make sense? Anything else that's better?\nThanks","AnswerCount":2,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":79,"Q_Id":60716815,"Users Score":0,"Answer":"You could use a JavaMail API to send email. \nFor formatting you can use the html formatting inside your code, which serves like your draft. \nRead the contacts from a file and replace the variable in \"Dear\" $varName. \nAnd to trigger it multiple times you could use java.util.Timer class. \n\nHope it helps.","Q_Score":0,"Tags":"python,selenium,automation,smtp,gmail","A_Id":60717386,"CreationDate":"2020-03-17T05:44:00.000","Title":"How to send a draft email that I've prepared in several to several contacts in gmail one by one?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I have written a python script that logs into a webpage and the script itself stores my credentials. I am trying to determine whether running this script could somehow expose my credentials? Or is it just as secure as opening my browser myself and typing in my credentials? I would have presumed it would be no different to doing it manually, but I don't necessarily understand how my script communicates with the webdriver and if in that process my credentials could possibly be exposed. The Selenium documentation refers to a Selenium Server, but exactly sure how that works. An official reference would be appreciated!","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":211,"Q_Id":60728992,"Users Score":1,"Answer":"your credentials is safe. selenium server acts as http proxy and sends your script http data , pass through selenium server before running in browser.","Q_Score":0,"Tags":"python,security,webdriver","A_Id":60729149,"CreationDate":"2020-03-17T19:25:00.000","Title":"Selenium Chrome Webdriver Security","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"Im new at this discord.py thing. I've just done a discord.py bot, it works ok, but sometimes the bot repeats constantly the command messages.\nI googled this problem and found out that maybe is for running the script over and over again (like when you save and run after edited or added functions).\nSo I want to stop running the process, just like when I restart windows, the bot is offline (if I run the script after restarting windows, the bot acts normaly).\nPls help\nIf someone needs the code, I can paste it here then.\nPD: I made the bot exact as a tutorial...","AnswerCount":5,"Available Count":3,"Score":0.0,"is_accepted":false,"ViewCount":21516,"Q_Id":60750793,"Users Score":0,"Answer":"just type exit in visual studio code output terminal and your bot will be disconnected. Enjoy the day","Q_Score":3,"Tags":"python,sublimetext3,discord.py","A_Id":70487529,"CreationDate":"2020-03-19T04:08:00.000","Title":"How to stop running discord bot process (python)","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"Im new at this discord.py thing. I've just done a discord.py bot, it works ok, but sometimes the bot repeats constantly the command messages.\nI googled this problem and found out that maybe is for running the script over and over again (like when you save and run after edited or added functions).\nSo I want to stop running the process, just like when I restart windows, the bot is offline (if I run the script after restarting windows, the bot acts normaly).\nPls help\nIf someone needs the code, I can paste it here then.\nPD: I made the bot exact as a tutorial...","AnswerCount":5,"Available Count":3,"Score":0.0,"is_accepted":false,"ViewCount":21516,"Q_Id":60750793,"Users Score":0,"Answer":"Easy way to solve this is by regenerating your bot token. Doing so will shutdown all active scripts. For anyone new to the Discord API, if you ever get an error that says something on the lines of 'Access Denied,' this solution should also help you.","Q_Score":3,"Tags":"python,sublimetext3,discord.py","A_Id":66404102,"CreationDate":"2020-03-19T04:08:00.000","Title":"How to stop running discord bot process (python)","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"Im new at this discord.py thing. I've just done a discord.py bot, it works ok, but sometimes the bot repeats constantly the command messages.\nI googled this problem and found out that maybe is for running the script over and over again (like when you save and run after edited or added functions).\nSo I want to stop running the process, just like when I restart windows, the bot is offline (if I run the script after restarting windows, the bot acts normaly).\nPls help\nIf someone needs the code, I can paste it here then.\nPD: I made the bot exact as a tutorial...","AnswerCount":5,"Available Count":3,"Score":0.0399786803,"is_accepted":false,"ViewCount":21516,"Q_Id":60750793,"Users Score":1,"Answer":"This message really is not discord.py specific and applies to every script which runs indefinetely.\nYou are running multiple instances of your bot. If you run it in an IDE envoirment, then there should be a stop button somewhere. If you are running it in console, closing the console window will close the bot.\nEdit: If you are running it in sublime3 like your tags suggest, every time you want to close your bot, go to \"Tools\" and then \"Cancel Build\" (hotkey: CTRL + Break). As soon as you run another instance of your bot, sublime \"decouples\" the current script in favour of the new one and this method does not work anymore. Then you have to manually go through your running processes (command line or Task Manager) and search for any \"Python\" processes.\nIn general I reccomend running the script in the commandline instead as you have more control over it.","Q_Score":3,"Tags":"python,sublimetext3,discord.py","A_Id":60756179,"CreationDate":"2020-03-19T04:08:00.000","Title":"How to stop running discord bot process (python)","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"My case: \nI want to display the meal plan from my University on my own online \"Dashboard\". I've written my python script to scrape that data and I get the data I need (plain Text). Now I need to put it on my website but I don't know how to start. On my first searching sessions, I have found something with CGI but I have no clue how to use it:( Is there maybe an even easier way to solve my problem? \nThanks","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":400,"Q_Id":60778980,"Users Score":0,"Answer":"I suggest you to use the Django, if you don't want to use django, you can edit your output in HTML formate and publish html page, directly.","Q_Score":0,"Tags":"python,html,cgi","A_Id":60779048,"CreationDate":"2020-03-20T17:35:00.000","Title":"Displaying data from my python script on a webpage","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"so I recently got started with pythonanywhere but was bummed out to notice there is no web socket support. My question today isn't about possible hacks but instead possible alternatives. Is there a solutions I could use for free or close to free specifically for carrying out my socketio microservice? It would only need to setup the socket since I would be using socketio P2P functionality, therefore this server would only be a fallback in the eventuality the client's browser does not support WebRTC. All replies are greatly appreciated. Thanks","AnswerCount":1,"Available Count":1,"Score":0.1973753202,"is_accepted":false,"ViewCount":357,"Q_Id":60786244,"Users Score":1,"Answer":"According to the socketio docs, it uses websockets when they're available, but uses long polling when websockets are not available. If you only need the connection for a short time, then long polling is fine on PythonAnywhere.","Q_Score":0,"Tags":"socket.io,webrtc,pythonanywhere","A_Id":60791135,"CreationDate":"2020-03-21T09:24:00.000","Title":"Pythonanywhere socketio alternative","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I have a Discord bot I use on a server with friends.\nThe problem is some commands use web scraping to retrieve the bot response, so until the bot is finished retrieving the answer, the bot is out of commission\/can't handle new commands.\nI want to run multiple instances of the same bot on my host server to handle this, but don't know how to tell my code \"if bot 1 is busy with a command, use bot 2 to answer the command\"\nAny help would be appreciated!","AnswerCount":2,"Available Count":1,"Score":-0.0996679946,"is_accepted":false,"ViewCount":4363,"Q_Id":60790811,"Users Score":-1,"Answer":"async function myFunction () {}\nthis should fix your problem\nhaving multiple instances could be possible with threads,\nbut this is just a much more easy way","Q_Score":2,"Tags":"python,bots,discord","A_Id":67353378,"CreationDate":"2020-03-21T17:12:00.000","Title":"Efficient Way to Run Multiple Instances of the Same Discord Bot (Discord)","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I've created a script using selenium and chromedriver on pycharm that works absolutely fine. I want to create an exe of that script so that I can run it on other PCs without installing python or the other dependencies. I've tried using pyinstaller but it is not working. Can someone guide me how can I convert that script into exe.\nThanks!!!","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":113,"Q_Id":60791397,"Users Score":0,"Answer":"You are getting any error ? as pyinstaller is the best option to turn your script in to executables.","Q_Score":1,"Tags":"python-3.x,selenium-webdriver,pycharm,selenium-chromedriver","A_Id":62485901,"CreationDate":"2020-03-21T18:01:00.000","Title":"How to convert a selenium webscraping script to exe file?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":1,"System Administration and DevOps":0,"Web Development":0},{"Question":"Im trying to selenium in python 3.8.0 but it doesn't work I get this error: Fatal error in launcher: \n\nUnable to create process using '\"c:\\program files (x86)\\python37-32\\python.exe\" \"C:\\Program Files (x86)\\Python37-32\\Scripts\\pip.exe\" install selenium'","AnswerCount":1,"Available Count":1,"Score":-0.1973753202,"is_accepted":false,"ViewCount":68,"Q_Id":60803447,"Users Score":-1,"Answer":"Try including your pip into your environmental variables and also your Scripts directory","Q_Score":0,"Tags":"python,selenium","A_Id":64872393,"CreationDate":"2020-03-22T18:37:00.000","Title":"Cant install selenium in python","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"How can i connect to multiple websockets in a loop using websocket_client module in python. I tried with to connect under a loop it just connect first of them.","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":43,"Q_Id":60807880,"Users Score":0,"Answer":"If are you using websocket_client you can use threading for connecting multiple websockets at a time.","Q_Score":0,"Tags":"python,websocket","A_Id":60882962,"CreationDate":"2020-03-23T04:16:00.000","Title":"Connecting to multiple Websockets","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I'm generating .docx files from Word documents that have changed Fonts but maintain the same format, style and information content. All has been done via the python-docx API.\nThere are still working-arounds not within the scope of the API (such as duplicating numeric bullets from source and inserting non-strings into Headers\/Footers). I'm approaching these via lxml.\nThe XML of the original file and the generated file, while similar, are not identical irrespective of the lack of <w:numPr> tags. Why is that? The output .docx files look as expected.\nThis complicates working with low level lxml fixes.","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":42,"Q_Id":60817879,"Users Score":1,"Answer":"Your assumption that there would be only a single way to represent a document in as complicated of a format as OOXML, especially generated from independently written code bases, is very much invalid.\nSo the answer to your question is that multiple OOXML representations can yield the same appearance in Microsoft Word (or any other DOCX application); it is not safe to assume that any given library will write any given OOXML exactly the same as any given DOCX application would.","Q_Score":0,"Tags":"python,xml,lxml,openxml,python-docx","A_Id":60818319,"CreationDate":"2020-03-23T16:42:00.000","Title":"Why a Word file duplicated with python-docx has different XML than the original","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I am trying to get some data using API calls in my Python script. I am currently using the requests library which takes about 47 mins for 5K rows. Is there any other library in python faster than requests?","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":183,"Q_Id":60822609,"Users Score":0,"Answer":"For synchronous calls, pycurl is the fastest.","Q_Score":0,"Tags":"python-2.7,api,python-requests","A_Id":62533345,"CreationDate":"2020-03-23T22:48:00.000","Title":"Python library for API call which is faster than requests","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I executed an AWS Glue Job via API Gateway to start the job run. The job run is successful. But the result of the Script (print of a result) has not gotten through the execution. Only job run ID comes as the response. Is there any way to get the result of the job through an API?","AnswerCount":1,"Available Count":1,"Score":0.1973753202,"is_accepted":false,"ViewCount":253,"Q_Id":60825670,"Users Score":1,"Answer":"For glue anything you print or log goes into cloud watch\nYou have an option of adding a handler in your logger that writes to a stream and push that stream to a file in s3. Or better yet, create a StringIO object , store your result to it and then send that to s3","Q_Score":0,"Tags":"python-3.x,amazon-web-services,aws-api-gateway,aws-glue","A_Id":60843757,"CreationDate":"2020-03-24T05:59:00.000","Title":"How to get results of AWS Glue Job when executing via API?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":1,"Web Development":0},{"Question":"I had a different question here, but realized it simplifies to this:\nHow do you detect when a client disconnects (closes their page or clicks a link) from a page (in other words, the socket connection closes)? I want to make a chat app with an updating user list, and I\u2019m using Flask on Python. When the user connects, the browser sends a socket.emit() with an event and username passed in order to tell the server a new user exists, after which the server will message all clients with socket.emit(), so that all clients will append this new user to their user list. However, I want the clients to also send a message containing their username to the server on Disconnect. I couldn\u2019t figure out how to get the triggers right. Note: I\u2019m just using a simple html file with script tags for the page, I\u2019m not sure how to add a JS file to go along with the page, though I can figure it out if it\u2019s necessary for this.","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":2837,"Q_Id":60879701,"Users Score":6,"Answer":"Figured it out. socket.on('disconnect') did turn out to be right, however by default it pings each user only once a minute or so, meaning it took a long time to see the event.","Q_Score":6,"Tags":"python,python-3.x,sockets,flask,socket.io","A_Id":61133214,"CreationDate":"2020-03-27T03:41:00.000","Title":"SocketIO + Flask Detect Disconnect","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I've installed selenium in vscode but when I try to run the program I get an error saying module isn't found. Also, when using from selenium import webdriver, I get messages saying that it is unable to import. I have installed selenium using pip. Is anyone able to help me understand what I'm doing wrong?","AnswerCount":2,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":1700,"Q_Id":60895455,"Users Score":0,"Answer":"This happened to me, searching for a fix I found that there is a bug that gives you the \"Module Isn't Found\" message. After checking your path and you have the proper interpreter selected close VS Code and re-open it. This fixed it for me.","Q_Score":1,"Tags":"python,selenium,visual-studio-code","A_Id":64273734,"CreationDate":"2020-03-27T23:25:00.000","Title":"Selenium installed in visual studio code but module isn't found","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":1,"System Administration and DevOps":0,"Web Development":0},{"Question":"I am building a chat application using flask socketio and I want to send to a specific singular client and I'm wondering how to go about this. \nI get that emit has broadcast and include_self arguments to send to all and avoid sending oneself, but how exactly would I go about maybe emitting to a single sid?\nI've built this application using standard TCP\/UDP socket connection where upon client connecting, there socket info was stored in a dictionary mapped to their user object with attributes that determined what would be sent and when I wanted to emit something to the clients I would iterate through this and be able to control what was being sent. \nI'm hoping some mastermind could help me figure out how to do this in flask socket io","AnswerCount":2,"Available Count":1,"Score":0.0996679946,"is_accepted":false,"ViewCount":45,"Q_Id":60897681,"Users Score":1,"Answer":"I ended up figuring it out. Using the flask request module, you can obtain the users sid using request.sid, which can be stored and emitted to within the room parameter emit(..... room=usersid","Q_Score":0,"Tags":"python,flask,socket.io","A_Id":60908544,"CreationDate":"2020-03-28T05:40:00.000","Title":"How to emit different messages to different users based on certain criteria","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I would like to ask about deploying a bot in 2 different servers. Is there any way to differentiate the two bot instances? For example, I would like to run a bot for serverA and another instance of the same bot in serverB (testing server). I would like to have it so that the bot would respond to a command like \"!ping\" only once in serverB when testing even if serverA and serverB both have running instances of my bot (meaning there are 2 different running scripts of the same code). My dilemma is that if I have two of the same code running in 2 different terminals, when I try to call for !ping command in serverB, it performs the action twice since there are 2 instances.","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":226,"Q_Id":60921774,"Users Score":0,"Answer":"Running a bot in 2 terminals is not the same as having your bot join 2 servers. You'll only have 1 instance of the bot running but it's basically listening to both servers.\nDifferentiating between different servers is generally not needed because the bot will be responding in the context of the server it got a command from. However, if you want to keep track of certain data with files you can do so by including the server name in the file name. This way you can store data for each server separately.","Q_Score":1,"Tags":"python-3.x,client,discord.py","A_Id":60932793,"CreationDate":"2020-03-29T22:54:00.000","Title":"Discord.py differentiating between two different servers","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I had this error while I was working on my assignment using python 3.7 in Visual Studio Code. I tried to comment in some previous stackoverflow posts but it doesn't allow me to comment. So I'm sharing the solution I found in this post.","AnswerCount":2,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":1354,"Q_Id":60922896,"Users Score":0,"Answer":"In my case it was a bit different. My vscode's interpreter was pointing to another version of python.\nTo change the python3 interpreter, open command palette(ctrl+shift+p) and search for, Python : Select Interpreter. Now change that to the correct path and it is good to go.","Q_Score":1,"Tags":"python,selenium,pylint","A_Id":64180272,"CreationDate":"2020-03-30T01:32:00.000","Title":"Unable to import 'selenium' pylint(import-error) (solution)","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":1,"System Administration and DevOps":0,"Web Development":0},{"Question":"I am trying to submit a job to EMR cluster via Livy. My Python script (to submit job) requires importing a few packages. I have installed all those packages on the master node of EMR. The main script resides on S3 which is being called by the script to submit job to Livy from EC2. Everytime I try to run the job on a remote machine (EC2), it dies stating Import Errors(no module named [mod name] )\nI have been stuck on it for more than a week and unable to find a possible solution. Any help would be highly appreciated.\nThanks.","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":1096,"Q_Id":60931595,"Users Score":1,"Answer":"These packages that you are trying to import. Are they custom packages ? if so how did you package them. Did you create a wheel file or zip file and specify them as --py-files in your spark submit via livy ?\nPossible problem.\nYou installed the packages only on the master node. You will need to log into your worker nodes and install the packages there too. Else when u provision the emr , install the packages using bootstrap actions\nYou should be able to add libraries via \u2014py-files option, but it\u2019s safer to just download the wheel files and use them rather than zipping anything yourself.","Q_Score":1,"Tags":"python,python-3.x,amazon-emr,livy","A_Id":60941345,"CreationDate":"2020-03-30T13:26:00.000","Title":"Python packages not importing in AWS EMR","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I am currently facing a weird problem when trying to access a server via Python Requests:\nThis server is using a certificate generated by a CA and the client uses a certificate signed by that same CA. The CA is signed by a self-signed root CA. When accessing a server using a hostname like this\nrequests.get(\"https:\/\/example.com\/\", verify=\"root.pem\", timeout=3)\nit works without any problems whatsoever. When the same code is running against a server that is only accessible via a public IP and also has a certificate with the common name and subject alternative name set to this IP, then this code fails with the error message\nrequests.get(\"https:\/\/<IP>\/\", verify=\"root.pem\", timeout=3)\nrequests.exceptions.SSLError: HTTPSConnectionPool(host='<IP>', port=443): Max retries exceeded with url: \/test\/ (Caused by SSLError(SSLCertVerificationError(\"hostname '<IP>' doesn't match '<IP>'\")))\neven though for example the OpenSSL s_client does not report any errors when connecting. \nOnline I could not find anyone else reporting or solving this issue where there were no hostnames involved but only plain IPs. The certificates themselves are also not at issue, since checking them against one another produces no errors. Especially the error message confuses me, since the IPs are clearly the same.","AnswerCount":2,"Available Count":1,"Score":0.0996679946,"is_accepted":false,"ViewCount":473,"Q_Id":60934651,"Users Score":1,"Answer":"I actually found the error now, it stemmed from a wrong assumptions, that an IP address should also be documented in the certificate using the DNS name extension. I was unaware that there is an IP address extension available for such purposes. Using this resolved the issue for me.","Q_Score":1,"Tags":"ssl,https,python-requests","A_Id":60938094,"CreationDate":"2020-03-30T16:06:00.000","Title":"Problem when accessing hosts using Python Requests via HTTPS via their IP","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"Tried using Python\nOption with document.body.style.zoom= '150%', not suitable.\nAnd with document.body.style.zoom= '1.5' also doesn't work.","AnswerCount":1,"Available Count":1,"Score":0.1973753202,"is_accepted":false,"ViewCount":435,"Q_Id":60941346,"Users Score":1,"Answer":"I'm sorry I almost immediately found a solution after the post of the question)\nwebdriver.execute_script(\"document.body.style.MozTransform='scale(1)';\")","Q_Score":1,"Tags":"python-3.x,selenium,geckodriver","A_Id":60941355,"CreationDate":"2020-03-30T23:52:00.000","Title":"How to zoom in Firefox-geckodriver by selenium?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I'm working right now on project (not mine to clarify) which scraps some sites using urllib3 to make requests and some of them are under CF protection. I found some cfscrape (etc etc list of similar names) library that is a wrapper of requests.Session which may help with circumventing antibot measures of CF but there is a catch, I need proxies which are fetched by API and put into ProxyManager objects. In devenv I have no access to those proxies because of policy. Is there an easy way to get proxy url and auth from ProxyManager or do I need to add some square wheels (aka save them somewhere else as second copy) to integrate that library into project with little work as possible without degrading performance by that much? Don't really want to rewrite urllib3 usage to requests.Session","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":47,"Q_Id":60972361,"Users Score":0,"Answer":"To close the question - ProxyManager does have easy access tho it's kinda strange that I couldn't find anything in the docs (maybe I overlooked it)","Q_Score":0,"Tags":"python,urllib3","A_Id":61211372,"CreationDate":"2020-04-01T13:47:00.000","Title":"How to access proxy_url\\auth in urllib3.ProxyManager","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I made an Amazon Web Scraper and I want it work as an extension. What will happen is that it will display the price if it is lower than the previously recorded price. I don't know Javascript. I went through things like Transcrypt but didn't understand much","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":406,"Q_Id":60972721,"Users Score":0,"Answer":"You cannot. Chrome extensions are written in JS.\nThe only way to accomplish what you want is to use the extension as a bridge from users browser to your script. You'll need to convert the script into a server of some kind that can accept requests from the extension and respond.","Q_Score":1,"Tags":"python,selenium-webdriver,google-chrome-extension","A_Id":60976107,"CreationDate":"2020-04-01T14:04:00.000","Title":"How do I package a Python web scraper as a Chrome Extension?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"i have created a web scraping project in python with the help of beautiful soup but i want my project to work as a desktop application.\nSo, that i can run that application and can perform web scraping on a specific url.\nIs it possible using python and beautiful soup or i have to make the project in different language?","AnswerCount":4,"Available Count":2,"Score":0.0,"is_accepted":false,"ViewCount":757,"Q_Id":61031816,"Users Score":0,"Answer":"I highly suggest using ElectronJS (check out electronjs.org). \nIf you are proficient enough in HTML\/CSS and some Javascript, you can use those markup languages to easily create a UI (which you would do on a normal website), which would look much better than any interface made using Python GUI libraries (such as tkinter, pygame etc.). \nThe Electron app itself is a browser but it works as a desktop application. It is also cross-platform.","Q_Score":0,"Tags":"python,web-scraping,beautifulsoup","A_Id":61032465,"CreationDate":"2020-04-04T17:13:00.000","Title":"How to convert web scraping python project into desktop application?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"i have created a web scraping project in python with the help of beautiful soup but i want my project to work as a desktop application.\nSo, that i can run that application and can perform web scraping on a specific url.\nIs it possible using python and beautiful soup or i have to make the project in different language?","AnswerCount":4,"Available Count":2,"Score":0.0,"is_accepted":false,"ViewCount":757,"Q_Id":61031816,"Users Score":0,"Answer":"If your intention is how to turn it into an executable you can download pyinstaller.\nIf you want to build a GUI you have a few options:\n\nUse a python module like: pygame, tkinter, PyQt5, wxPython, BeeWare.\nUse an JavaScript framework like electron.js.\nMake a Web-App and use python as a back end with a framework like Flask or Django.","Q_Score":0,"Tags":"python,web-scraping,beautifulsoup","A_Id":61032223,"CreationDate":"2020-04-04T17:13:00.000","Title":"How to convert web scraping python project into desktop application?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I am running a python script that runs on startup of Windows 10 due to a shortcut of it being in the startup folder. This script is checking constantly for requirements on whether or not to open a web-page. If a page is launched and chrome has not previously been closed (yes closed, not just opened), the program just \"dies\" and freezes. After debugging my program the loop is no longer running, reading and writing to files stops, and even if another web-page should open ... it doesn't. I'm guessing this is a Chrome issue or a problem with this code windowsChromePath = 'C:\/Program Files (x86)\/Google\/Chrome\/Application\/chrome.exe %s'\nwebbrowser.get(windowsChromePath).open(url)\nAnyone know why this could be?\nEdit: Forgot to mention I have had problems in this same project with read and write. Debugging leads me to believe it's not this, however if you think it could be a read\/write problem it might be","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":29,"Q_Id":61052115,"Users Score":0,"Answer":"Turns out webbrowser \"blocks\" waiting for an exit response from the browser. There is no way to make a non-block call. Use multi threading to run the open as a separate task.\nEdit: Make sure to use try on webbrowser since it returns a connect timeout error by default if the windows is not closed. Except then raise if you really want to see the error.","Q_Score":0,"Tags":"python,windows,google-chrome","A_Id":61070619,"CreationDate":"2020-04-06T02:27:00.000","Title":"~~Running Python Script on Startup That Opens Chrome Causing Unexpected Problems~~ SOLVED Multi-threading","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I'm writing a discord bot for educational purposes and out of boredom. In many tutorials command functionality is realized through on_message() but I found some that use @bot.command too. I don't have enough experience to tell which will be better in the long run and I didn't find any such information in discord.py docs. Which one should I use and why?","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":69,"Q_Id":61096498,"Users Score":2,"Answer":"Which one should I use and why?\n\nI would depend what you want to do specifically. on_message is completely open-ended, so if you need to process arbitrary messages and have your bot react then that's the one to use e.g. maybe you want to reply to people using words of more than 4 syllables and tell them their long words hurt or something.\nCommands are much more structured and used for explicitly interacting with bots.\nSo by default you'd use commands, and if your use case doesn't fit commands, then you'd use the more freeform on_message.","Q_Score":0,"Tags":"python,discord.py","A_Id":61096913,"CreationDate":"2020-04-08T08:32:00.000","Title":"Should I use on_message() or @bot.command?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"so I am working with angular at the moment and I wanted to know if its possible to use Web scraping and providing the information I get to my Angular Webpage.Is there a way to do that and if so is there any good practice you could recommend? Can I combine my Python code or more specific the outcome with my Angular Project? \nThank you very much for helping guys.","AnswerCount":4,"Available Count":2,"Score":0.049958375,"is_accepted":false,"ViewCount":1068,"Q_Id":61099132,"Users Score":1,"Answer":"There are many ways to do this, what I'd recommend for smaller projects is:\n\nSet up lambda functions in AWS that runs your scraping. On lambda you can choose to run node.js, python, and plenty of other languages and frameworks. I prefer selenium with python.\nWith cloudwatch events, set this lambda to scrape every X minutes.\nPut the scraped data into a mongoDB database, made simple with pymongo.\nIn angular, use a server to connect to the mongodb database.\n\nYou can set up your website on heroku.com for free, and add a DB from there. I have used this setup for hittavaran.com","Q_Score":1,"Tags":"python,angular,web-scraping","A_Id":61099326,"CreationDate":"2020-04-08T11:00:00.000","Title":"Web crawling and providing the data to my Angular Webpage","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"so I am working with angular at the moment and I wanted to know if its possible to use Web scraping and providing the information I get to my Angular Webpage.Is there a way to do that and if so is there any good practice you could recommend? Can I combine my Python code or more specific the outcome with my Angular Project? \nThank you very much for helping guys.","AnswerCount":4,"Available Count":2,"Score":1.2,"is_accepted":true,"ViewCount":1068,"Q_Id":61099132,"Users Score":0,"Answer":"Angular is a front-end framework.\nIf you want to display the same scraped data to all users, you should use a back-end framework that your angular app can request the data from.\nThere are many options for frameworks in python, such as flask, to start up an interface that any front-end application can request data from.","Q_Score":1,"Tags":"python,angular,web-scraping","A_Id":61099218,"CreationDate":"2020-04-08T11:00:00.000","Title":"Web crawling and providing the data to my Angular Webpage","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"My program is saying that clientsocket doesn't exist. Even though it has be defined in a different subprogram (that the user has to go through). So my question is do I have to have multiple sockets?\nFrom what I can see, people usually close their sockets then make a new one, but I am wondering if I can just disconnect my socket and then connect it to a different server, this is so I can just define one socket at the start of my program.","AnswerCount":1,"Available Count":1,"Score":0.1973753202,"is_accepted":false,"ViewCount":28,"Q_Id":61142732,"Users Score":1,"Answer":"... but I am wondering if I can just disconnect my socket and then connect it to a different server,\n\nThis is not possible with a TCP socket. You can only connect a socket once.","Q_Score":0,"Tags":"python,sockets","A_Id":61144017,"CreationDate":"2020-04-10T14:44:00.000","Title":"Can I use one socket for the client?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"i've got selenium to run and hit 2 sub categories via the X PATH already which are buttons and now i keep receiving the error above here again - \nole=\"button\">... is not clickable at point (555, 55). Other element would receive the click: ...\n (Session info: chrome=80.0.3987.163)\nmy code looks like this \n\"#More Faq page sub-categories\ntime.sleep(3)\nWaysToWatch = driver.find_element_by_xpath('\/\/*[@id=\"main\"]\/div\/div\/section\/div\/div[2]\/a')\nWaysToWatch.click()\n\"#Set-pin - THIS ONE DOESN'T WANT TO WORK \ntime.sleep(5)\nSetPin = driver.find_element_by_xpath('\/\/*[@id=\"main\"]\/div\/div\/section\/div\/div[3]\/a')\nSetPin.click()\nI would appreciate a fix as I have quite a few of these I need to run through","AnswerCount":3,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":90,"Q_Id":61149695,"Users Score":0,"Answer":"driver.find_element_by_tag_name('body').send_keys(Keys.HOME) \nUse send_keys(Keys.HOME) to scroll up to the top of page","Q_Score":1,"Tags":"python,selenium-webdriver","A_Id":61247397,"CreationDate":"2020-04-10T22:40:00.000","Title":"Python how to click multiple sub categories, get this error button\">... is not clickable at point (555, 55) Other element would receive the click","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I am using Windows10 and my python version 3.6.5\nIts giving me error 'win32api'\nimport win32com.client\nTraceback (most recent call last):\nFile \"\", line 1, in \nimport win32com.client\nFile \\Programs\\Python\\Python36-32\\lib\\site-packages\\win32com__init__.py\", line 5, in \nimport win32api, sys, os\nModuleNotFoundError: No module named 'win32api'\npypiwin32 is already installed","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":130,"Q_Id":61153515,"Users Score":0,"Answer":"It seems we need to restart the Python shell after installing the package pypiwin32. After restarting the shell, it worked.","Q_Score":0,"Tags":"python,python-3.x","A_Id":61154944,"CreationDate":"2020-04-11T07:39:00.000","Title":"ModuleNotFoundError for 'win32api' even pypiwin32 is installed","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I am trying to get input from the user and send this input to all bot subscribers.\nso I need to save his input in variable and use it after this in send_message method but I don't know how to make my bot wait for user input and what method I should use to receive user input \nthanks :]","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":289,"Q_Id":61185980,"Users Score":0,"Answer":"If you want to get an user input, the logic is a bit different. I suppose you are using longpolling.\nWhen the bot asks the user for input, you can just save a boolean\/string in a global variable, let's suppose the variable is user_input:\nYou receive the update, and ask the user for input, then you set user_input[user's id]['input'] = true\nThen when you receive another update you just check that variable with an if (if user_input[userid]['input']: # do something).\n\nIf your problem is 403 Forbidden for \"user has blocked the bot\", you can't do anything about it.","Q_Score":1,"Tags":"python,input,telegram,chatbot,telegram-bot","A_Id":61230604,"CreationDate":"2020-04-13T10:32:00.000","Title":"how i ask for input in my telegram chat bot python telebot","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"In my login First place I wanted to send OTP and second place I wanted to verify the OTP and then return the token.\nI am using rest_framework_simplejwt JWTAuthentication. First place I am verifying the user and sending the OTP, not returning the token and second place I am verifying the OTP and returning the token.\nLet me know If this is the correct way to use? If not how can I implement this using JWTAuthentication.\nOR If this is not correct way to use, can I implement like first place use Basic authentication to verify the user and second place jwt authentication to verify the OTP and send the tokens. Let me know your solution.","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":231,"Q_Id":61222312,"Users Score":0,"Answer":"What I understood?\nYou need to send an OTP to the current user who is hitting your send_otp route after checking if the user exists or not in your system and then verify_otp route which will verify the OTP that the user has sent in the API alongwith it's corresponding mobile_number\/email_id.\nHow to do it?\n\nsend_otp - Keep this route open, you don't need an authentication for this, not even Basic Auth (that's how it works in industry), just get the mobile_number from the user in the request, check whether it exists in the DB, and send the OTP to this number, and set the OTP to the corresponding user in your cache maybe for rechecking (redis\/memcache). Use throttling for this route so that nobody will be able to exploit this API of yours.\nverify_otp - This route will also be open (no authentication_class\/permission_classes), get the mobile_number\/email id + OTP from the user, verify it in cache, if verified, generate the token using TokenObtainPairSerializer and send the refresh + access token in the response, if the OTP is incorrect, send 401.","Q_Score":1,"Tags":"django,python-3.x,django-rest-framework,jwt,django-rest-framework-jwt","A_Id":61266385,"CreationDate":"2020-04-15T06:06:00.000","Title":"Can I use JWTAuthentication twice for a login authentication?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I have python script that use selenium to login website, you should insert the username and password and captcha for submit button to login.after this login webpage have send button for send form information with post request, how can i bypass this clicking in button and send the post request without clicking on the button ?","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":40,"Q_Id":61227582,"Users Score":0,"Answer":"If you mean that you want to try and bypass the captcha and go straight to the send button, I doubt that's possible. If you need to solve recaptchas, check out 2captcha.com and use their API to solve it - which will unlock the send button, theoretically.","Q_Score":0,"Tags":"python,selenium-webdriver","A_Id":61233892,"CreationDate":"2020-04-15T11:29:00.000","Title":"send request in selenium without clicking on the send button in python","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I'm programming a Telegram bot. I want to search for a specific message in a telegram channel and get its message id. Is it possible? Thx in advance.","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":1011,"Q_Id":61228148,"Users Score":3,"Answer":"As far as I know, you aren't able to search though messages (in a specific chat) by just using Telegram's bot API. You need an MTProto client to do that. You can use pyrogram or telethon to interface with MTProto then use messages.search.\nBut, there is a catch. If the message you're searching is in a channel, you can webscrape https:\/\/t.me\/s\/CHANNELUSERNAME with a library like BeautifulSoup.","Q_Score":1,"Tags":"python,api,bots,telegram,py-telegram-bot-api","A_Id":61230228,"CreationDate":"2020-04-15T11:58:00.000","Title":"How can i search for a specific message in a chat in python? (pyTelegramBotAPI)","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I am trying to download songs from youtube using python 3.8 and youtube_dl 2020.3.24.\nBut the weird thing is that most songs I try to download don't get downloaded. I'm talking 99% of them.\nThe ones that do get downloaded get the following Error from youtube_dl:\nERROR: unable to download video data: HTTP Error 403: Forbidden\nIt is worth saying that this happened overnight and I did not change any code.\nbefore this everything worked fine.\nI have friends who ran the same code and they did not get this Error","AnswerCount":2,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":10705,"Q_Id":61249612,"Users Score":29,"Answer":"Same problem many times ..\nsolution: youtube-dl --rm-cache-dir\nCause of the problem: Sometimes I download playlists of large videos and I force it to stop downloading, the next time I run the command to resume the download, the 403 problem arises\nAt the moment, the cache directory is used only to store youtube players for obfuscated signatures. Since all videos in playlist use simple signatures\nPlaylist caching is an obvious way to detect changed titles or changed playlists in general","Q_Score":19,"Tags":"python,youtube-dl","A_Id":61264406,"CreationDate":"2020-04-16T11:55:00.000","Title":"ERROR: unable to download video data: HTTP Error 403: Forbidden while using youtube_dl","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I have a python script for fetching graph api data within power bi. Occasionally it gives the following error on refresh - \nA problem occurred while processing your Python script. \nHere are the technical details: Process must exit before requested information can be determined.\nDetails:\nDataSourceKind=Python\nDataSourcePath=Python\nA problem occurred while processing your Python script.\nHere are the technical details: Process must exit before requested information can be determined.\nErrorCode=-2147467259\nExceptionType=Microsoft.PowerBI.Scripting.Python.Exceptions.PythonUnexpectedException\nEven after extensive search haven't been able to find the reason for the same.\nSteps already tried - \n 1. I have tried switching between anaconda and normal python installed.\n 2. All the libraries are updated with the latest version.\n 3. I have tried clearing the cache of internal power bi desktop version.\n 4. The python script runs perfectly outside power bi.\n 5. Used privacy level for the data source as public.\n 6. I even uninstalled everything and reinstalled powerbi desktop version and also python and anaconda but the issue remains the same.\n 7. Tried on a different system but the issue remains the same.","AnswerCount":1,"Available Count":1,"Score":0.1973753202,"is_accepted":false,"ViewCount":348,"Q_Id":61259101,"Users Score":1,"Answer":"As of today the error does not appear for Python v3.6 but for higher version. Seems there has to be a fix from Microsoft's end. This is a workaround solution that I have found and also raised the concern with the microsoft team but haven't received a reply from their end as of now.","Q_Score":0,"Tags":"python,powerbi","A_Id":62301097,"CreationDate":"2020-04-16T20:05:00.000","Title":"Python script for graph api gives error on refresh in power bi","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I have made a web application that uses Google Sheets as a database. I want to upload my project on github. But since it uses google sheets API for fetching data, I was wondering if is it safe to upload because I will also have to upload API credentials as well?\nI have seen a lot of questions like this on Stack Overflow but none of them addressed this question clearly. Also, my application\/database is nothing confidential or anything like that. My only concern is if uploading API credentials can cause any harm to my google account?","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":43,"Q_Id":61262038,"Users Score":2,"Answer":"You should never upload private\/secret credentials to a public forum like GitHub.\nThere will always be people looking to exploit free credentials like that and it's impossible to know the damage that could be caused. What if someone deletes data on your account? What if they spam calls and get your account suspended? \nThe best practice is to put some kind of placeholder in the code that you put on GitHub and add a section to the ReadMe explaining where the credentials need to be provided.","Q_Score":1,"Tags":"python,django,github,google-sheets-api","A_Id":61262110,"CreationDate":"2020-04-16T23:56:00.000","Title":"Can we upload google sheets api credentials on github?","Data Science and Machine Learning":0,"Database and SQL":1,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I use python boto3\nwhen I upload file to s3,aws lambda will move the file to other bucket,I can get object url by lambda event,like\nhttps:\/\/xxx.s3.amazonaws.com\/xxx\/xxx\/xxxx\/xxxx\/diamond+white.side.jpg\nThe object key is xxx\/xxx\/xxxx\/xxxx\/diamond+white.side.jpg \nThis is a simple example,I can replace \"+\" get object key, there are other complicated situations,I need to get object key by object url,How can I do it?\nthanks!!","AnswerCount":2,"Available Count":1,"Score":0.0996679946,"is_accepted":false,"ViewCount":984,"Q_Id":61265226,"Users Score":1,"Answer":"You should use urllib.parse.unquote and then replace + with space.\nFrom my knowledge, + is the only exception from URL parsing, so you should be safe if you do that by hand.","Q_Score":0,"Tags":"python,amazon-web-services,amazon-s3,aws-lambda,boto3","A_Id":61267789,"CreationDate":"2020-04-17T06:16:00.000","Title":"how to get s3 object key by object url when I use aws lambda python?or How to get object by url?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I want to send some data from the server to the client. I thought about a socket connection but is this really save? What do I have to consider with the socket connection? I mean when the client is connected to the server he could also send a virus to the server. Is there a way to avoid this?","AnswerCount":2,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":32,"Q_Id":61266902,"Users Score":0,"Answer":"That depends on what type of data a client wants to send. During the socket connection server and client sending each other bytes data. If you know exactly what should these bytes include you can discard other that not match your app.","Q_Score":0,"Tags":"python,sockets,virus","A_Id":61266962,"CreationDate":"2020-04-17T08:10:00.000","Title":"Python send data","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"When I access a specific webpage, it sends a specific POST request, the response to which I want to parse. How do I make my script, receiving only the webpage's URL, parse that specific request's response?\n(Ideally, in Python.)","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":27,"Q_Id":61268171,"Users Score":0,"Answer":"So, I've found out that the 'seleniumwire' library for Python is one way to access requests made by a browser when loading a page.","Q_Score":0,"Tags":"python,web-scraping","A_Id":61516105,"CreationDate":"2020-04-17T09:25:00.000","Title":"How do I make my script, receiving only the webpage's URL, parse its POST request's response?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"Am working on a project where i need to gather information from different websites ( Internal to company) and also few information from an windows application developed in Java.\nI was able to pull all the information from websites using Selenium\/Python, now I have an Window application where i need to fetch some information. Since selenium is not supporting Window application am not able to continue with that. I tried using AutoIt but my application is developed on JAVA and autoit windows spy is not able to detect many buttons on the application.\nAny suggestion\/Idea based on your experience how to proceed further ?\nI have 80% of my information in websites which I have already scripted to pull only pending is this applications.","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":212,"Q_Id":61269426,"Users Score":1,"Answer":"If the Java application is using DB (it almost certainly does) you can try getting the data from there directly instead of extracting it from the GUI.","Q_Score":0,"Tags":"python,selenium,user-interface,autoit","A_Id":61285451,"CreationDate":"2020-04-17T10:29:00.000","Title":"Python\/Selenium code to open an application and login ( GUI Automation)","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I'm trying to use Selenium via Python on a desktopless server. When I try to open chromium it says:\nWebDriverException: Message: unknown error: Chrome failed to start: exited abnormally\n (Driver info: chromedriver=2.35.528139 (47ead77cb35ad2a9a83248b292151462a66cd881),platform=Linux 4.15.0 x86_64)\nWhat can I do to make it work, without using a desktop?\nThanks to all of you!","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":781,"Q_Id":61275429,"Users Score":0,"Answer":"I'm running selenium remotely without issues.\nAre the versions of the browser and the chromedriver compatible? You can check this from the command line with which chromium-browser for instance. \nYour driver should also be set to run without a window: options.add_argument('--headless')","Q_Score":0,"Tags":"python,selenium,selenium-webdriver,python-webbrowser","A_Id":61278491,"CreationDate":"2020-04-17T15:38:00.000","Title":"How to use Python Selenium via SSH","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I'm very begginer with python and I have task to parse xsd schema to python code, fill nodes with values and create final XML file from it. Is it possible with generateDS tool? I can't find clear anwser in documentation. Maybe theres another way to do it? Thanks for anwsers","AnswerCount":2,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":310,"Q_Id":61282235,"Users Score":0,"Answer":"Maybe someone in future will have same problem as I had, so there's a tip: to create objects from Schema use generateDStool, it's a way easier to use rather than pyxb","Q_Score":0,"Tags":"python,xml,xsd,xml-parsing,python-generateds","A_Id":61291165,"CreationDate":"2020-04-17T23:17:00.000","Title":"How to parse xsd and create XML's from it with python?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I recently made a script that allow me to create multiple Browser sessions targeting one URL. I would like to add proxy support to it in order to not get banned when running it. I tried to use the Proxy lib from selenium but it just get ignored. \nMy Question : How can I add proxy support into this script while using Selenium in python ? (each session will get a random proxy) \nHere is my code","AnswerCount":2,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":653,"Q_Id":61306701,"Users Score":0,"Answer":"Here is an updated version of my code including the new proxy support elements, I want it to use proxies form a txt file via a proxies = read_from_txt(\"proxies.txt\") and randomly rotate between them using random lib :\nThank you for quick replies, really appreciate it.","Q_Score":1,"Tags":"python,python-3.x,selenium,proxy,selenium-chromedriver","A_Id":61308104,"CreationDate":"2020-04-19T15:14:00.000","Title":"Adding proxy support to Selenium multi session opening script in Python","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I am trying to get response from lambda in AWS API Gateway. I always get no data message with 200. But I return different message in lambda. Gateway and Lambda connected with non-proxy, because I have to add X-AMZ-Invocation-Type : event for async response. How can I change body of response using lambda?","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":811,"Q_Id":61316870,"Users Score":1,"Answer":"The message you see is not coming from lambda. At async invocation; api gateway returns the response first, then make a request to lambda \"asynchronously\". it is normal you to see different messages between lambda and api gateway test. You can't change response the body of response using lambda. \nNo response will be return with async integration (couldn't find it documentation to support this - but tried several times)","Q_Score":2,"Tags":"python-3.x,amazon-web-services,aws-lambda,async-await,aws-api-gateway","A_Id":61317826,"CreationDate":"2020-04-20T07:09:00.000","Title":"Always getting \"no data\" response in AWS Gateway with success code 200","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I'm using the stripe subscription API to provide multi tier accounts for my users. but about 50% of the transactions that i get in stripe are declined and flagged as fraudulent. how can i diagnose this issue knowing that i'm using the default base code provided in the stripe documentation (front end) and using the stripe python module (backend).\nI know that i haven't provided much information, but that is only because there isn't much to provide. the code is known to anyone who has used stripe before, and there isn't any issue with it as there are transaction that work normally.\nThank you !","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":284,"Q_Id":61343164,"Users Score":0,"Answer":"After contacting stripe support, i found that many payments were done by people from an IP address that belongs to a certain location with a card that is registered to a different location.\nfor example if someone uses a French debit card from England. i did ask stripe to look into this issue.","Q_Score":0,"Tags":"python,stripe-payments","A_Id":61781112,"CreationDate":"2020-04-21T12:05:00.000","Title":"Stripe too many high risk payments","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I am developing a login validator in Python 3 using selenium webdriver. I have a CSV with 10 username and passwords that the program will validate. Now, the website on which I have to validate is protected by reCAPTCHA. So after two or three entries, I am considered a bot and asked to \"prove I am human\". I am using Chrome browser.\nCan someone suggest me ways of performing this task smoothly. Are there other options compared to selenium webdriver?","AnswerCount":1,"Available Count":1,"Score":0.1973753202,"is_accepted":false,"ViewCount":120,"Q_Id":61348816,"Users Score":1,"Answer":"One of reCAPTCHA's purposes for existing is to prevent Selenium and similar programs from doing exactly what you're trying to do. My best advice would be to test against a development instance of your website without the reCAPTCHA protection feature enabled.","Q_Score":1,"Tags":"python,selenium,selenium-webdriver","A_Id":61350903,"CreationDate":"2020-04-21T16:52:00.000","Title":"Python - Selenium Webdriver and reCAPTCHA","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I am making a uwsgi web app with python and came across a requirement where I have to limit the no. of requests that the server should accept. Here , I came across the uwsgi option max-requests. But the help says help: reload workers after the specified amount of managed requests.\nSince I am not an native english speaker , I could not understand what that meant exactly.\nMy current understanding is :\nSuppose I set max-requests to 100 , the 10 workers.\nI get 150 requests , 100 requests are served and the rest are put in some queue . (?)\nAfter the 1st request is done , the 101st request is accepted by the worker. (?)\nPlease clarify\/help.","AnswerCount":1,"Available Count":1,"Score":0.1973753202,"is_accepted":false,"ViewCount":1464,"Q_Id":61359392,"Users Score":1,"Answer":"So your Python code is run in workers. There are usually several of them in parallel. The max-requests specifies that, after a certain number of requests per worker, it is reloaded. Basically, your Python code is restarted every max-requests.\nIn you example, if you have 150 requests with 10 workers, nothing will happen (because 150\/10 < 100).\nBut after 1000 requests, workers will be reloaded (which will take some time, but it's probably fine because there are 10 workers so they won't be all reloaded at the same time). There are no queues there.","Q_Score":1,"Tags":"python,nginx,server,uwsgi","A_Id":61359757,"CreationDate":"2020-04-22T07:20:00.000","Title":"How does max-requests in uwsgi actually work","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I need my discord bot to send a message whenever a specific youtube channel uploads a video. How would I do this in python? I just want examples of the different ways it can be done.","AnswerCount":1,"Available Count":1,"Score":-0.1973753202,"is_accepted":false,"ViewCount":1093,"Q_Id":61375754,"Users Score":-1,"Answer":"For this you need to use the Google data api, aka the youtube api, and combine it with the discord api. the discord api is pretty easy to understand, but the youtube one is pretty hard. so i sugesst you dig deep in how apis work and how the youtube data api functions","Q_Score":1,"Tags":"python,discord,discord.py","A_Id":62121296,"CreationDate":"2020-04-22T21:55:00.000","Title":"How to make a discord bot send a message in a channel when a video is uploaded by a specific youtuber","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I have some python scripts on PythonAnywhere.com that I am trying to download into my computer. Whenever I click the download icon next to the file, it opens into another web tab (Chrome) instead. I'm not sure what I did that caused this because I've been able to download files to my computer for the last couple of weeks but suddenly I cannot. Any help would be appreciated - thank you.","AnswerCount":1,"Available Count":1,"Score":0.1973753202,"is_accepted":false,"ViewCount":78,"Q_Id":61377853,"Users Score":1,"Answer":"Use your browser right-click menu on the download link to save the file to your disk.","Q_Score":0,"Tags":"pythonanywhere","A_Id":61384071,"CreationDate":"2020-04-23T01:30:00.000","Title":"Trying to download some python scripts from PythonAnywhere to my computer but it keeps opening another browser tab instead","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I try to run a python cobra sdk script to do a query in cisco aci apic database for class\u00a0l1PhysIf to obtain a port inventory, code seems to be ok, but when I execute code I get error:\ncobra.mit.request.QueryError: Unable to process the query, result dataset is too big\nin my code I use a for loop for append query's obtained data to a empty list\nmy fabric has about 6900 ports so query results can be big, how can handle the code for obtain the data?","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":159,"Q_Id":61381917,"Users Score":-1,"Answer":"use paging \/ pagination to reduce the size of query results","Q_Score":0,"Tags":"python-2.7,sdk,cisco,apic,cobra","A_Id":61382691,"CreationDate":"2020-04-23T07:47:00.000","Title":"cobra.mit.request.QueryError: Unable to process the query, result dataset is too big","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"ok; I wrote a code and use python selenium and then with pyqt5 made it graphical.\nbut when I push the button start webdriver opened and I can't access to my windows's code any more, It got frozen , but I should input some valuable or click on some button as the code running.\nwhat should I do ?","AnswerCount":2,"Available Count":2,"Score":0.0,"is_accepted":false,"ViewCount":40,"Q_Id":61382216,"Users Score":0,"Answer":"What I did was using a Process, so it would seperate the GUI-Loop and everything what is being done by selenium in a Process. By doing so you'll be able to use your GUI parallel to running Selenium in the Background by your process.","Q_Score":0,"Tags":"python,selenium,selenium-webdriver,webdriver,pyqt5","A_Id":61382538,"CreationDate":"2020-04-23T08:04:00.000","Title":"how to work with code when web driver is open","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"ok; I wrote a code and use python selenium and then with pyqt5 made it graphical.\nbut when I push the button start webdriver opened and I can't access to my windows's code any more, It got frozen , but I should input some valuable or click on some button as the code running.\nwhat should I do ?","AnswerCount":2,"Available Count":2,"Score":0.0,"is_accepted":false,"ViewCount":40,"Q_Id":61382216,"Users Score":0,"Answer":"I've faced the same problem before. My solution was to use the time.sleep(duration) when the web driver opened the Window. For your specific situation, I think that you can use javascript on the page to check the user input is complete or not. Once the input fields are filled then trigger the process behind.","Q_Score":0,"Tags":"python,selenium,selenium-webdriver,webdriver,pyqt5","A_Id":61382460,"CreationDate":"2020-04-23T08:04:00.000","Title":"how to work with code when web driver is open","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I'm trying to scrape data within the iFrame.\nI have tried webdriver in Chrome as well as PhantomJS with no success. There are source links contained within the iframe where I assume its data is being pulled from, however, when using these links an error is generated saying \"You can't render widget content without a correct InstanceId parameter.\" \nIs it possible to access this data using python (PhantomJS)?","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":27,"Q_Id":61387511,"Users Score":0,"Answer":"Go to network tools in your browser and investigate what data go to the server and just scrape via simple requests.","Q_Score":0,"Tags":"python,iframe,selenium-chromedriver,phantomjs,forex","A_Id":61387643,"CreationDate":"2020-04-23T12:42:00.000","Title":"Python Scrape Financial Data From iFrame","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I am a student and I am learning to create a Twitter bot using PyCharm. When I used the command: pip install tweepy\nthe terminal displays this statement (after some other lines) :\nInstalling collected packages: idna, urllib3, chardet, certifi, requests, six, PySocks, oauthlib, requests-oauthlib, tweepy\nfollowed by this error:\nERROR: Could not install packages due to an EnvironmentError: [Errno 2] No such file or directory: 'c:\\\\users\\\\name\\\\desktop\\\n\\folder1\\\\folder2\\\\folder3\\\\folder4\\\\our first twitter bot\\\\venv\\\\L\nib\\\\site-packages\\\\oauthlib\\\\oauth2\\\\rfc6749\\\\grant_types\\\\__pycache__\\\\resource_owner_password_credentials.cpython-37.pyc'\nI cannot import tweepy in my program now. Please help me solve this problem. Thank you.","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":63,"Q_Id":61400938,"Users Score":0,"Answer":"You have to enable long file names in windows.\nOpen Regedit and change HKEY_LOCAL_MACHINE\\SYSTEM\\CurrentControlSet\\Control\\FileSystem\\LongPathsEnabled to 1. Restart or sign out to apply for new settings.","Q_Score":1,"Tags":"python-3.x,pycharm,tweepy","A_Id":65447755,"CreationDate":"2020-04-24T04:03:00.000","Title":"How to fix Environment Error while installing tweepy in PyCharm","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":1,"System Administration and DevOps":0,"Web Development":0},{"Question":"I tried to run the telegram API through my Jupyter Notebook but while importing it showed the following error\nfrom telethon import TelegramClient\n----> 2 from telethon.errors.rpc_errors_401 import SessionPasswordNeededError\n 3 \n 4 # (1) Use your own values here\n 5 api_id = 11111\nModuleNotFoundError: No module named 'telethon.errors.rpc_errors_401'","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":922,"Q_Id":61403979,"Users Score":0,"Answer":"The module to be used for SessionPasswordNeededError is telethon.errors.rpcerrorlist. Try using from telethon.errors.rpcerrorlist import SessionPasswordNeededError","Q_Score":0,"Tags":"python,api,jupyter-notebook,telegram,telethon","A_Id":66260235,"CreationDate":"2020-04-24T08:14:00.000","Title":"Telegram API (ModuleNotFoundError: No module named 'telethon.errors.rpc_errors_401')","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"When I visit a particular website from my main browser, I don't get any captcha but when I do it from selenium, I'm getting ReCaptcha.\nIs there a way to make the website think that selenium is my main browser or something like that? \nI asked the same question previously but it was closed. However, I found a similar question that suggests changing the user agent. I tried implementing that but in my case, it doesn't work.\nYour help would be much appreciated.","AnswerCount":1,"Available Count":1,"Score":-0.1973753202,"is_accepted":false,"ViewCount":271,"Q_Id":61440889,"Users Score":-1,"Answer":"Try passing cookie of that website to selenium driver. Selenium always creates a fresh browser instance so it would not keep any trackers\/cookies, that's why captcha is appearing.\nSince I do not know how the website tracks that so above solution may not work. If so try creating a new firefox profile and login to that website manually. Then make sure captcha is not appearing for that profile. After that try using the same firefox profile for the automation. \nHope this will help you.","Q_Score":1,"Tags":"python,selenium,selenium-webdriver,selenium-chromedriver","A_Id":61450737,"CreationDate":"2020-04-26T12:35:00.000","Title":"Is there a way to not encounter captcha in selenium in python (changing user agent doesn't work)?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"So I just created a simple script with selenium that automates the login for my University's portal. The first reaction I got from a friend was: ah nice, you can put that on my pc as well. That would be rather hard as he'd have to install python and run it through an IDE or through his terminal or something like that and the user friendliness wouldn't be optimal.\nIs there a way that I could like wrap it in a nicer user interface, maybe create an app or something so that I could just share that program? All they'd have to do is then fill in their login details once and the program then logs them in every time they want. I have no clue what the possibilities for that are, therefore I'm asking this question. \nAnd more in general, how do I get to use my python code outside of my IDE? Thusfar, I've created some small projects and ran them in PyCharm and that's it. Once again, I have no clue what the possibilities are so I also don't really know what I'm asking. If anyone gets what I mean by using my code further than only in my IDE, I'd love to hear your suggestions!","AnswerCount":1,"Available Count":1,"Score":0.1973753202,"is_accepted":false,"ViewCount":45,"Q_Id":61445838,"Users Score":1,"Answer":"The IDE running you program is the same as you running your program in the console. But if you dont want them to have python installed (and they have windows) you can maybe convert them to exe with py2exe. But if they have linux, they probably have python installed and can run you program with \"python script.py\". But tell your friends to install python, if they program or not, it will always come in handy","Q_Score":1,"Tags":"python,ide,share","A_Id":61445921,"CreationDate":"2020-04-26T18:20:00.000","Title":"Are there any other ways to share \/ run code?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":1,"System Administration and DevOps":0,"Web Development":0},{"Question":"I want to create a web app with Django and integration of Selenium and Python-Docker API.\nWhere inside the Django project is the correct place to put the Selenium scripts so users will be able the run it by a puse of a button?\nI've seen some thred saying inside a 'view', but is is a lot of logic that I've sprated into multipale .py files.","AnswerCount":1,"Available Count":1,"Score":0.1973753202,"is_accepted":false,"ViewCount":226,"Q_Id":61457373,"Users Score":1,"Answer":"You must import Selenium package inside another file such as sel-functions.py inside your app. \nThen you define your class or functions inside this file. Finally you can import all those functionality to your views.py file. \nThis is best and cleanest way to work with another package or functionality in Django project. \nAlso if multiple apps of your project need to sel-functions.py you can put that in your project directory beside settings.py. \n\nAlso, if your code has grown so much that it has multiple files, you\n need to define a package for your intended work and define it as an\n application in your project.","Q_Score":1,"Tags":"python,django,selenium","A_Id":61458639,"CreationDate":"2020-04-27T11:00:00.000","Title":"Django and pure Python","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"i have a data of 100 nodes and 165 relations to be inserted into one keyspace. My grakn image have 4 core CPU and 3 GB Memory. While i try to insert the data i am getting an error [grpc-request-handler-4] ERROR grakn.core.server.Grakn - Uncaught exception at thread [grpc-request-handler-4] java.lang.OutOfMemoryError: Java heap space. It was noticed that the image used 346 % CPU and 1.46 GB RAM only. Also a finding for the issue in log was Caused by: com.datastax.oss.driver.api.core.AllNodesFailedException: Could not reach any contact point, make sure you've provided valid addresses (showing first 1, use getErrors() for more: Node(endPoint=\/127.0.0.1:9042, hostId=null, hashCode=3cb85440): io.netty.channel.ChannelException: Unable to create Channel from class class io.netty.channel.socket.nio.NioSocketChannel)\nCould you please help me with this?","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":173,"Q_Id":61461481,"Users Score":0,"Answer":"It sounds like Cassandra ran out of memory - currently, Grakn spawns to processes: one for Cassandra and one for Grakn server. You can increase your memory limit with the following flags (unix):\nSERVER_JAVAOPTS=-Xms1G STORAGE_JAVAOPTS=-Xms2G .\/grakn server start\nthis would give the server 1GB, and the storage engine (cassandra) 2gb of memory, for instance. 3 GB may be a bit on the low end once your data grows so keep these flags in mind :)","Q_Score":4,"Tags":"python-3.x,vaticle-typedb","A_Id":61476936,"CreationDate":"2020-04-27T14:42:00.000","Title":"Java Heap Space issue in Grakn 1.6.0","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I'm currently trying to host my first discord bot on heroku. In order to get the bot up and running I have to pass several tokens for discord authentication such as authentication for some APIs I'm using. \nDuring development I was using a config.json file which contains all the information I need for the bot to work. This works fine when hosted on my local machine, however I'm not sure if I should upload a file which contains passwords to a remote server like Heroku. \nWhat would be a proper and secure way for me to pass my API credentials to Heroku?","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":88,"Q_Id":61480468,"Users Score":0,"Answer":"the way to provide credentials\/tokens\/pwd is to use Config Vars.\nOnly the application admin has access to this info (in the Heroku Dashboard).\nIt is acceptable to trust the hosting infrastructure with this data as your applications are running on it and require it. For top security applications you would probably use a different hosting solution (private cloud) or run your own server.","Q_Score":1,"Tags":"python-3.x,heroku,discord","A_Id":61481664,"CreationDate":"2020-04-28T12:43:00.000","Title":"How do I securely pass API credentials to a remote Heroku server that my python script is hosted on?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I am working with selenium for the first time, and I'm trying to play a video on another website.\nI've already managed to play the video, but now I'm stuck trying to make it full-screen. The full-screen button is hidden unless I hover with my mouse over it.\nSo I searched for a solution and everybody suggests to use switch_to.frame() and then access the button, but it seems like no frame surrounds the video in my case. I'm not an expert in HTML so maybe I am getting this wrong, but is there a way to click that full-screen button when there's no frame surrounding the video?\nThanks in advance for any help :)","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":628,"Q_Id":61485665,"Users Score":1,"Answer":"If there isn't an iframe containing the video, you should be able to click the button like you would any other on the screen. To get the hover menu to appear you can use ActionChains(driver).move_to_element(element).perform(). \nIf you're having issues finding the button, you may also be able to video_player.send_keys(\"f\"). Many players will use that shortcut to switch to fullscreen.","Q_Score":0,"Tags":"javascript,python,html,selenium,selenium-webdriver","A_Id":61486320,"CreationDate":"2020-04-28T16:56:00.000","Title":"Python Selenium - how to click full screen button of a video without a frame","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I am trying to build a simple kivy app for mobiles \/ desktops which uses Twitter to authenticate users at the login screen. I am using Tweepy with Python to authenticate the users.\nThe authentication process works fine when I run the code from command line. It generates a redirect url which I open in a browser to authenticate the user and then it generates a pin. The pin is copied and entered into the command line interface of the parent Python program as an input and the authentication process completes successfully.\nIs there a way to copy the generated pin from the browser's html content and directly use it as an input in the program without any human intervention? This would enhance the user experience and people would not get confused as to what to do with the PIN. I was playing around with the weburl, urllib and htmlparser libraries in Python and was wondering if there any way to achieve this behaviour?\nPlease help.","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":43,"Q_Id":61538233,"Users Score":2,"Answer":"You need to create an intermediate web app which runs on a server with a static ip address. The authentication code \/ PIN should be returned by Twitter to this intermediate app, which in turn can exchange it with your mobile\/desktop app.","Q_Score":0,"Tags":"python-3.x,kivy,tweepy,kivy-language","A_Id":62116704,"CreationDate":"2020-05-01T07:05:00.000","Title":"How to achieve authentication with Twitter in Kivy App without manual intervention","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I have around 10 websites I need to scrape some data off from. I'm using Scrapy, and I need to know if I have to make 10 separate projects (one for each website) or 10 spiders in a single project?","AnswerCount":2,"Available Count":2,"Score":0.0,"is_accepted":false,"ViewCount":44,"Q_Id":61556851,"Users Score":0,"Answer":"In theory both is possible, you can have multiple spiders in one project or set up multiple projects. However with 10 websites chances are quite high that you need distinct pipelines, middleware-settings and so on. For that reason I would generally suggest using multiple projects. Also your items.py might grow very large with multiple spiders and so you might loose track.","Q_Score":2,"Tags":"python-3.x,scrapy","A_Id":61563582,"CreationDate":"2020-05-02T09:02:00.000","Title":"Is there a way to have more than 2 spiders crawling different websites in one project using Scrapy?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I have around 10 websites I need to scrape some data off from. I'm using Scrapy, and I need to know if I have to make 10 separate projects (one for each website) or 10 spiders in a single project?","AnswerCount":2,"Available Count":2,"Score":0.0,"is_accepted":false,"ViewCount":44,"Q_Id":61556851,"Users Score":0,"Answer":"If you can share any code between spiders (item base class, middleware, etc.), I would go for a single project.\nIf you have spider-specific items, pipelines or middleware, you can always define those on the spider Python files themselves.","Q_Score":2,"Tags":"python-3.x,scrapy","A_Id":61632248,"CreationDate":"2020-05-02T09:02:00.000","Title":"Is there a way to have more than 2 spiders crawling different websites in one project using Scrapy?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"Hi everyone i want to create a new telegram bot similar to @usinifobot.\nthis is how it works:\nyou just send someone username to this bot and it will give you back his\/her chat id.\ni want to do same thing but i dont khow how to do it with python-telegram-bot.","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":943,"Q_Id":61562744,"Users Score":0,"Answer":"python-telegram-bot can't help with this, because telegram bot api can't provide ids of unrelated entities to the bot - only ones with which bot iteracts\nyou need telegram client api for this and to hold a real account (not a bot) online to make checks by another library\nand with bot library like python-telegram-bot you could only get ids of users who write to bot, channels and groups bot is in and reposted messages from other users\nI created similar bot @username_to_id_bot for getting ids of any entity and faced all these moments","Q_Score":0,"Tags":"python,bots,telegram,telegram-bot","A_Id":61562993,"CreationDate":"2020-05-02T16:30:00.000","Title":"How can i get someone telegram chat id with python?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I want to develop a IoT application using Python that runs on ships and sends data occasionally when a network connection is available. This is the case in about 10% of the time.\nIs there any pattern \/ implementation available who handles exactly this use case? I already read about MQTT and RabbitMQ\/AMQP but the implementations expect that the client reaches the broker at the time of sending the message. What I need is some type of local persistance that builds up a local queue and sends the messages when a connection is detected. \nI know that I could handle that for myself and us a SQLite DB but I think that this is no rare use case so I am sure there must be a solution for this and I just used the wrong search terms.","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":46,"Q_Id":61576742,"Users Score":0,"Answer":"The pattern you're looking for is already implemented in the standard PAHO MQTT Client, as part of the Quality Of Service (QOS) flow.\nAll you need to do, is to call the method 'publish' with the parameter 'qos' being equal to 2. The framework will take care of everything else for you, and automatically buffer the messages till a connection is available.\nThere are 2 points to notice however:\n- the buffer is an in-memory buffer, that is in a case of a power outage, all messages will be lost. If the messages cannot be lost in such a scenario, you have to implement persistent storage of messages into a local file \/ database.\n\nAs an in-memory buffer it is limited by the total available memory for your python code. So for very large number of messages or message size, possibly exceeding available memory, you would also have to implement persistent local storage.","Q_Score":0,"Tags":"python,rabbitmq,mqtt,iot","A_Id":61641332,"CreationDate":"2020-05-03T15:09:00.000","Title":"Pattern for handling message delivery in IoT applications when not online 24\/7","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I\u2019m creating a bot app in python using selenium, for people, but I would need to change the xpath code every week, how do I do this once I distribute the app to people?\nThanks In advance","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":46,"Q_Id":61580812,"Users Score":0,"Answer":"I think the best approach is to locate selectors using Id rather than XPath since there won't be any change to Id selector once a new feature(adding a table\/div to the HTML)is added. Also, this reduces the rework effort to a large extend.","Q_Score":1,"Tags":"python,selenium,selenium-webdriver,pycharm","A_Id":61584978,"CreationDate":"2020-05-03T19:56:00.000","Title":"Updating code for my application every week","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"In examples, I saw of socket programming projects (most of which were chat projects), they often saved all the clients in one array, and when a message was received from a client, in addition to saving it in the database, to all clients also was sent.\nThe question that comes to my mind is: How can this message received from the client and saved in the database and send to clients when number of clients is very large? (I mean, the number of customers is so large that a single server can't meet their demand alone, and several servers are needed to connect sockets).\nIn this case, not all clients can be managed through the array. So how do you transfer a message that is now stored on another server (by another customer) to a customer on this server? (Speed \u200b\u200bis important).\nIs there a way to quickly become aware of database changes and provide them to the customer? (For example, Telegram.)\nI'm looking for a perspective, not a code.","AnswerCount":2,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":217,"Q_Id":61581434,"Users Score":0,"Answer":"You should use your database as your messaging center. Have other servers watch for changes in the database either by subscription or by pulling on a time interval. Obviously subscription would be fastest possible. \nWhen a message is inserted, each server picks this up and sends to their list of clients. This should be quite fast for broadcasting messages.","Q_Score":0,"Tags":"python,sockets,serversocket","A_Id":61584872,"CreationDate":"2020-05-03T20:50:00.000","Title":"How to managing a large number of clients in the socket programming","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I need to get response from sending request to the AWS. I have the secretkey\/AccessKey of AWS.\nWhat is the method\/syntax to access the aws APi, for example GetCredentialReport is an API of AWS, How to access this api?","AnswerCount":2,"Available Count":1,"Score":0.0996679946,"is_accepted":false,"ViewCount":969,"Q_Id":61586015,"Users Score":1,"Answer":"You can use get_credential_report() API in boto3 for credential repot\nget_credential_report()\nRetrieves a credential report for the AWS account. For more information about the credential report, see Getting Credential Reports in the IAM User Guide .\nSee also: AWS API Documentation\nRequest Syntax\nresponse = client.get_credential_report()\nResponse Structure\n(dict) --\nContains the response to a successful GetCredentialReport request.\nContent (bytes) --\nContains the credential report. The report is Base64-encoded.\nReportFormat (string) --\nThe format (MIME type) of the credential report.\nGeneratedTime (datetime) --\nThe date and time when the credential report was created, in ISO 8601 date-time format.\nTry this code\nimport os\nimport boto3\nfrom dotenv import load_dotenv\nload_dotenv()\nAWS_ACCESS_KEY_ID = os.getenv('AWS_ACCESS_KEY_ID')\nAWS_SECRET_ACCESS_KEY = os.getenv('AWS_SECRET_ACCESS_KEY')\nclient = boto3.client(\n 'iam',\n aws_access_key_id=AWS_ACCESS_KEY_ID,\n aws_secret_access_key=AWS_SECRET_ACCESS_KEY\n)\nresponse = client.generate_credential_report()","Q_Score":1,"Tags":"python,amazon-web-services,amazon-iam,rest","A_Id":61586228,"CreationDate":"2020-05-04T06:00:00.000","Title":"How to get response and access the AWS Api with python?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"In Python3 I created a socket with s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM) in order to send UDP to a certain destination in the network. Now the classical way would be to s.sendto(my_data, (ip, port)). But I would like to handle the socket as an io file object. Therefore I created one by f = s.makefile(mode='wb'). Now I can use f.write(my_data) to send data. But wait ... I never had to specify IP and port. Needless to say, the data does not arrive at the destination. With TCP there is no problem because with s.connect((ip, port)) I can specify ip and port before I create a file object.\nIs it possible, and if yes how, to send UDP with a socket as file object?","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":208,"Q_Id":61594796,"Users Score":0,"Answer":"okey...as I was writing the question it occurred to me that I could try calling s.connect((ip, port)) on the UDP socket although there is no connection to establish. Voil\u00e0 it works. Maybe this helps somebody.","Q_Score":1,"Tags":"python-3.x,sockets,makefile,udp","A_Id":61645036,"CreationDate":"2020-05-04T14:38:00.000","Title":"Python3: Is a write to UDP socket as file object even possible?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I have a Single file spider which crawls only a site. Which informations can the owner of the site can see? \nHere, I am not using custom settings. (So that's default setting).\nExample: \n1.Spider name\n2.IP\n3.OS","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":37,"Q_Id":61602294,"Users Score":1,"Answer":"The site owner can see this info.\n\nSpider name. You can change the to anything you want\nIP address you can use a VPN or proxies to change it.\nBrowser you are using. To change this go to settings and go look for USER_AGENT\nremove the comment and you can easily go on google to search for other user-agents.\nCookies. Scrapy has cookies enables by default. Go to settings and look for COOKIES_ENABLED Remove the comment so it is set to False. This would make the scraping job more stealthy.\nConcurrent_Requests. This is so also in the settings. It is how many requests you make simultaneously. It is better to lower the number so it is more friendly to the website.","Q_Score":0,"Tags":"python,scrapy,web-crawler","A_Id":61603285,"CreationDate":"2020-05-04T21:54:00.000","Title":"Which information a site can get when scraped by a scrapy spider?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I have string with html tags and I would like to pass formatting to power point.\nThe only idea I have now is to split it using some xml library and add bunch of ifs adding formatting to run depending on a tag.\nDid you encounter similar problem or have better idea how to approach it?","AnswerCount":1,"Available Count":1,"Score":0.1973753202,"is_accepted":false,"ViewCount":680,"Q_Id":61636863,"Users Score":1,"Answer":"I don't think there is a method of doing this. For a start, some HTML elements and attributes aren't likely to translate.\nI've done a very limited amount of this - and actually it was mostly Markdown I was translating. (The HTML relevance is that I did work with entity references and also <br\/>.)\nI'm sorry to say my code is useless to you.\nMy advice would be to support a small subset of HTML. Perhaps <span> with some limited styling and things like <br\/>.","Q_Score":1,"Tags":"python,python-pptx","A_Id":61922406,"CreationDate":"2020-05-06T13:48:00.000","Title":"Python pptx - pass html formatted text to paragraph in powerpoint","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I'm automating WhatsApp messaging on WhatsApp Web using selenium in python.\nThe code is working all fine. It's sending out messages.\nBut is there any way, i can use it to send Bulk messages using WhatsApp.\nI tried, but I ended up whatsapp blocking my number.","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":571,"Q_Id":61655531,"Users Score":0,"Answer":"For sending the bulk message you can use a loop and can use a time delay for sending the messages in a specific time delay.\nOr for getting help with code you have to share your code.\nYou can use any virtual number to use for your bot, So you not have to worried for getting blocked by whatsapp. There are some apps which are providing free virtual numbers.\nBy the way I have also created a whatsapp bot for automate whatsapp messaging and I have used that number for 1 year to test the working and progress of automation and have used much and still that number is not get blocked by whatsapp.","Q_Score":1,"Tags":"python,selenium,automation,block,whatsapp","A_Id":70091979,"CreationDate":"2020-05-07T10:28:00.000","Title":"WhatsApp Automation using Selenium","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I'm trying to run a Python script to request some webpages from a server. I'm using the requests library and pretty much using default arguments except for custom headers.\nI've tested this in my main computer (lets call it A), which is running Ubuntu 18.04 desktop. Everything works fine and the responses from the server are correct.\nNow, I've been trying to execute the same script in an old laptop at home that I use for this kind of stuff (lets call it B). I've replicated the same environment: same Python version, same packages. This laptop is running Debian 10.3 without a desktop environment.\nBoth A and B share the same public IP address because they are connected to the local network.\nThe issue is, requests sent by computer B are systematically identified as a bot and get blocked. I've tried this for several days and it's consistent: all requests from A work fine, all requests from B don't. I can even run both of them simultaneously and the behavior remains the same.\nNow, this has me puzzled because:\n\nI understand that, from the webserver point of view, the requests from both A and B are identical: same request, same headers, same IP.\nThe workload is clearly not a problem: I'm launching about one request every 10 minutes.\n\nWhat could be the reason B is getting blocked but A is not?","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":159,"Q_Id":61662211,"Users Score":0,"Answer":"You may want to check the 'requests' versions on both the machines and try out different versions.\nI had the exact same problem. My local machine had requests 2.22.0 installed and it worked okay, but my VM had 2.24.0 installed, and the requests sent from there were unsuccessful.\nLater I downgraded to 2.22.0 (on VM) and it worked like charm.","Q_Score":1,"Tags":"python,web-scraping,python-requests,webserver","A_Id":68950214,"CreationDate":"2020-05-07T15:55:00.000","Title":"Same HTTP requests, different responses?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I am trying to come up with a way to scrape information on houses on Zillow and I am currently using xpath to look at data such as rent price, principal and mortgage costs, insurance costs. \nI was able to find the information using xpath but I wanted to make it automatic and put it inside a for loop but I realized as I was using xpath, not all the data for each listing has the same xpath information. for some it would be off by 1 of a list or div. See code below for what I mean. How do I get it more specific? Is there a way to look up for a string like \"principal and interest\" and select the next value which would be the numerical value that I am looking for?\nworks for one listing:\ndriver.find_element_by_xpath(\"\/html\/body\/div[1]\/div[6]\/div\/div[1]\/div[1]\/div[1]\/ul\/li[1]\/article\/div[1]\/div[2]\/div\")\na different listing would contain this:\ndriver.find_element_by_xpath(\"\/html\/body\/div[1]\/div[6]\/div\/div[1]\/div[1]\/div[2]\/ul\/li[1]\/article\/div[1]\/div[2]\/div\")","AnswerCount":2,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":77,"Q_Id":61672860,"Users Score":0,"Answer":"selecting multiple elements using xpath is not a good idea. You can look into \"css selector\". Using this you can get similar elements.","Q_Score":0,"Tags":"python,selenium","A_Id":61673109,"CreationDate":"2020-05-08T05:52:00.000","Title":"Selenium scraping","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"Some people were using my bot on a server I am a part of, and for some reason, the bot suddenly started duplicating responses to commands. Basically, instead of doing an action once, it would do it twice. I tried restarting it multiple times which didn't work, and I know it isn't a problem with my code because it was working perfectly well a few seconds ago. It probably wasn't lag either, because only a couple of people were using it. Any ideas on why this may be and how to fix it? I am also hosting it on repl.it, just so you know what ide Im using","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":280,"Q_Id":61683552,"Users Score":0,"Answer":"Its probably because you run the script and run the host at them same time so it sends the command thorught host and code. If you dont run the code but just the host and it still dupliactes it might be an error with the host or it runs somewhere else in the backround.","Q_Score":0,"Tags":"python,discord.py","A_Id":61711270,"CreationDate":"2020-05-08T16:25:00.000","Title":"Why is my python discord bot suddenly duplicating responses to commands","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I am stuck at this error while importing from ldap3 import Server, Connection, ALL.\nPlease help","AnswerCount":2,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":2695,"Q_Id":61686948,"Users Score":0,"Answer":"I stumbled above the same error. In my case my executed python file was named like the ldap3 python library (ldap3.py). So i renamed my file and it worked.","Q_Score":1,"Tags":"python-3.x,ldap3","A_Id":66463764,"CreationDate":"2020-05-08T19:47:00.000","Title":"How to resolve ImportError: cannot import name 'PyAsn1UnicodeDecodeError' in ldap3 connection","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":1,"System Administration and DevOps":0,"Web Development":0},{"Question":"I've already seen multiple posts on Stackoverflow regarding this. However, some of the answers are outdated (such as using PhantomJS) and others didn't work for me.\nI'm using selenium to scrape a few sports websites for their data. However, every time I try to scrape these sites, a few of them block me because they know I'm using chromedriver. I'm not sending very many requests at all, and I'm also using a VPN. I know the issue is with chromedriver because anytime I stop running my code but try opening these sites on chromedriver, I'm still blocked. However, when I open them in my default web browser, I can access them perfectly fine. \nSo, I wanted to know if anyone has any suggestions of how to avoid getting blocked from these sites when scraping them in selenium. I've already tried changing the '$cdc...' variable within the chromedriver, but that didn't work. I would greatly appreciate any ideas, thanks!","AnswerCount":1,"Available Count":1,"Score":0.1973753202,"is_accepted":false,"ViewCount":164,"Q_Id":61707706,"Users Score":1,"Answer":"Obviously they can tell you're not using a common browser. Could it have something to do with the User Agent?\nTry it out with something like Postman. See what the responses are. Try messing with the user agent and other request fields. Look at the request headers when you access the site with a regular browser (like chrome) and try to spoof those.\nEdit: just remembered this and realized the page might be performing some checks in JS and whatnot. It's worth looking into what happens when you block JS on the site with a regular browser.","Q_Score":2,"Tags":"python,selenium,web-scraping,webdriver,selenium-chromedriver","A_Id":61708614,"CreationDate":"2020-05-10T05:44:00.000","Title":"How to be undetectable with chrome webdriver?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"When I try to run my discord python bot:\nqbasty@cody-discord-bot:~\/matrixcraft-cody$ pm2 start Cody.py --interpreter=python\n[PM2][ERROR] Interpreter python is NOT AVAILABLE in PATH. (type 'which python' to double check.)\nI get that error.\nUbuntu 20 on DigitalOcean.","AnswerCount":1,"Available Count":1,"Score":1.0,"is_accepted":false,"ViewCount":3701,"Q_Id":61728462,"Users Score":10,"Answer":"Here is what worked for me :\npm2 start bot.py --name bot --interpreter python3","Q_Score":4,"Tags":"python,discord","A_Id":64982779,"CreationDate":"2020-05-11T11:19:00.000","Title":"[PM2][ERROR] Interpreter python is NOT AVAILABLE in PATH. (type 'which python' to double check.)","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I'm developing a program in python that plays audio files from various websites, mostly mp3s. My first thought to play these files was to try streaming them with requests then decoding the chunks, seeking would be some sort of range in the request header.\nSo I've tried doing some tests with this and ran into some problems converting small chunks of data to the playable form. Before I dig in and try to fix them, I was wondering if streaming is even necessary. Is that how programs like vlc deal with it? How would you go about dealing with it?\nI did a lot of google searching and didn't come up with anything useful.","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":36,"Q_Id":61736687,"Users Score":1,"Answer":"Yes, if you're playing back as you're downloading, streaming is the way to go.\nThis generally doesn't require any special requests on your part. Simply make a request, decode the data as it comes in, buffer it, and put backpressure on the stream if your buffers are getting full.\nWhat ends up happening is that the TCP window size will be reduced, slowing the speed at which the server transmits to you, until it matches the rate of playback. (In practice, this means that the window slams to zero pretty quickly, then spurts open for a few packets and back to zero again, since internet connections these days are typically much faster than required.)\nNow, you still may want to handle ranged requests if you lose your connection. That is, if I'm listening to audio for several minutes and then lose my connection (such as when changing from WiFi to LTE for example), your app can reconnect and request all bytes from the point at which it left off. Browsers do this. It becomes more important when using common HTTP CDNs which are less tolerant of connections that stay open for long periods of time. Typically, if the TCP window size stays at zero for 2 minutes, expect that TCP connection to close.\nYou might download a copy of Wireshark or some other packet sniffer and watch what happens over the wire while you play one of these HTTP streams in VLC. You'll get a better idea of what's going on under-the-hood.","Q_Score":0,"Tags":"python,audio-streaming","A_Id":61736925,"CreationDate":"2020-05-11T18:28:00.000","Title":"Should I Try to be Streaming mp3s","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I am using discord.py to make a bot, and there are more commands than can fit on one page for my custom help command. I want the bot to add 2 reactions, back and forward, then the user that sent the help message can pick one, and go onto different pages of the help command. I want the bot to be able to edit the message to show the second page, and if they go back, then edit back to the original first page. Could anyone help with this? This is similar to the owobot definitions, where you can scroll back and forth between definitions.","AnswerCount":2,"Available Count":1,"Score":-0.1973753202,"is_accepted":false,"ViewCount":9726,"Q_Id":61787520,"Users Score":-2,"Answer":"If you're using client.command() instead of bot.command() replace the two variables bot to client.","Q_Score":7,"Tags":"python,discord.py","A_Id":65416751,"CreationDate":"2020-05-14T01:08:00.000","Title":"I want to make a multi-page help command using discord.py","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I have a json file with over 16k urls of images, which I parse using a python script and use urllib.request.urlretrieve in it to retrieve images. I uploaded the json file to google drive and run the python script in google Colab.\nThough the files were downloaded (I checked this using a print line in the try block of urlretrieve) and it took substantial time to download them, I am unable to see where it has stored these files. When I had run the same script on my local machine, it stored the files in the current folder.\nAs an answer to this question suggests, the files may be downloaded to some temporary location, say, on some cloud. Is there a way to dump these temporary files to google drive?\n(*Note I had mounted the drive in the colab notebook, still the files don't appear to be stored in google drive)","AnswerCount":2,"Available Count":1,"Score":0.0996679946,"is_accepted":false,"ViewCount":1309,"Q_Id":61798141,"Users Score":1,"Answer":"Colab stores files in some temp location which is new every time you run the notebook. If you want your data to persist across sessions you need to store it in GDrive. For that you need to map some GDrive folder in your notebook and use it as path. Also, you need to give the Colab permissions to access your GDrive","Q_Score":0,"Tags":"python,url,google-colaboratory","A_Id":61798564,"CreationDate":"2020-05-14T13:07:00.000","Title":"dump files downloaded by google Colab in temporary location to google drive","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"Context: I recently realized that YouTube has this new chapter thing feature, I listen to a lot of music mixes and such and those have the song name as chapters, and on linux I print the currently playing YouTube song anyway, so I tought that instead of displaying the name of the mix if I could display the song name in a mix using this new chapter feature.\nQuestion: My question would be, is there any way Python can communicate with a Chrome extension written in JS? Because I made this song name finding extension in JS, so only what it does is extract the name of the song with its Xpath then outputs it with console.log(X). I would need this output inside python where I could work with this given song name.\nUsing selenium and stuff like this, is not an option I think because those are completely separate from the browser","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":211,"Q_Id":61819396,"Users Score":0,"Answer":"You can either work on translating your Python code to JS and adding it to the extension, or running a web server built in Python that you can make requests to.","Q_Score":0,"Tags":"javascript,python,google-chrome","A_Id":61819420,"CreationDate":"2020-05-15T12:36:00.000","Title":"Is there a way that Python can communicate with a Chrome extension?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"Using DynamoDB, I know I can specify a conditional expression to control updates, for example, using attribute_not_exists() to prevent overwriting an existing item. However, is there any way to check the result of this? I.e. if there was indeed an existing item and the create failed, I'd like to know that so I can return an error code in my HTTP response. However, looking at the documentation in Boto3, by default put_item returns nothing, so I'm unsure of how I'd be able to monitor the success of the operation. Anyone found a way to do this?","AnswerCount":2,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":225,"Q_Id":61825481,"Users Score":0,"Answer":"Found it, ConditionalCheckFailedException is thrown. Disappointed the docs don't mention this, other kinds of exceptions are detailed in the boto3 docs, but not this one!","Q_Score":1,"Tags":"python,amazon-web-services,amazon-dynamodb","A_Id":61825683,"CreationDate":"2020-05-15T17:54:00.000","Title":"DynamoDB Python API: Way to check result of conditional expression?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"My script opens Crome's page and presses \"down\" button. It occures in headless mode. If I open chrome window script focuses on it. How can I avoid that situation?","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":31,"Q_Id":61832958,"Users Score":0,"Answer":"For automalte your keyboard or mouse clicks you can use \"pyautogui\" module in python. You can find more about \"pyautogui\" module in it's docs and can contact me for more info.\nMail me on this address : gamingbomb911@gmail.com","Q_Score":0,"Tags":"python,selenium,google-chrome,keyboard,focus","A_Id":61833440,"CreationDate":"2020-05-16T06:46:00.000","Title":"How can I press keyboard buttons via selenium only for current Chrome session in Python","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I have a discord.py bot using the datetime and random libraries ( and discord.py of course ). My question is how can i run it even when my computer is off. I think the answer is a rented server but i think there are cheeper options","AnswerCount":2,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":85,"Q_Id":61842467,"Users Score":0,"Answer":"You'll either have to run it on a machine you don't turn off. Or deploy it to a server. You can get cheap servers through Linode, Digital Ocean and others.","Q_Score":1,"Tags":"python","A_Id":61842486,"CreationDate":"2020-05-16T19:30:00.000","Title":"How do i run a python project nonstop ( even when i close the computer )","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I am new to programming and learned a little bit of Python in the previous two weeks and wrote an automation script that scrapes Categories and articles in each category from a website. Now I want to test that script on my WordPress blog and post articles in their respective categories in the WordPress blog. Can someone direct me to any thorough guide to executing this process? I have done some research on this but there is very little information available on the internet.","AnswerCount":1,"Available Count":1,"Score":0.1973753202,"is_accepted":false,"ViewCount":55,"Q_Id":61846902,"Users Score":1,"Answer":"The best way is to write a php code (in your site) where you will upload the json file. the script will then save data to your database .Modify the python script to save the scraped data in json(many languages easily support it + has more usage and is faster) then upload it to the php file.From database you can display your data","Q_Score":0,"Tags":"python,php,wordpress,web-scraping","A_Id":61847623,"CreationDate":"2020-05-17T04:14:00.000","Title":"Can I post Python scraped articles to my wordpress site using XML-RPC?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I wonder if it is possible to get client's IP address from http request header using Python? I was working on a weather project, it would be very good if I could show weather info of his own location.","AnswerCount":2,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":4819,"Q_Id":61860592,"Users Score":0,"Answer":"If the request happens to contain X-Forwarded-For, X-Real-IP then you might get the client's actual IP address. However, most of the time, if a user doesn't want you to know their actual IP by using a proxy there is no way for a server to tell. And as the other comments pointed out, if you are only working on the HTTP layer it's impossible to get the IP address. The two headers I mentioned are usually only added to the request if you are using a reverse proxy\/load balancer.","Q_Score":1,"Tags":"python,http,python-requests,request-headers,python-requests-html","A_Id":61861079,"CreationDate":"2020-05-18T00:27:00.000","Title":"Is it possible to get a client's IP address from HTTP request header?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"Im new to Javascript so I would like to keep it at the bare minimum. Is there a way that I can use the Electron to communicate with python script without having node.js? My app is just a basic app that takes some input from users from a html page and I need this text input to be processed in python and write an excel file. So there is not much happening in html so is there a simple way to transfer the input to python file? I want to use Electron because I need this html to be my UI and also I need to distribute this app.","AnswerCount":2,"Available Count":1,"Score":0.0996679946,"is_accepted":false,"ViewCount":1855,"Q_Id":61866120,"Users Score":1,"Answer":"Recently i have done it with some sort of trick hope it will help you and there are the following step which i followed- \n\nCreated a stand alone python exe using pyinstaller and the exe has flask server internally then i put the flask server inside my node application.\nNow we have to initiate our flask server and send a request to it for processing, i have done this with the help of \"execFile\" function as a child process, for which i have created a function and the code was something like that- \nasync function callFlask(){\n var child = require('child_process').execFile;\n child('path_to_python_exe ', function(err, data) {\n if(err){\n console.error(err);\n return;\n }\n });\n }\nNow we have initiated our flask server then will send the request with the help of fetch request like\nawait callFlask().then(\nawait fetch('host_ip_defined_in_flask'+encodeURIComponent('data'))\n\nNow further we can extend our then chain to get response from python if any and proceed further forexample - \n\nawait callFlask().then(\nawait fetch('host_ip_defined_in_flask'+encodeURIComponent('data'))\n.then(res => res.text())\n.then(body => console.log(body)))\n\nHere, your output data which python return will be printed in console then you can make your node application behave differently depending on output returned by it.\n\nAlso you can package your app with available packagers for electron like electron-packager it will work like a charm.\n\nAlso there is are some disadvantage for using python as like it will increase your package size and the process will be difficult to kill from electron after processing so it will increase burden on host machine.\nI am assuming that Explaining to create a flask server is not the scope of this question instead if you face any issues let me know, i hope it will help...","Q_Score":2,"Tags":"python,electron,electron-builder,electron-packager","A_Id":62085290,"CreationDate":"2020-05-18T09:08:00.000","Title":"Can I use Electron without node.js","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I'm making a twitterbot for an honors project and have it almost completed. However, when I scrape the website for a specific URL, the href refers to a link that looks like this:\n?1dmy&urile=wcm%3apath%3a%2Fohio%2Bcontent%2Benglish%2Fcovid-19%2Fresources%2Fnews-releases-news-you-can-use%2Fnew-restartohio-opening-dates\nWhen inspecting the html and hovering over the href contents above, it shows that the above is actually the tail-end of the link. Is there any way to take this data and make it into a usable link? Other links within the same carousal provide full links on the same website, so I'm not sure why this is different than the others.\nI tried searching for answers to this question but came up short: sorry if this is a repeat.","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":28,"Q_Id":61876033,"Users Score":0,"Answer":"BeautifulSoup is showing you what the HTML of the page has. If the link is relative, you need the base URL for the page. That should come back in your request result, not in the HTML itself.","Q_Score":0,"Tags":"python,web-scraping,twitter,beautifulsoup,href","A_Id":61876137,"CreationDate":"2020-05-18T18:02:00.000","Title":"Beautifulsoup4: How do you extract a usable link from href when it only provides parameters","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"When I try to open Python it gives me an error saying:\nIDLE's subprocess didn't make connection. See the 'startup failure' section of the IDLE doc online\nI am not sure how to get it to start. I am on the most recent version of windows, and on the most recent version of python.","AnswerCount":3,"Available Count":2,"Score":0.0,"is_accepted":false,"ViewCount":711,"Q_Id":61880099,"Users Score":0,"Answer":"Open cmd and type python to see if python was installed. If so fix you IDE. If not download and reinstall python.","Q_Score":0,"Tags":"python","A_Id":61880601,"CreationDate":"2020-05-18T22:30:00.000","Title":"Python not starting: IDLE's subprocess didn't make connection","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":1,"Web Development":0},{"Question":"When I try to open Python it gives me an error saying:\nIDLE's subprocess didn't make connection. See the 'startup failure' section of the IDLE doc online\nI am not sure how to get it to start. I am on the most recent version of windows, and on the most recent version of python.","AnswerCount":3,"Available Count":2,"Score":0.0,"is_accepted":false,"ViewCount":711,"Q_Id":61880099,"Users Score":0,"Answer":"I figured it out, thanks. All I needed to do was uninstall random.py.","Q_Score":0,"Tags":"python","A_Id":61880823,"CreationDate":"2020-05-18T22:30:00.000","Title":"Python not starting: IDLE's subprocess didn't make connection","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":1,"Web Development":0},{"Question":"We have a small website with API connected using AJAX.\nWe do not ask for usernames and passwords or any authentication like firebase auth.\nSo it's like open service and we want to avoid the service to be misused.\nOAuth 2 is really effective when we ask for credentials to the user.\nCan you suggest the security best practice and how it can be implemented in this context using python?\nThanks","AnswerCount":1,"Available Count":1,"Score":0.1973753202,"is_accepted":false,"ViewCount":63,"Q_Id":61883103,"Users Score":1,"Answer":"Use a firewall\nAllow for third-party identity providers if possible\n Separate the concept of user identity and user account","Q_Score":0,"Tags":"python,api,security,certificate,backend","A_Id":61883428,"CreationDate":"2020-05-19T04:10:00.000","Title":"Python backend -Securing REST APIs With Client Certificates","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I am trying to deploy a simple user bot on Google App Engine Flexible Environment running python3.7, but I get the following error. Can anyone help me with suggestions of solving this?\n\"File \"\/env\/lib\/python3.7\/site-packages\/telethon\/client\/auth.py\", line 20, in phone: typing.Callable[[], str] = lambda: input('Please enter your phone (or bot token): '), EOFError: EOF when reading a line\"\nThank you for your time","AnswerCount":3,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":780,"Q_Id":61884537,"Users Score":0,"Answer":"Telethon (obviously) requires you to login and make a normal user session on your account, which natively requires you to input your number when asked and enter the code received but since Google App engine doesn't allow input as @Sashidhar mentioned, depending on your userbot implementation, you can try using the userbot.session method for authentication, it can be generated locally and placed in the Google App Engine.","Q_Score":0,"Tags":"google-app-engine-python,telethon","A_Id":62506639,"CreationDate":"2020-05-19T06:25:00.000","Title":"Hosting a Telethon User BOT on google-app-engine","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":1,"Web Development":0},{"Question":"I have a HTML code having a div tag with class=\"some_class\". And there is a p tag inside this div tag:\n<div class=\"some_class\">\n<p>blah blah blah...<\/p>\n<\/div>\nI need to get the p tag's content and store it in a variable.\nI did it in two steps:\n1)Get the div tag using findAll('div', class_='some_class'),\n2)Using split at the p tag\nIs there any direct way of doing this without having to do the complex splitting?","AnswerCount":2,"Available Count":1,"Score":0.0996679946,"is_accepted":false,"ViewCount":1001,"Q_Id":61894438,"Users Score":1,"Answer":"If you have only one div tag and only one p tag in it, you can do this:\ncontent = soup.findAll('div', class_= 'some_class')[0].findAll('p')[0].contents[0]","Q_Score":2,"Tags":"python,html,web-scraping,beautifulsoup","A_Id":61895043,"CreationDate":"2020-05-19T15:11:00.000","Title":"Getting the content of a tag inside a class=\"some_class\"> tag using Beautiful Soup in Python","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I've recently been working on a little project and I want to host it on the cloud so it can run all the time, rather than constantly on my local machine. However I'm kind of stumped when it comes to choosing a hosting solution, and I was wondering if you guys could give me some advice.\nSome details about the project:\n\nIt's a Python script, not a web app, and should be running constantly (actions are scheduled once per day)\nIt uses a few fairly large external dependencies, namely Selenium and Chromedriver, which requires binaries to be installed on the machine and grabbable (not sure if that's a word) by path from the script\nIt uses a few static files (mp3) as assets, which need to be stored on a file system somehow and grabbed by the script\nIt writes video files to the local filesystem\n\nDo you guys have any recommendations for hosting solutions which satisfy these details? I've tried with Heroku, which satisfies details 1-2, but since Heroku has like a weird \"ephemeral file system\" I can't grab the static files nor write video files to the file system. \nThanks for the help!","AnswerCount":1,"Available Count":1,"Score":0.1973753202,"is_accepted":false,"ViewCount":151,"Q_Id":61899261,"Users Score":1,"Answer":"I suggest you to look at AWS (since is the only platform I've been working with).\nWith it's free acount you can use an EC2, where you can store your script and dependencies, and your output can be written in an S3 instance (free for 30GB if I remeber well).\nAs you are going to use Selenium, you will need a GUI for your system, so your choices are a Windows machine for EC2 or Linux (does not came with a graphical environment by default). It is not difficult to install a GUI on Linux machines and configure to acces by RDP, but I suggest to use instead a simulated graphical environment such as XVFB. I used it and is as simple as launching your script with xvfb run first. That way you are saving some processor load for your machine.\nI hope it helps,\nRegards.","Q_Score":1,"Tags":"python,selenium,filesystems,hosting","A_Id":61899537,"CreationDate":"2020-05-19T19:32:00.000","Title":"How to host selenium Python script with file system access?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"If I develop a REST service hosted in Apache and a Python plugin which services GET, PUT, DELETE, PATCH; and this service is consumed by an Angular client (or other REST interacting browser technology). Then how do I make it scale-able with RabbitMQ (AMQP)? \nPotential Solution #1\n\nMultiple Apache's still faces off against the browser's HTTP calls.\nEach Apache instance uses an AMQP plugin and then posts message to a queue\nPython microservices monitor a queue and pull a message, service it and return response\nResponse passed back to Apache plugin, in turn Apache generates the HTTP response \n\nDoes this mean the Python microservice no longer has any HTTP server code at all. This will change that component a lot. Perhaps best to decide upfront if you want to use this pattern as it seems it would be a task to rip out any HTTP server code.\nOther potential solutions? I am genuinely puzzled as to how we're supposed to take a classic REST server component and upgrade it to be scale-able with RabbitMQ\/AMQP with minimal disruption.","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":241,"Q_Id":61900319,"Users Score":2,"Answer":"I would recommend switching wsgi to asgi(nginx can help here), Im not sure why you think rabbitmq is the solution to your problem, as nothing you described seems like that would be solved by using this method.\nasgi is not supported by apache as far as I know, but it allows the server to go do work, and while its working it can continue to service new requests that come in. (gross over simplification)\nIf for whatever reason you really want to use job workers (rabbitmq, etc) then I would suggest returning to the user a \"token\" (really just the job_id) and then they can call with that token, and it will report back either the current job status or the result","Q_Score":0,"Tags":"python,rabbitmq,amqp","A_Id":61900432,"CreationDate":"2020-05-19T20:36:00.000","Title":"How to interpose RabbitMQ between REST client and (Python) REST server?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"There is an API that I am using from another company that returns the ID-s of the last 100 purchases that have been made in their website.\nI have a function change_status(purchase_id) that I would like to call whenever a new purchase has been made. I know a workaround on how to do it, do a while True loop, keep an index last_modified_id for the last modified status of a purchase and loop all purchases from the latest to the earliest and stop once the current id is the same as last_modified_id and then put a sleeper for 10 seconds after each iteration. \nIs there a better way on how to do it using events in python? Like calling the function change_status(purchase_id) when the result of that API has been changed. I have been searching around for a few days but could not find about about an event and an API. Any suggestion or idea helps. Posting what I have done is usually good in stackoverflow, but I don't have anything about events. The loop solution is totally different from the events solution.\nThank you","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":23,"Q_Id":61921432,"Users Score":1,"Answer":"The only way to do this is to keep calling the API and watching for changes from the previous response, unless...\nThe API provider might have an option to call your API when something is updated on their side. It is a similar mechanism to push notifications. If they provide a method to do that, you can create an endpoint on your side to do whatever you need to do when a new purchase is made, and provide them the endpoint. However, as far as I know, most API providers do not do this, and the first method is your only option.\nHope this helps!","Q_Score":1,"Tags":"python,api,events","A_Id":61921484,"CreationDate":"2020-05-20T19:33:00.000","Title":"Call function when new result has been returned from API","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"Attempting to use GCP Authorized Session get method in Python to query data in GCP. The doc shows:\nget(url, **kwargs)\nSends a GET request. Returns Response object.\nParameters: \nurl \u2013 URL for the new Request object.\n**kwargs \u2013 Optional arguments that request takes.\nReturn type:\nrequests.Response\nI set up a kwarg dict as kwargs = {'page_size': 2000'}\nHowever, when I passed it to the call authsess.get('someurl', kwargs), I get the following error...\nTypeError: get() takes 2 positional arguments but 3 were given.\nIt works fine using just the url, but, I will need to add parms (specifically a nextPageToken as if one is returned)\nCan someone please give me a little guidance on what I am missing here? Do I need to encode the url and parms together? or? I have tried various options here, but, just can't get it to work and not sure yet just 'how' to interpret\/implement what I find in the doc for making http requests...\nThanks in advance.","AnswerCount":1,"Available Count":1,"Score":0.1973753202,"is_accepted":false,"ViewCount":131,"Q_Id":61924362,"Users Score":1,"Answer":"OK, sorry if I wasted anyone's time. The answer is that I needed to add 'params=' to the beginning of the parameters string... i.e., authsess.get('someurl', params=kwargs)\nReally wish that had been mentioned in the doc....","Q_Score":0,"Tags":"python,google-cloud-platform,parameters,get,request","A_Id":61925609,"CreationDate":"2020-05-20T22:59:00.000","Title":"GCP authorized session get method using python 3","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I am doing web scraping with python in some pages and I have been blocked from some of them. When I have tried to check it also through the TOR Browser I have seen that I cannot access to the pages neither, so I think that these pages have been able to track all my IP or I dont have well configurated TOR (and I think not cause I have checked my IP address with Chrome and TOR and are different), so, any one knows why? \nAlso, I am trying to do a function or method in my python code to change mi IP automatically. What I have seen is that the best is to do it through the TOR browser (using it as the search engine to get data from pages) but I am not able to make it work. Do you have any recommendation to create this function?\nThank you!","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":1529,"Q_Id":61938512,"Users Score":3,"Answer":"I would expect anti scrape protection to also block visits from known Tor exit nodes. I dont think they know it is you. Some websites hire\/implement state of the art scrape protection services.\nYou could setup your own proxies at friends and family and use a very conservative crawl rate or maybe search for commercial residential proxy offerings.","Q_Score":2,"Tags":"python,web-scraping,ip,tor","A_Id":61940997,"CreationDate":"2020-05-21T15:53:00.000","Title":"How to hide my IP doing web scraping in python?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I wonder if is there any way in Selenium to verify or check if item was saved to sql database via python language ? Thanks for help","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":18,"Q_Id":61987434,"Users Score":0,"Answer":"Not in Selenium itself - you would have to code it, using a Python library to connect to the SQL database you want, and perform the queries required for what you want.","Q_Score":0,"Tags":"python,sql,database,testing,web","A_Id":61988632,"CreationDate":"2020-05-24T14:28:00.000","Title":"Selenium, item sent do database, python","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I have been using Puppeteer to log into a bank website and scrape data of my transactions in a C# application, and it works fine.\nHowever, I am looking for a way to do the same in a simple Azure function. I was able to get it working by wrapping it in a docker instance, but the expense of the Azure Premium Subscription in order to run it makes it not cost-effective.\nI have looked at trying to log into logging into the site and scraping the HTML, except it uses CryptoJS to alter the login credentials, as well as create some unique form properties on the POST message. \nSo my question is, is it possible to pull out the JS from the website to replicate the encoding with python, so I don't require a full headless browser to log in?","AnswerCount":2,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":61,"Q_Id":62014689,"Users Score":0,"Answer":"After further investigation, the site itself is a Javascript Renderable site, which requires requests-html to render, which in itself uses pyppeteer which is a headless chromium, which won't work in Azure anyway. \nSo I'm just going to look at alternative hosting and build it as a web API instead of Azure Functions.","Q_Score":0,"Tags":"python-3.x,azure,cryptojs","A_Id":62032684,"CreationDate":"2020-05-26T04:29:00.000","Title":"Unable to log into encrypted Banking website with Python","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I've been learning HTML and CSS for about a year, I\u2019m a massive noob at Python and I would like someone to clear some things up for me if possible.\nI'm wanting to use a browser extension to automate some work tasks, there's a few good ones that work with JS but I'd like to focus on Python.\nI understand that python would require a interpreter to work, I've ran some basic scripts on my PC but I don't understand how you'd execute scrips externally on the web.\nDo web browsers have a built in interpreter?\nAre there such extensions that exist?\nAnd if so do they have interpreters built in?\nPython is a brand new concept to me and I apologise if this is a stupid question. I aim to get the basics down first and then pick up a Raspberry Pi for some project work.","AnswerCount":1,"Available Count":1,"Score":0.1973753202,"is_accepted":false,"ViewCount":43,"Q_Id":62023510,"Users Score":1,"Answer":"Do web browsers have a built in interpreter?\n\nThey have a JavaScript console in dev tools instead of python","Q_Score":0,"Tags":"python,web,browser,automation,scripting","A_Id":68939635,"CreationDate":"2020-05-26T13:42:00.000","Title":"Noob Question About Python Web Automation","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I am new to webscraping, and I am unsure what is the most efficient way of going about this problem. This project is mostly in Python, but I am open to using other languages for the sake of efficiency.\nImagine some js driven value on a website, something like a real-time visitor count, or the price of some financial asset. that after the website is loaded, constantly updates as new information comes in. What I want is to sub-second sample one such value, but naturally, I don't want to bombard the server with requests, which would ruin accuracy anyway.\nI was thinking of something along the lines of loading a webpage with selenium, or the PyQt webkit, and then running a simple loop that monitors the value and updates a database as changes occur. I am not looking for code, that I can figure out on my own, but I'm unsure if that's the right way to go about solving this problem, or if there are more efficient ways.\nThank you","AnswerCount":2,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":313,"Q_Id":62030579,"Users Score":0,"Answer":"I have done exactly what you are referring to by using Python and Selenium exclusively. I don't see why Python would not be the language to use for your specific use case. The only issue I could see would be if the webpage is constantly (or slowly) changing. \nI came across an issue where the webpage was redesigned and it through off my Xpath. All I had to do was correct the Xpath to reflect the new location and it still worked.\nAnother solution could be to check if their is already an existing API for the webpage you are trying to scrape. If this is the case, you may not have to do any web-scraping, and could just call the API and get your value. This would be the most efficient way, but obviously not possible if there is not API to call.\nSide-Note: I am most comfortable with Python and usually default to it when trying to automate a task. There may be more performant languages to use, but my choice would be Python.","Q_Score":1,"Tags":"javascript,python,web-scraping","A_Id":62030866,"CreationDate":"2020-05-26T20:09:00.000","Title":"How do I monitor a constantly updating value on a website?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I have a dataframe with a single column \"Cntr_Number\" with x no of rows.\nWhat i trying to achieve is using selenium to copy and paste the data into the web page textarea.\nThe constraint is that the web page text area only accept 20 rows of data per submission.\nSo how can i impplment it using while loop or other method.\n\nCopy and paste the first 20 rows of data and click on the \"Submit\"\nbutton \nCopy and paste the next 20 rows of data and click on the\n\"Submit\" button\n\nrepeat the cycle until the last row.\nSorry i dont have any sample code to show but this is what I'm trying to achieve.\nAppreciate if could have some sample code on how to do the implmentation.","AnswerCount":1,"Available Count":1,"Score":0.1973753202,"is_accepted":false,"ViewCount":93,"Q_Id":62042753,"Users Score":1,"Answer":"The better approach will be capture all the the data in a List, Later while pasting it you can check the length of the list, and later iterate through the list and paste the data 20 at a time in the text area. I hope this will solve your problem.","Q_Score":0,"Tags":"python-3.x,pandas,selenium,rpa","A_Id":62043005,"CreationDate":"2020-05-27T12:06:00.000","Title":"How to copy and paste dataframe rows into a web page textarea field","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I wrote a python application and it works fine on my private network.\nThe application uses requests package and constantly fetching data from Yahoo.finance.\nHowever when my computer is connected to my work network (either through VPN or where i am actually @work),\nThe application is not working.\nIs there a way to make it work?\nAsaf","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":680,"Q_Id":62059849,"Users Score":0,"Answer":"Look into the firewall settings, or company list over blocked URLs.\nIt could be that your work has blocked the Yahoo.finance domain or specific applications from having outbound traffic as well.\nYou might also have success with privilege escalation to admin user which could grant you wider access.\nCheck if you can ping Yahoo.finance from a terminal or simply open it manually from a few different browsers (Edge, Chrome, Firefox etc.).\nThen try to see if you from inside your Python application can fetch any URL with the request module, just to troubleshoot.","Q_Score":2,"Tags":"python,python-requests","A_Id":62059901,"CreationDate":"2020-05-28T08:04:00.000","Title":"python application is not working where i use vpn or @work","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I have a Python application that accesses Youtube-Data-API v3.\nAfter the program runs for an hour, it throws an error suggesting the access token has expired.\nHow can I persist the token for a longer period?","AnswerCount":2,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":361,"Q_Id":62071059,"Users Score":0,"Answer":"When you create O-Auth2 credentials, you need to select \"Web App\" which is what I think you're trying to create. (A website, correct?).\nThe \"Desktop-App\" option is for if you want to make a desktop application, not a website.\nDesktop applications and web applications handle redirect uris differently, which is what's causing your issue.","Q_Score":0,"Tags":"python,youtube-data-api,refresh-token","A_Id":62071124,"CreationDate":"2020-05-28T17:39:00.000","Title":"Persist Youtube Data API access token in Python desktop app","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":1,"System Administration and DevOps":0,"Web Development":0},{"Question":"I am working on a web app in which some content is generated in the form of text (and will soon include images as well). I want to add a share button which will allow the user to add this content to their Instagram story. \nSo the flow of this is going to be:\n\n\nUser does something on the website\nThe website generates some content based on user input\nUser clicks on \"Share on Instagram\" and posts it on his\/her story \n\n\nIs there a way to do this using Javascript or an API call like the Twitter share option directly from the HTML ?\nI am using Python on the backend (Flask) and JS on the frontend","AnswerCount":2,"Available Count":1,"Score":1.0,"is_accepted":false,"ViewCount":7688,"Q_Id":62071327,"Users Score":10,"Answer":"OK, lets make this simple.\nThe direct answer as found on the official Facebook developers page is no. You can't trigger an API to create an instagram story, as for now only the \"read-mode\" is supported through API.\nBut as most things in life you can hack your way arround. Let's then copy a funcional method.\nYou create a button that says, \"share on your instagram stories\" and here's what it is gonna do:\n\nTrigger a function on the back-end that creates a screenshot\/image or a video of the content you want to share\nSend that content to the user so it's the last content they have in the gallery\nTrigger the open story camera in Instagram app with this link: instagram:\/\/story-camera. (the last content they have will be the content u sent and will be the first thing they see for posting a story)\n\nIt goes without saying that this will work when the website is opened in a phone and not in a desktop browser as in desktop you can't post stories.\nPS: I'm sure there is a way to send that content directly to the editing in the stories, so you don't have to select the image you sent to the client, but I didn't figure that out yet. Hope this works for you, cheers.","Q_Score":8,"Tags":"javascript,python,html,api,instagram","A_Id":65893635,"CreationDate":"2020-05-28T17:53:00.000","Title":"Share content of a webpage to Instagram story","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I have a query string that will return data successfully if I run it in Postman but if I pull the data via Python3 (http.client.HTTPSConnection), the target returns 'bad request'.\nThe problem:\n\/v4_6_release\/apis\/3.0\/service\/tickets?customFieldConditions=caption=\"Escalated To\" AND value =\"Cloud Operations\"\nI have tried putting +AND+ or %20and%20 or %20AND%20 instead of AND but no success. For example:\n\/v4_6_release\/apis\/3.0\/service\/tickets?customFieldConditions=caption=\"Escalated To\"%20AND%20value=\"Cloud Operations\"\n^ This works fine in postman but returns <hr><p>HTTP Error 400. The request is badly formed.<\/p> when via Python3. \nIs there anything else wrong with the url encoding here?","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":29,"Q_Id":62079936,"Users Score":1,"Answer":"You have to encode all whitespaces in you URL. So the URL should be \/v4_6_release\/apis\/3.0\/service\/tickets?customFieldConditions=caption=\"Escalated%20To\"%20AND%20value=\"Cloud%20Operations\"","Q_Score":0,"Tags":"python,python-3.x","A_Id":62080734,"CreationDate":"2020-05-29T06:30:00.000","Title":"Issue with URL encoding","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I recently installed webdriver-manager 2.5.2 to work with chromedriver for selenium in python. It was working fine, but now suddenly i am getting the below error-\nPermissionError: [Errno 13] Permission denied: 'C:\\Users\\mrudu\\.wdm\\drivers\\chromedriver\\83.0.4103.39\\win32\\chromedriver.exe'\nHow to resolve this issue? I am new to programming so not sure how the permissions work in windows 10 and Pycharm.\nPlease help","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":1678,"Q_Id":62084879,"Users Score":0,"Answer":"It worked when i did this-\nThere is some process that is holding this driver. Close all Chrome drivers, and try to find all processes with name chromedriver.exe.\nSolution provided by SergeyPirogov github owner for webdriver-manager","Q_Score":1,"Tags":"python,selenium,selenium-webdriver,permissions,webdriver-manager","A_Id":62086147,"CreationDate":"2020-05-29T11:27:00.000","Title":"PermissionError: [Errno 13] Permission denied to access selenium python through webdriver-manager","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"Ok, so I decided to use selenium for some code, but it doesn't work. When I put the chromedriver in my path, manually, it still doesn't work. can anyone help. PS: I already pip installed selenium.\nAlso, I watch\/read the other article on stack overflow about this problem, but it didn't help me.","AnswerCount":3,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":325,"Q_Id":62086554,"Users Score":0,"Answer":"If you put chromedriver in the home directory of the user running it, that should work. Would be helpful if you stated in your question what OS\/version you are using.","Q_Score":1,"Tags":"python","A_Id":62086589,"CreationDate":"2020-05-29T12:55:00.000","Title":"Message: 'chromedriver.exe' executable needs to be in PATH. Please see https:\/\/sites.google.com\/a\/chromium.org\/chromedriver\/home","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I crawled and saved the user's website usage lists.\nI want to analyze the results of the crawl, but I wonder how there is a way.\nFirst of all, what I thought was Word Cloud.\nI am looking for a way to track user's personal preferences with user's computer history. \nI want a way to visualize personal tendencies, etc. at a glance. Or I'm looking for a way to find out if there's no risk of suicide or addiction as a result of the search.\nthank you.","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":18,"Q_Id":62100079,"Users Score":0,"Answer":"If you want to visualize data and make analysis on it matplotlib would be good start , again it depends a lot on your data. Matplotlib and seaborn are plotting libraries that are good for representing quantitative data and get some basic analysis at least.","Q_Score":0,"Tags":"python,word-cloud","A_Id":62100135,"CreationDate":"2020-05-30T08:51:00.000","Title":"How to analyze crawl results","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"Yesterday I made a bot and everything was working, today when I try to run my code I have this error:\n\nClientConnectorCertificateError(aiohttp.client_exceptions.ClientConnectorCertificateError:\n Cannot connect to host discordapp.com:443 ssl:True\n [SSLCertVerificationError: (1, '[SSL: CERTIFICATE_VERIFY_FAILED]\n certificate verify failed: certificate has expired (_ssl.c:1108)')]\n\nI am using win10, python 3.8, and pycharm. \nI saw a similar problem on StackOverflow but the solution Is not working for me.","AnswerCount":3,"Available Count":1,"Score":0.1325487884,"is_accepted":false,"ViewCount":8071,"Q_Id":62108183,"Users Score":2,"Answer":"For Mac Users, If they are facing the same problem, You can do the following :-\n\nGo to MacintoshHD -> Applications -> Python3.x Folder.\nDouble click on the \"Install Certificates.command\".","Q_Score":1,"Tags":"python,bots,discord.py","A_Id":71559132,"CreationDate":"2020-05-30T20:17:00.000","Title":"Discord.py bot dont have certificate","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"python request.get always throws status code 200 . This is coming even after adding an user agent in the request.get( ) .\nAs status_code is always 200 , a file from that website is downloaded even if doesn't exists there . \nHow to get correct atatus_code ?","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":98,"Q_Id":62108533,"Users Score":0,"Answer":"200 - \"The HTTP 200 OK success status response code indicates that the request has succeeded.\" This means it worked! Now, on to the next steps... :)","Q_Score":1,"Tags":"python","A_Id":62108608,"CreationDate":"2020-05-30T20:52:00.000","Title":"python request.get always throws status code 200","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I am trying to install \/ run Head-Less Chrome in AWS Lambda but getting the following Error when I test it through Lambda.\n\"errorMessage\": \"Message: Service \/tmp\/bin\/chromedriver unexpectedly exited. Status code was: 127\\n\",\n \"errorType\": \"WebDriverException\"\nI am using ChromeDriver 83.0.4103.39 for Linux.\nSteps performed in Python:\n\nCreated \/tmp\/bin in Lambda\nCopy Chrome Driver to \/tmp\/bin\nChanged the permission to 0777\n\nPython Code:\nThrowing error on the following line -\ndriver = webdriver.Chrome(chrome_options=chrome_options, executable_path= \"\/tmp\/bin\/\" + chrome_linux_driver)\nCode is working fine when I am running under windows with same windows Chrome driver.\nAny help is really appreciated.","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":1096,"Q_Id":62110569,"Users Score":0,"Answer":"The error might come from the fact that the chromedriver version and the headless chromium binary you are using aren't the same so make sure to download the same version for both\nanother reson could be that you are not using linux chromedriver and headless chromium and since lambda work on linux the code won't work\nhopefully this was useful.","Q_Score":8,"Tags":"python,selenium,aws-lambda,selenium-chromedriver","A_Id":68178317,"CreationDate":"2020-05-31T01:31:00.000","Title":"Headless Chrome Python 3.8 on AWS Lambda - unexpectedly exited. Status code was: 127","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"Im building tools for a business I am starting, and one of the tools I need is a live reader of scrap prices to put into my website as well as deciphering overhead\/profit. How would I take that information and then put it into a live data table?\nVery amateur programmer, but better to do it myself.","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":23,"Q_Id":62141465,"Users Score":0,"Answer":"You can use flask and Beautiful Soup. With the flask you will be able to save the data after obtaining the information from the site after manipulating the html with beautiful soup.","Q_Score":0,"Tags":"python,interpretation","A_Id":62142140,"CreationDate":"2020-06-01T21:44:00.000","Title":"How do I extract data from a website for a data table using python?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":1,"System Administration and DevOps":0,"Web Development":0},{"Question":"How do I enable method autocompletion for discord.py in PyCharm? Until now I've been doing it the hard way by looking at the documentation and I didn't even know that autocomplete for a library existed. So how do I enable it?","AnswerCount":2,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":1231,"Q_Id":62159638,"Users Score":1,"Answer":"The answer in my case was to first create a new interpreter as a new virtual environment, copy over all of the libraries I needed (there is an option to inherit all of the libraries from the previous interpreter while setting up the new one) and then follow method 3 from above. I hope this helps anyone in the future!","Q_Score":1,"Tags":"python,python-3.x,pycharm,discord,discord.py","A_Id":62162550,"CreationDate":"2020-06-02T19:12:00.000","Title":"How to enable PyCharm autocompletion for imported library (Discord.py)","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I'm currently making a small hobby project where I use a Raspberry Pi Zero to control a set of RGB LED's.\nI've got a basic python app so far that is built on Flask that allows me to set up a web server so that the LED colours can be set remotely.\nMy main concern is that I'm self-taught as far as programming goes and I don't know squat about security.\nThe plan is essentially to be sending any port 80 traffic to the raspberry pi on my home network and give friends my IP. I may eventually get a domain name to simplify things.\nShould I have any security concerns when I set this up live full-time? I don't know if it's possible to access my private network via the raspberry pi or if I'm just being paranoid.","AnswerCount":2,"Available Count":1,"Score":0.0996679946,"is_accepted":false,"ViewCount":199,"Q_Id":62162817,"Users Score":1,"Answer":"You can try putting your Raspberry PI on separate vlan and put your home devices on another vlan. Please note you need a router which supports vlan and configure it in a way that the both vlans cant talk to each other \nAlso, try using HTTPS for your webserver and don't run the webserver process as root user. If you want to go more crazy you can put a firewall.\nThese are generic suggestions for hardening the security for any web app.","Q_Score":1,"Tags":"python,security,flask,raspberry-pi","A_Id":62162893,"CreationDate":"2020-06-02T23:12:00.000","Title":"Raspberry Pi as a web server security","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I have a python script that hits the \/v3\/surveys?pagenum end point. My plan was to calculate the number of responses in the Json and the number of pages (rounded up from 50\/page) and compare this with data already brought in from that endpoint. My thinking was if the number of responses in the DB and the Site differed, I could just bring in the last page or pages from the SurveyMonkey API\nThe code works but what I've found is the new Surveys don't necessarily appear on the last pages They can be on any page. How do I just bring in new Content without re downloading everything. Is this the case with every other endpoint as well? I've looked through the api on the Survey Monkey Site but didn't see anything geared towards getting new content\nI have a limited number of api calls and don't really want to download everything each time as this data changes frequently. Any guidance would be appreciated.","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":229,"Q_Id":62163671,"Users Score":0,"Answer":"Register a webhook. It will tell you when there is a new response.","Q_Score":1,"Tags":"python,surveymonkey","A_Id":62272844,"CreationDate":"2020-06-03T01:09:00.000","Title":"Survey Monkey - Retrieve new Responses only via API","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I created a aws windows instance (linux is sadly no option in this particular case) on which i deployd a python script using flask-restful. \nI added inbount traffic rules to allow http traffic on port 80. \nThe Flask api is being hostet on host=0.0.0.0, port=80 and debug is False. \nThe api is reachable via browser on the server running it.\nSadly im not able to reach the api from external servers. I thought it might be a internal issue between public and internal ip so i tryd reverse proxing that with nginx with no result as well. \nDoes somebody have an idea what i could do.","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":35,"Q_Id":62184248,"Users Score":1,"Answer":"Based on the comments, the issue was windows firewall. \nThe solution was to allow the connection in the firewall.","Q_Score":1,"Tags":"python,windows,amazon-web-services,flask,firewall","A_Id":62184660,"CreationDate":"2020-06-03T23:12:00.000","Title":"Allow traffic on AWS windows instance with flask restful","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I searched about this a lot but I could not find anything. I haven't tried much with python sockets, just made a system where a host and a client can message each other. What I want to know is when a client is connected to a host via python socket, does it make a connection outside the program too? Like can we play any lan game after being connected via sockets for example ??","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":212,"Q_Id":62184433,"Users Score":0,"Answer":"The sockets are not specific to any language; they are the logical endpoint offered by the underlying API to the userspace. You can create TCP\/IP sockets in any language which supports networking.\n\"What I want to know is when a client is connected to a host via python socket, does it make a connection outside the program too?\"\nThat depends on which layer you are looking at. Consider the first three layers of the TCP\/IP model from the top (Application, Transport, Network).\n\nApplication Layer: All the underlying of the internet are abstracted from the Application layer. Now, if you see whether a connection is established outside your program, that purely depends on what protocol they are following. If you consider HTTP protocol, it is mostly stateless or connectionless (If cookies are ignored). But if you look at SSL, some handshakes happen, and the session state is maintained in both the client-side and server-side.\nTransport Layer: The two famous protocols in this layer are TCP and UDP. With TCP before any data transfer, a TCP connection is established by performing a three-way handshake. But in the case of UDP, no such term like a connection is defined. \nInternet Layer: Though the well known and most commonly used network architecture is the Internet, there are some other architectures like ATM CBR and ATM ABR. The Internet architecture falls under the category of datagram networks, meaning there is no state maintained, the link-layer switches, or routers just forward the packets. In contrast, ATM architecture falls under the category of Virtual-circuit networks. Each connection at the network layer is monitored to deliver the services. e.g., in the case of ATM CBR, the connections are monitored to deliver a constant bit rate service, and in the case of ATM ABR, the connections are monitored to deliver a minimum bit rate service.\n\nHere I just summarized the connection setup for the first three layers, and the same things can happen with the other layers as well. \nTo conclude, whether the connection is established outside your programs? It depends on all these above factors, what application layer protocol you are using, what transport layer protocol you are using, and what is the underlying network architecture... etc. And you can also see a scenario where the connection establishment happens at the network layer but not in the application layer, that's where all these network abstractions come. You can blissfully ignore the underlying details!","Q_Score":0,"Tags":"python,sockets,networking,connection,vpn","A_Id":62190050,"CreationDate":"2020-06-03T23:31:00.000","Title":"Does python socket basically connect host and client over internet like a computer to computer VPN?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I'm trying to retrieve versions of all packages from specific index. I'm trying to sending GET request with \/user\/index\/+api suffix but it not responding nothing intresting. I can't find docs about devpi rest api :( \nHas anyone idea how could I do this?\nBest regards, Matt.","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":218,"Q_Id":62194793,"Users Score":2,"Answer":"Simply add header Accept: application\/json - it's working!","Q_Score":2,"Tags":"python,python-3.x,devpi","A_Id":62222635,"CreationDate":"2020-06-04T12:37:00.000","Title":"Devpi REST API - How to retrieve versions of packages","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"Can I get the clarification for the below question?\nGiven that 2 nodes one is R\/W and another was a backup node, the backup node can have out of sync data, I have give few APIs like load, extract metadata, delete, update.\nCan any tell me that what is algorithm can be used that the end of the algorithm both nodes have the same data","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":108,"Q_Id":62198740,"Users Score":0,"Answer":"There are a few different ways you can do this.\n\nEvery write to the R\/W node you write to the backup node before returning.\n\nPros\n\nLow complexity\nGood if your application issues way more writes than reads.\nConsistent backup\n\nCons\n\nSlow writes\n\n\nWith every write to the to the R\/W node, asynchronously write to the backup node (you can return the write to the R\/W node before the writing to the backup completes).\n\nPros\n\n\nFaster writes\n\nCons\n\n\nData on backup may be out of sync for a few miliseconds\/seconds.\n\n\n\nIt is normally okay for your backup to have out of sync data for a few seconds. Depending on your application requirements, it won't be a big deal if you miss the last few milliseconds of data. If strong consistency between the backup and your R\/W node is absolutely necessary then go for option 1.","Q_Score":0,"Tags":"python,algorithm,data-structures,distributed-system","A_Id":62289748,"CreationDate":"2020-06-04T15:45:00.000","Title":"How to sync data between master and slave node in distributed systems","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I am using the pho MQTT client library successfully to connect to AWS. After the mqtt client is created, providing the necessary keys and certificates is done with a call to client.tls_set() This method requires file paths to root certificate, own certificate and private key file.\nAll is well and life is good except that I now need to provide this code to external contractors whom should not have direct access to these cert and key files. The contractors have a mix of PC and macOS systems. On macOS we have keychain I am familiar with but do not know how to approach this with python - examples\/library references would be great. On the PC I have no idea which is the prevalent mechanism to solve this.\nTo add to this, I have no control over the contractor PCs\/Macs - i.e., I have no ability to revoke an item in their keychain. How do I solve this?\nSorry for being such a noob in security aspects. No need to provide complete examples, just references to articles to read, courses to follow and keywords to search would be great - though code examples will be happily accepted also of course.","AnswerCount":1,"Available Count":1,"Score":0.1973753202,"is_accepted":false,"ViewCount":22,"Q_Id":62201091,"Users Score":1,"Answer":"Short answer: you don't.\nLonger answer:\nIf you want them to be able connect then you have no choice but to give them the cert\/private key that identifies that device\/user.\nThe control you have is issue each contractor with their own unique key\/cert and if you believe key\/cert has been miss used, revoke the cert at the CA and have the broker check the revocation list.\nYou can protect the private key with a password, but again you have to either include this in the code or give it to the contractor.\nEven if the contractors were using a device with a hardware keystore (secure element) that you could securely store the private key in, all that would do is stop the user from extracting the key and moving it to a different machine, they would still be able to make use of the private key for what ever they want on that machine.\nThe best mitigation is to make sure the certificate has a short life and control renewing the certificate, this means if a certificate is leaked then it will stop working quickly even if you don't notice and explicitly revoke it.","Q_Score":0,"Tags":"python,security,mqtt,tls1.2","A_Id":62203164,"CreationDate":"2020-06-04T17:52:00.000","Title":"How to prevent direct access to cert files when connecting MQTT client with Python","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I have a network composed of nodes and edges' weights. I know all of them. I get a \"packet\" which I know is travelling from A to B currently on node C. How to get minimal path it can be presently traversing (next and previous nodes)?","AnswerCount":2,"Available Count":2,"Score":0.0996679946,"is_accepted":false,"ViewCount":78,"Q_Id":62205637,"Users Score":1,"Answer":"Update:\nDespite the claim in the accepted answer mine would work. Because\n\nIf there is only one shortest path from A to B, then C is on the shortest path. The accepted answer itself clearly claims it with the phrases:\n\nThe shortest path A -> C -> B must have the shortest A->C and C->B subpaths.\n\nand\n\nIt is trivial to prove.\n\nSo the only possibility for the C not to be on the shortest path is if there are several of them. The accepted answer doesn't address this issue at all. So it will give if not wrong but potentially misleading result. While here is clearly stated that\n\n\nyou need to find all shortest paths\n\nWhich approach is simplest or fastest is totally another matter. Because performance will depend on specific implementation of a specific algorithm and on the net topology. It's not necessary true that finding one long path is slower than finding two smaller ones (without caching... and cache will add complexity).\nAnyway, checking the fact that there are several shortest paths is more general approach as it allows to make predictions on the implementation of the algorithm used to calculate the packet path.\nAnd if, by any chance, @PakUula meant that the shortest path between three points A->C->B is not necessary the same as the shortest path between two points A->B then his answer is incorrect. Exactly because path A->C->B is not the shortest between two points, and as a result C did not travel the shortest path, so we have no idea about the algorithm used to construct the path of the packet and\/or our knowledge about net's weights is wrong. In that case see theoretical exercise and the end of this answer.\n\n\nThe answer itself:\nI suppose we assume that packets always trying to follow the shortest path.\nAs a packet, before it starts it's journey from A to B, needs to know how to make it first move it already needs to know the shortest path. So you just find this shortest path between A and B, and after that you can find where is the C node on it.\nShortest path can be found with, for example, the Dijkstra\u2019s algorithm, or some other. Them you can find on the internet with a request similar to \"graph shortest path\".\nAs you've said that edges' weights are known. Then there is nothing more to it. The C node will always be on the shortest path. Because there is only one shortest path... Well actually - no. There can be several paths with the same weight.\nSo you task is a little bit different - you need to find all shortest paths. And then you'll need to find which of then include C node. Again, there can be several paths with node C. So you can't answer your question with 100% certainty in these cases.\n\nAnd just as a theoretical exercise: what if there is no C node on any of the shortest paths? That means that maybe there were some \"obstacles\" on the way and the plan had to be changed.\nSo what can be done about it?\nIt's quite easy to find the next node(s): just find the shortest path(s) between C and B. But still the answer won't always be definitive, because of the possibility of several routes of same weight.\nBut it becomes impossible to find the previous node without info about \"obstacles\". Because depending on them the packet can come from any direction. That essentially means that weights of the graph have changed and you don't know them.","Q_Score":0,"Tags":"python,networking,path-finding","A_Id":62357716,"CreationDate":"2020-06-04T23:13:00.000","Title":"How to obtain a path we are currently \"on\" the graph while knowing all graph nodes and weights?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I have a network composed of nodes and edges' weights. I know all of them. I get a \"packet\" which I know is travelling from A to B currently on node C. How to get minimal path it can be presently traversing (next and previous nodes)?","AnswerCount":2,"Available Count":2,"Score":1.2,"is_accepted":true,"ViewCount":78,"Q_Id":62205637,"Users Score":3,"Answer":"The shortest path A -> C -> B must have the shortest A->C and C->B subpaths. It is trivial to prove. Note. The shortest path A -> B might not traverse C. Therefore the answer by @x00 might fail if you don't have apriory guarantee that C is selected correctly from the shortest path.\nThe previous node is the last segment on the path A->C and the next node is the first segment of the path C->B.\nThe shortest paths could be computed using Dijkstra algorithm.","Q_Score":0,"Tags":"python,networking,path-finding","A_Id":62357953,"CreationDate":"2020-06-04T23:13:00.000","Title":"How to obtain a path we are currently \"on\" the graph while knowing all graph nodes and weights?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"How to receive data when clients are behind NAT e.g. router?\nI have simple app that can send data and receive data, it works great on LAN.\nI want it to be decentralized. Without need to manually configure the router.","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":49,"Q_Id":62214933,"Users Score":0,"Answer":"You should find some kind of setting in your router, that allows you to route incoming requests to specific ports, to chosen IPs in your local network.\nThe keyword you are looking for is Port Forwarding.\nSince you now specified, that you do not want change anything in the router, the only possibility is, that the person behind the NAT establishes a connection with the person not behind a NAT. The other way is not possible without accessing the router. If both, are behind a NAT, it is impossible.","Q_Score":0,"Tags":"python,networking","A_Id":62214965,"CreationDate":"2020-06-05T12:03:00.000","Title":"How to receive data when clients are behind NAT e.g. router - Python","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I am trying to string together lambda functions. The first lambda function is on a 30 minute timer to scrape and put the data in a S3 bucket, the next lambda function retrieves and parses that data and puts it in a seperate S3 bucket, and the last function performs analysis on that data and sends the user (in this case myself) an email of the results via pythons smtplib module.\nInstead of having the last two lambda functions running on timers, I want the second function to be triggered when the first function is done, and the last function to be triggered when the second function is done. As well as, deleting the two folders in the first S3 bucket and the contents in the second S3 bucket to save on memory and processing time.\nIs there a way to do this totally in the AWS web interface rather than rewrite the python code I already have?","AnswerCount":5,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":732,"Q_Id":62220738,"Users Score":0,"Answer":"All the answers here work. To summarize, you could:\n\nTrigger the lambdas when a file arrives in an S3 bucket\nUse Step Functions co-ordinate the 3 lambda functions, one after another\nUse Lambda destinations (on success) to orchestrate the workflow\n\nAll of these accomplish what you wish for, and here's some extra info.\n\nFor the S3 option you can specify which events trigger the lambda (e.g. PutObject), and even the prefix and suffix of the file(s).\nFor Step functions, this is my favorite options. It's an expensive option (typically 5-10x more than lambda) but, it actually orchestrate everything, and the view you get from the console is amazing. It's very easy to triage issues if they occur.\nFor Lambda destinations, you can also set a onFailure mode, so that the you can get alerted if the function fails. onSuccess will of course lead to the next function in question.\n\nHope that helps.","Q_Score":2,"Tags":"python,amazon-web-services,amazon-s3,aws-lambda","A_Id":62231258,"CreationDate":"2020-06-05T17:11:00.000","Title":"AWS Lambda function completion triggers another lambda function","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I wonder and i've been trying everything to get my program with python sockets to work remotely. When I say remotely is like, I run the server in my computer and my friend at his house can run the client and connect to my server. Is this possible without using Hamachi? Pls let me know, because I'm already dying by trying so many things and installing and uninstalling programs.","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":41,"Q_Id":62232490,"Users Score":0,"Answer":"You have to activate port forwarding on your router so that everything that comes on the specific port is forwarded to your local IP (and the port should be open).","Q_Score":0,"Tags":"python,sockets,ip,remote-server,portforwarding","A_Id":62326793,"CreationDate":"2020-06-06T13:33:00.000","Title":"Socket ssh connections remotely?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I am trying to run a simulation with Sumo using TRACI in python as a way to perform some actions in the simulation. What I am doing is basically using netconvert to extract a portion of the map.\nnetconvert --keep-edges.in-boundary minX,minY,maxX,maxY -s large.net.xml -o small.net.xml\nSo basically I am trying to perform an evaluation in the original net(large.net.xml) and in some point of the simulation transfer a vehicle from the original simulation to my small network(small.net.xml) with the option:\nconn2.vehicle.add(vehID='0',routeID='R0',depart=time,departPos=conn1.vehicle.getLanePosition('0'),departSpeed=conn1.vehicle.getSpeed('0'))\nwhere time is the simulation time in my original simulation(conn1). This should take the vehicle in simulation conn1 and start that vehicle in conn2(Simulation with small.net.xml) in that exact same position that it was in conn1, but it seems that my cropped network has different sizes regarding that lane as the vehicle start in a position a little different. Is there a way to avoid these changes in the size of the lanes? I have tried this with transferring the vehicle to the same original network in conn2 and it appears in the desired position.","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":111,"Q_Id":62240620,"Users Score":0,"Answer":"The easiest solution is probably to make the cropped network a little larger (containing all edges directly adjacent to the current cropped network) in order to ensure that the edges of interest have still the same junction geometries on the beginning and the end node.","Q_Score":0,"Tags":"python,sumo,network-traffic,traffic-simulation","A_Id":62286901,"CreationDate":"2020-06-07T03:50:00.000","Title":"Simulation in SUMO with netconvert","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I have a project and i was wondering how can i make a button in my web that when it is being clicked it can display a string in my python terminal\nThank you in advance","AnswerCount":2,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":172,"Q_Id":62255012,"Users Score":0,"Answer":"You need to create an http server in your Python, and call it with fetch in JavaScript. You can pass data in the query parameters.","Q_Score":0,"Tags":"python,html","A_Id":62255040,"CreationDate":"2020-06-08T04:26:00.000","Title":"How can i make a button in my web that when it is being clicked. it can send a data in my python script","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"How should I save the path of a image uploaded in html page so that the path can be used as input to the python script","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":41,"Q_Id":62274672,"Users Score":0,"Answer":"Using Local Storage (require HTML 5) or cookie will solve your problem","Q_Score":0,"Tags":"javascript,python,html","A_Id":62274748,"CreationDate":"2020-06-09T03:44:00.000","Title":"is there a way to save the path of the file uploaded in html page","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I have received a link with .ipynb link. I am new to Python and Jupyter and I need to open the link to work on the details inside.\nThe link opens in my internet browser and I couldn't properly see the contents and bring it in to a Jupyter notebook. \nCould anyone please give me a tip how to handle such links for Python\/Jupyter?","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":100,"Q_Id":62285116,"Users Score":0,"Answer":"Adding to Vinzee's answer: jupyter notebook starts in your home folder and you can't move up from there; only down into subfolders. Open jupyter to see what folder it starts in, and make sure that you put the .ipynb file in that folder or one of its subfolders.","Q_Score":0,"Tags":"python,jupyter-notebook","A_Id":62285954,"CreationDate":"2020-06-09T14:36:00.000","Title":"How to bring a web browser with .ipybn link opened in my Jupyter?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":1,"System Administration and DevOps":0,"Web Development":1},{"Question":"I have just started python. I am trying to web scrape a website to fetch the price and title from it. I have gone through multiple tutorial and blog, the most common libraries are beautiful soup and scrapy. My question is that is there any way to scrape a website without using any library?\nIf there is a way to scrape a website without using any 3rd party library like beautifulsoup and scrapy. It can use builtin libraries\nPlease suggest me a blog, article or tutorial so that I can learn","AnswerCount":2,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":417,"Q_Id":62287967,"Users Score":2,"Answer":"Instead of using scrapy you can use urllib.\nInstead of beautifulsoup you can use regex.\nBut scrapy and beautifulsoup do your life easier.\nScrapy, not easy library so you can use requests or urllib.","Q_Score":1,"Tags":"python,web-scraping,beautifulsoup,scrapy,libraries","A_Id":62288047,"CreationDate":"2020-06-09T16:59:00.000","Title":"Scrape websites with python","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I want to know why we use from module import module?\nFor example from BeautifulSoup import BeautifulSoup instead of only import BeautifulSoup.\nAre both of those different?\nAnd if yes, then how?","AnswerCount":2,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":63,"Q_Id":62289256,"Users Score":2,"Answer":"Yes, they are different.\nThere are several ways to import a module\n\nimport module\nThe module is imported, but all its functions, classes and variables remain within the name space of 'module . To reference them, they must be prepended with the module name:\nmodule.somefunction() module.someclass() module.somevariable = 1\nimport module as md\nThe module is imported, but given a new name space 'md'. To reference the functions, classes and variables, they must be prepended with the new namespace:\nmd.somefunction() module.someclass() module.somevariable = 1\nThis keeps the namespaces separated, but provides a shorthand notation which makes the code more readable.\nfrom module import *\nAll functions, classes and variabes from the module are imported into the current namespace as if they were defined in the current module. They can be called with their own name:\nsomefunction() someclass() somevariable = 1\nThe disadvantage is that there might be overlapping names!\nfrom module import something\nfrom module import something, somethingelse\nThis imports only 'something' (and 'somethingelse') a function, a class, a variable into the current namespace.\nThis is not only more efficient, but it also reduces the risk of overlapping names.\n\nA module name and a class inside the module may have the same name. Don't let that confuse you:\nimport BeautifulSoup reference: BeautifulSoup.BeautifulSoup()\nfrom BeautifulSoup import BeautifulSoup reference: BeautifulSoup()\nimport BeautifulSoup as bs reference: bs.BeautifulSoup()","Q_Score":0,"Tags":"python,python-3.x,python-requests","A_Id":62289410,"CreationDate":"2020-06-09T18:12:00.000","Title":"Python packages import","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":1,"System Administration and DevOps":0,"Web Development":0},{"Question":"I have a script that is throwing a CERTIFICATE_VERIFY_FAILED on one of the PCs available to me, but not the other. The certificate was updated last week. How do I check if there is a cached version of the certificate somewhere, and get it refreshed? \nI tried manually updating the .pem file in the lib\/site-packages\/certifi folder but that doesn't seem to make a difference.","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":53,"Q_Id":62313105,"Users Score":0,"Answer":"Try running command pip install certifi","Q_Score":0,"Tags":"python,certificate,urllib2","A_Id":62316490,"CreationDate":"2020-06-10T21:03:00.000","Title":"Python: CERTIFICATE_VERIFY_FAILED, Windows","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I'm trying to get Python and selenium to store the \"1292\" in the following html script and cant figure out why it won't work. I've tried using find_element_by_xpath as well as placing a wait before it and I keep getting this error \"Message: no such element: Unable to locate element:\"\nAny ideas on how else I can accomplish this? Thanks\n<text x=\"76.76666666666667\" y=\"141.42345774407445\" style=\"font-family:"Lucida Grande", "Lucida Sans Unicode", Verdana, Arial, Helvetica, sans-serif;font-size:12px;color:#4572A7;font-size:11px;fill:#4572A7;\" text-anchor=\"middle\" zIndex=\"1\">\n <tspan x=\"76.76666666666667\">1292<\/tspan>\n <\/text>","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":20,"Q_Id":62316046,"Users Score":0,"Answer":"You can try:\ndriver.find_element_by_xpath(\"\/\/tspan[text()='1292']\").text\nto obtain the string \"1292\".","Q_Score":0,"Tags":"python,html,selenium","A_Id":62316368,"CreationDate":"2020-06-11T02:21:00.000","Title":"Need help getting data using Selenium","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I want to develop a python script to retrieve data from the Primavera API for my firm. I do not have any idea where to start.I saw a couple of articles where getting data from API's was fairly simple task using few lines of python code. However this doesn't seem to be the case here.\nCan anyone please help!","AnswerCount":1,"Available Count":1,"Score":0.1973753202,"is_accepted":false,"ViewCount":1383,"Q_Id":62326127,"Users Score":1,"Answer":"Primavera P6 API uses Java, so you have 3 choices:\n\nUse Python Java bridge\nWrite the code in Java\nUse Python to connect to Primavera P6 Web Services (another Primavera integration technology)","Q_Score":0,"Tags":"python-3.x,primavera,product-management","A_Id":62350601,"CreationDate":"2020-06-11T13:58:00.000","Title":"Accessing the Primavera data using Primavera Integration API and a python script","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":1,"System Administration and DevOps":0,"Web Development":0},{"Question":"from selenium import webdriver\ndriver = webdriver.Firefox(executable_path=\"C:\\Users\\yhtiy\\Documents\\chromedriver_win32\\geckodriver.exe\")\ndriver.get(\"https:\/\/instagram.com\")\nAnd the output is: \n\nProcess finished with exit code 0\n\nI have tried to change the browser to Chrome and Firefox but the issue remains. I have checked that Selenium is installed.","AnswerCount":2,"Available Count":2,"Score":0.0,"is_accepted":false,"ViewCount":107,"Q_Id":62329117,"Users Score":0,"Answer":"webdriver.Firefox(executable_path=\"C:\/Users\/yhtiy\/Documents\/chromedriver_win32\/geckodriver.exe\")\nThis should work","Q_Score":0,"Tags":"python-3.x,selenium,selenium-webdriver","A_Id":62329403,"CreationDate":"2020-06-11T16:33:00.000","Title":"Selenium webdriver not opening neither Chrome nor Firefox browser, I downloaded chromedriver, geckodriver in pycharm","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"from selenium import webdriver\ndriver = webdriver.Firefox(executable_path=\"C:\\Users\\yhtiy\\Documents\\chromedriver_win32\\geckodriver.exe\")\ndriver.get(\"https:\/\/instagram.com\")\nAnd the output is: \n\nProcess finished with exit code 0\n\nI have tried to change the browser to Chrome and Firefox but the issue remains. I have checked that Selenium is installed.","AnswerCount":2,"Available Count":2,"Score":0.0,"is_accepted":false,"ViewCount":107,"Q_Id":62329117,"Users Score":0,"Answer":"You are probably running a script, when the script finishes, the process exits, nothing is wrong. If you want to see what's being obtained, save driver.page_source to a file or print it out.","Q_Score":0,"Tags":"python-3.x,selenium,selenium-webdriver","A_Id":62329526,"CreationDate":"2020-06-11T16:33:00.000","Title":"Selenium webdriver not opening neither Chrome nor Firefox browser, I downloaded chromedriver, geckodriver in pycharm","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"thanks for reading! I've created a python program using selenium that uploads a file onto a website automatically. I've placed the .py file and chrome driver in a folder on my desktop and set my executable path to \"chromedriver.exe\". When I run the program from IDLE, the program runs without any issue. When I try to run it from the command line, however, the program cannot find the chromedriver. I've placed them in that folder because I'd like to share the thing with my co-workers and plan on using a VBA script from an Excel to call the python program and pass some variables. My assumption is that since all the files are in the same folder, path would not be an issue.\nHow do I make it so that all path dependencies are eliminated?","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":226,"Q_Id":62355838,"Users Score":0,"Answer":"You should try to specify full path to chromedriver.exe like C:\\users\\username\\Desktop\\projectfolder\\chromedriver.exe","Q_Score":0,"Tags":"python,selenium,selenium-chromedriver,filepath","A_Id":62359791,"CreationDate":"2020-06-13T05:17:00.000","Title":"Python program can't find chromedriver from when run from cmd prompt","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"Stuck on a single step and doc searching isn't helping...\nIn Connect I have:\n\nStore customer input : \"What's your age?\"\nSet contact attribute: age -> system -> stored customer input\n\nI know the above works because I've had a Play Prompt read back the attribute. \nI want Lambda to handle somethings with the age. I setup a function, I attached it to the Connect instance. I added it to the flow with a simple value pair response and that works (the Play Prompt plays back the non-dynamic response from Lambda).\nWhen I try to make it dynamic breaks.\nMy understanding is that I should be able to get to this attribute in Lambda by using the passed JSON.... so I'm doing this:\nage = event[\"Details\"][\"ContactData\"][\"Attributes\"][\"age\"]\nConnect starts saying it has trouble accessing the attribute and ends the call. What am I doing wrong? The rest of the Python code is fine because if I hard set age (age = 24) the code runs fine.","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":175,"Q_Id":62363320,"Users Score":0,"Answer":"There was nothing wrong my code as it pertains to the lambda \/ connect integration... it was with something that was just causing an error. I started monitoring the function in cloudwatch and that helped!","Q_Score":0,"Tags":"python-3.x,amazon-web-services,aws-lambda","A_Id":62411434,"CreationDate":"2020-06-13T17:22:00.000","Title":"Passing attribute from AWS Connect to Lambda using Python","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I get this error in discord.py bot what this means - error details 400 Bad Request (error code: 50035): Invalid Form Body\nIn content: Must be 2000 or fewer in length.\nThe fun fact is that i get this sometimes and sometimes not in the same code .","AnswerCount":2,"Available Count":2,"Score":0.0,"is_accepted":false,"ViewCount":4698,"Q_Id":62381210,"Users Score":0,"Answer":"So, even if you have Nitro and can send 4k, that doesn't extend to your tuppers? Because I just sent a pretty long RP Post in a server and got this error code. I figured that was the issue. The message sent, just from me and not my tupper. It's not a big deal, I'm just new to the whole Nitro thing and am still not sure how everything words lol","Q_Score":1,"Tags":"python,discord.py","A_Id":69421980,"CreationDate":"2020-06-15T04:10:00.000","Title":"error details 400 Bad Request (error code: 50035): Invalid Form Body In content: Must be 2000 or fewer in length","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I get this error in discord.py bot what this means - error details 400 Bad Request (error code: 50035): Invalid Form Body\nIn content: Must be 2000 or fewer in length.\nThe fun fact is that i get this sometimes and sometimes not in the same code .","AnswerCount":2,"Available Count":2,"Score":0.0996679946,"is_accepted":false,"ViewCount":4698,"Q_Id":62381210,"Users Score":1,"Answer":"2K is the maximum length of characters you can send. It's the default restriction in the discord. It would be due to you are sending a list or dictionary containing more than the recommended limit.","Q_Score":1,"Tags":"python,discord.py","A_Id":62381721,"CreationDate":"2020-06-15T04:10:00.000","Title":"error details 400 Bad Request (error code: 50035): Invalid Form Body In content: Must be 2000 or fewer in length","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I have an Excel sheet where one column has XMLs (each row is a different xml). I am trying to use Pyspark and spark-xml to parse these, by doing df = spark.read.format('xml').options(rowTag = 'book').load(___). \nThe load works fine when you specify an xml file, but is it possible to read in the Excel sheet and loop in those xmls to be parsed without having to convert each one to its own xml file?","AnswerCount":1,"Available Count":1,"Score":0.1973753202,"is_accepted":false,"ViewCount":128,"Q_Id":62388441,"Users Score":1,"Answer":"You need to first read the Excel file as a Spark DataFrame. Then you can further parse a String column containing XML with the from_xml function. That gives a new column with the parsed data from the XML column.","Q_Score":2,"Tags":"python,excel,xml,pyspark","A_Id":63676518,"CreationDate":"2020-06-15T12:41:00.000","Title":"Spark XML - Use XMLs from Excel","Data Science and Machine Learning":1,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I am using selenium python and was wondering how one effectively scrapes over 50,000 data points in under 24 hours. For example, when I search for products on the webpage 'insight.com' it takes about 3.5 seconds for the scraper to search for the product and grab its price, meaning that with large amounts of data it takes the scraper several days. A part from using threads to simultaneously look up several products at the same time, how else can I speed up this process?\nI only have one laptop and will have to simultaneously scrape six other similar websites so therefore do not want too many threads and the speed at which the computer operates will slow down significantly.\nHow do people achieve to scrape large amounts of data in such short periods of time?","AnswerCount":2,"Available Count":2,"Score":0.1973753202,"is_accepted":false,"ViewCount":130,"Q_Id":62438727,"Users Score":2,"Answer":"If you stop using the selenium module, and rather work with a much more sleek and elegant module, like requests, you could get the job done in a matter of mere minutes.\nIf you manage to reverse engineer the requests being handled, and send them yourself, you could pair this with threading to scrape at some 50 'data points' per second, more or less (depending on some factors, like processing and internet connection speed).","Q_Score":2,"Tags":"python,selenium,web-scraping","A_Id":62438810,"CreationDate":"2020-06-17T21:33:00.000","Title":"How to scrape over 50,000 data points from dynamically loading webpage in under 24 hours?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I am using selenium python and was wondering how one effectively scrapes over 50,000 data points in under 24 hours. For example, when I search for products on the webpage 'insight.com' it takes about 3.5 seconds for the scraper to search for the product and grab its price, meaning that with large amounts of data it takes the scraper several days. A part from using threads to simultaneously look up several products at the same time, how else can I speed up this process?\nI only have one laptop and will have to simultaneously scrape six other similar websites so therefore do not want too many threads and the speed at which the computer operates will slow down significantly.\nHow do people achieve to scrape large amounts of data in such short periods of time?","AnswerCount":2,"Available Count":2,"Score":0.0996679946,"is_accepted":false,"ViewCount":130,"Q_Id":62438727,"Users Score":1,"Answer":"Find an API and use that!!! The goal of both web scraping and APIs is to access web data.\nWeb scraping allows you to extract data from any website through the use of web scraping software. On the other hand, APIs give you direct access to the data you\u2019d want.\nAs a result, you might find yourself in a scenario where there might not be an API to access the data you want, or the access to the API might be too limited or expensive.\nIn these scenarios, web scraping would allow you to access the data as long as it is available on a website.\nFor example, you could use a web scraper to extract product data information from Amazon since they do not provide an API for you to access this data. However, if you had access to an API, you could grab all the data you want, super, super, super fast!!! It's analogous to doing a query in a database on prem, which is very fast and very efficient, vs. refreshing a webpage, waiting for ALL elements to load, and you can't use the data until all elements have been loaded, and then.....do what you need to do.","Q_Score":2,"Tags":"python,selenium,web-scraping","A_Id":63663707,"CreationDate":"2020-06-17T21:33:00.000","Title":"How to scrape over 50,000 data points from dynamically loading webpage in under 24 hours?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I now have an HTML file and I want to send it as a table, not an attachment by using outlook. The code that I found online only sends the file as an attachment. Can anyone give me ideas on how to do it properly?","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":33,"Q_Id":62442067,"Users Score":0,"Answer":"You can use the HTMLBody property of the MailItem class to set up the message body.","Q_Score":0,"Tags":"python,html,email,outlook","A_Id":62479266,"CreationDate":"2020-06-18T03:50:00.000","Title":"How to send a HTML file as a table through outlook?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I have been trying to do web automation using selenium,Is there any way to use browser like chrome,firefox without actually installing then, like using some alternate options, or having portable versions of them.If I can use portable versions how do i tell selenium to use it?","AnswerCount":2,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":3184,"Q_Id":62442465,"Users Score":0,"Answer":"If you install pip install selenium\nit comes with the protable chrome browser, no need to install any browser for this.\nthe chrome has a tag \"chrome is controlled by automated test software\" near search bar","Q_Score":1,"Tags":"python,selenium,google-chrome,browser,automation","A_Id":62442630,"CreationDate":"2020-06-18T04:35:00.000","Title":"Using Selenium without using any browser","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I am trying to create a UDF in PySpark. The function takes in an input string which is an xml. It then uses lxml to parse it and returns a list of dictionaries with the attributes. I created the function parse_xml, but when I try the line spark.udf.register(\"parse_xml\", parse_xml) but this gives the error: PicklingError: Could not serialize object: TypeError: can't pickle lxml.etree.XMLParser objects. \nIt seems as if lxml objects are not serializable, but the input is a string and the output is a list\/dictionary -- is there any way to create a UDF like this?","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":382,"Q_Id":62449753,"Users Score":0,"Answer":"The library used by pyspark is cpickle which stands for cloud pickle and serializing libraries written in C is not supported by it yet.\nIf you want to parse xml use databricks XML parser instead and it'll be fast as well.","Q_Score":1,"Tags":"python,apache-spark,pyspark,lxml,user-defined-functions","A_Id":62455447,"CreationDate":"2020-06-18T12:15:00.000","Title":"Unable to serialize PySpark UDF","Data Science and Machine Learning":1,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I am trying to write a script to automate a task I'd need to do. So I need this script to check if python is on the \/login page and if he is, to enter the username and password for logging in.\nFor username when logging in, in HTML, it is an input type=\"text\" name=\"username\" and for the password it is an input type=\"password\" name=\"password\". I did search and saw that I should use requests but I only saw them executing POST methods and I'm not sure if this is the case here as well.","AnswerCount":2,"Available Count":2,"Score":0.0996679946,"is_accepted":false,"ViewCount":169,"Q_Id":62487636,"Users Score":1,"Answer":"I'd suggest considering using selenium Framework","Q_Score":0,"Tags":"python,python-3.x,python-requests","A_Id":62487663,"CreationDate":"2020-06-20T15:14:00.000","Title":"Python script to enter data in website inputs","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I am trying to write a script to automate a task I'd need to do. So I need this script to check if python is on the \/login page and if he is, to enter the username and password for logging in.\nFor username when logging in, in HTML, it is an input type=\"text\" name=\"username\" and for the password it is an input type=\"password\" name=\"password\". I did search and saw that I should use requests but I only saw them executing POST methods and I'm not sure if this is the case here as well.","AnswerCount":2,"Available Count":2,"Score":1.2,"is_accepted":true,"ViewCount":169,"Q_Id":62487636,"Users Score":1,"Answer":"Yes selenium works well for this, just to give you some hints. I've done this type of program in the past:\nselenium allows you to click on buttons on a website (you can use this to navigate to the login page).\nThen it allows you to enter data into text fields (this is using a .send_keys() function which you will use to input your name\/password).\nLastly, to actually login you can use the click() function to click on the login button or use send_keys() to \"click\" enter on the keyboard.\nOnce you get it working its actually really fun to watch.","Q_Score":0,"Tags":"python,python-3.x,python-requests","A_Id":62487964,"CreationDate":"2020-06-20T15:14:00.000","Title":"Python script to enter data in website inputs","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"can i run multiple python http servers on one machine to receive http post request from a webpage?\ncurrently i am running an http server on port 80 and on the web page there is a HTML form which sends the http post request to the python server and in the HTML form i am using the my server's address like this : \"http:\/\/123.123.123.123\" and i am receiving the requests\nbut i want to run multiple servers on the same machine with different ports for each server.\nif i run 2 more servers on port 21200 and 21300 how do i send the post request from the HTML form on a specified port , so that the post request is received and processed by correct server??\ndo i need to define the server address like this : \"http:\/\/123.123.123.123:21200\" and \"http:\/\/123.123.123.123:21300\" ?","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":912,"Q_Id":62490441,"Users Score":3,"Answer":"Yes can run multiple webservers on one machine.\nuse following commands to run on different ports:\npython3 -m http.server 4000\n4000 is the port number, you can replace it with any port number here.","Q_Score":0,"Tags":"python,html,http-post","A_Id":62490625,"CreationDate":"2020-06-20T19:06:00.000","Title":"is it possible to run multiple http servers on one machine?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"im just trying to use an vpn extension with selenium. I have the extension running , but i need to click in the button and enable the vpn so it can works, there's a way to do that with selenium? im thinking to use another similar option like scrapy or pyautogui...","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":86,"Q_Id":62500391,"Users Score":0,"Answer":"No there is no way to enable the VPN on your extension\nIf you want to use your VPN extension you have to set a profile (otherwise selenium will create a new profile without installed extension)","Q_Score":0,"Tags":"python,selenium,google-chrome","A_Id":62502290,"CreationDate":"2020-06-21T15:07:00.000","Title":"how can i use an chrome extension in my selenium python program?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I wonder if there is any example or suggested solution to use BeautifulSoup to scrape the data on a site that could be accessed by clicking on a hyperlink on the main webpage.\nThank you so much!!","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":24,"Q_Id":62507917,"Users Score":0,"Answer":"Sure. Can you inspect element to get the link and then scrape it? That could also be done programmatically.","Q_Score":0,"Tags":"python,web-scraping,beautifulsoup,hyperlink,jupyter-notebook","A_Id":62507932,"CreationDate":"2020-06-22T05:22:00.000","Title":"Scrape data from new site by following link","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"Is it possible to define whether pip uses TCP or UDP to make connections? Currently I observe that traffic is blocked by my firewall because only TCP is allowed but pip tries to use UDP.\nI am using pip inside of conda from the Windows Command Prompt.\nAny help would be much appreciated!","AnswerCount":1,"Available Count":1,"Score":0.1973753202,"is_accepted":false,"ViewCount":120,"Q_Id":62512827,"Users Score":1,"Answer":"Pip always uses TCP, as http \/ https via UDP does not exist.\nTry to post the exact error message and the exact command that you type, that causes this error.\nThe cause must be something else, than you assume (perhaps a DNS setup issue, perhaps the need for an http proxy, ...)","Q_Score":0,"Tags":"python,pip","A_Id":62513415,"CreationDate":"2020-06-22T10:55:00.000","Title":"Installing Python Packages (pip) via TCP","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":1,"System Administration and DevOps":0,"Web Development":0},{"Question":"I am looking for the best solution for Python to autopost to instagram from a database of images. I have already made a script using instabot that does this but cant figure out how to post a carousel of images, instead of just one.\nI am wondering what the current best solution for Python Instagram coding is as there is quite a lot of outdated information floating around the web...","AnswerCount":1,"Available Count":1,"Score":0.1973753202,"is_accepted":false,"ViewCount":307,"Q_Id":62529675,"Users Score":1,"Answer":"You Can Use A Combination Of Selenium and PyAutoGUI And It Will Be Really Easy! :D\nHope It Helps,\nYash","Q_Score":1,"Tags":"python,instagram","A_Id":62529791,"CreationDate":"2020-06-23T07:39:00.000","Title":"Automatic Instagram Posting in Python","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I want to automate the process of importing XML file data in MS Excel using Python or any other programming language or by any other method which is possible. How can I achieve that?","AnswerCount":2,"Available Count":1,"Score":0.0996679946,"is_accepted":false,"ViewCount":240,"Q_Id":62554581,"Users Score":1,"Answer":"The solution to your problem is mentioned in the steps below:\n\nImports required library\nread and parse XML file from the path\nextract the root element\nmake a map after finding the required elements from the root elements\ncreate Excel file from the created map object\nsave that file to the destined path\nclose all the stream if any remain opened.\n\nNow code the above steps in any prefer language python, java, or any other.","Q_Score":2,"Tags":"java,python,xml,automation,xlsx","A_Id":62554832,"CreationDate":"2020-06-24T12:00:00.000","Title":"Importing xml file in MS Excel - How can we automate this flow using Python or any other programming language?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I am working on a scraper built in RSelenium. A number of tasks are more easily accomplished using Python, so I've set up a .Rmd file with access to R and Python code chunks.\nThe R-side of the scraper opens a website in Chrome, logs in, and accesses and scrapes various pages behind the login wall. (This is being done with permission of the website owners, who would rather users scrape the data ourselves than put together a downloadable.)\nI also need to download files from these pages, a task which I keep trying in RSelenium but repeatedly come back to Python solutions.\nI don't want to take the time to rewrite the code in Python, as it's fairly robust, but my attempts to use Python result in opening a new driver, which starts a new session no longer logged in. Is there a way to have Python code chunks access an existing driver \/ session being driven by RSelenium?\n(I will open a separate question with my RSelenium download issues if this solution doesn't pan out.)","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":28,"Q_Id":62560581,"Users Score":0,"Answer":"As far as I can tell, and with help from user Jortega, Selenium does not support interaction with already open browsers, and Python cannot access an existing session created via R.\nMy solution has been to rewrite the scraper using Python.","Q_Score":0,"Tags":"python,r,selenium,web-scraping,selenium-chromedriver","A_Id":63379878,"CreationDate":"2020-06-24T17:15:00.000","Title":"Can a Python + R file share a webdriver session between the languages?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I've included this library as a layer to my lambda function but when I go to test it I get the error: cannot import name 'etree' from 'lxml'\nThere are multiple posts about people having this issue and some say to that I need to build it to compile some c libraries. Most posts say to look for another file or folder name 'lxml' which I've verified is not the issue.\nI'm able to run the same code I've deployed to my layer on my local linux workstation and it runs without an issue.","AnswerCount":1,"Available Count":1,"Score":0.3799489623,"is_accepted":false,"ViewCount":146,"Q_Id":62564463,"Users Score":2,"Answer":"Turns out my lambda was running python version 3.8 and that version is not compatible with the version of lxml I am using 4.5.1. Changing the run time to 3.7 fixed the issue. Hope this helps someone.","Q_Score":0,"Tags":"python-3.x,aws-lambda,lxml","A_Id":62564464,"CreationDate":"2020-06-24T21:37:00.000","Title":"lxml library in AWS Lambda","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I am trying to download youtube videos using python and for the code to work I need to install pytube3 library but I am very new to coding so I am not sure how to do it.","AnswerCount":3,"Available Count":1,"Score":0.0665680765,"is_accepted":false,"ViewCount":41,"Q_Id":62573330,"Users Score":1,"Answer":"You could use\npython3 -m pip install pytube3","Q_Score":0,"Tags":"python","A_Id":62573608,"CreationDate":"2020-06-25T10:32:00.000","Title":"How do you download online libraries on python?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":1,"System Administration and DevOps":0,"Web Development":0},{"Question":"From Medium API docs I can see how to retrieve publications and create posts, but it doesn't mention retrieving reading list. Is retrieving reading list for a user currently possible using the Medium's API?","AnswerCount":2,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":702,"Q_Id":62576452,"Users Score":0,"Answer":"I don't think it is currently possible to retrieve reading list. According to the documentation api's it is not part of that list.","Q_Score":1,"Tags":"python,api,medium.com","A_Id":66298868,"CreationDate":"2020-06-25T13:25:00.000","Title":"How to retrieve Medium reading list for a user from the API using python?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":1,"System Administration and DevOps":0,"Web Development":0},{"Question":"I'm asking this question because I've noticed that a lot of (if not all) discord bots go offline every few hours, then come back 2 seconds later. After making my own bot, it does the exact same thing (and it's only in 4 servers). Every few hours, it goes offline for 2 seconds then comes back. Does anyone have an explanation as to why this happens?\nDebugging\nWhen the problem occurs, it also triggers on_ready(), which seems to me to mean that it isn't discord, but the client.\nOther Information\n\nI am using discord.py-rewrite library, but I've seen this happen with the discord.js library as well\nThis happens in active servers\nUnless most bot developers make the same mistake, this is not a problem with my code.\nThis is not a pressing issue for me. I am just mostly curious why and am not necessarily looking for a solution, though I would appreciate it if you had one.","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":327,"Q_Id":62595104,"Users Score":1,"Answer":"It could be possible that the Raspberry Pi loses connection to the discord client if it's connected wirelessly to the internet. Mine does the same for other servers unless it's connected through ethernet.","Q_Score":0,"Tags":"python,python-3.x,discord.py,discord.py-rewrite","A_Id":62688949,"CreationDate":"2020-06-26T12:49:00.000","Title":"Discord.py bot goes offline repeatedly","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I am using chatterbot, I want to send clickable link and Mail as per message sent by the user. I cant find any link or reference on how to do this","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":72,"Q_Id":62599758,"Users Score":0,"Answer":"Try using linkify.... pip install autolink... linkify (bot.get_response(usr_text))","Q_Score":0,"Tags":"python,flask,bots,chatterbot","A_Id":62600578,"CreationDate":"2020-06-26T17:16:00.000","Title":"How to send clickable link and Mail in Chatterbot flask app","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I want to use flask-Socketio. But the problem is that the sessionId (sid) is transmitted in the URL. Which means that anybody sniffing can steal the sessionId.\nHow can flask send the sessionId backin a Post?\nThanks in advance for any help!","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":187,"Q_Id":62605638,"Users Score":0,"Answer":"Your production server should use SSL so that nobody can see the contents of the requests and responses that are exchanged.\nThe session ID in the URL is part of the Socket.IO protocol, not a choice of the Flask-SocketIO package. To be accurate, the server sends the session id in a JSON response, and then the client adds it to the URL in follow-up requests. That's how the protocol works.\nData included in the JSON response can also be picked up by intermediaries if not encrypted, this isn't a problem specific to the URL. The solution is to use encryption.","Q_Score":0,"Tags":"security,socket.io,flask-socketio,flask-session,python-socketio","A_Id":62621757,"CreationDate":"2020-06-27T03:50:00.000","Title":"flask-SocketIO sessionId in URL not secure","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I am new to JavaScript and noticed when I install libraries using npm the node_modules folder is large and it's taking much time and that's not the case in python when I install Django or any library for example!\nso how NodeJs is different from Python at installing packages?\nI searched on Quora, YouTube, reddit and didn't found a clear comparison between both","AnswerCount":1,"Available Count":1,"Score":0.1973753202,"is_accepted":false,"ViewCount":1268,"Q_Id":62610987,"Users Score":1,"Answer":"npm uses local dependencies, so the same module is installed over and over again for every project and every transitive dependency. pip uses global dependencies (at least global to the virtualenv, if used) which avoids this.\nApart from that, pythons standard library is much more larger than the one found in javascript. Thus, much more packages are needed in JS compared to python. Additionally, aspects like transpilation (Babel) are more prevalent in JS than in python.","Q_Score":0,"Tags":"python,node.js,npm,pip,node-modules","A_Id":62611148,"CreationDate":"2020-06-27T14:01:00.000","Title":"why node_modules are so heavy and python dependencies are light?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":1,"System Administration and DevOps":0,"Web Development":1},{"Question":"I'm currently working on a Benchmark project, where I'm trying to stress the server out with zmq requests.\nI was wondering what would be the best way to approach this, I was thinking of having a context to create a socket and push it into a thread, in which I would send request and wait for responses in each thread respectively, but I'm not too sure this is possible with python's limitations.\nMore over, would it be the same socket for all threads, that is, if I'm waiting for a response on one thread (With it's own socket), would it be possible for another thread to catch that response?\nThanks.\nEDIT:\nTest flow logic would be like this:\nClient socket would use zmq.REQ.\nClient sends message.\nClient waits for a response.\nIf no response, client reconnects and tries again until limit.\nI'd like to scale this operation up to any number of clients, preferring not to deal with Processes unless performance wise the difference is significant..\nHow would you do this?","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":161,"Q_Id":62624686,"Users Score":0,"Answer":"Q : \"...can I have one context and use several sockets?\"\n\nOh sure you can.\nMoreover, you can have several Context()-instances, each one managing ... almost... any number of Socket()-instances, each Socket()-instance's methods may get called from one and only one python-thread ( a Zen-of-Zero rule: zero-sharing ).\nDue to known GIL-lock re-[SERIAL]-isation of all the thread-based code-execution flow, this still has to and will wait for acquiring the GIL-lock ownership, which in turn permits a GIL-lock owner ( and nobody else ) to execute a fixed amount of python instructions, before it re-releases the GIL-lock to some other thread...","Q_Score":0,"Tags":"python,message-queue,benchmarking,zeromq,pyzmq","A_Id":62626820,"CreationDate":"2020-06-28T15:37:00.000","Title":"pyzmq - zmq_req can I have one context and use several sockets?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":1,"System Administration and DevOps":0,"Web Development":0},{"Question":"I wrote a Python script that allows me to retrieve calendar events from an externally connected source and insert them into my Google Calendar thanks to the Google Calendar's API. It works locally when I execute the script from my command line, but I would like to make it happen automatically so that the externally added events pop up in my Google Calendar automatically.\nIt appears that a cron job is the best way to do this, and given I used Google Calendar's API, I thought it might be helpful to use Cloud Functions with Cloud Scheduler in order to make it happen. However, I really don't know where to start and if this is even possible because accessing the API requires OAuth with Google to my personal Google account which is something I don't think a service account (which I think I need) can do on my behalf.\nWhat are the steps I need to take in order to allow the script which I manually run and authenticates me with Google Calendar run every 60 seconds ideally in the cloud so that I don't need to have my computer on at all times?\nThings I\u2019ve tried to do:\nI created a service account with full permissions and tried to create an http-trigger event that would theoretically run the script when the created URL is hit. However, it just returns an HTTP 500 Error.\nI tried doing Pub\/Sub event targets to listen and execute the script, but that doesn\u2019t work either.\nSomething I\u2019m confused about:\nwith either account, there needs to be a credentials.json file in order to login; how does this file get \u201cdeployed\u201d alongside the main function? Along with the token.pickle file that gets created when the authentication happens for the first time.","AnswerCount":1,"Available Count":1,"Score":0.3799489623,"is_accepted":false,"ViewCount":391,"Q_Id":62648913,"Users Score":2,"Answer":"The way a service account works is that it needs to be preauthorized. You would take the service account email address and share a calendar with it like you would with any other user. The catch here being that you should only be doing this with calendars you the developer control. If these are calendars owned by others you shouldnt be using a service account.\nThe way Oauth2 works is that a user is displayed a consent screen to grant your application access to their data. Once the user has granted you access and assuming you requested offline access you should have a refresh token for that users account. Using the refresh token you can request a new access token at anytime. So the trick here would be storing the users refresh tokens in a place that your script can access it then when the cron job runs the first thing it needs to do is request a new access token using its refresh token.\nSo the only way you will be able to do this as a cron job is if you have a refresh token stored for the account you want to access. Other wise it will require it to open a web browser to request the users consent and you cant do that with a cron job.","Q_Score":0,"Tags":"python,google-cloud-functions,google-calendar-api,google-oauth,google-cloud-scheduler","A_Id":62652102,"CreationDate":"2020-06-30T01:16:00.000","Title":"How do I use a cron job in order to insert events into google calendar?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":1,"Web Development":0},{"Question":"I have been working on two scripts where one of the script (we can call generate_data.py) that generates the data (looks for new data 24\/7) and I have notifications.py that receives the data from generate_data , convert it to a json and post it to my server through a requests.\nThere is some problems that I have and one of them is that I need it to run simultaneously. That means whenever the generate_data.py is looking for new data, when it sends the data to notifications.py it should NOT wait for the notifications.py to be done. Instead it should work all the time and only send data to notifications.py when new found and continue looking for new data.\nScenario that can happen is that generate_data.py can get new 5 data at the same time and is needing to be send to notifications.py that needs to send the requests to my server as soon as possible without any delay\/stopping\/blocking.\nI got few suggestions such as using listener, queue, rcp, threading, multiprocessing but here I am. My question is, What would be my best option to use here where I have generate_data.py that is on 24\/7 and notifications.py should only post once it got the data from generate_data and post it to my server as fast as possible and without being blocked\/stopped and if there is any example would also be appreciated it! (Using python)","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":68,"Q_Id":62671632,"Users Score":1,"Answer":"so as generate_data script generates data and sends it to notification script, while notification script is in processing the data which has sent meanwhile the data generated by generate_data where you are storing it to push to the notification script later ?\nyou can do one thing as you asked only with python..\nyou can generate data in to a file always and notifications script read the content and delete only content which is read, so that you can do both parallel.","Q_Score":0,"Tags":"python,multithreading,queue,listener,rcp","A_Id":62671801,"CreationDate":"2020-07-01T07:22:00.000","Title":"What is the best option to send data to another script without be blocked","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I am currently doing Selenium testing with Pytest. There will be around 10 tests, however, a user may only have 70-80% of the data available to test (which is fine).\nIs there a way to then say, if 7\/10 tests have passed, pass the overall pytest, but still show the failing tests\nThis will interact with a CI\/CD pipeline, and it seems that if even 1 tests fails, that the Jenkins just sees red.","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":73,"Q_Id":62677374,"Users Score":0,"Answer":"As suggested, if there isn't enough data for tests. Then don't run them.\nI was able to find a database table which only contains the users with the relevant data. Now all tests are passing 100%.\nThanks everyone for the help.","Q_Score":0,"Tags":"python,selenium,jenkins,testing,pytest","A_Id":62699363,"CreationDate":"2020-07-01T12:56:00.000","Title":"Pytest pass overall if >50% of Selenium tests pass","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I have a python script that I have converted into a attributable executable. This script uses selenium to look up a part number on my company website. My end goal is to integrate this into Power Bi so that any part number in a Power Bi list can be easily looked up. My idea is to make a URL that can be a clickable link in Power Bi. This URL would call the executable from a network drive as well as pass the part number to the script. To incorporate the part number I would like to add it onto the end of the URL and have the script pick it up from there.\nIs this just a dream or can a relatively new programmer do this? I have found a way to make a web-page that contains a python script, but not one that I can pass information to without user interaction. Is there a tool out there that will allow me to click a URL that launches a local .exe and passes information to it while running in the background?\nAny recommendations would be greatly appreciated.","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":55,"Q_Id":62680494,"Users Score":0,"Answer":"You can create a simple flask app that has a GET view and pass information to it with GET or POST parameters, in the view you would then execute whatever you want.","Q_Score":0,"Tags":"python,powerbi,exe","A_Id":62680798,"CreationDate":"2020-07-01T15:43:00.000","Title":"How to Launch python EXE from URL and Pass Information Through the URL","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I am trying to set up my python web scraper to run at a set time using task scheduler but I am getting this error PermissionError: [Errno 13] Permission denied: 'geckodriver.log' I have no idea why because it works fun when running the python script by using the batch file or just by double-clicking the .py file. Anyone have any ideas about why it only happens with task scheduler.\nFYI: I am using a batch file that calls the python.exe and the .py script and this what task scheduler is using to trigger the script. I am using selenium and firefox.\nMany thanks.","AnswerCount":2,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":481,"Q_Id":62699600,"Users Score":0,"Answer":"I fixed this issue by checking the Run with highest privileges and configure for Windows 10","Q_Score":1,"Tags":"python,python-3.x,selenium,geckodriver","A_Id":62701037,"CreationDate":"2020-07-02T15:14:00.000","Title":"Geckodriver PermissionError: [Errno 13] Permission denied: 'geckodriver.log' only with task scheduler","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I want to download several large files from the internet (specifically Reddit monthly submissions from the site PushShift) into an S3 bucket. I am SSHed into an EC2 instance and have a Jupyter notebook running.\nIdeally I want to be able to write a Python script in Jupyter notebook of my EC2 instance that downloads the file from the internet and then pushes it to my S3 bucket. How would I go about doing this?","AnswerCount":1,"Available Count":1,"Score":0.3799489623,"is_accepted":false,"ViewCount":200,"Q_Id":62702724,"Users Score":2,"Answer":"It is not possible to \"download data from the Internet into Amazon S3\".\nAmazon S3 is an object storage service. You can upload data to S3 and download data from S3, but it is not possible to tell S3 to download data from some other location and store it.\nYou will need a program running somewhere that obtains the data from the Internet, then uploads it (creates an object) in Amazon S3. Such a program could be clever enough to 'stream' the data to S3 by downloading content in-memory and then sending it to S3, without having to save to disk in between, but you would need to write that code.\nAs to 'where' such a program might run, it would be most efficient to run such code either as an AWS Lambda function or on an Amazon EC2 instance that is in the same region as the Amazon S3 bucket.\nSince you are running a Jupyter notebook on an Amazon EC2 instance, it would be easiest to download the file to local storage, then upload it to S3.","Q_Score":0,"Tags":"python,amazon-web-services,amazon-s3,amazon-ec2,jupyter-notebook","A_Id":62706422,"CreationDate":"2020-07-02T18:20:00.000","Title":"How do I download data from the internet to an S3 bucket via EC2?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I know it's possible to disable InsecureRequestWarning but I need the opposite, I want to either catch it or make the request abort and throw an exception if this warning is present.","AnswerCount":1,"Available Count":1,"Score":0.1973753202,"is_accepted":false,"ViewCount":139,"Q_Id":62710891,"Users Score":1,"Answer":"I've solved it with warnings.filterwarnings(\"error\"), which turns warnings into errors so they can be caught with try catch.","Q_Score":0,"Tags":"python,python-3.x,python-requests","A_Id":62712580,"CreationDate":"2020-07-03T07:52:00.000","Title":"Is there a way to catch InsecureRequestWarning from python request","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I'm new to tcp\/ip programming and I'm trying to establish a tcp\/ip connection that sends a byte from my python server to a c++ client. Is this possible or does the client and server have to be written in the same programming language?","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":72,"Q_Id":62739874,"Users Score":1,"Answer":"Tcp connection does not depend on programming language you could make connection between two processes one of them should be listen for another(server) and the other should send sync packet to request a connection(client). You could implement two mentioned components with every language programming.If you need more help let us know.","Q_Score":0,"Tags":"python,c++,sockets,tcp","A_Id":62740048,"CreationDate":"2020-07-05T11:09:00.000","Title":"Can i establish a tcp\/ip socket with a python server but a c++ client?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I have to devise a python program to communicate with a number, lets say 1000 devices via TCP in parallel. Each connection establishes a client connection and should start to communicate on a stream based way.\nUntil now I successfully employed select.select() to get notified when something arrives for reading or when writing has completed.\nNow I learned a little about Python Async IO using async function definitions. I think it could write code more elegantly using this framework instead of re-using my old pattern.\nHowever, I'm not sure, whether I should invest some time to learn it or just be satisfied with the old fashioned approach.\nFrom a rough view I came to the conclusion that I could actually solve all I need with asyncio too.\nHowever, to convince other people, what would be the real advantage beside using a \"cool\" framework?\nEDIT:\nI will never understand SO's policy regarding closing questions. Is this only a forum for experts on the highest top-level? I know that my question doesn't target a specialized and\/or detailed question, but this is not a reason at all to close it.","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":95,"Q_Id":62740254,"Users Score":1,"Answer":"Async IO is not a part of python only. It is a programming paradigm, which is universally used across all programming languages. For instance, Async IO is one of the most highlighted feature of NodeJs. Therefore, in order to answer your question, yes, it is a long term investment if you study the Async IO as it is mandatory in most of the communication applications.","Q_Score":0,"Tags":"python,python-3.x,asynchronous,async-await,python-asyncio","A_Id":62740636,"CreationDate":"2020-07-05T11:46:00.000","Title":"What would be the benefit of using Async IO in my Python project?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":1,"System Administration and DevOps":0,"Web Development":0},{"Question":"Due to AWS deprecating Signature Version 3 in Oct 2020 for SES, I want to handle this issue with AWS boto (Python). But I didn't see any doc related to boto supporting signature version 4 for SES.\nIs anyone having similar issue and have solutions?","AnswerCount":2,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":4142,"Q_Id":62757921,"Users Score":3,"Answer":"My recommendation is that you migrate from boto, which is essentially deprecated, to boto3 because boto3 supports signature v4 by default (with the exception of S3 pre-signed URLs which has to be explicitly configured).","Q_Score":8,"Tags":"python,amazon-web-services,boto,signature","A_Id":62762520,"CreationDate":"2020-07-06T14:12:00.000","Title":"Is AWS boto (python) supporting SES signature version 4?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I would like to request for the best python package to be able to send SNMP Request(Get,Getbulk..etc)\nI am using pysnmp with multiprocessing and it is working fine for 60 nodes however when i am trying to use it for 100+ nodes with multiprocessing i could not ( I have upgraded my CPU to be sufficient enough)\nmy assumption the issue is either the pysnmp package or the Port itself which is 161 that can't handle to send or establish many session or send many requests at the same time.\nif it is the package, which snmp package which i try to use?","AnswerCount":1,"Available Count":1,"Score":0.1973753202,"is_accepted":false,"ViewCount":549,"Q_Id":62763042,"Users Score":1,"Answer":"After long evaluation, it turns out pysnmp is the problem, because it has dependencies on other packages and SNMP MIB within Linux or Windows.\nThe best package is puresnmp, since there are no dependencies.\nI have been using that packages for almost 2 months and it's working very fast for 100+ nodes.","Q_Score":1,"Tags":"python,snmp,pysnmp","A_Id":63976067,"CreationDate":"2020-07-06T19:22:00.000","Title":"What is the best SNMP Package to use with python for over 100 nodes to monitor at the same time","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I have this Instagram bot that is made using Python and Selenium, It log into Instagram, goes to a profile, select the last post and select the \"other x people liked this photo\" to show the complete list of the people that liked the post(it can be done with the follower of the page too).\nNow I am stuck because I don't know how can i make the bot click only the profiles that have active stories and how to make it scroll down (the problem is that the way that i found to click on the profiles works just with the first one profile because when I click on the profile it opens the stories and closes the post, so when i reopen the post and the list of like on this post it will reclick on the same profile that I have already seen the stories of).\nDoes someone know how to do that or a similar thing maybe something even better that I didn't thinked of?\nI don't think code is needed but if you need I will post it, just let me know.","AnswerCount":1,"Available Count":1,"Score":0.1973753202,"is_accepted":false,"ViewCount":462,"Q_Id":62774102,"Users Score":1,"Answer":"Have you tried to use the \"back\" button on your browser window? Or open the page in a new tab, so you have still the old one to go back to.","Q_Score":1,"Tags":"python,selenium,bots,instagram,instagram-story","A_Id":62774176,"CreationDate":"2020-07-07T11:19:00.000","Title":"Python Selenium bot to view Instagram stories | How can i click the profiles of people that have active stories?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I have a server which continuously sends data block (sample) at a sampling rate of 100Hz. My client receives it and performs some calculation on the sample to visually display a real-time tracking and then receives the next sample and so on. The time taken for it to receive and calculate until the next sample is received is 0.1s.\nThis means that I am missing on the in between samples, if I am not wrong. How can change the code such that I receive 'all samples' and perform the calculation in order to enable a real-time tracking?\nThank you in advance!","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":231,"Q_Id":62779087,"Users Score":0,"Answer":"From the comment section:\n\nThe server is a software from where I am receiving the samples. And the manual of the software says the sampling rate of the samples is 100Hz. My client code receives it and checks if the data within the sample is above the threshold value or not and displays an error on the screen when it is. To be able to check for every sample that is being streamed by the software, my receiving and computation must be faster. Could you explain how could I use buffer to better my case? I am sorry if my question is very silly. But this is my first attempt in TCP programming as well as Python\n\nFor a sampling rate of 100 Hz the network latency has to be below 10ms.\nFirst test: Determine the network latency...\nOnly receive the data and log the times (including milliseconds) to a file. If you have gaps of more than 10ms between the datapackets, the network is probably not fast enough.\nIf the gaps are far below the 10ms, data processing is too slow.\nServer-side buffering (high latency):\nIf you can change the server, consider sending multiple packets at once, but at a slower frequency. The client would have more time to process the data instead of waiting for the I\/O.\nClient-side buffering (low latency):\nIf the server sends packets faster than you can process them, consider a FIFO buffer for the client. Incoming data is appended to the buffer, while processing code fetches its packets from the queue, ideally simultaneously.\nLastly, some warm words of warning: You [say you] are new to a programming language and network programming - it's probably too early for a realtime data processing project and I would suggest to start with a simpler projects for training purposes...","Q_Score":0,"Tags":"python,tcp,runtime,client-server,rate","A_Id":62780063,"CreationDate":"2020-07-07T15:48:00.000","Title":"Server sending data at a sampling rate 100 Hz and the client receiving and processing time is around 0.1s using Python","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I'm looking for a way to scan a website and instantly detect an update, without having to refresh the page. So when a new post is pushed to the webpage, I'd like to be instantly notified. Is there a way to do that without having to refresh the page constantly?\nCheers","AnswerCount":2,"Available Count":2,"Score":0.0996679946,"is_accepted":false,"ViewCount":65,"Q_Id":62780541,"Users Score":1,"Answer":"What browser are you using? Chrome has an auto refresh extension. Try doing a Google search for the extension. It's very easy to set up. It's more of a timed refresh that you can program. But it works for situations like what you are asking.","Q_Score":1,"Tags":"python,html,jquery,google-chrome","A_Id":62780663,"CreationDate":"2020-07-07T17:13:00.000","Title":"How to get notified when new content is pushed to a webpage","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I'm looking for a way to scan a website and instantly detect an update, without having to refresh the page. So when a new post is pushed to the webpage, I'd like to be instantly notified. Is there a way to do that without having to refresh the page constantly?\nCheers","AnswerCount":2,"Available Count":2,"Score":0.0996679946,"is_accepted":false,"ViewCount":65,"Q_Id":62780541,"Users Score":1,"Answer":"Without knowing a bit more about your task, it's hard to give you a clear answer. Typically you would set up some kind of API to determine if data has been updated, rather than scraping the contents of a website directly. See if an API exists, or if you could create one for your purpose.\nUsing an API\nWrite a script that calls the API every minute or so (or more often if necessary). Every time you call the API, save the result. Then compare the previous result to the new result - if they're different then the data has been updated.\nScraping a Website\nIf you do have to scrape a website, this is possible. If you execute an HTTP GET request against a webpage, the response will contain the DOM of the webpage. You can then traverse the DOM to determine the contents of a webpage. Similar to the API example, you can write a script that executes the HTTP request every minute or so, saves the state, and compares it to the previous state. There are numerous libraries out there to help preform HTTP request and traverse the DOM, but without knowing your tech stack I can't really recommend anything.","Q_Score":1,"Tags":"python,html,jquery,google-chrome","A_Id":62780866,"CreationDate":"2020-07-07T17:13:00.000","Title":"How to get notified when new content is pushed to a webpage","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"We handle 10-20 payments daily using Paypals IPN system.\nThe past hour or so Paypal has been sending the IPNs in the wrong order.\nOur payments are subscriptions.\nSo generally they send.\nSubscr_signup\n& shortly after that\nsusbcr_payment.\nBut today they've been sending the subscr_payment before the subscr_signup every time. Which messed up our flow since the payment tries to link itself with a subscriber which it can't so it throws an error. This has forced us to manually add all the information which is a headache.\nAny ideas on how I should solve this?","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":34,"Q_Id":62780899,"Users Score":0,"Answer":"IPN is an asynchronous, best-effort delivery service. The \"Instant\" in its name, can be misleading, since that service was created a long time ago, and it was instant relative to, say, waiting for an email and processing things manually.\nSince payments and subscription profile creations are separate events in separate systems, you may receive notifications of each event in any order.\nYou should be able to create new\/temporary\/placeholder subscriber objects with the first subscr_payment event alone -- and if necessary fill in any missing information (or activate the placeholder) when you do receive a subscr_signup even.","Q_Score":0,"Tags":"python,django,paypal","A_Id":62781905,"CreationDate":"2020-07-07T17:36:00.000","Title":"Paypal IPN sent in the wrong order","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I have been trying to install discord on my raspberry pi, but its saying error code 1 and I need a more updated version of python (3.4.5). However, Thonny Python says it's version is 3.7.3.","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":3322,"Q_Id":62798274,"Users Score":0,"Answer":"Python on raspberry pi comes preinstalled with version 2 and version 3 of python\nTry using pip3 install discord and see if that fixes it","Q_Score":0,"Tags":"python,discord","A_Id":62805382,"CreationDate":"2020-07-08T15:26:00.000","Title":"Install discord on raspberry pi","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I am doing a python project that would use multiple python API (At least 3) to obtain datas from a lot of countries and I am wondering if it is more efficient to for the data using the APIs or to use the APIs once to make a file and access it later, when needed.","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":23,"Q_Id":62807126,"Users Score":0,"Answer":"It's always better to handle data management in your back end to minimize the performance loss in your front end.","Q_Score":0,"Tags":"python,database,api","A_Id":62807197,"CreationDate":"2020-07-09T03:50:00.000","Title":"Using multiple API vs reading from a file","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":1,"System Administration and DevOps":0,"Web Development":0},{"Question":"I have done my Dialogflow voice conversation chatbot using python, But currently I am running the conversation chatbot using loop. So my conversation is running till the loop end. The cloud is developed by using webhook and dialogflow intent.\nSo how can I end the conversation once the chat is end in cloud to end notification in python application.","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":213,"Q_Id":62827534,"Users Score":0,"Answer":"I have added the following print in my python code, so there I have got the end_conversation intent, below I added the end_conversation log.\nprint('Fulfillment text: {} \\n'.format(query_result))\nEnd conversation log:\nintent_detection_confidence: 1.0\ndiagnostic_info {\nfields {\nkey: \"end_conversation\"\nvalue {\nbool_value: true\n}\n}","Q_Score":0,"Tags":"python-requests,dialogflow-es","A_Id":62949215,"CreationDate":"2020-07-10T04:51:00.000","Title":"How the dialogflow voice chatbot end conversation","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I'm looking for a good solution to implement a handshake between a python backend server and a react frontend connected through a websocket.\nThe frontend allows the user to upload a file and then send it to the backend for processing. Now it might be possible that the processing encounters some issues and likes the user's confirmation to proceed or cancel - and that's where I'm stuck.\nMy current implementation has different \"endpoints\" in the backend which call then different function implementations and a queue which is continuously processed and content (messages) is sent to the frontend. But these are always complete actions, they either succeed or fail and the returned message is accordingly. I have no system in place to interupt a running task (e.g. file processing), send a request to the frontend and then wait for response before I continue the function.\nIs there a design pattern or common approach for this kind of problem?","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":43,"Q_Id":62828807,"Users Score":0,"Answer":"How long it takes to process? Maybe a good solution is set up a message broker like RabbitMQ and create a queue for this process. In the front-end you have to create a panel to see the state of the process, which is running in an async task, and if it has found some issues, let the user know and ask what to do.","Q_Score":0,"Tags":"python,reactjs,websocket,system-design","A_Id":62829294,"CreationDate":"2020-07-10T06:54:00.000","Title":"Interupt backend function and wait for frontend response","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"So I'm doing this python basics course and my final project is to create a card game. At the bottom of the instructions I get this\n\nFor extra credit, allow 2 players to play on two different computers that are on the same network. Two people should be able to start identical versions of your program, and enter the internal IP address of the user on the network who they want to play against. The two applications should communicate with each other, across the network using simple HTTP requests. Try this library to send requests:\n\n\nhttp:\/\/docs.python-requests.org\/en\/master\/\n\n\nhttp:\/\/docs.python-requests.org\/en\/master\/user\/quickstart\/\n\n\nAnd try Flask to receive them:\n\n\nhttp:\/\/flask.pocoo.org\/\n\n\nThe 2-player game should only start if one person has challenged the other (by entering their internal IP address), and the 2nd person has accepted the challenge. The exact flow of the challenge mechanism is up to you.\n\nI already investigated how flask works and kind of understand how python-requests works too. I just can't figure out how to make those two work together. If somebody could explain what should I do or tell me what to watch or read I would really appreciate it.","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":195,"Q_Id":62841949,"Users Score":0,"Answer":"it would be nice to see how far you've come before answer (as hmm suggested you in a comment), but i can tell you something theorical about this.\nWhat you are talking about is a client-server application, where server need to elaborate the result of clients actions.\nWhat i can suggest is to learn about REST API, that you can use to let client and server to communicate in a easy way. Your clients will send http requests to server exposed APIs.\nFrom what you wrote, you have a basically constraints that should be respected during client and server communication, here reasumed:\n\nSomeone search for your ip and send you a challenge request\n\nYou have received a challenge that you refuse or accept; only if you accept the challenge you can start the game\n\n\nAs you can see from the project specifications the entire challenge mechanism is up to you, so you can decide the best for you.\nI would begin start thinking to a possible protocol that make use of REST API to start initial communication between client and server and let you define a basic challenge mechanism.\nEnjoy programming :).","Q_Score":0,"Tags":"python,flask,python-requests,ip","A_Id":62848649,"CreationDate":"2020-07-10T20:37:00.000","Title":"Python same Network Card Game","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I have a website for work and I need to go through a list of numbers and determine if the user associated with the number is still active. The website requires a sign in so I can't use requests. Is there a way I can run it through my chrome browser to get the information I require?\nIf I can get the HTML then I am fine from there onward with the code.\nAny help would be greatly appreciated","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":34,"Q_Id":62844688,"Users Score":0,"Answer":"Can I have the webpage you are trying to access?","Q_Score":0,"Tags":"python,python-3.x,web-scraping","A_Id":63064700,"CreationDate":"2020-07-11T02:49:00.000","Title":"Is there a way to get HTML text through python and a web browser?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I Am using Scrapy with scrapySelenium but the scrapy selenium starts with a headless browser but for some purpose i need to start it with head so i can keep seeing what my scrapy does in the browser how to turn off headless arguments passed in settings of scrapy\nthis is what i copied in the settings it is headless as you can see but want to see the browser open up with Scrapy Selenium is there anyway\nfrom shutil import which\nSELENIUM_DRIVER_NAME = 'chrome'\nSELENIUM_DRIVER_EXECUTABLE_PATH = which('chromedriver')\nSELENIUM_DRIVER_ARGUMENTS=['-headles'] # '--headless' if using chrome instead of firefox","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":1744,"Q_Id":62870900,"Users Score":2,"Answer":"Delete the SELENIUM_DRIVER_ARGUMENTS = ['--headless'] from the settings.py.\nWith selenium webdriver you can pass arguments and this argument makes sure that it is a headless browser.","Q_Score":1,"Tags":"python,selenium,selenium-webdriver,web-scraping,scrapy","A_Id":62872252,"CreationDate":"2020-07-13T07:13:00.000","Title":"How to use chrome driver with head in Scrapy Selenium","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I am very new to python, and I am trying to create a chatbot with python for a school project.\nI am almost done with creating my chatbot, but I don't know how to create a website to display it, I know how to create a website with Flask but how can I embed the chatbot code into the website?","AnswerCount":2,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":293,"Q_Id":62874661,"Users Score":0,"Answer":"In your flask code you can also embed the chatbot predict-functions into specific routes of your flask app. This would require following steps:\nJust before you start the flask server you train the chatbot to ensure its predict function works propperly.\nAfter that you can specifiy some more route-functions to your flask app.\nIn those functions you grab input from the user (from for example route parameters), send it through the chatbots predict function and then send the respons (probably with postprocessing if you wish) back to the requester.\nSending to the requester can be done through many different ways.\nTwo examples just of my head would be via display (render_template) to the webpage (if the request came in over GET-Request via usual browser site-opening request) or by sending a request to the users ip itself.\nAs a first hand experience i coupled the later mechanism to a telegram bot on my home-automation via post-request which itself then sends the response to me via telegram.","Q_Score":0,"Tags":"python,chatbot","A_Id":72057057,"CreationDate":"2020-07-13T11:13:00.000","Title":"How to embed my python chatbot to a website","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"Below you can find an example of the problem that I'm facing:\n\nproject:\n\npackage:\n\nxml.py\n\n\n\n\nxml.py:\nimport xml.etree.ElementTree as Et\n\nMy module named xml.py is getting confused with the xml package when I import it, now my question is:\nCan I name my module in such a way that won't interfere with the package name, but at the same time would be compliant with the PEP8 styling guide (e.g: xml_.py. I have seen this naming convention for variables but I am not sure it is a good practice to name modules like this).","AnswerCount":2,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":535,"Q_Id":62878577,"Users Score":0,"Answer":"modules (filenames) should have short, all-lowercase names, and they can contain underscores;\nPep8 documentation, so yes you can :)","Q_Score":1,"Tags":"python,module,package,naming,pep","A_Id":62878642,"CreationDate":"2020-07-13T14:59:00.000","Title":"Can module names in Python end with an underscore?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":1,"System Administration and DevOps":0,"Web Development":0},{"Question":"I have a config file in an S3 bucket that needs to be modified by adding the response from ec2.describe_subnets and sent to an api endpoint. Is there a way to automate this process using Lambda to get the file from S3 without having to save it to a temp directory?","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":230,"Q_Id":62883144,"Users Score":0,"Answer":"Is there a way to automate this process using Lambda to get the file from S3 without having to save it to a temp directory?\n\nIf you're asking about modifying the contents of the S3 Object, then the answer is no, it's not possible because S3 doesn't support that kind of operation. The only thing you can do is overwrite the entire object (ie, not just parts of it).\nIf you're asking about overwriting the S3 Object with new contents, then yes, it is possible to do it \"without having to save it to a temp directory\" if you do it in memory, for example.\nDownload the object from S3 without writing it to storage, make the changes in memory, and re-upload it to S3. If the file size is too big to fully store in memory, you can do it in a streaming fashion, too (i.e., initiate the download, for each chunk you download you make the necessary changes, and upload the modified chunk with a multipart upload, clear up the memory, repeat, etc).\nAs a final note, do keep in mind that S3 supports only eventual consistency for updates. This means that after you update the object, subsequent reads may still download the previous version. If whatever is consuming the file cannot properly deal with that, you'll probably need a different approach (i.e., don't overwrite, but write a new object with a new key, and send that new key to the consumer; or just use storage system that does support strong consistency, such as DynamoDB).","Q_Score":0,"Tags":"python-3.x,amazon-web-services,amazon-s3,aws-lambda","A_Id":62884456,"CreationDate":"2020-07-13T19:44:00.000","Title":"How can I edit an xml file that is stored in S3 without using a temp directory?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I'm creating a chatbot with Amazon Lex and the idea is that a user can drop an Excel file into a Slack channel and the bot sends the file to the appropriate Lambda function. What is the best way to achieve this goal?\nIt would also be acceptable if the Lex bot uploads the file somewhere else accessible (s3, EC2, Github, etc.) and then sends the address to the Lambda functions.\nI do not see any SlotTypes that support this type of input.","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":482,"Q_Id":62896237,"Users Score":1,"Answer":"Lex PostText or PostContent method doesn't support the upload load files.\n\nwhere as in slack you will get the file url in payload when user uploads a file.\nCreate slot type Eg.- File. keep the enumeration value as uploaded.\nWhen slotToElicit is File check whether url is exists or not in the payload. if exists send the URL to lex using the session attributes and fill the slot value as Uploaded(Default value). If user doesn't upload a file fill the slot value with some random text to elicit the slot again.\nWith the help of Session attributes you can have access to file and upload it to S3.","Q_Score":0,"Tags":"python,amazon-web-services,slack,aws-lex","A_Id":64303167,"CreationDate":"2020-07-14T13:22:00.000","Title":"Uploading a file to Amazon Lex","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I have created a Machine Learning Application using Python. I want to host it online so that it can be used from anywhere. What are my best options?","AnswerCount":1,"Available Count":1,"Score":0.1973753202,"is_accepted":false,"ViewCount":45,"Q_Id":62898336,"Users Score":1,"Answer":"Personally, I use pythonanywhere.com, lets you run any python code from anywhere, and even host your websites.","Q_Score":0,"Tags":"python","A_Id":62898372,"CreationDate":"2020-07-14T15:13:00.000","Title":"How to host Python Web application online?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"Wondering can someone point me in the right direction on how to convert base64 image string to its original image URL.\nMy code is scraping top 5 news from google based on my search string.\nImages are in one big massive base64 string. Images are printing ok on my outlook email (my code extract the news and send out an email in outlook) but when I forward that email on to different email account can't see any image but a message The linked image cannot be displayed. The file may have been moved, renamed or deleted. Just to check that, I copied the image from my outlook email and tried to paste on word document; all I can see is an empty box but no image.\nany advice, please?","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":521,"Q_Id":62914205,"Users Score":0,"Answer":"You can't get a URL from those. Those base64 encoded strings are fully embedded images. You could base64 decode them and save it to a file or just take the base64 encoded string an attach it to another image tag like in the incoming email.\nIf you have some specific code I could be of more help.","Q_Score":0,"Tags":"python,base64","A_Id":62914260,"CreationDate":"2020-07-15T11:51:00.000","Title":"Get original image url from base64 Image String using Python?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"After having installed libgdcm-tools (2.6.6-3) on my system (via apt install libgdcm-tools) I am not longer able to import pydicom in Python. When running import pydicom as pdc I got the following error\nAttributeError: module 'gdcm' has no attribute 'DataElement'.\nRemoving libgdcm-tools does not solve the issue.\nI am working with Python 3.7 and pydicom 2.0.0 in Ubuntu 18.04.","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":598,"Q_Id":62933483,"Users Score":0,"Answer":"Indeed @scaramallion's comment pointed me in the right direction. I didnt't have a gcdm directory in the working directory, but in my home directory. Removing that directory solved the problem.","Q_Score":2,"Tags":"python,pydicom,gdcm","A_Id":62990094,"CreationDate":"2020-07-16T11:14:00.000","Title":"Why importing pydicom rise AttributeError: module 'gdcm' has no attribute 'DataElement'","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I am trying log to AWS cloud watch and I am getting this error. same code was working all from yesterday without issues.\nI looked into some article and I installed certifi package as well but this also did not help\nI am using Python 3.5 and watchtower to log data to AWS cloud watch.\nI am running my python code in windows 10 OS.\nCan you please help me with this?\n\n\nTraceback (most recent call last):\nFile \"C:\\Users\\BBB\\AppData\\Local\\Programs\\Python\\Python35\\lib\\site-packages\\urllib3\\connectionpool.py\", line 677, in urlopen\nchunked=chunked,\nFile \"C:\\Users\\BBB\\AppData\\Local\\Programs\\Python\\Python35\\lib\\site-packages\\urllib3\\connectionpool.py\", line 381, in _make_request\nself._validate_conn(conn)\nFile \"C:\\Users\\BBB\\AppData\\Local\\Programs\\Python\\Python35\\lib\\site-packages\\urllib3\\connectionpool.py\", line 976, in validate_conn\nconn.connect()\nFile \"C:\\Users\\BBB\\AppData\\Local\\Programs\\Python\\Python35\\lib\\site-packages\\urllib3\\connection.py\", line 370, in connect\nssl_context=context,\nFile \"C:\\Users\\BBB\\AppData\\Local\\Programs\\Python\\Python35\\lib\\site-packages\\urllib3\\util\\ssl.py\", line 377, in ssl_wrap_socket\nreturn context.wrap_socket(sock, server_hostname=server_hostname)\nFile \"C:\\Users\\BBB\\AppData\\Local\\Programs\\Python\\Python35\\lib\\ssl.py\", line 376, in wrap_socket\n_context=self)\nFile \"C:\\Users\\BBB\\AppData\\Local\\Programs\\Python\\Python35\\lib\\ssl.py\", line 747, in init\nself.do_handshake()\nFile \"C:\\Users\\BBB\\AppData\\Local\\Programs\\Python\\Python35\\lib\\ssl.py\", line 983, in do_handshake\nself._sslobj.do_handshake()\nFile \"C:\\Users\\BBB\\AppData\\Local\\Programs\\Python\\Python35\\lib\\ssl.py\", line 628, in do_handshake\nself._sslobj.do_handshake()\nssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:646)\nDuring handling of the above exception, another exception occurred:\nTraceback (most recent call last):\nFile \"C:\\Users\\BBB\\AppData\\Local\\Programs\\Python\\Python35\\lib\\site-packages\\botocore\\httpsession.py\", line 263, in send\nchunked=self._chunked(request.headers),\nFile \"C:\\Users\\BBB\\AppData\\Local\\Programs\\Python\\Python35\\lib\\site-packages\\urllib3\\connectionpool.py\", line 725, in urlopen\nmethod, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]\nFile \"C:\\Users\\BBB\\AppData\\Local\\Programs\\Python\\Python35\\lib\\site-packages\\urllib3\\util\\retry.py\", line 379, in increment\nraise six.reraise(type(error), error, _stacktrace)\nFile \"C:\\Users\\BBB\\AppData\\Local\\Programs\\Python\\Python35\\lib\\site-packages\\urllib3\\packages\\six.py\", line 734, in reraise\nraise value.with_traceback(tb)\nFile \"C:\\Users\\BBB\\AppData\\Local\\Programs\\Python\\Python35\\lib\\site-packages\\urllib3\\connectionpool.py\", line 677, in urlopen\nchunked=chunked,\nFile \"C:\\Users\\BBB\\AppData\\Local\\Programs\\Python\\Python35\\lib\\site-packages\\urllib3\\connectionpool.py\", line 381, in _make_request\nself.validate_conn(conn)\nFile \"C:\\Users\\BBB\\AppData\\Local\\Programs\\Python\\Python35\\lib\\site-packages\\urllib3\\connectionpool.py\", line 976, in validate_conn\nconn.connect()\nFile \"C:\\Users\\BBB\\AppData\\Local\\Programs\\Python\\Python35\\lib\\site-packages\\urllib3\\connection.py\", line 370, in connect\nssl_context=context,\nFile \"C:\\Users\\BBB\\AppData\\Local\\Programs\\Python\\Python35\\lib\\site-packages\\urllib3\\util\\ssl.py\", line 377, in ssl_wrap_socket\nreturn context.wrap_socket(sock, server_hostname=server_hostname)\nFile \"C:\\Users\\BBB\\AppData\\Local\\Programs\\Python\\Python35\\lib\\ssl.py\", line 376, in wrap_socket\ncontext=self)\nFile \"C:\\Users\\BBB\\AppData\\Local\\Programs\\Python\\Python35\\lib\\ssl.py\", line 747, in init\nself.do_handshake()\nFile \"C:\\Users\\BBB\\AppData\\Local\\Programs\\Python\\Python35\\lib\\ssl.py\", line 983, in do_handshake\nself.sslobj.do_handshake()\nFile \"C:\\Users\\BBB\\AppData\\Local\\Programs\\Python\\Python35\\lib\\ssl.py\", line 628, in do_handshake\nself.sslobj.do_handshake()\nurllib3.exceptions.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (ssl.c:646)\nDuring handling of the above exception, another exception occurred:\nTraceback (most recent call last):\nFile \".\\test_main.py\", line 2, in \nimport test_provision_api\nFile \"C:\\Users\\BBB\\Documents\\Drive D\\GIT_Local\\NBNBNN\\Python_Logging\\V2.1\\test_provision_api.py\", line 39, in \nobj_PdAdmin.call_provisionadmin(pv_url1, WssVersion_list[0], master_AffId[0], master_Culture[0])\nFile \"C:\\Users\\BBB\\Documents\\Drive D\\GIT_Local\\NBNBNN\\Python_Logging\\V2.1\\test_provision_api.py\", line 26, in call_provisionadmin\nHttpJsonlogger.info('PD Admin - \/ProductDefinition', extra=pd_admin_extra)\nFile \"C:\\Users\\BBB\\AppData\\Local\\Programs\\Python\\Python35\\lib\\logging_init.py\", line 1279, in info\nself.log(INFO, msg, args, **kwargs)\nFile \"C:\\Users\\BBB\\AppData\\Local\\Programs\\Python\\Python35\\lib\\logging_init.py\", line 1415, in log\nself.handle(record)\nFile \"C:\\Users\\BBB\\AppData\\Local\\Programs\\Python\\Python35\\lib\\logging_init.py\", line 1425, in handle\nself.callHandlers(record)\nFile \"C:\\Users\\BBB\\AppData\\Local\\Programs\\Python\\Python35\\lib\\logging_init.py\", line 1487, in callHandlers\nhdlr.handle(record)\nFile \"C:\\Users\\BBB\\AppData\\Local\\Programs\\Python\\Python35\\lib\\logging_init.py\", line 855, in handle\nself.emit(record)\nFile \"C:\\Users\\BBB\\AppData\\Roaming\\Python\\Python35\\site-packages\\watchtower_init.py\", line 174, in emit\nlogStreamName=stream_name)\nFile \"C:\\Users\\BBB\\AppData\\Roaming\\Python\\Python35\\site-packages\\watchtower_init.py\", line 20, in _idempotent_create\n_callable(args, **kwargs)\nFile \"C:\\Users\\BBB\\AppData\\Local\\Programs\\Python\\Python35\\lib\\site-packages\\botocore\\client.py\", line 316, in _api_call\nreturn self._make_api_call(operation_name, kwargs)\nFile \"C:\\Users\\BBB\\AppData\\Local\\Programs\\Python\\Python35\\lib\\site-packages\\botocore\\client.py\", line 622, in _make_api_call\noperation_model, request_dict, request_context)\nFile \"C:\\Users\\BBB\\AppData\\Local\\Programs\\Python\\Python35\\lib\\site-packages\\botocore\\client.py\", line 641, in _make_request\nreturn self._endpoint.make_request(operation_model, request_dict)\nFile \"C:\\Users\\BBB\\AppData\\Local\\Programs\\Python\\Python35\\lib\\site-packages\\botocore\\endpoint.py\", line 102, in make_request\nreturn self._send_request(request_dict, operation_model)\nFile \"C:\\Users\\BBB\\AppData\\Local\\Programs\\Python\\Python35\\lib\\site-packages\\botocore\\endpoint.py\", line 137, in _send_request\nsuccess_response, exception):\nFile \"C:\\Users\\BBB\\AppData\\Local\\Programs\\Python\\Python35\\lib\\site-packages\\botocore\\endpoint.py\", line 256, in _needs_retry\ncaught_exception=caught_exception, request_dict=request_dict)\nFile \"C:\\Users\\BBB\\AppData\\Local\\Programs\\Python\\Python35\\lib\\site-packages\\botocore\\hooks.py\", line 356, in emit\nreturn self._emitter.emit(aliased_event_name, **kwargs)\nFile \"C:\\Users\\BBB\\AppData\\Local\\Programs\\Python\\Python35\\lib\\site-packages\\botocore\\hooks.py\", line 228, in emit\nreturn self._emit(event_name, kwargs)\nFile \"C:\\Users\\BBB\\AppData\\Local\\Programs\\Python\\Python35\\lib\\site-packages\\botocore\\hooks.py\", line 211, in _emit\nresponse = handler(kwargs)\nFile \"C:\\Users\\BBB\\AppData\\Local\\Programs\\Python\\Python35\\lib\\site-packages\\botocore\\retryhandler.py\", line 183, in call\nif self._checker(attempts, response, caught_exception):\nFile \"C:\\Users\\BBB\\AppData\\Local\\Programs\\Python\\Python35\\lib\\site-packages\\botocore\\retryhandler.py\", line 251, in call\ncaught_exception)\nFile \"C:\\Users\\BBB\\AppData\\Local\\Programs\\Python\\Python35\\lib\\site-packages\\botocore\\retryhandler.py\", line 277, in _should_retry\nreturn self._checker(attempt_number, response, caught_exception)\nFile \"C:\\Users\\BBB\\AppData\\Local\\Programs\\Python\\Python35\\lib\\site-packages\\botocore\\retryhandler.py\", line 317, in call\ncaught_exception)\nFile \"C:\\Users\\BBB\\AppData\\Local\\Programs\\Python\\Python35\\lib\\site-packages\\botocore\\retryhandler.py\", line 223, in call\nattempt_number, caught_exception)\nFile \"C:\\Users\\BBB\\AppData\\Local\\Programs\\Python\\Python35\\lib\\site-packages\\botocore\\retryhandler.py\", line 359, in _check_caught_exception\nraise caught_exception\nFile \"C:\\Users\\BBB\\AppData\\Local\\Programs\\Python\\Python35\\lib\\site-packages\\botocore\\endpoint.py\", line 200, in _do_get_response\nhttp_response = self._send(request)\nFile \"C:\\Users\\BBB\\AppData\\Local\\Programs\\Python\\Python35\\lib\\site-packages\\botocore\\endpoint.py\", line 269, in _send\nreturn self.http_session.send(request)\nFile \"C:\\Users\\BBB\\AppData\\Local\\Programs\\Python\\Python35\\lib\\site-packages\\botocore\\httpsession.py\", line 281, in send\nraise SSLError(endpoint_url=request.url, error=e)\nbotocore.exceptions.SSLError: SSL validation failed for https:\/\/logs.-**-.amazonaws.com\/ [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:646)","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":508,"Q_Id":62941750,"Users Score":0,"Answer":"I don't know the specific about your case but certificate validation always end up with the following solutions:\n\nthe certificate is expired\nthe certificate is not trusted because you don't have the public key or the certificate authority in your trust file\/truststore\nthe domain is not covered by the certificate\n\nThe message provided seems to tell the domain https:\/\/logs.-**-.amazonaws.com is not covered by the certificate\nSo download the certificate, get the details and check if:\n\n\"Delivered to\" field cover the domain name\none of \"Subject Alternative Name (SAN)\" cover the domain name\n\nEDIT:\nTo download the certificate either use a browser to load the url and use the \"lock\" icon to download the certificate or use openssl utility to download the certificate from the url. To get the details, save the certificate with the \".cer\" extension and double click on it or use openssl utility to get the details from a certificate file.\nGood luck","Q_Score":0,"Tags":"python-3.x,amazon-cloudwatchlogs","A_Id":62942895,"CreationDate":"2020-07-16T18:56:00.000","Title":"ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:646)","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":1,"Web Development":0},{"Question":"Let us say, a gRPC client makes two requests R1 and R2 to gRPC server, one after the other (assume without any significant time gap, i.e R2 is made when R1 is still not served). Also, assume that R1 takes much more time than R2.\nIn this case, should I expect R2's response first as it takes less time or should I expect R1's response first as this request is made prior to R2? What will happen and why?\nAs far as what I have observed, I think requests are served in FCFS fashion, so, R1's response will be received by the client first and then R2's, but I am not sure.","AnswerCount":2,"Available Count":2,"Score":0.1973753202,"is_accepted":false,"ViewCount":3160,"Q_Id":62949357,"Users Score":2,"Answer":"Theoretically nothing discourages server and client process gRPC requests in parallel. GRPC connection is made over HTTP\/2 one that can handle multiple requests at once. So yes - if server doesn't use some specific synchronization or limitation mechanisms then requests would be processes with overlapping. If server resources or policy doesn't allow it then they should be processed one by one. Also I can add than request can have a Timeout after which it would be cancelled. So long wait can lead to cancellation and non-processing at all.","Q_Score":0,"Tags":"web,protocol-buffers,grpc,rpc,grpc-python","A_Id":62951845,"CreationDate":"2020-07-17T07:42:00.000","Title":"How does gRPC handle multiple overlapping requests?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"Let us say, a gRPC client makes two requests R1 and R2 to gRPC server, one after the other (assume without any significant time gap, i.e R2 is made when R1 is still not served). Also, assume that R1 takes much more time than R2.\nIn this case, should I expect R2's response first as it takes less time or should I expect R1's response first as this request is made prior to R2? What will happen and why?\nAs far as what I have observed, I think requests are served in FCFS fashion, so, R1's response will be received by the client first and then R2's, but I am not sure.","AnswerCount":2,"Available Count":2,"Score":-0.1973753202,"is_accepted":false,"ViewCount":3160,"Q_Id":62949357,"Users Score":-2,"Answer":"All requests should be processed in parallel. The gRPC architecture for the Java implementation for example, it is divided into 2 \"parts\":\n\nThe event loop runs in a thread work group - It is similar to what we have to reactive implementations. One thread per core to handle the incoming requests.\n\nThe request processing is done in a dedicated thread which will be created using the CachedThreadPool system by default.\n\n\nFor single-thread languages like Javascript, I am not sure how they are doing it, but I would guess it is done in the same thread and therefore it would end up queuing the requests.","Q_Score":0,"Tags":"web,protocol-buffers,grpc,rpc,grpc-python","A_Id":63014195,"CreationDate":"2020-07-17T07:42:00.000","Title":"How does gRPC handle multiple overlapping requests?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"My API code is in python. It is hosted and works perfectly when called through postman. But when I try calling the same calls through my frontend react application, most of them respond 400 Bad request. Any idea as to why is this happening?\nMy application uses amazon cognito for authentication.","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":162,"Q_Id":62950954,"Users Score":0,"Answer":"My GET requests were not working because I was sending content-type: \"application\/json\" in headers which was not required. Now its working fine.","Q_Score":1,"Tags":"python,reactjs,api,amazon-cognito,http-status-code-400","A_Id":63298179,"CreationDate":"2020-07-17T09:17:00.000","Title":"Most of my APIs are returning 400 bad request error while being called on frontend","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I found that most modern browsers support HSTS and switch over to HTTPS if they find it in the HTTP\/HTTPS headers for the domain. Browsers would implemented appropriate response to teh HSTS header.\n\nWhat about client applications that are making http\/https calls to the web servers?\nDo standard http clients like apache commons library ot python requests already support that inherently? Or some flags have to bee set? How can that be done?","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":180,"Q_Id":62955043,"Users Score":0,"Answer":"There's no way to answer for libraries \"like\" A or B. But typically libraries implement the protocol, but don't store any data by themselves (they have no way to persist stuff), so it'll be in the responsibility of the application using the library in question to store the HSTS information somewhere.\nUnless you're using a library that has this separate store. I'm not aware that any does (or doesn't), you'll have to check with the one that you intend to use. But my expectation is: You'll have to do it yourself.","Q_Score":0,"Tags":"python-requests,apache-commons-httpclient,hsts","A_Id":62956347,"CreationDate":"2020-07-17T13:30:00.000","Title":"Do applications have to explicitly implement action to HHTPS HSTS?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I have a python script that runs a discord bot. The bot only works when the script is running, which means that in order to use it I must constantly be running a python script which is something I don't want to do. I've looked into cloud services to deploy it on and I keep getting talk about event loops and other mumbo jumbo. Is there a safe and easy way to deploy a python script over a cloud of some sort.\nAny help is very welcome.","AnswerCount":2,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":167,"Q_Id":62989198,"Users Score":1,"Answer":"I personally suggest Heroku, I'm using it for running multiple scripts for free. You can use scheduler addon for scheduling scripts to run at particular time as well. Please refer to heroku documentation & get started!","Q_Score":2,"Tags":"python,python-3.x,cloud,discord.py","A_Id":62989248,"CreationDate":"2020-07-20T05:34:00.000","Title":"Can I deploy python scripts over a cloud instead of running them on my own computer?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I have one simple question, is there a easy way to know the type of API's response?\nFox example:\nUsing requests post method to send api requests, some apis will return data format as .xml type or .json type,\nhow can i know the response type so i can choose not to convert to .json use json() when response type is .xml?","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":1648,"Q_Id":62991425,"Users Score":1,"Answer":"Use r.headers.get('content-type') to get the response type","Q_Score":0,"Tags":"python-3.x,python-requests","A_Id":62991684,"CreationDate":"2020-07-20T08:27:00.000","Title":"How to know the response data type of API using requests","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"i just started working with python and with some scripts to automate some tasks. I manage to create a web scraper using selenium that notifies me daily with the latest news via a mail . Now the script works great if i run it manually , but my question is : How can i make the script to run automatically every day , and what option would you recomand . I know that you can use TaskScheduler and program it , but i was wondering if you know any options to run it directly in the cloud ? Also , i found some tutorials where they say to wrap all the code in an infinite loop and add a timer , and once you run the script it will stay active as a background process,\nI just started working with scripts and automation , so i would really like to know your opinion on the best way to run the script automatically . Thanks a lot","AnswerCount":3,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":663,"Q_Id":63005252,"Users Score":0,"Answer":"Im not sure if that is what you are looking for, but (if you are on windows) you could attach your program to your autostart. Probably your Startup Folder: C:\\ProgramData\\Microsoft\\Windows\\Start Menu\\Programs\\Startup","Q_Score":1,"Tags":"python-3.x,selenium,web-scraping,automation,background","A_Id":63005523,"CreationDate":"2020-07-20T23:34:00.000","Title":"Python automation project to run in the background","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I want to automate downloading using selenium python which in turn carries the link to IDM. However, the thing is I can't get to download using IDM.","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":195,"Q_Id":63012062,"Users Score":1,"Answer":"Thisis not good practice in selenium automation\nWhilst it is possible to start a download by clicking a link with a browser under Selenium\u2019s control, the API does not expose download progress, making it less than ideal for testing downloaded files. This is because downloading files is not considered an important aspect of emulating user interaction with the web platform. Instead, find the link using Selenium (and any required cookies) and pass it to a HTTP request library like libcurl.\nPlease refer seleniumhq site","Q_Score":2,"Tags":"python,selenium,automation","A_Id":63012470,"CreationDate":"2020-07-21T10:02:00.000","Title":"How can I automate downloads using IDM?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I used import twitter and then tried authenticating using the following command - Twitter(auth=OAuth(access_token, access_token_secret, consumer_key, secret_key))\nOn running the program I get the error: name Twitter not defined. When I use from twitter import *, it works. Why is it so?\nI am asking this because if i use tweepy instead, a simple import tweepyworks. Also, doesn't import twitter work the same as from twitter import *?","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":29,"Q_Id":63019030,"Users Score":2,"Answer":"Try twitter.Twitter (auth=OAuth(access_token, access_token_secret, consumer_key, secret_key))","Q_Score":0,"Tags":"python,twitter","A_Id":63019081,"CreationDate":"2020-07-21T16:36:00.000","Title":"error: name Twitter not defined on using import twitter","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"How do I wait for all the new elements that appear on the screen to load after clicking a specific button? I know that I can use the presence_of_elements_located function to wait for specific elements, but how do I wait until all the new elements have loaded on the page? Note that these elements might not necessarily have one attribute value like class name or id.","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":193,"Q_Id":63021972,"Users Score":2,"Answer":"Well in reality you can't, but you can run a script to check for that.\nHowever be wary that this will not work on javascript\/AJAX elements.\nself.driver.execute_script(\"return document.readyState\").equals(\"complete\"))","Q_Score":0,"Tags":"python,selenium,selenium-chromedriver","A_Id":63022098,"CreationDate":"2020-07-21T19:42:00.000","Title":"Selenium(Python): After clicking button, wait until all the new elements (which can have different attributes) are loaded","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I work in tech support and currently have to manually keep our manuals for products updated manually by periodically checking to see if it has an update and if it does replacing the current one saved on our network.\nI was wondering if it would be possible to build a small program to quickly download all files on a suppliers website and have them automatically download and be sorted into the given folders for those products, replacing the current PDF's in that file. I must also note that the website is password protected and is sorted into folders.\nIs this possible with Python? I figured a small program I could perhaps run once a week or something to automatically update our manuals would be super useful (and a learning experience).\nApologies if I haven't explained the requirement well, any questions let me know.","AnswerCount":2,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":77,"Q_Id":63022562,"Users Score":1,"Answer":"It's certainly possible. As the other answer suggests you will want to use libaries like Requests (Handle HTTP requests) or Selenium (AUtomated browser activity) to navigate through the login.\nYou'll need to sort through the links on a given page, could be done with beautifulsoup ideally (An HTML parser) but could be done with selenium (Automated Browser activity).You'll need to check out libraries like requests (To handle HTTP requests) for downloading the pdf's, the OS module for sorting the folders out into specific folders and replacing files.\nI strongly urge you to think through the steps, But I hope that gives an idea about the libaries that you'll need to learn abit about. The most challenging thing to learn will be using selenium, so if you can use requests to do the login that is much better.\nIf you've got a basic grasp of python the requests, OS module and beautifulsoup libraries are not difficult things to pick up.","Q_Score":0,"Tags":"python,web-scraping,download","A_Id":63028786,"CreationDate":"2020-07-21T20:26:00.000","Title":"Scraping PDF's from a password protected website","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I'm using an API in python and when I make a request to this API, the server must send a audio mensage in a channel.\nIn a text channel I use a webhook, that gives me the context of the channel. I need something like that to be abble to play the audio.\nThere is any way to do this?","AnswerCount":2,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":940,"Q_Id":63044586,"Users Score":0,"Answer":"You can not play the audio file after sending it through bot.\nWhat you can do is download the audio file and play it using FFMPEG","Q_Score":0,"Tags":"python,api,discord,discord.py,discord.py-rewrite","A_Id":63314438,"CreationDate":"2020-07-22T23:27:00.000","Title":"How use a API call with discord.py?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I can get to the user information using the API but I cannot access course information. Can someone explain what I need to do to make the correct call for course information?","AnswerCount":2,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":267,"Q_Id":63046118,"Users Score":0,"Answer":"The easiest way to answer these questions is to try it in Postman. Highly recommended.","Q_Score":1,"Tags":"python,api","A_Id":68505713,"CreationDate":"2020-07-23T02:51:00.000","Title":"Schoology API understanding","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I created an Python Web Scraper app and it works fine on my pc but sadly not for my friend who I sent it to. I used Pyinstaller to create the application and got two folders, \"build\" and \"dist\". I sent the information of both of them to my friend. Still it doesn't work...can anyone help me please?\nStay healty!","AnswerCount":1,"Available Count":1,"Score":0.3799489623,"is_accepted":false,"ViewCount":33,"Q_Id":63056522,"Users Score":2,"Answer":"Have you added the webdriver binary to pyinstaller?","Q_Score":0,"Tags":"python,selenium-webdriver,tkinter,pyinstaller","A_Id":63057233,"CreationDate":"2020-07-23T14:22:00.000","Title":"Python (with Tkinter and Selenium Webdriver, created with pyinstaller) app doesn't work on other pc's","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I want to develop a python script to create topics in MSK using AWS SDK boto3. I don't see any methods to create a topic in Kafka.client. I can use kafka-python but wanted to use AWS SDK or CDK\nPlease suggest.","AnswerCount":1,"Available Count":1,"Score":0.3799489623,"is_accepted":false,"ViewCount":1175,"Q_Id":63060816,"Users Score":2,"Answer":"Amazon MSK APIs are meant for operations for managing an Amazon MSK cluster. For operations with Apache Zookeeper or Apache Kafka you can use kafka-python or execute Kafka\/Zookeeper APIs from a jumpbox to MSK brokers directly.","Q_Score":1,"Tags":"python-3.x,boto3,aws-msk","A_Id":63061731,"CreationDate":"2020-07-23T18:26:00.000","Title":"Creating MSK topic using python script and boto3","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I'm pretty new to python and scrapy. Right now I'm using a spider that is utilizing selenium to load JS pages. I'm running a windows computer through Bootcamp and I have about 10GB of storage left, however, when I run my scrapy spider, within 30 minutes of running it takes up almost half of that storage and I'm not sure why. I turned HTTP Cache off in settings.py but it still is eating up several GBs of storage.\nI'm using the -o command in my cmd to save the output to a csv, but that's the only thing I can think of that would take up storage. Any advice would be greatly appreciated!","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":39,"Q_Id":63083380,"Users Score":0,"Answer":"You have a couple of options with your information given. Most notably, run the python script on your mac instead of the Bootcamp windows. If your storage is being eaten up, you can try writing to a .txt instead of .csv. Maybe try giving an example of what you are loading up \/ trying to do so I could replicate","Q_Score":0,"Tags":"python,selenium,web-scraping,scrapy","A_Id":63083404,"CreationDate":"2020-07-25T01:30:00.000","Title":"Scrapy using significant computer storage","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I'm running Python's http.server with this command:\npy -m http.server --bind 0.0.0.0\nI can access the server on the same computer by typing bob:8000 in a browser's address bar (I may or may not have named my computer Bob), but when I try to access the server in the same way from a different computer on the same network, it says\n\nThis site can\u2019t be reached\nbob\u2019s server IP address could not be found.\n\nI was able to access this server previously, but I haven't been able to since upgrading my internet and router. How can I access this server from another computer?","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":572,"Q_Id":63083674,"Users Score":2,"Answer":"hey there you should be using your IPv4 address if you are accessing it from another computer.You can only access the 0.0.0.0:8000 from your computer similarly you can only use the following address from your computer\n\n127.0.0.0:anyport\nlocalhost:anyport\n0.0.0.0:anyport\n\nYou can always find out your IPv4 address by typing ipconfig in cmd the the IPv4 address would be looking like this\n192.168.xxx.xxx,10.0.xxx.xxx\nAfter you try this you should be able to access the page from another computer\nGood luck!","Q_Score":3,"Tags":"python,networking,server","A_Id":63083771,"CreationDate":"2020-07-25T02:25:00.000","Title":"Why can't I access a simple Python server on a different computer?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"Is it possible to set a prefix for each cog?\nFor example the Cog with admin commands has the prefix pa! and the cog for some fun commands pf!.\nWithout using on_message.\nEdit:\nI guess I have to go more in detail:\n\nMy cog for serverstats has a command called 'stats', this command should get\ntriggered when I'm using pa!\nMy cog for fun has a 'stats' command too. This should be triggered when I'm\nusing pf!\n\nMy only thought up to now was to set the default prefix to p! and then call the command astats. But I want to have a better way.","AnswerCount":4,"Available Count":2,"Score":0.0,"is_accepted":false,"ViewCount":681,"Q_Id":63105582,"Users Score":0,"Answer":"If all of your commands are in the on_message event, you can split the message content with spaces (\" \") and check the first element of the splitted list for the prefix and put the avaibale commands under an if statement. This could work, but will be slighly resource heavy depending on the amount of the commands","Q_Score":3,"Tags":"python,discord.py","A_Id":70622633,"CreationDate":"2020-07-26T20:53:00.000","Title":"Different prefix for each Cog?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"Is it possible to set a prefix for each cog?\nFor example the Cog with admin commands has the prefix pa! and the cog for some fun commands pf!.\nWithout using on_message.\nEdit:\nI guess I have to go more in detail:\n\nMy cog for serverstats has a command called 'stats', this command should get\ntriggered when I'm using pa!\nMy cog for fun has a 'stats' command too. This should be triggered when I'm\nusing pf!\n\nMy only thought up to now was to set the default prefix to p! and then call the command astats. But I want to have a better way.","AnswerCount":4,"Available Count":2,"Score":0.0,"is_accepted":false,"ViewCount":681,"Q_Id":63105582,"Users Score":0,"Answer":"I think this is an interesting question as I have had the same one for some time. I had an idea for implementing this myself although it does use the on_message function but not just that.\nBasically my thought was to override the cog class and add a way for it to denote what prefix it should have. I thought use a custom argument in @commands.command to signify what prefix to use but this was for commands themselves.\nThen in the on_message function you will need to determine there what command to call. Possibly the use of a global dict that correlates what prefix to what command.\nThis was just my own personal brainstorming and you will need to keep in mind if you allow each server to set their own prefixes and how to go about allowing that plus allowing them to set custom prefixes for each command. Good luck I hope this helps","Q_Score":3,"Tags":"python,discord.py","A_Id":63105679,"CreationDate":"2020-07-26T20:53:00.000","Title":"Different prefix for each Cog?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I have a discord bot written in python. It doesn't do much, however if you spam commands while it is in the middle to replying to another command, it freaks out. is there a way to make it refuse other commands until it has finished with the current process?","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":47,"Q_Id":63127434,"Users Score":1,"Answer":"Maybe some global variable \"ready\" and just allow to run commands if ready is true?\nAt start of command set ready to false and at the end to true.","Q_Score":1,"Tags":"python,discord,bots","A_Id":63129807,"CreationDate":"2020-07-28T05:13:00.000","Title":"(discord.py) How can I make it so my bot that doesn't reply to other commands until the current command is finished","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"New to Python and getting errors when importing modules. I have the following structure (not sure if this is a good way to show folder structure):\n\necommerce\n\ncustomer\n\n__init__.py\ncontact.py\n\n\nshopping\n\n__init__.py\nsales.py\n\n\n\n\n__init__.py\n\n\nI want to import contact.py from customer into sales.py in shopping but get presented with: ModuleNotFoundError: No module named 'eccommerce'.\nI'm using this:\nfrom ecommerce.customer import contact at start of the sales.py file.\nAny ideas?\nVSCode, MacOS 10.14.6","AnswerCount":2,"Available Count":2,"Score":0.0,"is_accepted":false,"ViewCount":162,"Q_Id":63129976,"Users Score":0,"Answer":"from ecommerce.customer import contact would work if 'customer' was a class in the ecommerce.py file, but in your case, those are different files so it doesn't work...\nThe correct syntax would be:\nfrom customer import contact\nbut you need to adjust those files.\nAnother way around would be to import os, navigate to that ecommerce folder like so: os.getcwd('insert dir path here'), and then import customer.","Q_Score":0,"Tags":"python,python-3.x,import,directory","A_Id":63130145,"CreationDate":"2020-07-28T08:27:00.000","Title":"ModuleNotFoundError - Python3","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":1,"System Administration and DevOps":0,"Web Development":0},{"Question":"New to Python and getting errors when importing modules. I have the following structure (not sure if this is a good way to show folder structure):\n\necommerce\n\ncustomer\n\n__init__.py\ncontact.py\n\n\nshopping\n\n__init__.py\nsales.py\n\n\n\n\n__init__.py\n\n\nI want to import contact.py from customer into sales.py in shopping but get presented with: ModuleNotFoundError: No module named 'eccommerce'.\nI'm using this:\nfrom ecommerce.customer import contact at start of the sales.py file.\nAny ideas?\nVSCode, MacOS 10.14.6","AnswerCount":2,"Available Count":2,"Score":0.0,"is_accepted":false,"ViewCount":162,"Q_Id":63129976,"Users Score":0,"Answer":"One thing to pay attention to is where is your current working directory located. If you have the cwd set to be in ecommerce than you should be able to access the contact with the syntax you inputted.\nI ran into this issue several times and in the python documentation this setup should technically work, init files being parsed by the python parser as individual packages which you can then access. It seems though that it won't work with the standard python interpreter and I haven't found the answer why. One way to get around this is to do as @Sory suggests, at the beginning of your package entry, add the path to the environment variable. This is a work around though and can lead to problems later on.\nAnother option is to use a separate python interpreter, i use for example the IPython interpreter from Jupyter which runs with this setup perfectly fine. This will give you another external dependency though.\nBe sure though that the current working directory is set to be the root folder though as that is usually the first problem.","Q_Score":0,"Tags":"python,python-3.x,import,directory","A_Id":63132230,"CreationDate":"2020-07-28T08:27:00.000","Title":"ModuleNotFoundError - Python3","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":1,"System Administration and DevOps":0,"Web Development":0},{"Question":"I have created an API using AWS Lambda function (using Python). Now my react js code hits this API whenever an event fire. So user can request API as many times the events are fired. Now the problem is we are not getting the response from lambda API sequentially. Sometime we are getting the response of our last request faster than the previous response of previous request.\nSo we need to handle our response in Lambda function sequentially, may be adding some delay between 2 request or may be implementing throttling. So how can I do that.","AnswerCount":3,"Available Count":2,"Score":1.2,"is_accepted":true,"ViewCount":363,"Q_Id":63149889,"Users Score":0,"Answer":"You can use SQS FIFO Queue as a trigger on the Lambda function, set Batch size to 1, and the Reserved Concurrency on the Function to 1. The messages will always be processed in order and will not concurrently poll the next message until the previous one is complete.\nSQS triggers do not support Batch Window - which will 'wait' until polling the next message. This is a feature for Stream based Lambda triggers (Kinesis and DynamoDB Streams)\nIf you want to streamlined process, Step Function will let you manage states using state machines and supports automatic retry based off the outputs of individual states.","Q_Score":0,"Tags":"python-3.x,reactjs,amazon-web-services,api,aws-lambda","A_Id":63158828,"CreationDate":"2020-07-29T08:54:00.000","Title":"How to set intervals between multiple requests AWS Lambda API","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I have created an API using AWS Lambda function (using Python). Now my react js code hits this API whenever an event fire. So user can request API as many times the events are fired. Now the problem is we are not getting the response from lambda API sequentially. Sometime we are getting the response of our last request faster than the previous response of previous request.\nSo we need to handle our response in Lambda function sequentially, may be adding some delay between 2 request or may be implementing throttling. So how can I do that.","AnswerCount":3,"Available Count":2,"Score":0.0665680765,"is_accepted":false,"ViewCount":363,"Q_Id":63149889,"Users Score":1,"Answer":"Did you check the concurrency setting on Lambda? You can throttle the lambda there.\nBut if you throttle the lambda and the requests being sent are not being received, the application sending the requests might be receiving an error unless you are storing the requests somewhere on AWS for being processed later.\nI think putting an SQS in front of lambda might help. You will be hitting API gateway, the requests get sent to SQS, lambda polls requests concurrently (you can control the concurrency) and then send the response back.","Q_Score":0,"Tags":"python-3.x,reactjs,amazon-web-services,api,aws-lambda","A_Id":63150857,"CreationDate":"2020-07-29T08:54:00.000","Title":"How to set intervals between multiple requests AWS Lambda API","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I have written a spider in scrapy. Everytime i try to scrape a specific website, the spider is running quite well at the start. but when i goes down to like 500-600 request, the spider start to crawl very slowly.\nI have checked memory and cpu settings but that is not the problem. Im quite sure that the website is slowing down my requests.\nHow do i adjust scrapy to scrape fast and consistently?","AnswerCount":1,"Available Count":1,"Score":0.1973753202,"is_accepted":false,"ViewCount":77,"Q_Id":63161677,"Users Score":1,"Answer":"If the website is slowing down your requests, it's probably best to be more careful with the requests you're sending.\nA couple of things you can try:\n\nSet 'AUTOTHROTTLE_ENABLED' to True: when scrapy sees the server is responding slowly, it will adjust the crawling speed\nLimit the number of concurrent requests yourself (by setting CONCURRENT_REQUESTS_PER_DOMAIN), and change the DOWNLOAD_DELAY to wait a bit between requests\nUse proxies: most likely the target website is limiting you based on the IP. Using proxies will overcome this, but this will not help in case the target server is overloaded by your requests","Q_Score":0,"Tags":"python,scrapy","A_Id":63167234,"CreationDate":"2020-07-29T20:08:00.000","Title":"Scrapy: website is slowing down my requests","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I'm making a discord bot that kicks bots that are invited to the server who were not invited by users with specific roles. I am struggling to find out who invited a bot to the server. I ran a few tests and came to the conclusion that an invite code\/link is not created when someone invites a bot. I know I can see who invited a bot in the AuditLogs but I want to kick a bot automatically if it is invited by an admin who should not be able to invite them.","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":97,"Q_Id":63163142,"Users Score":0,"Answer":"One way to solve this issue is to just not allow people other than admins to invite people to the server. However, if you want people to be ale to invite others, just not bots, here is what you can do: When a bot joins the server, you can use your bot to check iterate over the audit logs and check if the action is an invite and the target is the bot they you can check who invited the bot. From there on just check if who invited the bot is an admin, if not, kick the bot.","Q_Score":0,"Tags":"python,discord.py","A_Id":63207891,"CreationDate":"2020-07-29T22:06:00.000","Title":"Discord.py I want to have something in my code that gives me the person who invited a bot","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"When my colleague's Java(Client) TCP socket program send a package (500KB~1MB) to my Python(Server) TCP socket program, there are cases of subcontracting and sticking. I want to know, is there a universally valid solution to this situation? And what people usually do in this case?\nI think if it is possible to manually refresh the buffer zone requested by socket function recv(buffer_zone).If I can do that, the program will be free from sticking packages at least.","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":41,"Q_Id":63169189,"Users Score":0,"Answer":"It's me, and I've solved this problem by Head-Tail Package Mode.\nSpecifically, when send a serial of data to the server, the client may send a head package including stamp information and data length. When the server receive the head package sent from client, it can know the length that client will send to server, and the server can preper the same length cache for incoming data. When the client send the tail (data) package to the server, the server can depend on the length of the data it knows in advance and take out the data. That's my solution for now.","Q_Score":0,"Tags":"java,python,sockets,tcp","A_Id":63408074,"CreationDate":"2020-07-30T08:29:00.000","Title":"Unexpected Subcontract and Stick happened in TCP Socket between Python (Server) and Java (Client)","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I have selenium installed and chrome driver in a python project, when i create a new project is says it can't recognize the selenium module.\nDoes this mean with each project I'll need to install selenium again?","AnswerCount":4,"Available Count":2,"Score":0.0,"is_accepted":false,"ViewCount":671,"Q_Id":63189662,"Users Score":0,"Answer":"You don't need every time , you need use by giving path and locate to folder in your machine","Q_Score":0,"Tags":"python,selenium,selenium-webdriver,module,webdriver","A_Id":63190408,"CreationDate":"2020-07-31T09:58:00.000","Title":"Do we need to install Selenium each time we open and execute a new project using Python?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":1,"System Administration and DevOps":0,"Web Development":0},{"Question":"I have selenium installed and chrome driver in a python project, when i create a new project is says it can't recognize the selenium module.\nDoes this mean with each project I'll need to install selenium again?","AnswerCount":4,"Available Count":2,"Score":0.0,"is_accepted":false,"ViewCount":671,"Q_Id":63189662,"Users Score":0,"Answer":"yes you need to. In cases where you create virtual environment for each projects. It is better to have it install into the project virtual environment. This will save you a lot of stress debugging your code later in future.","Q_Score":0,"Tags":"python,selenium,selenium-webdriver,module,webdriver","A_Id":63190588,"CreationDate":"2020-07-31T09:58:00.000","Title":"Do we need to install Selenium each time we open and execute a new project using Python?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":1,"System Administration and DevOps":0,"Web Development":0},{"Question":"I want to delete files in Sharepoint 2016 from its recycle bin.\nI am able to delete files from my folder using requests.delete. However, not able to do the same from recycle bin.\nCan we achieve that?","AnswerCount":3,"Available Count":1,"Score":0.0665680765,"is_accepted":false,"ViewCount":1566,"Q_Id":63214420,"Users Score":1,"Answer":"Did you try Powershell script? There is a Sharepoint function for that","Q_Score":1,"Tags":"python,sharepoint,recycle-bin,sharepoint-2016","A_Id":63338021,"CreationDate":"2020-08-02T09:32:00.000","Title":"Delete Sharepoint files using Python","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I have issue with scraping page and getting json from it.\n<script type=\"text\/x-magento-init\"> inside is json that I'am trying to get but when I try with .find('script',{'type':'text\/x-magento-init'})I recive first json that I don't need. My question is how to find the 8th json with that same name? There is no other name\/id etc.","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":24,"Q_Id":63216006,"Users Score":0,"Answer":"you need to use re.findall(), that will return to you a list of matches. Then to get the 8th element you can go by result[7]","Q_Score":0,"Tags":"python,json","A_Id":63216032,"CreationDate":"2020-08-02T12:29:00.000","Title":"Getting json from html with same name","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I need to extract common data from different websites. Like I want to scrape 100 event websites and extract the same information like event name, price, location etc. Every website has a different layout so I'm writing scraping rules by hand. There are some services like diffbot that can extract this automatically. They are using some sort of AI\/ML model. I was wondering if this can be a Named Entity task or maybe LSTM can be used.","AnswerCount":2,"Available Count":2,"Score":0.0996679946,"is_accepted":false,"ViewCount":156,"Q_Id":63228794,"Users Score":1,"Answer":"Without more details on the structure\/format of your targeted websites, it's difficult to go beyond a generic answer.\nIf these are mostly text based (ie natural text not semi structured with table and all), then it seems like a classic information extraction (IE) of named entities. LSTM is an architecture that could be used for this like the ones in spacy. Many other classic NLP libraries like stanfordNLP can also be of use (not always with deep learning).\nHow to make the choice? It will depends on the type of language in these pages. If it's more natural English, then DL models could be better. If this is a domain jargon (small dataset to learn), you might need to investigate more grammar based analysis.","Q_Score":0,"Tags":"python,machine-learning,web-scraping,nlp,named-entity-recognition","A_Id":63229451,"CreationDate":"2020-08-03T11:42:00.000","Title":"Machine Learning model to parse webpage data and extract fields","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I need to extract common data from different websites. Like I want to scrape 100 event websites and extract the same information like event name, price, location etc. Every website has a different layout so I'm writing scraping rules by hand. There are some services like diffbot that can extract this automatically. They are using some sort of AI\/ML model. I was wondering if this can be a Named Entity task or maybe LSTM can be used.","AnswerCount":2,"Available Count":2,"Score":0.0,"is_accepted":false,"ViewCount":156,"Q_Id":63228794,"Users Score":0,"Answer":"To add to the previous response, don't forget to check if the websites you scrape have an API, which could highly reduce time spent coding, and be more reliable if the websites change their layouts.\nYou probably already checked it, but that doesn't hurt to remind this.","Q_Score":0,"Tags":"python,machine-learning,web-scraping,nlp,named-entity-recognition","A_Id":63229525,"CreationDate":"2020-08-03T11:42:00.000","Title":"Machine Learning model to parse webpage data and extract fields","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I have a python selenium script that runs after every 5 minutes inside the windows server and using a chrome driver. It takes 3-4 minutes to complete.\nIs there any possibility that it causes the server to stop working or make it down?\nbecause in my case it was running fine for 4-5 days after that, we found the server stop working and we require to restart it.","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":78,"Q_Id":63249101,"Users Score":0,"Answer":"Are you closing the browser with driver.quit() at the end of your script? Check if you have one multiple instances of chromedriver.exe running when your script is not actually running.","Q_Score":0,"Tags":"python,windows,selenium,selenium-chromedriver","A_Id":63774016,"CreationDate":"2020-08-04T14:30:00.000","Title":"Can selenium slow down or stop the windows server?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I'm working with a json API using python requests, the company who issued the API doesn't clarify in its documentation how many requests I can make per day\/hour\/minute.\nSometimes I am getting connection errors with it, is there any way to systematically deal with that? find out what the limits are or some way to test what they are efficiently?\nThank you.","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":58,"Q_Id":63253894,"Users Score":0,"Answer":"I am not an expert in this and your question was pretty vague so I can't really tell in what way you need to systematically deal with the problems that arise.\nIf it is something that is already running and only occasionaly fails due connection errors or request limits then I would suggest programming something that helps you track the requests and answers from the server by writing them into a text file so you can check later what the error message was and why it happened. To make it easier to go over this data you can have it as an object in your code and save it as a json or xml file so that you can open it and have it as an object again in python.\nFor example you could have an object that saves the number of requests you have sent, the number of errors you have gotten, a list of requests you have sent, a list of answers you have gotten from the server and a list with numbers of which requests have failed.\nIf it is something small scale or something that isn't working yet then I would suggest just going over the errors manually and just try to see if you can fix them. You could send alot of requests manually to check the request limit and you could try to find out what happens when a connection error occurrs and try to recognize that in python and send the request again.","Q_Score":0,"Tags":"python,api,python-requests","A_Id":63254238,"CreationDate":"2020-08-04T19:31:00.000","Title":"What's the best way to manage many requests from the same API?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":1,"System Administration and DevOps":0,"Web Development":0},{"Question":"Is it possible to send formatted message from Viber Bot with Markdown or HTML tags?\nIf yes - then how? What attributes to specify?","AnswerCount":2,"Available Count":2,"Score":1.2,"is_accepted":true,"ViewCount":1402,"Q_Id":63267388,"Users Score":3,"Answer":"Response from Viber Developer Support:\n\nformatting text with our API is only possible for keyboard or rich media buttons, not for text messages.","Q_Score":1,"Tags":"bots,viber,viber-api,viber-bot,viber-bot-python","A_Id":63446335,"CreationDate":"2020-08-05T14:30:00.000","Title":"How to send markdown or HTML message via Viber bot","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"Is it possible to send formatted message from Viber Bot with Markdown or HTML tags?\nIf yes - then how? What attributes to specify?","AnswerCount":2,"Available Count":2,"Score":0.0,"is_accepted":false,"ViewCount":1402,"Q_Id":63267388,"Users Score":0,"Answer":"Good news from future, add a character on both sides without space:\nbold with *,\nitalic with _,\nwasted with ~,\nmonospace with ```.\nBut the formatted view is rendered only in the mobile version.","Q_Score":1,"Tags":"bots,viber,viber-api,viber-bot,viber-bot-python","A_Id":66764341,"CreationDate":"2020-08-05T14:30:00.000","Title":"How to send markdown or HTML message via Viber bot","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I was trying to test run my python solution in Foobar platform using verify solution.py option . But I am continuously getting this error : 'CSRF Failed: CSRF token missing or incorrect.'\nShould I change my cookie preferences ?","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":167,"Q_Id":63270317,"Users Score":0,"Answer":"Try enabling cookies in your browser and reloading the page.","Q_Score":0,"Tags":"python,python-3.x,google-chrome,csrf","A_Id":65066245,"CreationDate":"2020-08-05T17:17:00.000","Title":"CSRF Failed: CSRF token missing or incorrect. error while submitting the foobar solution in Google's foobar platform","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I made a python program for PC remote controlling and when I use the socket.gethost() to get the ip for the server it works fine but if I use 192.168... it does not even connect (the PC's are on the same wifi though).\nI think the code is correct because if I try to connect the same PC as a host and as a client it works normally, so I wanted to know what ip I should use and if I want to use it on different Wi-Fi's what should ip should I use?\nIf you need the code just ask.\nI use Kali.","AnswerCount":1,"Available Count":1,"Score":0.1973753202,"is_accepted":false,"ViewCount":34,"Q_Id":63272740,"Users Score":1,"Answer":"for the bind address on the server file use 0.0.0.0 if you want to connect to it from a different device. then use its ip 192.168... to connect from the other device.","Q_Score":1,"Tags":"python,python-3.x,sockets,remote-control","A_Id":63272784,"CreationDate":"2020-08-05T20:03:00.000","Title":"socket not connecting","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I mean, how am I supposed to do something like this, when I click a button, a browser opens, on the page it url gives?\nMy guess is that a module is needed here, unless it can be done in the Tkinter module, because that's where I want this button (hyperlink, as in html)","AnswerCount":2,"Available Count":1,"Score":0.0996679946,"is_accepted":false,"ViewCount":355,"Q_Id":63314105,"Users Score":1,"Answer":"Try using the webbrowser module. You can have a look at this link for more information-https:\/\/docs.python.org\/2\/library\/webbrowser.html\nThe webbrowser module provides a high-level interface to allow displaying Web-based documents to users. Under most circumstances, simply calling the open() function from this module will do the right thing.\nUnder Unix, graphical browsers are preferred under X11, but text-mode browsers will be used if graphical browsers are not available or an X11 display isn\u2019t available. If text-mode browsers are used, the calling process will block until the user exits the browser.\nIf the environment variable BROWSER exists, it is interpreted to override the platform default list of browsers, as an os.pathsep-separated list of browsers to try in order. When the value of a list part contains the string %s, then it is interpreted as a literal browser command line to be used with the argument URL substituted for %s; if the part does not contain %s, it is simply interpreted as the name of the browser to launch. 1\nFor non-Unix platforms, or when a remote browser is available on Unix, the controlling process will not wait for the user to finish with the browser, but allow the remote browser to maintain its own windows on the display. If remote browsers are not available on Unix, the controlling process will launch a new browser and wait.\nThe script webbrowser can be used as a command-line interface for the module. It accepts a URL as the argument. It accepts the following optional parameters: -n opens the URL in a new browser window, if possible; -t opens the URL in a new browser page (\u201ctab\u201d). The options are, naturally, mutually exclusive.","Q_Score":2,"Tags":"python,hyperlink","A_Id":63314182,"CreationDate":"2020-08-08T10:01:00.000","Title":"What module for opening links in python?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":1,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"The code\ntext = pic.image_to_string(image)\nwebbrowser.open('https:\/\/www.google.com\/search?q='(text))\ni need to place the variable test in url so i can look up every image text scanned","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":14,"Q_Id":63332928,"Users Score":0,"Answer":"Using f-strings, for Python >= 3.6\nwebbrowser.open(f'https:\/\/www.google.com\/search?q={text}')\nUsing format\nwebbrowser.open('https:\/\/www.google.com\/search?q={}'.format(text))","Q_Score":0,"Tags":"python,python-3.x","A_Id":63333022,"CreationDate":"2020-08-10T01:18:00.000","Title":"need help to place a verable within a python webbrowser.open","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"When I'm done with automating, the chrome webdriver stays running in the background, is there anyway to automatically close it without the need of doing it via task manager?","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":59,"Q_Id":63333628,"Users Score":0,"Answer":"driver.close() => It will shut the focused browser window.\ndriver.quit() => It will shut all opened browser window and terminates the WebDriver session.","Q_Score":1,"Tags":"python,google-chrome,background,webdriver,taskmanager","A_Id":63333644,"CreationDate":"2020-08-10T03:11:00.000","Title":"Chrome webdriver - Python","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I wanna add members of a specific telegram group to another group. But there is an issue, I have a problem with flood waiting of Telegram.\nIs it possible to add whole members of the group with their user_id to my contacts then adding them to my group?","AnswerCount":1,"Available Count":1,"Score":-0.1973753202,"is_accepted":false,"ViewCount":1788,"Q_Id":63337120,"Users Score":-1,"Answer":"This is not possible because you can not add a member who did not save the ID in your Telegram contact.\nit is like to create a new group.\nTelegram does not allow you","Q_Score":0,"Tags":"javascript,python,telegram,telegram-bot","A_Id":63337325,"CreationDate":"2020-08-10T09:00:00.000","Title":"How can I move member from telegram group to another group","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I have 2 Raspberry Pi 4 units.Both have been updated and upgraded and both flashed with rasbian lite. I created a really simply script, on the first raspi which connects to my pc on port 5999 and performs a basic function of grabbing the time and sending that on said port. My pc accepts it, prints it and writes it to a file. The second raspi has the exact same script with same permissions etc and the script does not work. No errors etc returned. I would for example run the server side on my pc, run the client side on the first raspi, works 100%, disconnect and run the same script on second raspi and no response on server side or client side. I have check \"netstat -a\"command, port listening etc on pc, have changed the port numbers on one and both raspis' but still cannot get the first raspi to send data.\nAny similar experiences? Thank you in advance.","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":22,"Q_Id":63338245,"Users Score":0,"Answer":"Problem solved, sorry guys, thx anyway for reply, omitted the 6 in the LAN IP address in the code on one of the raspberry Pi units.\nIf anyone wants the code then I will post, client and server working on PI4 units. Just let me know in comments.","Q_Score":0,"Tags":"python,raspberry-pi,raspberry-pi4,python-sockets","A_Id":63339650,"CreationDate":"2020-08-10T10:16:00.000","Title":"Two Raspi4 different responses on Socket Programming to PC","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I'd like to make a function which converts Google Drive videos into VLC streamable links (e.g. vlc:\/\/https:\/\/WEBSITE.com\/FILE_ID.mkv.\nI've tried methods which were shared on stack overflow, such as modifying the Google Drive link to:\nhttps:\/\/drive.google.com\/uc?export=download&id=FILE_ID\nAll the methods I've tried seem to not work anymore. Any ideas?","AnswerCount":2,"Available Count":1,"Score":-0.0996679946,"is_accepted":false,"ViewCount":1963,"Q_Id":63341586,"Users Score":-1,"Answer":"Just make the file public and copy your ID.\nYou can find it here: \/file\/d\/YOUR ID\/view?usp=sharing.\nCopy your ID and paste it in this:\ndrive.google.com\/uc?export=view&id=YOUR ID","Q_Score":4,"Tags":"python,google-drive-api","A_Id":70487792,"CreationDate":"2020-08-10T13:53:00.000","Title":"How do I get the direct link for a Google Drive video (Google Drive API v3)","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I just Started using Google Drive API using python to download File. I'm seeing it requires authentication for the first time and generates a token.pickle file. From next time onwards we can use this token file for API call.\nNote : The file token.pickle stores the user's access and refresh tokens.\nWhat I needed to know is when this token file will expire ? If it expires then how to extend it's validity ?\nI've gone through many google documentations but couldn't find anything clear regarding this.\nIf anyone can suggest me any info on this, that will be most welcomed.","AnswerCount":2,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":2220,"Q_Id":63354260,"Users Score":0,"Answer":"If your app's publishing status is in 'Testing', then the token will last for 2 weeks only.\nIf its publishing status is 'in Production' then the token has no expiry. However if any sensitive scopes are there and not verified in your app, then the number of users also limited.\nThese are all tested but seems no official documentation is available yet.","Q_Score":3,"Tags":"python,google-api,google-drive-api,google-oauth","A_Id":68140154,"CreationDate":"2020-08-11T08:21:00.000","Title":"When does Refresh Token expires for Google Drive API?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I have Energy Meter connected through RTU and i am able to get holding registers data through simple RTU Code.\nNow i want to make Convert this RTU to TCP through Forwarder. I want to send data to TCP which forwards the command to RTU and fetches data of RTU connected device for me.\nI have implement the Forwarder code just dont know how to fetch the holding register of RTU through it.","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":88,"Q_Id":63356967,"Users Score":0,"Answer":"If you are working with real device, make sure you are explicitly initialising the remote slave context with the unit Id.\ncontext = RemoteSlaveContext(serial_client, unit=<unit-id-of-slave>)\nThe default is 0 which works fine with simulated slaves but would be considered as a broadcast address with the real devices generally and no response would be returned back.","Q_Score":0,"Tags":"python,python-3.x,pymodbus,pymodbus3","A_Id":63386969,"CreationDate":"2020-08-11T11:12:00.000","Title":"Read RTU Pymodbus data through TCP\/Forwarder","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":1,"Web Development":0},{"Question":"Sorry for this really vague question but I am just sort of stuck on how to phrase this question so I haven't really been able to find a proper answer just by googling.\nEssentially, I have this python script that alters a csv file in a very specific way. Right now it runs fine on my local machine, but I need this to work with the web. Essentially what I want to be possible is for someone to upload their CSV to my site, it would automatically send to my server, my script would see this, manipulate it in the way necessary, and then send it back once it was finished. Does anyone know any methods of doing this that I should look into?\nAgain, sorry this is so poorly researched sounding but I am just having a lot of trouble properly phrasing it so I haven't been able to find much.\nThanks!","AnswerCount":1,"Available Count":1,"Score":0.1973753202,"is_accepted":false,"ViewCount":26,"Q_Id":63380960,"Users Score":1,"Answer":"Multiple ways to make it happen.\nUse synchronous flow\nYou would upload the file from the browser, that would be handled by the Flask handler, the handler would process the file and return the updated file in the response. Here you can also use polling to poll the server to see if the processing is done or not using ajax along with polling you can use a loader.\nProblem: processing large file, slow internet connection\nUse asynchronous flow\n\nYou would upload the file from the browser, that would be processed later using some Celery task, other background processor or crontabs that would manipulate the file in the background.\nPost manipulation of the file, you can either give an option in the user profile to download the file, send an email attachment or provide download a link, etc.","Q_Score":0,"Tags":"python,flask","A_Id":63381411,"CreationDate":"2020-08-12T16:32:00.000","Title":"How would one make it so that I can send a file to a server and then have my python script manipulate it, and then send it back?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"Is there any way to get the person who invited the bot to the server? My point is to dm that person to tell her what to do so things go smoothly and I don't want the bot to just write it in a random text channel that everybody could see. Thanks in advance","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":210,"Q_Id":63386784,"Users Score":1,"Answer":"There isn't any way yet to know who invited the bot.\nWhat you can do is DM the Server Owner when its added to a Server.","Q_Score":1,"Tags":"python,python-3.x,discord,discord.py","A_Id":63386809,"CreationDate":"2020-08-13T01:23:00.000","Title":"Get person who invited the discord bot discord.py","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I have code on aws ec2. Right now, it accepts input and output files from s3. Its an inefficient process. I have to upload input file to s3, copy s3 to ec2, run program, copy output files from ec2 to s3, then download locally.\nIs there a way to run the code on ec2 and accept a local file as input and then have the output saved on my local machine?","AnswerCount":2,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":435,"Q_Id":63403563,"Users Score":0,"Answer":"@John Rotenstein we have solved the problem of loading 60MB+ models into Lambdas by attaching AWS EFS volumes via VPC. Also solves the problem with large libs such as Tensorflow, opencv etc. Basically lambda layers almost become redundant and you can really sit back and relax, this saved us days if not weeks of tweaking, building and cherry picking library components from source allowing us to concentrate on the real problem. Beats loading from S3 everytime too. The EFS approach would require an ec2 instance obviously.","Q_Score":0,"Tags":"python,amazon-web-services,amazon-s3,amazon-ec2","A_Id":63439343,"CreationDate":"2020-08-13T21:37:00.000","Title":"How to run code on aws ec2 with local input and output","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I am working on building a movie recommendation engine on aws sagemaker environment and i plan to show working demonstration for the same. I am trying to use the streamlit library for running the app.\nAfter running the command: streamlit run app.py, it provides me 2 urls i.e. Network and External urls.\nThe issues is none of the url works. It results in connection timed out, the server does not respond.\nPlease help","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":557,"Q_Id":63427965,"Users Score":0,"Answer":"You can try the following url https:\/\/{NOTEBOOK_URL}\/proxy\/8501\/. You may have to change the proxy based on which proxy is used in the Network and External urls","Q_Score":2,"Tags":"python,user-interface,jupyter,amazon-sagemaker,streamlit","A_Id":65145807,"CreationDate":"2020-08-15T15:48:00.000","Title":"Streamlit Browser App does not open from Sagemaker Terminal","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":1,"Web Development":1},{"Question":"I am using Google Cloud Document AI's Form Parser API. After i do the request to the API , I get a response with type google.cloud.documentai.v1beta2.types.document.Document. I tried to write it to JSON using json.dumps() but it gives JSONDecodeError because JSON.dumps() dont know how to serialize object of type google.cloud.documentai.v1beta2.types.document.Document.\nI am confused how to convert this to JSON\nAny Help Appreciated!","AnswerCount":2,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":439,"Q_Id":63428499,"Users Score":0,"Answer":"I solved my problem.\nbasically you have to write a function that explores the Document object, and then assemble entire JSON yourself by code.","Q_Score":0,"Tags":"python,json,google-api,google-api-python-client","A_Id":63439205,"CreationDate":"2020-08-15T16:44:00.000","Title":"How do i convert 'google.cloud.documentai.v1beta2.types.document.Document' into JSON?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":1,"System Administration and DevOps":0,"Web Development":0},{"Question":"I have a discord bot I need to scale.\nThe main features of the bot is to fetch data from a 3rd party website and also keep a database with member info.\nThese 2 operations are quite time consuming and I wanted to have a separate worker\/process for each of them.\nMy constraints:\n\nThere is a limit of GET's per min with the 3rd party website.\nThe database can't be accessed simultaneously for same guild.\n\nI've been researching online for the best way to do this but I come into several libraries\/ways to implement this kind of solution. What are the options I have and their strengths and weaknesses?","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":69,"Q_Id":63428663,"Users Score":0,"Answer":"Since there is a limit on the amount of requests from the host, I would first try to run a synchronous program and check whether the limit is reached before the minute ends. If it does then there would be no need to concurrently run other workers. However if the limit is not reached, then I would recommend you use both asyncio and aiohttp to asynchronously get the requests. There's a ton of information out there on how to get started using these libraries.\nThe other option would be to use the good old threading module (or concurrent.futures for a higher level use case). Both options have their pros and cons. What I would do is first try the concurrent.futures (namely, the ThreadPoolExecutor context manager) module since you only have to add like one line of code. If it does not get the job done, then remember: use asyncio if you have to, and threading if you must. Both of these modules are easy to use and understand as well, but they do need to follow a general structure, which means you'll most likely have to change your code.","Q_Score":0,"Tags":"python,discord.py","A_Id":63428890,"CreationDate":"2020-08-15T17:01:00.000","Title":"Python Multiprocessing: worker pool vs process","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":1,"System Administration and DevOps":0,"Web Development":0},{"Question":"I'm trying to make a command where when someone types for example '!messages @mee6' it shows how many messages that person has said in the server. So say if I typed \"a\" \"b\" \"c\" then did \"!messages\"\nthe bot would reply with \"@user has sent 3 messages in this server. Does anyone know if this would be possible and if so how would I go about doing it in discord.py?","AnswerCount":4,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":56,"Q_Id":63456592,"Users Score":0,"Answer":"The principle is the following:\nYou'd have to take advantage of the Message Received event. Once a message is received, update the counts for the person, and when the !messages command is ran, display the amount.\nLooping through every channel in the server and filtering all messages from the user is extremely inefficient and time taking, as well as it can get your bot ratelimited.","Q_Score":0,"Tags":"python,discord.py","A_Id":63457199,"CreationDate":"2020-08-17T18:23:00.000","Title":"Is it possible to make a command !messages @user","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I'm using a raspberry pi as a server to host a program. Part of what this program does is interface with websites such as spotify, google sheets, and google forms through their respective APIs (or a library wrapper)\nIn order to interface with spotify, i've attempted to use a python module called 'spotipy' which is a wrapper for the spotify API. However, i'm having trouble getting it to work, and I have a feeling its because the API requires authentication which utilizes the systems browser, i.e. chromium on the pi, but chromium does not have support for open.spotify.com due to its limited capabilities.\nIs it reasonable to say that a web API wont work without access to the website you are attempting to interface with?\nHopefully this is provides enough information, thanks.","AnswerCount":1,"Available Count":1,"Score":-0.1973753202,"is_accepted":false,"ViewCount":68,"Q_Id":63461337,"Users Score":-1,"Answer":"github.com\/plamere\/spotipy#quick-start\nto answer your question, you don't need chromium to use APIs. According to the quickstart that I linked above\n\nyou need to register your raspberry pi as an App on spotify dev site and acquire the \"credentials\"\nthen declare those credentials in your script","Q_Score":0,"Tags":"python,api,raspberry-pi,spotipy","A_Id":63461488,"CreationDate":"2020-08-18T03:11:00.000","Title":"Using web APIs with limited website access (raspberry pi)","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I just finished working on a PyCharm project (using python 3.7), which uses libraries downloaded using PyCharm's built in functionality. I just want to send this project to my boss so that he can simply run it by clicking on it, without having to go through downloading PyCharm and downloading the applicable packages. Please tell me that there is a way to do this. Thank you","AnswerCount":3,"Available Count":2,"Score":0.0,"is_accepted":false,"ViewCount":750,"Q_Id":63473594,"Users Score":0,"Answer":"You could solve this issue by converting the .py file to an executable file or .exe file....\nHopefully it helped..thx","Q_Score":0,"Tags":"python,pycharm","A_Id":63473611,"CreationDate":"2020-08-18T17:18:00.000","Title":"Sending a PyCharm project to someone","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":1,"System Administration and DevOps":0,"Web Development":0},{"Question":"I just finished working on a PyCharm project (using python 3.7), which uses libraries downloaded using PyCharm's built in functionality. I just want to send this project to my boss so that he can simply run it by clicking on it, without having to go through downloading PyCharm and downloading the applicable packages. Please tell me that there is a way to do this. Thank you","AnswerCount":3,"Available Count":2,"Score":0.0,"is_accepted":false,"ViewCount":750,"Q_Id":63473594,"Users Score":0,"Answer":"A simple way to that would be sharing it on github with a requirements.txt file containing all the dependencies which then can be installed using python -m pip install -r requirements.txt.\nOr You can use pyinstaller --onefile <yourfile.py> to create a .exe file which could be run without any requirements on windows.","Q_Score":0,"Tags":"python,pycharm","A_Id":63473710,"CreationDate":"2020-08-18T17:18:00.000","Title":"Sending a PyCharm project to someone","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":1,"System Administration and DevOps":0,"Web Development":0},{"Question":"I currently have a bot that pings people if certain criteria are met. The issue is that the bot will ping spam if the content is repeated. For example, say user JohnSmith#1234 says \"ping me if someone says 'apple' or 'apples' in the chat.' It'll do so. However, say there are 4 messages in quick succession:\n\nUser1: \"Hey, do you like apples?\"\nUser2: \"Yeah, I like apples\"\nUser1: \"What's your favorite kind of apple?\"\nUser2: \"I don't really have a favorite apple, but I like Granny Smith\"\n\nthen the bot will say:\n\n@JohnSmith#1234 for \"apple\"\n@JohnSmith#1234 for \"apple\"\n@JohnSmith#1234 for \"apple\"\n@JohnSmith#1234 for \"apple\"\n\nbecause \"apple\" was mentioned 4 times. Is there a way to tell the bot \"if you pinged JohnSmith#1234 in the last 10 seconds, don't ping him again?\" I basically want an action to not execute if it's been done recently.\nEdit:\nWhile the answers shown seem to work for just 1 user, they don't seem to work for multiple. The answers will accommodate JohnSmith#1234 just fine, but as of now, it'll render the bot useless until JohnSmith#1234's delay is done.\nI'm not sure how to tell the bot \"keep doing what you're doing, but run the delay loop on the side.\" Here's what the bot is currently doing, continuing off of the example I provided previously:\n1: Say @JohnSmith#1234 for 'apple'\n2: Puts John Smith into a waiting dictionary until 10 seconds is up. (Note: the bot won't do anything else but wait. This is bad because it should ping users besides John Smith).\n3: If those 10 seconds are up, it'll ping John Smith again. Otherwise, it'll keep doing stuff for other users.","AnswerCount":3,"Available Count":1,"Score":0.0665680765,"is_accepted":false,"ViewCount":380,"Q_Id":63476864,"Users Score":1,"Answer":"Create a cooldown dictionary where you can store people the bot has already pinged and the end of the cooldown. Ex: cooldows = {\"JohnSmith#1234\": 1597786825}.\nThen when your bot tries to ping someone just check if they are in the dictionary. if they are check the epoch timestamp, if the time now is greater than the time listed, remove the user from the dictionary and ping him, if its not don't ping them. If they are not in the dictionary the bot can ping them","Q_Score":0,"Tags":"python,discord.py","A_Id":63477038,"CreationDate":"2020-08-18T21:27:00.000","Title":"Is there a way to stop a Discord.py bot from doing a repeat action if it's been done in the past X seconds?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"Maybe is my question a little bit weird, but I am looking for an answer since couple days.\nThe only thing what I found were sockets and asyncio(dont really know what it is) , but with sockets you can establish a connection just in your localhost.\nIs there any otherways to creat a connection between two quite different pcs or between a client and server wireless.\nIf you have any ideas just write, maybe it could be helpful. As a beginner in python, I would be happy if you guys could show me some way to find easy tutorials (maybe some websides) and important things to learn. So everything could be helpful, I am just curious. :)","AnswerCount":3,"Available Count":2,"Score":0.0,"is_accepted":false,"ViewCount":39,"Q_Id":63478715,"Users Score":0,"Answer":"what you can do is create two seperate python files: client.py and server.py\nThe client.py will be trying making the connection. the server.py will be trying to receive that connection. You mentioned you weren't familiar with socket. Socket is a low-level networking interface when you import the library socket to a file, you are able to make system calls between different computers to interact with each other.","Q_Score":0,"Tags":"python,sockets,networking,server,client","A_Id":63478833,"CreationDate":"2020-08-19T01:20:00.000","Title":"How to establish a connection between two computers wirelessly?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"Maybe is my question a little bit weird, but I am looking for an answer since couple days.\nThe only thing what I found were sockets and asyncio(dont really know what it is) , but with sockets you can establish a connection just in your localhost.\nIs there any otherways to creat a connection between two quite different pcs or between a client and server wireless.\nIf you have any ideas just write, maybe it could be helpful. As a beginner in python, I would be happy if you guys could show me some way to find easy tutorials (maybe some websides) and important things to learn. So everything could be helpful, I am just curious. :)","AnswerCount":3,"Available Count":2,"Score":1.2,"is_accepted":true,"ViewCount":39,"Q_Id":63478715,"Users Score":1,"Answer":"You can create a server \/ client pair, through executables running on both PCs. TCP and UDP sockets can be used to communicate outside 'localhost'. You can do this rather easily in many programming languages like C, C#, python amongst others.","Q_Score":0,"Tags":"python,sockets,networking,server,client","A_Id":63483150,"CreationDate":"2020-08-19T01:20:00.000","Title":"How to establish a connection between two computers wirelessly?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I am working on a web scraping project. In this project, I have written the necessary code to scrape the required information from a website using python and selenium. All of this code resides in multiple methods in a class. This code is saved as scraper.py.\nWhen I execute this code, the program takes sometime between 6 and 10 seconds to extract all the necessary information from the website.\nI wanted to create a UI for this project. I used django to create the UI. In the webapp, there is a form which when submitted opens a new browser window and starts the scraping process.\nI access the scraper.py file in django views, where depending on the form inputs, the scraping occurs. While this works fine, the execution is very slow and takes almost 2 minutes to finish running.\nHow do I make the execution of the code faster using django faster? can you point me some tutorial on how to convert the scraper.py code into an api that django can access? will this help in making the code faster?\nThanks in advance","AnswerCount":1,"Available Count":1,"Score":0.3799489623,"is_accepted":false,"ViewCount":614,"Q_Id":63480479,"Users Score":2,"Answer":"Few tiny tips,\n\nHow is your scraper.py working in the first place? Does it simply print the site links\/details, or store it in a text file, or return them? What exactly happens in it?\nIf you wish to use your scraper.py as an \"API\" write your scraper.py code within a function that returns the details of your scraped site as a dictionary. Django's views.py can easily handle such dictionaries and send it over to your frontend HTML to replace the parts written in Jinja2.\nFurther speed can be achieved (in case your scraper does larger jobs) by using multi-threading and\/or multi-processing. Do explore both :)","Q_Score":0,"Tags":"python,django,selenium,django-views","A_Id":63480617,"CreationDate":"2020-08-19T05:22:00.000","Title":"How to scrape data faster with selenium and django","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I am trying to send json collections from a Flask server to remote Perforce depot using P4Python.\nIs there a way to do that through Python?","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":31,"Q_Id":63482517,"Users Score":0,"Answer":"The strategy for doing this via P4Python is going to be about the same as doing it via the command line or one of the other scripting APIs: you'll need to create a workspace, write the file to the workspace, and use p4 add and p4 submit.\nThere isn't an API that lets you pipe data directly into the server to be converted into a depot file.","Q_Score":0,"Tags":"python,flask,pymongo,perforce,p4python","A_Id":63489549,"CreationDate":"2020-08-19T08:00:00.000","Title":"Sending json collections to Perforce depot","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"We are developing REST APIs using python framework called FASTAPI. The code security test failed for html injection. They are sending some html tag code in the post payload, we are inserting that in DB and sending same in GET Response. Is there any way to prevent this HTML injection while processing request in FASTAPI.","AnswerCount":1,"Available Count":1,"Score":0.761594156,"is_accepted":false,"ViewCount":1102,"Q_Id":63488278,"Users Score":5,"Answer":"yes. No framework is going to magically change the content you get without you being explicit about it. (Imagine if it was a REST API for recording HTML snippets from an internal system to be used in rendering web pages in another endpoint: you'd need the HTML as is)\nIt is just a matter of calling a escape function on your input data, before putting that on the db.\nPython's standard library html.escape function suffices in this case.\nThere is no code in your question, and I don't know FASTAPI by heart - but if it puts the payload in the DB without going through any code you write, then you should either customize that and put in this call to preprocess your data, or add a triggered stage (that is, an event subscriber) that will do that for you.","Q_Score":2,"Tags":"python,fastapi,html-injections","A_Id":63488487,"CreationDate":"2020-08-19T13:39:00.000","Title":"How can we prevent html injection in FASTAPI?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I had the idea to synchronise the equal scripts from a python program\nthat Im running on different computers in parallel, by creating a\nglobal timestamp based of the script that startet first.\nSo for a better explanation, it should work like this:\nThe Script runs on PC 1 and creates a timestamp.\nEverything the script has to do after that will only be executed once\nthe time in the timestamp + 5 minutes is reached.\nFor the script on PC 2 or more, basically the same is happening but without\ncreating a new timestamp, since the starting time wouldnt be the\nsame anymore then.\nNow I already build a Python webserver that creates the timestamp and I am\nalso able to retrieve the time for my script. But I have to make sure that I\ncan reach the webserver without beeing in my own network and\nusing \"localhost:...\" as an adress. So now I wonder how I can make the webserver\npublic in the most easy way possible. The other option would be an\nalready public site, that can create custom timestamps on the basis of my configuration, so that I only create 1 timestamp for the first visitor on the site.\nDo you by chance know any of these sites or can tell me how I can make my HTML web server public?\nThanks","AnswerCount":4,"Available Count":2,"Score":0.049958375,"is_accepted":false,"ViewCount":3943,"Q_Id":63494148,"Users Score":1,"Answer":"You could create a proxy server on the same network with the pc you have the script running on and connect to it from the other pcs","Q_Score":0,"Tags":"python,timestamp,ip,webserver,public","A_Id":63494207,"CreationDate":"2020-08-19T19:58:00.000","Title":"How to make Python Web Server public?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I had the idea to synchronise the equal scripts from a python program\nthat Im running on different computers in parallel, by creating a\nglobal timestamp based of the script that startet first.\nSo for a better explanation, it should work like this:\nThe Script runs on PC 1 and creates a timestamp.\nEverything the script has to do after that will only be executed once\nthe time in the timestamp + 5 minutes is reached.\nFor the script on PC 2 or more, basically the same is happening but without\ncreating a new timestamp, since the starting time wouldnt be the\nsame anymore then.\nNow I already build a Python webserver that creates the timestamp and I am\nalso able to retrieve the time for my script. But I have to make sure that I\ncan reach the webserver without beeing in my own network and\nusing \"localhost:...\" as an adress. So now I wonder how I can make the webserver\npublic in the most easy way possible. The other option would be an\nalready public site, that can create custom timestamps on the basis of my configuration, so that I only create 1 timestamp for the first visitor on the site.\nDo you by chance know any of these sites or can tell me how I can make my HTML web server public?\nThanks","AnswerCount":4,"Available Count":2,"Score":0.049958375,"is_accepted":false,"ViewCount":3943,"Q_Id":63494148,"Users Score":1,"Answer":"You could use an amazon service like elastickbeanstalk its free to some extent and will make your server public","Q_Score":0,"Tags":"python,timestamp,ip,webserver,public","A_Id":63494268,"CreationDate":"2020-08-19T19:58:00.000","Title":"How to make Python Web Server public?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"My bot is in a few servers now and one of the main bits of feedback I've gotten is that server admins would like to block the bot from responding in certain channels without having to go through Discord's permissions manager. However I am unsure of where to start with this so I thought I'd reach out here and see if I can get any advice or code snippets to use!\nBasically the admin would use like !fg ignore 'channel name or id' and then somewhere the bot would store this and not respond, and then similarly if they use !fg unignore 'channel name or id' it would then remove that from the list or where ever its stored.\nAny help would be greatly appreciated, thanks!","AnswerCount":2,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":493,"Q_Id":63514605,"Users Score":0,"Answer":"You'll need to keep the channels ids in a list then in the bots on_message function check if the message is not in that channel and if not then run your commands.","Q_Score":0,"Tags":"python,python-3.x,discord,discord.py","A_Id":63514633,"CreationDate":"2020-08-20T23:47:00.000","Title":"Discord.py - How would I go about making a command that will allow server admins to block the bot from responding in specified channels?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I have a trading bot that trades multiple pairs (30-40). It uses the previous 5m candle for the price input. Therefore, I get 5m history for ALL pairs one by one. Currently, the full cycle takes about 10 minutes, so the 5m candles get updated once in 10m, which is no good.\nAny ideas on how to speed things up?","AnswerCount":2,"Available Count":1,"Score":0.0996679946,"is_accepted":false,"ViewCount":11818,"Q_Id":63515267,"Users Score":1,"Answer":"Just to follow up on that answer. You can see the candle closing as the websocket return data for every tick has a boolean property for if the candle is closed or not i.e. on a 5min timeframe if the candle closed on the 5min mark","Q_Score":12,"Tags":"python,api,algorithmic-trading,trading,binance","A_Id":67761679,"CreationDate":"2020-08-21T01:30:00.000","Title":"How to get ALL (or multiple) pair's historical klines from Binance API in ONE request?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I need to know which user is connected with the organization via OAuth2.0 .I noticed in the xero developer site it contains some information but not clear enough. Is there anyone who is having python code written for this.\nReason : Currently I am getting all the organisation informations from the API connection.I need to filter the content of these output based on the user type who is connected(Ex: Manager).So for that I need to know which user(user email address) is connected with Oauth2.0","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":56,"Q_Id":63536726,"Users Score":2,"Answer":"To find out the email address of the authorizing user you need to ask for the openid profile email scopes during authorization. You will then receive an id token back (along with the access and refresh token) which you can decode to get the user's name and email address.\nAny OpenID Connect library will allow you to do this.","Q_Score":1,"Tags":"python,xero-api","A_Id":63551620,"CreationDate":"2020-08-22T13:30:00.000","Title":"Oauth2.0 Expose which user connected the organization via OAuth","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I'm using scrapy to crawl all the links and selenium to scrape all the pages. Selenium scraped most of the pages but left a few pages as the page took time to load.\nI tried with timeout() , but didn't seem to work, then I tried with execute_script\n\ndriver.execute_script(\"return document.readyState==\"complete\";\")\n\nthis also didn't seem to work, then I tried with expected_conditions\n\nWebDriverWait.until(expected_conditions.execute_script(\"return\ndocument.readyState==\"complete\";\"))\n\nbut didn't seem to work\nI m using firefox browser, phantomJs for Headless\nTried using Chrome driver so installed using brew cask install chromedriver but I'm facing this error\n\nraise WebDriverException(\"Can not connect to the Service %s\" %\nself.path) selenium.common.exceptions.WebDriverException: Message: Can\nnot connect to the Service chromedriver\n\nso back to phantomjs.\nThank you!","AnswerCount":3,"Available Count":2,"Score":0.0,"is_accepted":false,"ViewCount":76,"Q_Id":63545051,"Users Score":0,"Answer":"Make use of sleep function which will help you in delaying the running of code in that time webpage will be loaded","Q_Score":0,"Tags":"python,selenium,web-scraping,scrapy,macos-catalina","A_Id":63545075,"CreationDate":"2020-08-23T08:45:00.000","Title":"How to make selenium to scrape pages after the page gets loaded","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I'm using scrapy to crawl all the links and selenium to scrape all the pages. Selenium scraped most of the pages but left a few pages as the page took time to load.\nI tried with timeout() , but didn't seem to work, then I tried with execute_script\n\ndriver.execute_script(\"return document.readyState==\"complete\";\")\n\nthis also didn't seem to work, then I tried with expected_conditions\n\nWebDriverWait.until(expected_conditions.execute_script(\"return\ndocument.readyState==\"complete\";\"))\n\nbut didn't seem to work\nI m using firefox browser, phantomJs for Headless\nTried using Chrome driver so installed using brew cask install chromedriver but I'm facing this error\n\nraise WebDriverException(\"Can not connect to the Service %s\" %\nself.path) selenium.common.exceptions.WebDriverException: Message: Can\nnot connect to the Service chromedriver\n\nso back to phantomjs.\nThank you!","AnswerCount":3,"Available Count":2,"Score":0.0,"is_accepted":false,"ViewCount":76,"Q_Id":63545051,"Users Score":0,"Answer":"I had this problem before. I used a while loop with try and except in it. Loop will keep trying to finish the work you have completed. If the page is not loaded then it will go into except which will just pass. But when it enters the try block and executes successful then at the end of the try block you can use a break to cone out of the loop. This worked 100% of the time for me.","Q_Score":0,"Tags":"python,selenium,web-scraping,scrapy,macos-catalina","A_Id":63545837,"CreationDate":"2020-08-23T08:45:00.000","Title":"How to make selenium to scrape pages after the page gets loaded","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I am setting up a discord game which uses folders as profiles, how do I fix the problem that if someone changes their username the bot can no longer access their profile?","AnswerCount":2,"Available Count":1,"Score":0.0996679946,"is_accepted":false,"ViewCount":50,"Q_Id":63547065,"Users Score":1,"Answer":"Use the UserID instead of the Username.\nThis never changes unless they change to a different account, so you won\u2019t have to worry about usernames changing.","Q_Score":0,"Tags":"python,discord,discord.py","A_Id":63547104,"CreationDate":"2020-08-23T12:40:00.000","Title":"Python\/Discord Username changes prevents my bot from recognizing players","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I once knew the name of python3 module that after importing when gave the name of request file saved by burpsuite, it automatically generated python3 requests headers. What is the module name?\nEdit: I am not searching for bs4, instead i am searching for what sqlmap -r\ndoes. I want to implement functionality similar to sqlmap -r but it was a module rather than what sqlmap is doing, and it made a variable with all the HTTP header set such as cookie, host etc","AnswerCount":2,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":671,"Q_Id":63547162,"Users Score":0,"Answer":"if I understand your question, you are searching the module name that burp suite is using to display the text from the request. If it's that, bs4 (beautiful soup 4) could be the module.","Q_Score":0,"Tags":"python-3.x,parsing,request,burp","A_Id":63547203,"CreationDate":"2020-08-23T12:51:00.000","Title":"How to parse burp suite request from file in python3?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I have an assignment where i need to implement three attacks after socket connection with python. I already did socket connection and implemented ransomware and key logger. I need one more attack implementation. Can anyone suggest me one attack that i can perform on the client's\/ victim's side besides these two?","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":43,"Q_Id":63548701,"Users Score":0,"Answer":"Some ideas that I have right now are:\n\nIf you have root access, you can basically have fun with some files and change their permissions. Maybe create a new user to access those files only.\n\nYou can try making a virus that could for instance create multiple processes of itself and slow down the victim's machine.\n\nYou could also create a script that scans for files in a certain directory (ex: Downloads) and send them to your machine.\n\nYou could try to modify some browser settings (called browser hijacking).\n\n\nI might add more ideas later. Good luck","Q_Score":0,"Tags":"python,security,client-side-attacks","A_Id":63550580,"CreationDate":"2020-08-23T15:21:00.000","Title":"What attacks can be performed on victim's machine after socket connection?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"Is it possible to remove videos from Plex's \"Recently Added\" section? I'd like to add some old videos to my server for posterity but not have them appears as newly added.","AnswerCount":3,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":3924,"Q_Id":63551357,"Users Score":0,"Answer":"I was faced to this situation and while gathering documentations to modify the SQL db, I thought about another solution and it did the trick for me. As previously said, all support threads regarding this issue are closed elsewhere so I'm posting it here for others to eventually find it :\n\nStopped Plex Server\nDisabled the server (computer) internal clock automatic sync\nSet the internal clock manually to a year before\nStarted Plex Server\nAdded the movies\nScanned the library\nMovie got added to the library without showing into recently added BUT the metadata from moviedb where not able to be loaded (probably caused by the difference in time and date between my server and moviedb server)\nStopped Plex Server\nRestored the internal clock automatic sync\nStarted Plex server\nUpdated all metadata of the library (actually I did it manually for each as I didn't know if I had some custom metadata somewhere else in the library)\nEnjoyed !","Q_Score":2,"Tags":"python,sqlite,plex","A_Id":67075416,"CreationDate":"2020-08-23T19:50:00.000","Title":"Remove Video From Plex's \"Recently Added\" Section","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"So as the title says.. ''Is there a way to get the emails of new YouTube subscribers? Python''\ni am planning to build an emailing list wherein if a person subscribes to my youtube channel, i want to be able to automatically send them an email containing my latest blogs and video updates. so is there a way to do that? maybe using the youtube api? or external codes or websites?\ni know that i could just ask them for their email addresses using a google form and link that to a python script that sends emails using SMTP but i want to do it automatically when they subscribe to my youtube channel.\nany help will help alot\nthank you","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":222,"Q_Id":63571445,"Users Score":0,"Answer":"Yes, you can. You can use the youtube api. Simply call the api, and get whatsoever the element is, for new subscribers. Then, you can create a text and mail it to yourself.\nFor making it run all the time, you have to go to the terminal and keep it running...","Q_Score":0,"Tags":"python,python-3.x,email,youtube,youtube-data-api","A_Id":63576069,"CreationDate":"2020-08-25T03:00:00.000","Title":"Is there a way to get the emails of new YouTube subscribers? Python","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"To give an example of what I'm trying to do, let's say there is a website that displays results of a lottery every hour. The webpage itself is static with the surrounding text staying the same and only the numbers changing (input by human not updated dynamically).\nSomething like The lucky number is: X where X indicates a different number each hour.\nNow I want to run a python script that parses the number(s) each hour, and then at the end of the day would print out all the numbers in a nice format.\nI know how to get the webpage content and get only the text parts of it without html tags etc by using the BeautifulSoup and requests libraries, however I'm not quite sure how to get the target number.\nI was thinking something like a regex which would find a static word from the text e.g. 'number is:' in this case and then grab the word (number) right after it.\nIs this doable? and if yes, how?\nThank you in advance.","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":55,"Q_Id":63576453,"Users Score":0,"Answer":"It's possible with regex but if you know the string already and if it's static , use simple split on that string.\nLet's say\nvar='The lucky number is: 123'\nOut= int(var.split(':')[1])\nOut will be 123","Q_Score":0,"Tags":"python,web-scraping,beautifulsoup","A_Id":63576713,"CreationDate":"2020-08-25T10:01:00.000","Title":"How can I extract a dynamic number from a webpage content with python?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":1,"System Administration and DevOps":0,"Web Development":0},{"Question":"I got the following error message:\nAn HTTP Client raised and unhandled exception: name 'FEATURE_OCSP_MODE' is not defined.\nHowever, I cannot find any reference on the internet about this FEATURE_OCSP_MODE. I was calling describe_addresses() of boto3 using Pythan 3.8. The code was working until yesterday (8\/24\/2020).","AnswerCount":2,"Available Count":2,"Score":1.2,"is_accepted":true,"ViewCount":209,"Q_Id":63583851,"Users Score":2,"Answer":"I rolled back to boto3==1.14.46 and snowflake-connector-python==2.2.10 as I was getting errors in both within Airflow 1.10.10 on Python 3.6.\nThis did the trick.","Q_Score":2,"Tags":"python-3.x,boto3","A_Id":63586825,"CreationDate":"2020-08-25T17:06:00.000","Title":"boto3 refers FEATURE_OCSP_MODE when setting IP address, but cannot find document about it","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I got the following error message:\nAn HTTP Client raised and unhandled exception: name 'FEATURE_OCSP_MODE' is not defined.\nHowever, I cannot find any reference on the internet about this FEATURE_OCSP_MODE. I was calling describe_addresses() of boto3 using Pythan 3.8. The code was working until yesterday (8\/24\/2020).","AnswerCount":2,"Available Count":2,"Score":0.2913126125,"is_accepted":false,"ViewCount":209,"Q_Id":63583851,"Users Score":3,"Answer":"I have just faced the same problem and the root cause was the snowflake-connector-python version as mentioned by Ben Campbell. In version v2.3.0 they accidentally removed the 'FEATURE_OCSP_MODE' constant but in version v2.3.1 they added it back.\nThe solution is to use snowflake-connector-python==2.3.1 or a newer one.","Q_Score":2,"Tags":"python-3.x,boto3","A_Id":63599231,"CreationDate":"2020-08-25T17:06:00.000","Title":"boto3 refers FEATURE_OCSP_MODE when setting IP address, but cannot find document about it","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I have 150 GB file in s3,I would like to unzip and upload the each file back to s3.what is the best approach to do with python and EC2 ? I appreciate your response.","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":34,"Q_Id":63604509,"Users Score":0,"Answer":"Download it on your system\nUnzip the folder and it will create a normal folder say \"Unzipped_Folder\"\nAssuming you are using Windows, install aws-cli in that.\nCreate an IAM User with S3 write access to that bucket and create a Access and Secret Access Key.\nIn aws-cli add the credentials from command prompt.\n$ aws configure\nNow run the following command to send files to S3 bucket\n$ aws s3 cp your_directory_path s3:\/\/your_bucket_name --recursive","Q_Score":0,"Tags":"python,amazon-s3,amazon-ec2,boto3","A_Id":63795510,"CreationDate":"2020-08-26T19:29:00.000","Title":"how to unzip the file from s3 using EC2?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I want to write and receive data from a frequency response analyser. I am connecting to the instrument using a serial connection via a Serial-USB adapter. The manual says it will only transmit when CTS (pin8) is high, and only receive when DCD (pin1) is high, both are listed as input pins. Does anyone know if there is a way to send high signals to these pins when i want to transmit\/receive using pySerial?\nThanks.","AnswerCount":2,"Available Count":2,"Score":0.0,"is_accepted":false,"ViewCount":596,"Q_Id":63620277,"Users Score":0,"Answer":"In the end I used a wire to manually connect the CTS and DCD pins to a pin that constantly outputs a high signal.","Q_Score":0,"Tags":"python,serial-port,pyserial","A_Id":63628974,"CreationDate":"2020-08-27T16:22:00.000","Title":"Setting RS232 pins using python serial","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I want to write and receive data from a frequency response analyser. I am connecting to the instrument using a serial connection via a Serial-USB adapter. The manual says it will only transmit when CTS (pin8) is high, and only receive when DCD (pin1) is high, both are listed as input pins. Does anyone know if there is a way to send high signals to these pins when i want to transmit\/receive using pySerial?\nThanks.","AnswerCount":2,"Available Count":2,"Score":0.0,"is_accepted":false,"ViewCount":596,"Q_Id":63620277,"Users Score":0,"Answer":"Today, connecting a PC to a device through a serial port is often done in a peer-to-peer relationship.\nIn that case, a cable called a cross cable is used to connect the two, but there is no standard specification for the pin assignment, and there are many variations.\nDCD (and RI) is an input signal on both the PC and the device, and there is no corresponding output signal, so the RTS or DTR signal should be used as an alternative.\nHow you can handle them in software depends on the connection specifications of the cable hardware you choose.\nAfter confirming that the RTS\/DTR signal on the PC side is connected to the CTS\/DCD pin on the device side of the cable, by setting rts and dtr of PySerial to True by software, the CTS\/DCD on the device side turns on.\n\nIn response to comment:\nIf you fix the signal on the device side as described in the comment and the device side can operates in that state, the PC side will be able to send and receive at any time without additional control processing.","Q_Score":0,"Tags":"python,serial-port,pyserial","A_Id":63626195,"CreationDate":"2020-08-27T16:22:00.000","Title":"Setting RS232 pins using python serial","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I was just wondering if it was possible to tweet an mp4 file with tweepy (maybe using api.update_with_media). I've seen a post from a few years ago saying that it's not possible with the official tweepy, but I find it really hard to believe that there's no way to do it with the official tweepy. Can anyone please help?","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":703,"Q_Id":63625810,"Users Score":1,"Answer":"update_with_media was deprecated as a Twitter API method about 5 years ago. This pre-dated the ability to upload GIFs and video MP4s. If you want to post media on Twitter via the API, you must find a library that supports the media upload endpoints (and chunked uploads). The process is\n\nupload the media file (chunked if necessary depending on size and format)\nretrieve a media ID string\npost a Tweet, and add the media ID to the Tweet.\n\nNote that you can only post multiple images on a single Tweet; you can only post a single GIF or video file on a single Tweet, not multiple.\nI do not believe the default Tweepy release supports this but I could be wrong, you\u2019ll need to check the Tweepy documentation.\nDo NOT use update_with_media - it\u2019s very old and unsupported as an API path.\nAlso worth being aware that tweepy itself is a third party library - that we at Twitter LOVE - but this is not officially supported.","Q_Score":1,"Tags":"python,api,twitter,tweepy,twitterapi-python","A_Id":63625966,"CreationDate":"2020-08-28T00:24:00.000","Title":"Can I tweet mp4 files with tweepy?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I've worked through all the threads concerning this problem but haven't found anything.\nWhenever I try to download a file from Wasabi S3, I get a 403 forbidden error. The file is uploaded with a pre-signed url that was signed with the same access key as the one I am using for the download request.\nFor the signed url I tried setting the ACL to \"private\" as well as \"public-read\". But that hasn't changed anything. I did also experiment with different access keys with different permissions for signing and downloading. But also no change.\nAny more suggestions would be highly appreciated.","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":435,"Q_Id":63630434,"Users Score":0,"Answer":"Long story short, the endpoint_url parameter is not optional when initializing boto3.client.","Q_Score":0,"Tags":"python,amazon-s3,boto3,pre-signed-url,wasabi","A_Id":63686436,"CreationDate":"2020-08-28T08:46:00.000","Title":"boto3 upload_file() 403 forbidden with Wasabi S3","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I was trying an automation project with selenium 3.141.0 and chrome 84 driver. But the code was getting stuck despite adding correct id and details. I would like to know whether its a compatibility issue or something else.\nPlease help me!","AnswerCount":2,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":1397,"Q_Id":63638862,"Users Score":0,"Answer":"I was making a FB automation project.\nfrom selenium import webdriver\nfrom time import sleep\ndriver=webdriver.Chrome(r'C:\\Users\\samar\\OneDrive\\Desktop\\MAIN\\projects\\Jarvis\\chromedriver.exe')\ndriver.get('https:\/\/www.facebook.com')\nemail=driver.find_element_by_xpath('.\/\/*[@id=\"email\"]')\nemail.send_keys('username')\npassword=driver.find_element_by_xpath('.\/\/*[@id=\"pass\"]')\n'''with open('pass.txt','r') as myfile:\n password1=myfile.read().replace('\\n','')'''\npassword.send_keys('password')\nlogin=driver.find_element_by_id('u_0_b')\nlogin.click()\nsleep(3)\npostarea=driver.find_element_by_xpath('\/\/*[@id=\"mount_0_0\"]\/div\/div[1]\/div[1]\/div[3]\/div\/div\/div[1]\/div[1]\/div\/div[2]\/div\/div\/div[3]\/div\/div[2]\/div\/div\/div\/div[1]\/div')\npostarea.click()\nsleep(3)\nactivepostarea=driver.switch_to_active_element()\nactivepostarea.send_keys(\"Hello World\")\nBut as it reached the FB homepage after login. A message popped up for asking permission for notification i.e. allow or block. And then the following error came.\nraise exception_class(message, screen, stacktrace) selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {\"method\":\"xpath\",\"selector\":\"\/\/*[@id=\"mount_0_0\"]\/div\/div[1]\/div[1]\/div[3]\/div\/div\/div[1]\/div[1]\/div\/div[2]\/div\/div\/div[3]\/div\/div[2]\/div\/div\/div\/div[1]\/div\"} (Session info: chrome=84.0.4147.135)","Q_Score":0,"Tags":"python-3.x,selenium,selenium-webdriver,selenium-chromedriver,google-chrome-devtools","A_Id":63642995,"CreationDate":"2020-08-28T18:03:00.000","Title":"Is chrome 84 webdriver compatible with selenium 3.141.0?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I developed a small program with twilio a few days ago and it was working perfectly until today. When I run the code I get the error \"No module named 'twilio'\". I already tried uninstalling and reinstalling it, I also checked, and don't have any archive called twillio, the python version i'm using is the latest, the library seems to be correctly installed... Any idea of what may be happening?\nThe line of code that im trying to run is from twilio.rest import Client","AnswerCount":1,"Available Count":1,"Score":0.1973753202,"is_accepted":false,"ViewCount":361,"Q_Id":63649453,"Users Score":1,"Answer":"If you are installed 'twilio' in a virtual environment then you are probably not activating and using it properly.\nThe error can also be caused due to Circular import, which happens when you have a file named 'twilio' in the same directory as the program file. You can prevent that by changing the name to something else.\nIf you have more than one version of python you are probably using the wrong version, so select the correct version of the interpreter for python in your IDE.","Q_Score":0,"Tags":"python,twilio","A_Id":63649732,"CreationDate":"2020-08-29T16:43:00.000","Title":"ModuleNotFoundError: No module named 'twilio'","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"im trying to find a way to make my HTTP request in a exact time for my computer example 13:02:00:00.\nAny recomendation?","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":58,"Q_Id":63676253,"Users Score":0,"Answer":"in my opinion, it depends on your network speed. Anyway, I think that's impossible as when you make a GET request, your request will pass through many network devices such as a router, switch, and finally reaches a server where IP address of the site will be sent to you. Even request that is sent will travel with the speed of light you can't get it in exact time, as some switches will make ARP requests to get to know destination address.","Q_Score":0,"Tags":"python,python-3.x,python-requests,clock","A_Id":63676551,"CreationDate":"2020-08-31T18:27:00.000","Title":"Python HTTP Request in exact time?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"When establishing remote webdriver connection via https:\nself.driver = webdriver.Remote(command_executor='https:\/\/' + settings.BROWSERSTACK_USERNAME + ':' + settings.BROWSERSTACK_KEY + '@hub-cloud.browserstack.com\/wd\/hub', desired_capabilities=desired_cap)\nThis error message pops up:\nMaxRetryError: HTTPSConnectionPool(host='hub-cloud.browserstack.com', port=443): Max retries exceeded with url: \/wd\/hub\/session (Caused by SSLError(SSLError(\"bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')],)\",),))\nFrom Browserstack's FAQ:\nMy firewall\/IDP\/proxy is rejecting SSL\/HTTPS content from BrowserStack.com\nThis could be because of a missing certificate chain due to which your firewall\/IDP\/proxy may not recognize, or block the SSL\/HTTPS content. This usually happens if your firewall\/IDP\/proxy is not updated regularly. In this case please install the ValiCert Legacy Certificate Chain from the GoDaddy repository and try again.\nAm I on the right track? (Following Browserstack's FAQ)\n*OS: Fedora-30\nThank you in advance for any help....","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":754,"Q_Id":63678321,"Users Score":0,"Answer":"You could try changing the Hub URL from HTTPS to HTTP or whitelist *.browserstack.com over ports 80 and 443 over your network.\n\u200b","Q_Score":0,"Tags":"python,ssl,fedora,browserstack","A_Id":63678616,"CreationDate":"2020-08-31T21:19:00.000","Title":"How to resolve Browserstack Local Testing SSLError(\"bad handshake\")","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I wanted to read this article online and something popped and I thought that I want to read it offline after I have successfully extracted it... so here I am after 4 weeks of trials and all the problem is down to is I the crawler can't seem to read the content of the webpages even after all of the ruckus...\nthe initial problem was that all of the info was not present on one page so is used the button to navigate the content of the website itself...\nI've tried BeautifulSoup but it can't seem to parse the page very well. I'm using selenium and chromedriver at the moment.\nThe reason for crawler not being able to read the page seems to be the robot.txt file (the waiting time for crawlers for a single page is 3600 and the article has about 10 pages, which is bearable but what would happen if it were to say 100+)and I don't know how to bypass it or go around it.\nAny help??","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":82,"Q_Id":63678927,"Users Score":1,"Answer":"If robots.txt puts limitations then that's the end of it. You should be web-scraping ethically and this means if the owner of the site wants you to wait 3600 seconds between requests then so be it.\nEven if robots.txt doesn't stipulate wait times you should still be mindful. Small business \/ website owners might not know of this and by you hammering a website constantly it could be costly to them.","Q_Score":0,"Tags":"python,selenium,web-crawler,dynamic-pages","A_Id":63679106,"CreationDate":"2020-08-31T22:22:00.000","Title":"dynamic web page crawler using python","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I have encoded my program so I can sell it to customers.\nAlso I want to convert it to exe with the tool auto-py-to-exe so the customers won't have to install python.\nWhen I launch the encoded program from Visual Studio Code all works as intended.\nBut when I convert it to exe it returns an error message:\n\nTraceback (most recent call last):\nFile \"test.py\", line 8, in \nFile \"\", line 2, in \nModuleNotFoundError: No module named 'selenium'\n[11189] Failed to execute script test\nlogout\n\nWhen I do not encode my program I can convert it without any issues.\nAny idea what I might be missing here?","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":55,"Q_Id":63683604,"Users Score":0,"Answer":"Solution was, to add the \"import\" lines for all my modules on the top level of the .py\nWorking now.","Q_Score":0,"Tags":"python,selenium,base64,pyinstaller","A_Id":63683692,"CreationDate":"2020-09-01T08:00:00.000","Title":"Python Base64 Selenium module missing when converting to exe","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I am a 15-year student. I am a student as well as a growing programmer. I want to create a bot that reminds the teacher that \"It's your time to leave the class!\" I heard somewhere that \"running bots on websites is illegal\". So, I thought to ask you regarding this.\nPlease tell me if I can run the time remainder bot on Google Meet.\nThank you and have a great day.","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":151,"Q_Id":63686124,"Users Score":0,"Answer":"Making a bot would be possible, although the problem is that bots are against Google Rules, they would ban the bot.\nYou will not face any law complications, if this bot will be looking just as a normal user then it's fully possible.\nAnother thing - Running bots on websites is legal while owner approves your bot, but still 1 or 2 bots which will tell the teacher \"It's time to end the lesson\" aren't a reason for which a giant company would waste its time to start a court hearing.\nYou can do it, but if your bot gets banned don't make a new one.","Q_Score":0,"Tags":"python,selenium,automation,google-meet","A_Id":63686681,"CreationDate":"2020-09-01T10:44:00.000","Title":"Can i run a bot on google meet","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I know very little regarding python,I'm using it to get data from twitter.\nmy old laptop has python version 3.7.3 it works fine, but I didn't set it up.\nI installed python on my new laptop, but I think I also need to install some packages from my old python (laptop)\nthe error shows in my new laptop \"no module named twitter\"\nI don't know how this works.","AnswerCount":3,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":59,"Q_Id":63699030,"Users Score":0,"Answer":"First, you need to cd to your python project and then activate your virtual environment and run\npip install twitter","Q_Score":1,"Tags":"python,python-3.x,twitter,module","A_Id":63699273,"CreationDate":"2020-09-02T05:08:00.000","Title":"I have python 3.7.3 in my old laptop, I installed python 3.8 on my new laptop and it's not working anymore","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":1,"System Administration and DevOps":0,"Web Development":0},{"Question":"Tldr: Is it feasible to create a web tool\/app that relies on server side Selenium automation.\nCreated a local script that automates form filling for car insurance quote websites and returns the cost to insure. Ie fill one form and it auto fills every other providers quote form and returns quotes.\nBut now I want to extend that functionality to others via some sort of webapp [flask\/Django?] that handles a clients requests server side by fetching that information and returning it to the client based on their inputs.\nWhat I\u2019m struggling with is Selenium is limited to 5 web drivers (locally) I believe, and is resource intensive, so to me that means at most you can handle 5 website requests at once?","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":105,"Q_Id":63702116,"Users Score":0,"Answer":"The short answer is YES.\nThe idea to solve the problem is as follows:\n\nCreate a web app which can fetch user's inputs and return something back to user. Just like the most common website.\nCreate a service in the web app. The service can handle what you wanted using Selenium, such as, filling form of car insurance and getting the cost to insure.\nCreate a webpage or API in the web app. The webpage\/API calls the service mentioned above. When user use the webpage\/API, Selenium will automate to do something.\n\nSo, it's done.","Q_Score":0,"Tags":"python,selenium,server,client","A_Id":63718920,"CreationDate":"2020-09-02T08:58:00.000","Title":"Feasible? Form automation webapp with selenium - [server side]","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I'm implementing get text on my Discord bot and I have a few strings that come from configuration files for example: Regions where the game is from (Global server, Korean server, Japanese server, Asia server, etc.) that I want to also be translated into. Since these files are stored as JSON, I can't really use get text on them.\nSo I was wondering what can I do to get these strings translated? I came up with a few approaches that would solve my problem but they don't look that nice to me.\nMethod 1 - Use .py files for these settings\nThis one is self explanatory. I can use python modules to store the configuration for that, which would allow me to use the _(...) get text function which would allow xgettext to pick up the strings to be translated.\nMethod 2 - Hard code the strings\nThis would come in two ways, hardcoding them inside the module they're going to be used on or in a module where I would hard code all the strings that come from external data sources.\nIs there any better approach to tackle this?","AnswerCount":1,"Available Count":1,"Score":0.1973753202,"is_accepted":false,"ViewCount":40,"Q_Id":63713616,"Users Score":1,"Answer":"Extract the strings from the configuration files into an additional .pot file like config.pot. You can then pass this file as an additional input file to xgettext because xgettext will always recognize .po or .pot files as input regardless of the programming language.","Q_Score":0,"Tags":"python-3.x,translation,external,gettext,xgettext","A_Id":63914412,"CreationDate":"2020-09-02T21:12:00.000","Title":"Translating strings that are outside of the range of gettext","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":1,"System Administration and DevOps":0,"Web Development":0},{"Question":"I have written a python program to download YouTube videos. I want to add it as an extension to browser, but I have read that browser doesn't run python scripts. So how can I deploy it? Can I convert it into a software which when installed can be added as a plug-in to the browser? Or is there any other way of doing it?","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":97,"Q_Id":63716152,"Users Score":0,"Answer":"I assume you are aware of the chrome extension\/plugin development, If not, please go through online tutorials, Fast forward...\nAdding Python to the Chrome extension\nWe have two options to add Python into a chrome extension:\nMethod 1: Include\n\nBrython\n\nin an iframe (requires server)\nTo run Python in the browser you have several options including Brython and emcascripten.\nMethod 2: Compile Python to Javascript using\n\nRapydscript\n\n(best, serverless, pure extension.)\nThere are several tools to compile Python to Javascript. Rapydscript works fine, Pyjs does not work well with chrome.\nFurther, you can do your research.","Q_Score":0,"Tags":"python-3.x,youtube-api,browser-extension,browser-plugin","A_Id":63720439,"CreationDate":"2020-09-03T03:00:00.000","Title":"How to create a software that can be used as a browser extension in python?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I want to make a command that with a parameter of a role that lists everybody who has that role in a discord server. Is there a way to do this?","AnswerCount":3,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":83,"Q_Id":63728413,"Users Score":0,"Answer":"If you can get the list of people, you can iterate through the list of players and use playerList[index].roles[index] == \"Some Role\"","Q_Score":2,"Tags":"python,discord,discord.py,discord.py-rewrite","A_Id":63728512,"CreationDate":"2020-09-03T17:01:00.000","Title":"Is there a way to make a command that lists everybody who has a certain role in a discord server?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I'm making a discord bot using python. When I try to run it to connect my client to discord I get this error:\n\nraise ClientConnectorCertificateError(\naiohttp.client_exceptions.ClientConnectorCertificateError: Cannot\nconnect to host discord.com:443 ssl:True [SSLCertVerificationError:\n(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed:\ncertificate has expired (_ssl.c:****)')]\n\nHappens every time I try to run it.\nI saw a plausible fix for this which was:\n\nGo to discord.com with Internet Explorer (Ran as Administrator)\nClick the lock on the top right\nClick view certificates\nInstall one\n\nHowever, when I tried it and then tried to run my bot again I still got the same error.\nDont know if this is relevant but at the start of the error message in the Traceback section it says the first call that started the error is in the last line of my bot's code which is client.run(TOKEN).\nI'm using Python 3.8 on Windows 10.","AnswerCount":2,"Available Count":1,"Score":0.3799489623,"is_accepted":false,"ViewCount":3719,"Q_Id":63742983,"Users Score":4,"Answer":"TL;DR Your certificate has expired, and you need to download and install the new one to your local machine by downloading it from the website and installing it into your OS.\nThis solution is performed using Google Chrome and Windows 10.\n\nGo to the website you're trying to request\nClick the lock icon to the left of the URL entry field\nIf it says Certificate (Valid), you're in business. Click that\nIn the resulting window, click the Details tab\nClick Copy to File...\nIn the certificate export wizard, select Cryptographic Message Syntax Standard (.P7B)\nCheck the box under it, Include all certificates in the cert path if possible\nClick next, click Browse and choose a folder, create a name for the file and click Save\nNext, Finish\nOpen the file you just exported. If you're on windows, it will open certificate manager\nNavigate deeper into the folders to find the actual certs\nOne by one, double click each cert and choose Install Certificate, for the current user, and automatically select where it goes\n\nOnce you install all the certificates required to validate the HTTPS request, your script shouldn't encounter the certificate expired error anymore, until the new certs expire.\nYou can also check when each cert expires in certmgr.","Q_Score":3,"Tags":"python-3.x,ssl,certificate,discord,discord.py","A_Id":69550170,"CreationDate":"2020-09-04T14:22:00.000","Title":"\"Certificate verify failed: certificate has expired (_ssl.c:****)\" When attempting to connect my bot to discord","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I want to make a \/subscribe endpoint that will add the client to a list that will be notified with a webhook when an event happens. How can I get the data from flask needed to connect to the client later so I can then send a POST in requests later?","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":38,"Q_Id":63746471,"Users Score":0,"Answer":"You can not store the connection details.\n\nThere is nothing called CONNECTION. It is just an HTTP request from one server to the other.\nAll HTTP requests are just messages that you (Browser or a server) sends to another server asking for something like \"Can I view your website?\" or \"Can you send me this data?\" etc.,\n\n\nPossible solution for you:\n\nI have worked on webhooks for one of your products in our company and the best way to give the ability to create a webhook to your client is, asking for their endpoint.\n\n\nWhat I did:\n\n\nI ask the client for their endpoint so that I can call it whenever there is an event.\nAsk them to create and share a secret_key so that no one else can call that endpoint other than you.\nCall that endpoint using their secret_key whenever there is an event.\n\nAbove is just an outline of how it is done (Or at least how I did it).","Q_Score":0,"Tags":"python,flask,python-requests","A_Id":63747062,"CreationDate":"2020-09-04T18:38:00.000","Title":"Flask: How can I store the connection details of clients [Inside NAT], so I can POST them a webhook later?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I need to be able to run python code on each \"node\" of the network so that I can test out the code properly. I can't use different port numbers and run the code since I need to handle various other things which kind of force using unique IP addresses.","AnswerCount":2,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":171,"Q_Id":63756753,"Users Score":0,"Answer":"I think vmware or virtual box can help you.","Q_Score":1,"Tags":"python,networking,simulation,p2p","A_Id":63770811,"CreationDate":"2020-09-05T17:33:00.000","Title":"Is there a way to simulate a network so that I can test my p2p network code written in python?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I am using the django framework. I'll do the messaging part. I will do this with the socket. However, I always come across a channel library. Is there no other solution?","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":34,"Q_Id":63784910,"Users Score":1,"Answer":"Without writing your own socket library, Channels is as close as you're going to get to official support. You could use Tornado (a different framework) for lower-level handling of simpler cases, but as far as Django goes, Channels is the normal pathway.\nIs there a reason you didn't want to use Channels?","Q_Score":0,"Tags":"python,django,sockets,channels","A_Id":63785043,"CreationDate":"2020-09-07T22:18:00.000","Title":"Is channels required to use socket with django? Is there any other alternative or way?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I am new to web scraping and am not very familiar with requests and BeautifulSoup.\nI am trying to scrape an aspx website with BeautifulSoup. But to get the values I want to scrape, I first need to select a drop down value, input an ID, and then press submit.\nIs this even possible?\nAny help would be much appreciated!","AnswerCount":2,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":47,"Q_Id":63785020,"Users Score":0,"Answer":"Maybe try to input the url that you get after you have made all the selections from dropdowns. Otherwise.. using Selenium is best bet.","Q_Score":0,"Tags":"python,web-scraping,beautifulsoup,python-requests","A_Id":63785808,"CreationDate":"2020-09-07T22:34:00.000","Title":"Scraping with BeautifulSoup, but first have to input values?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"Is there a way to pass SSEKMSKeyId and ServerSideEncryption to scrapy FilesPipeline using AWS S3 or do I need to write a separate pipeline?","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":134,"Q_Id":63790967,"Users Score":0,"Answer":"Subclass scrapy.pipelines.files.S3FilesStore, and extend its HEADERS class attribute in your subclass to define the headers you want with the values you want.\nIn your case, the corresponding headers are: X-Amz-Server-Side-Encryption, X-Amz-Server-Side-Encryption-Aws-Kms-Key-Id (you can see the header-to-key mapping in the source code of the class, for additional header names).\n\nSubclass FilesPipeline, and edit the STORE_SCHEMES class attribute in your subclass to point s3 to your S3FilesStore subclass.\n\nUpdate your ITEM_PIPELINES setting to use your FilesPipeline subclass.","Q_Score":0,"Tags":"python,amazon-web-services,amazon-s3,scrapy","A_Id":63917828,"CreationDate":"2020-09-08T09:25:00.000","Title":"Is there a way to pass SSEKMSKeyId and ServerSideEncryption to scrapy FilesPipeline using AWS S3?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I wrote a python script which is deployed in a EC2 Instance and lets say this EC2 reside in AWS account A1. Now my script from A1 want to access 10 other AWS account.\nAnd remember I don't have any AWS_ACCESS_KEY or SECRET_KEY of any account cause using AWS_ACCESS_KEY or SECRET_KEY is strictly prohibited here.\nI can easily do that if I have access key. But I can't figure it out how can I do that without access key?\nIs there any possible way to do that?","AnswerCount":3,"Available Count":1,"Score":0.1325487884,"is_accepted":false,"ViewCount":234,"Q_Id":63810941,"Users Score":2,"Answer":"The EC2 should assume an IAM Role.\nThen log in to all your 10 other accounts and create roles there. These roles should give cross account access to the EC2 instance role. It is also in these roles that you define what permissions the EC2 instance should have.","Q_Score":0,"Tags":"python,python-3.x,amazon-web-services,boto3","A_Id":63811205,"CreationDate":"2020-09-09T11:43:00.000","Title":"Accessing multiple AWS accounts using boto3 without secret key?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"socketio.emit('text', 'text') not working inside dash callback where we write something back to the database.\nThese same emit is working inside other callbacks.","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":101,"Q_Id":63817272,"Users Score":0,"Answer":"Flask socket io stalls when there are long polling processes. We need to emit before the start of such processes otherwise there is no impact of the emit statement. Just emitting before writing to the db fixed the issue.","Q_Score":0,"Tags":"python,flask,socket.io","A_Id":63817273,"CreationDate":"2020-09-09T18:10:00.000","Title":"Flask socketio not emitting event in dash callback","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"Using beautifulsoup it's easy to fetch URLs that follow a certain numeric order. However how do I fetch URL links when it's organized otherwise such as https:\/\/mongolia.mid.ru\/en_US\/novosti where it has articles like\nhttps:\/\/mongolia.mid.ru\/en_US\/novosti\/-\/asset_publisher\/hfCjAfLBKGW0\/content\/24-avgusta-sostoalas-vstreca-crezvycajnogo-i-polnomocnogo-posla-rossijskoj-federacii-v-mongolii-i-k-azizova-s-ministrom-energetiki-mongolii-n-tavinbeh?inheritRedirect=false&redirect=https%3A%2F%2Fmongolia.mid.ru%3A443%2Fen_US%2Fnovosti%3Fp_p_id%3D101_INSTANCE_hfCjAfLBKGW0%26p_p_lifecycle%3D0%26p_p_state%3Dnormal%26p_p_mode%3Dview%26p_p_col_id%3Dcolumn-1%26p_p_col_count%3D1?\nWebsites such as these are weird because once you first open the link, you have \u00bb \u0411\u0443\u0441\u0430\u0434 \u043c\u044d\u0434\u044d\u044d button to go to the next page of articles. But once you click there, now you have Previous or Next button which is so unorganized.\nHow do I fetch all the news articles from websites like these (https:\/\/mongolia.mid.ru\/en_US\/novosti or https:\/\/mongolia.mid.ru\/ru_RU\/)?","AnswerCount":1,"Available Count":1,"Score":0.1973753202,"is_accepted":false,"ViewCount":22,"Q_Id":63822647,"Users Score":1,"Answer":"It seems that the \u00bb \u0411\u0443\u0441\u0430\u0434 \u043c\u044d\u0434\u044d\u044d button from https:\/\/mongolia.mid.ru\/ru_RU\/ just redirects to https:\/\/mongolia.mid.ru\/en_US\/novosti. So why not start from the latter?\nTo scrape all the news just go page through page using the link from the Next button.\nIf you want it to be more programatic, just check the differences in the query parameters and you'll see that _101_INSTANCE_hfCjAfLBKGW0_cur is set to the actual page's number (starting from 1).","Q_Score":0,"Tags":"python-3.x,beautifulsoup,scrapy","A_Id":63823640,"CreationDate":"2020-09-10T04:01:00.000","Title":"How to fetch website links when they're not numerically ordered","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I've been looking through examples of AuthnRequest messages and have yet to determine where the user and password data that I want to authenticate (with my installed, configured and tested SimpleSAMLphp IDP) is located in the actual request.\nI'm building a python-based SP which I want to talk to the IDP.","AnswerCount":2,"Available Count":1,"Score":0.1973753202,"is_accepted":false,"ViewCount":52,"Q_Id":63837480,"Users Score":2,"Answer":"The password is never in the authn request. It's possible to include the user name but not all identity provider support receiving the user name. The password is only input at the identity provider site.","Q_Score":0,"Tags":"python,saml,simplesamlphp","A_Id":63838959,"CreationDate":"2020-09-10T20:47:00.000","Title":"In SAML, when an user\/password pair is to be authenticated, where is the user\/passwd pair located in an AuthnRequest?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"With PRAW, I know you can do\nsubs = reddit.subreddit(\"Pics+Funny\")\nto get the streams of both r\/Pics and r\/Funny, and I know you can do this\nuser = reddit.redditor(\"spez\")\nto get one user, but is there a way to get multiple users at once? Like so:\nusers = reddit.redditor(\"spez+kn0wthing\")?\nI'm trying to read the comments of a large group of users, and I would like to know if something like this is possible. When I tried what I put right above this, it returned a 404 error. Could anyone help me out?","AnswerCount":3,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":422,"Q_Id":63837953,"Users Score":0,"Answer":"I don't believe there's a built-in way to do this. Either you'll need to loop through the users one at a time or create separate threads and merge the data.","Q_Score":1,"Tags":"python,reddit,praw","A_Id":63853475,"CreationDate":"2020-09-10T21:31:00.000","Title":"In PRAW, is there a way to get the comment streams for multiple users?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"Hello i want to make a chat between two different computer.\nComputer Number 1 is server and connected to the Wi-Fi Number 1.\nComputer Number 2 is client and connected to the Wi-Fi Number 2.\nBut my problems is: Computer Number 2 can't connect to the Computer Number 1 it actually for that the Computer Number 2 can't find the server IP.\nWhat should i do?","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":27,"Q_Id":63848326,"Users Score":0,"Answer":"You could use the broadcast adress to send first message to the server. Then the server would respond to client and establish a connection.","Q_Score":1,"Tags":"python,server,connection,client,chat","A_Id":63863406,"CreationDate":"2020-09-11T13:43:00.000","Title":"make a chat between computer with socket in the Python","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I need to use discord.py regardless of the events inside my program, and specifically get the user's id by his nickname (they are all on the same server). bot.loop.create_task doesn't work for me, as it needs to start an event loop and my program is not related to event handling.\nI will be glad to any of your help","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":58,"Q_Id":63875058,"Users Score":1,"Answer":"In theory there is no way, because discord.py is asynchronous.\nIn practice...\nyou can try discord.ext.tasks and setup something that looks like a chron task.","Q_Score":1,"Tags":"python-3.x,discord.py","A_Id":63883726,"CreationDate":"2020-09-13T20:04:00.000","Title":"How to use discord.py regardless of events?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"So, Currently if suppose I have an article of around 1500 words and I don't wanna use telegra.ph, neither I want to send two messages to send the complete article.\nI want to send the complete article in one message.\nSo for that, Can we group two messages with two different id's and link them through Inline Buttons - Next and Previous.\nThe next button will open the second message, and previous will return to the first.\nIs it possible, How can I do it if it is. And Do anyone has a better idea if it isn't.\nI am creating a group solely for writers. As stories, novels, articles, are some Times quite lengthy I want readers that feasibility to read such long writeups without having to go to Telegra.ph or reading multiple messages. Instead an ebook format for the readers to read the next page or chapter by just clicking the \"Next\" Inline button.","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":33,"Q_Id":63876043,"Users Score":0,"Answer":"Yes, you can do this by adding \"Next\" InlineButton.\nRead official documentation for the library you choose and examples there.","Q_Score":0,"Tags":"javascript,python,node.js,telegram-bot","A_Id":63880678,"CreationDate":"2020-09-13T22:08:00.000","Title":"Can we Add an EBook kinda functionality to Telegram bots for viewing lengthy messages?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I am trying to create a bot that randomly selects elements and generally browses around looks 'human'.\nI want to simply select any element that has an h3 tag at random, but I only seem to be able to select specific id strings for the bot to find and click on, but I haven't seen anyone just select something by just an h3 tag. Is this even possible?","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":32,"Q_Id":63910055,"Users Score":1,"Answer":"You can get all h3 elements with driver.find_elements_by_tag_name(\"h3\") in a list and use the random module's 'sample' method to get one of them at random.","Q_Score":0,"Tags":"python,python-3.x,selenium,google-chrome","A_Id":63910386,"CreationDate":"2020-09-15T21:24:00.000","Title":"Python: Is there a way to randomly select an element that has an h3 tag in selenium chromedriver?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"We're using the DocuSign SDK (version 3.3.0) and utilizing webhooks to receive the various envelope and recipient events.\nSimilar to DocuSign, we have given a option to delete an envelope in our application. Since there is no webhook notification once a envelope is moved to deleted bin. And also there is a field called deleted_date_time in get_envelope method (which provides the envelope details) and there is no status change, which are not updated once the envelope is being removed from the deleted bin.\nCan anyone please help me in how to make sure the envelope has been removed from the deleted bin?","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":197,"Q_Id":63931911,"Users Score":1,"Answer":"There's no means to permanently delete and envelope or remove it from the recycle bin folder. You cannot do that from the DocuSign Web App either as far as I can tell.\nVoiding an envelope is and deleting it is as good as it gets. There's no need to worry about it anymore.","Q_Score":0,"Tags":"python-3.x,docusignapi,webhooks","A_Id":63946598,"CreationDate":"2020-09-17T05:43:00.000","Title":"DocuSign - Delete Envelope","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"Curious, if there is way to avoid skipping messages sent from Telegram Bot while web-server that accepts Webhooks is down (because of redeploy, failure or maintenance).\nWhen you use polling - Telegram API sends messages starting from last retrieved and no message are skipped.\nBut how to be with Webhooks? Use polling or there are some special mechanism for that?","AnswerCount":2,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":856,"Q_Id":63933435,"Users Score":0,"Answer":"I had the same problem recently but I just resolved it by when the server starts save the started time to a variable and then use Telegrambot.Message.date and compare the time if it was sent before the server start time or not.","Q_Score":0,"Tags":"telegram,telegram-bot,python-telegram-bot,telegram-webhook,node-telegram-bot-api","A_Id":63948225,"CreationDate":"2020-09-17T07:43:00.000","Title":"How to get messages that were not delivered while Telegram Bot web-server was down?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"i have an NI USB Data Logger. when i plug-in the device, device sends same 10 byte data every 100 ms before opening the software. i captured this data using an USB packet sniffer software.(assume this data is device id). my question is how the device can send data without its software being open? and how can i find out with which endpoint this packets are sending?\nthank you","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":56,"Q_Id":63969316,"Users Score":0,"Answer":"I can't answer your first question (why a particular device could be sending 10-byte packets before its accompanying application software is launched). There are numerous possibilities that are in line with the USB spec.\nTo your 2nd question, you may use a tracing software like WireShark or Microsoft Event Analyzer (the latter has recently reached EOL, though) to find out what endpoints do the transfers belong to and even decode the content to produce an easily-readable outputs (for devices belonging to standard classes).","Q_Score":1,"Tags":"python,c#,c,usb,libusb","A_Id":64883324,"CreationDate":"2020-09-19T13:24:00.000","Title":"is there a way to send a USB data packet without host permission?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":1,"Web Development":0},{"Question":"I made a program that works with selenium, and it automates for posting comment to the some blogs' contents. I'm not familiar with the requests module of python. (working on it for just a week) The thing that I'm wondering is, my program with selenium is a bit slow for page loading, and it loads everything from ads to the images\/videos. If I'd made my program with requests module, would it save data and a bit faster according to the selenium module?\nI searched this issue at some forum-sites, generally they say request modules a bit faster, but not all. Also I couldn't find any info about saving data by comparing this modules?\nPlz don't give me directly the thumbs down. I need this answer with details.","AnswerCount":3,"Available Count":2,"Score":0.0,"is_accepted":false,"ViewCount":697,"Q_Id":63977623,"Users Score":0,"Answer":"If you are proficient with HTTP requests and verb (know how to make a POST request to a server with requests library), then choose requests. If you want to test your script, use selenium or BeautifulSoup.","Q_Score":1,"Tags":"python,selenium,python-requests","A_Id":63977703,"CreationDate":"2020-09-20T09:52:00.000","Title":"Comparing request module vs selenium in Python","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I made a program that works with selenium, and it automates for posting comment to the some blogs' contents. I'm not familiar with the requests module of python. (working on it for just a week) The thing that I'm wondering is, my program with selenium is a bit slow for page loading, and it loads everything from ads to the images\/videos. If I'd made my program with requests module, would it save data and a bit faster according to the selenium module?\nI searched this issue at some forum-sites, generally they say request modules a bit faster, but not all. Also I couldn't find any info about saving data by comparing this modules?\nPlz don't give me directly the thumbs down. I need this answer with details.","AnswerCount":3,"Available Count":2,"Score":0.0,"is_accepted":false,"ViewCount":697,"Q_Id":63977623,"Users Score":0,"Answer":"Selenium allows you to control a browser and execute actions on a webpage.\nrequests library is for making HTTP requests.\nSo, if you know how to write your program for posting comments with just using HTTP API then I\u2019d go with requests, Selenium would be an overhead in this case","Q_Score":1,"Tags":"python,selenium,python-requests","A_Id":63977682,"CreationDate":"2020-09-20T09:52:00.000","Title":"Comparing request module vs selenium in Python","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"How can we interact with temenos t24 TCServer on TCP port using Python libraries like telnet or socket? Has anyone worked on this area?","AnswerCount":2,"Available Count":2,"Score":0.0,"is_accepted":false,"ViewCount":231,"Q_Id":63981734,"Users Score":0,"Answer":"You can connect by telnet and then post \"tRun tSS [ofs source]\"\nwhere ofs source is an id from OFS.SOURCE T24 application (in older T24 command: \"tSS [ofs source]\" without \"tRun\"). It allows you to send OFS messages. With OFS you can do almost everything you want in T24 ;)","Q_Score":1,"Tags":"python,jboss,tcserver","A_Id":71040999,"CreationDate":"2020-09-20T17:21:00.000","Title":"T24 Telnet Connection on TCP Port Using Python","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"How can we interact with temenos t24 TCServer on TCP port using Python libraries like telnet or socket? Has anyone worked on this area?","AnswerCount":2,"Available Count":2,"Score":0.0,"is_accepted":false,"ViewCount":231,"Q_Id":63981734,"Users Score":0,"Answer":"Easy to connect with Java. Temenos gives standard java libraries to all clients with examples","Q_Score":1,"Tags":"python,jboss,tcserver","A_Id":64936964,"CreationDate":"2020-09-20T17:21:00.000","Title":"T24 Telnet Connection on TCP Port Using Python","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I can't seem to get the requests package to install correctly. Regardless of what I use to actually run my program, I get the same error:\nTraceback (most recent call last): File \"\/Users\/garce\/Desktop\/songlyrics\/getlyrics.py\", line 2, in <module> import requests ModuleNotFoundError: No module named 'requests'\nHowever, I've used pip to install requests. When I run pip show requests in a terminal window, it comes up with this:\nName: requests Version: 2.24.0 Summary: Python HTTP for Humans. Home-page: https:\/\/requests.readthedocs.io Author: Kenneth Reitz Author-email: me@kennethreitz.org License: Apache 2.0 Location: \/Users\/garce\/opt\/anaconda3\/lib\/python3.8\/site-packages Requires: urllib3, idna, certifi, chardet Required-by: Sphinx, jupyterlab-server, conda, conda-build, anaconda-project, anaconda-client\nI assume this is a path issue of some sort, but I can't figure out what's going wrong here. Any idea?","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":64,"Q_Id":64001564,"Users Score":1,"Answer":"Are you installing the package globally, or within a pipenv environment? If globally, have you tried pipenv? Not only is it considered best practices, but I find that using pipenv or Anaconda environments often resolves path issues I experience.","Q_Score":0,"Tags":"python,path,pip,python-requests,package","A_Id":64001836,"CreationDate":"2020-09-22T00:36:00.000","Title":"How do I get python requests package to work?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":1,"System Administration and DevOps":0,"Web Development":0},{"Question":"So I was selecting the port number for a Python backdoor. When I kept a large number like 5979543, it said it was invalid.\nSo... anyway to bypass that restriction?","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":20,"Q_Id":64030552,"Users Score":3,"Answer":"So... anyway to bypass that restriction?\n\nNo. The port number in the TCP and UDP header is 16 bit which allows for a maximum port number of 65535. No way to assign a higher port number since it cannot be represented in the communication packets.","Q_Score":0,"Tags":"python,sockets","A_Id":64031878,"CreationDate":"2020-09-23T14:48:00.000","Title":"Can I bypass number-limit while specifying the port for socket library in Python?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"Im making a webserver from scratch in python by simply supplying HTML content through a tcp socket. I want to be able to run gwt projects, is this possible by just echoing the contents of the generated index.html to a browser?","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":42,"Q_Id":64030792,"Users Score":3,"Answer":"If it is a client-side GWT application, then yes, you could serve it with a basic HTTP webserver that receives HTTP GET requests from a browser and serves files like HTML, JavaScript, images, etc.\nIf the GWT application includes server-side servlets then it wouldn't work when the client sent requests to the server part of the application.","Q_Score":0,"Tags":"python,gwt","A_Id":64056309,"CreationDate":"2020-09-23T15:00:00.000","Title":"Can i run a gwt project without using a webserver","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"My client is based on angular and electron, the server is written in python language.\nI am able to send json-rpc requests between python-python and angular-angular.\nI want to send a json object to client using json-rpc websocket from python to angular.\nI am a noob so not really sure if it is possible or if there's a library available as I cannot find anything cross platform. Any help will be appreciated :)","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":131,"Q_Id":64030846,"Users Score":0,"Answer":"It is possible, there seems only python 3.x rpc websocket support.\nLook for 'jsonrpcclient[websockets]'. There is a server available too.\nOn angular side, I can only suggest 'rpc-websockets'","Q_Score":0,"Tags":"python,angular,websocket,electron,json-rpc","A_Id":65813605,"CreationDate":"2020-09-23T15:03:00.000","Title":"Is it possible to receive rpc-websocket json request in angular from python?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":1,"System Administration and DevOps":0,"Web Development":0},{"Question":"I am working on a project using social media data. An important part of it is to understand the spatial distribution of the social data, like tweets. I used the getoldtweet3 to scrape tweets but all of them are devioded of geo data. Just wonder if there a way to get the spatial features of tweets; or there are other tools to scrape other social media data with geospatial feature.","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":37,"Q_Id":64037027,"Users Score":0,"Answer":"First of all, the library you are using is not accessing the Twitter API, you probably know this already. It\u2019s not official and it could stop working at any moment.\nSecondly and more importantly for your question, there is only about a 1% use of geo pinning in the API. There\u2019s no special way to go around this, it is up to a user to enable geo on their Tweets.\nDoes this answer your question?","Q_Score":0,"Tags":"python,web-scraping,twitter,geospatial","A_Id":64038568,"CreationDate":"2020-09-23T22:18:00.000","Title":"Is there a way to get the geospatail data of tweets?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I'm getting an error: twilio.base.exceptions.TwilioRestException: HTTP 401 error: Unable to create record: Authenticate.\n\nif anybody could help me then would be much appreciated.","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":2331,"Q_Id":64040084,"Users Score":0,"Answer":"I also faced the same problem while trying their tutorial, what seemed to be the error was that the value of auth_token was given under square brackets as ['*******************']. Removing the square brackets and keep it as a string solved it for me.","Q_Score":1,"Tags":"python,django-rest-framework,twilio,twilio-api","A_Id":64725394,"CreationDate":"2020-09-24T05:29:00.000","Title":"TwilioRestException: HTTP 401 error: Unable to create record : Python","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I created a Twitter App in order to control a specified account (post tweet, etc).\nI created this app with my personal account so I can only post tweets on my personal account.\nI'd like to share permissions with my other account so I can tweet on it.\nI heard about OAuth but I don't understand how to use it.","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":57,"Q_Id":64043496,"Users Score":0,"Answer":"You can use twurl to authenticate to another account. The account token and secret will be stored in the .twurlrc file in your home directory, you can use those to post from another account.","Q_Score":0,"Tags":"twitter,twitter-oauth,tweepy,twitterapi-python","A_Id":64045025,"CreationDate":"2020-09-24T09:27:00.000","Title":"How to Access Twitter App with another account?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I am new to selenium and while I was going through the docs and saw the below passage.\n\n5.2. Implicit Waits\n\n\nAn implicit wait tells WebDriver to poll the DOM for a certain amount of time when trying to find any element (or elements) not immediately available. The default setting is 0 (zero). Once set, the implicit wait is set for the life of the WebDriver object.\n\nDoes this mean that you cannot have multiple different implicit wait times in a program? Can you use implicit wait multiple times without it messing with the program?\nThank you","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":50,"Q_Id":64073645,"Users Score":1,"Answer":"You can have multiple driver.implicitly_wait() in your program with varying waits.","Q_Score":1,"Tags":"python,selenium","A_Id":64073738,"CreationDate":"2020-09-26T03:43:00.000","Title":"Are Selenium Implicit Wait times immutable?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"Here is my code:\ndriver.find_element_by_xpath(\"\/\/*[text()='Supreme\u00ae\/Vanson Leathers\u00ae Worn Leather Jacket']\", \"\/\/*[text()='Dark Purple']\") .click()\nBasically, I'm trying to use two two pieces of text to click on one object. However, I'm getting the error below:\nTypeError: find_element_by_xpath() takes 2 positional arguments but 3 were given\nHere's the HTML content of the website for reference:\n<a class=\"name-link\" href=\"\/shop\/jackets\/hfxyzecg4\/e6ap8ojhd\">Supreme\u00ae\/Vanson Leathers\u00ae Worn Leather Jacket<\/a>\n<a class=\"name-link\" href=\"\/shop\/jackets\/hfxyzecg4\/e6ap8ojhd\">Dark Purple<\/a>\nAny way to get around this?","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":40,"Q_Id":64073959,"Users Score":0,"Answer":"Each <a> anchor tag in HTML refers to a link. You can click on any one anchor tag at any point of time.\nIn Selenium any find_element method like find_element_by_id or find_element_by_xpath will allow to accept only one argument i.e., that elements value.\nUse the appropriate link you want click according to your flow and use that xpath in the find_element_by_xpath","Q_Score":0,"Tags":"python,selenium,xpath","A_Id":64076781,"CreationDate":"2020-09-26T04:38:00.000","Title":"I'm trying to get Selenium to click on one object using XPath on two different pieces of text, but there's an error","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"qs\nwhen run locust --master and client locust --master-host,\nlocust can not run api stress,why?\nenv\nserver: locust -f locustfile.py --master (centos)\nclient: locust -f locustfile.py --worker --master-host 10.200.6.1 (mac)\nstatus\nvisit webUI,worker show 1 worker,but start test,\nit will stop immediately after three seconds,\nrps and result still show None.\ncmd log\nerror log : lewis-test\/INFO\/locust.main: Starting Locust 1.2.3\npasslog :\nlewis-test\/INFO\/locust.main: Starting Locust 1.2.3\nlewis-test\/INFO\/locust.runners: Spawning 100\nusers at the rate 100 users\/s (0 users already running)...\nlewis-test\/INFO\/locust.runners: All users spawned: MyUser: 100 (0 already running","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":80,"Q_Id":64074975,"Users Score":0,"Answer":"it work\ndiferent locust version,ths","Q_Score":0,"Tags":"python,centos,locust","A_Id":64075278,"CreationDate":"2020-09-26T07:22:00.000","Title":"when run locust --master and client locust --master-host,locust can not run api stress,why?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"ok, I'm really confused...\nThere seems to be a fair few posts on this, but are all outdated as I have tried about 15 different python bots and code sections and cant get them to work.\nIt seems that the API's have changed in the past year and thus every developer who had a working bot is now left unable to.\nI'm confused at to why, as there are services like Later .com and others that allow people to post to their instagram account. So there has to be a programmable way to do this.\nI had a look at the API docs for their API but I can only seen to find 2 and none of them allow posting to your profile:\n\nInstagram Graph API\n\n\nThe API can be used to get their media, manage and reply to comments on their media, identify media where they have been @mentioned by other Instagram users, find hashtagged media, and get basic metadata and metrics about other Instagram Businesses and Creators.\n\nThis only says get their media, not publish.\n\nInstagram Basic Display API\n\n\nThe API can be used to access any type of Instagram account but only provides read-access to basic data. If you are building an app that will allow Instagram Businesses or Creators to publish media, moderate comments, identify @mentioned and hashtagged media, or get data about other Instagram users, use the Instagram Graph API instead.\n\nThis one says should I want to publish media then I should go to the Instagram Graph API, again which doesnt say I can publish.\nLet's say for the sake of this convo that the one I should use is the Instagram Graph API. The requirements for this are:\n\nBefore you can switch your app to Live Mode you must complete the App Review process. You can begin the process in the App Review > Permissions and Features tab within the App Dashboard. All Instagram Graph API permissions except for pages_show_list require App Review approval.\n\n\nOnce you complete App Review you will be asked to complete Business Verification. You must complete verification before your app will be able to use Advanced Access Levels.\n\nAll I want to do is automate the posting of my personal instagram account so I can bulk edit photos and do all that in one day without having to create a full working live app that people can use or a business.\nPlease, for the love of all that is automation, please point me in the right direction to do this..","AnswerCount":2,"Available Count":2,"Score":0.0,"is_accepted":false,"ViewCount":439,"Q_Id":64084202,"Users Score":0,"Answer":"With the new Facebook Graph API, you can make an Instagram post. But this API, hence the post functionality, works only with the Instagram Businesses and Creators users.\nIn your case, just switch to the Creators account.","Q_Score":2,"Tags":"python,automation,instagram,instagram-api","A_Id":65346143,"CreationDate":"2020-09-27T02:25:00.000","Title":"How to programatically post to instagram","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"ok, I'm really confused...\nThere seems to be a fair few posts on this, but are all outdated as I have tried about 15 different python bots and code sections and cant get them to work.\nIt seems that the API's have changed in the past year and thus every developer who had a working bot is now left unable to.\nI'm confused at to why, as there are services like Later .com and others that allow people to post to their instagram account. So there has to be a programmable way to do this.\nI had a look at the API docs for their API but I can only seen to find 2 and none of them allow posting to your profile:\n\nInstagram Graph API\n\n\nThe API can be used to get their media, manage and reply to comments on their media, identify media where they have been @mentioned by other Instagram users, find hashtagged media, and get basic metadata and metrics about other Instagram Businesses and Creators.\n\nThis only says get their media, not publish.\n\nInstagram Basic Display API\n\n\nThe API can be used to access any type of Instagram account but only provides read-access to basic data. If you are building an app that will allow Instagram Businesses or Creators to publish media, moderate comments, identify @mentioned and hashtagged media, or get data about other Instagram users, use the Instagram Graph API instead.\n\nThis one says should I want to publish media then I should go to the Instagram Graph API, again which doesnt say I can publish.\nLet's say for the sake of this convo that the one I should use is the Instagram Graph API. The requirements for this are:\n\nBefore you can switch your app to Live Mode you must complete the App Review process. You can begin the process in the App Review > Permissions and Features tab within the App Dashboard. All Instagram Graph API permissions except for pages_show_list require App Review approval.\n\n\nOnce you complete App Review you will be asked to complete Business Verification. You must complete verification before your app will be able to use Advanced Access Levels.\n\nAll I want to do is automate the posting of my personal instagram account so I can bulk edit photos and do all that in one day without having to create a full working live app that people can use or a business.\nPlease, for the love of all that is automation, please point me in the right direction to do this..","AnswerCount":2,"Available Count":2,"Score":0.0,"is_accepted":false,"ViewCount":439,"Q_Id":64084202,"Users Score":0,"Answer":"The instagram API does not support uploading images.\nI too was frustrated with how much effort one has to invest to discover this. After all, posting an image seems like a fundamental use case for an API. I wish there was a FAQ page, or bold disclaimer that makes it clear.\nThere are projects on github that support posting to Instagram in unsupported ways (e.g. using a \"private\" api, or simulating the Instagram app). I haven't personally used any of these solutions, but wanted to mention they exist.","Q_Score":2,"Tags":"python,automation,instagram,instagram-api","A_Id":64091366,"CreationDate":"2020-09-27T02:25:00.000","Title":"How to programatically post to instagram","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I am trying to figure out if there is any way to get the class name of some element when the user clicks on the element with selenium.\nThe way this would work is\n\nThe user opens a webpage with selenium\nThe user clicks on an element in the opened browser\nthe respective class name of the element is returned to the user\n\nI know we can search for elements in a webpage with their ids or classes using driver.find_element_by_id or driver.find_element_by_xpath but can we do the reverse with selenium?","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":178,"Q_Id":64087956,"Users Score":0,"Answer":"you mentioned user clicks on an element using selenium.\nIn that case, I would assume you already know the element to click on.\nAnd therefore can use element.getAttribute(\"class\") to get the class Name of the object.","Q_Score":0,"Tags":"python,selenium,markup,classname","A_Id":64088759,"CreationDate":"2020-09-27T11:30:00.000","Title":"Is there any way to get the class name of an element on click with selenium","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I am new to python.\nI created a simple Selenium bot that basically runs on a while loop.\nHowever, it runs on an external machine that I do not have access at all times.\nIt runs into unexpected crashes sometimes, and it takes hours and sometimes days until I check it again only to find a crashed bot. With that said, I'd like to know if a monitoring module or program I can implement into my bot, so I can easily check if the program is running or not.\nSome have suggested a WebSocket heartbeat, however, I do not know, after extensive research, how I would go about implementing such an intricate system. Thank you for taking the time to read my question.\nAny answers appreciated!","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":41,"Q_Id":64092971,"Users Score":0,"Answer":"You can just put a few lines of code at the appropriate point in your script (after the Selenium bot completes its work successfully) to write the current date and time into a file.\nA second script can be scheduled just to read that file and ring an alarm bell (e.g. send you an email) if it is X hours since the timestamp was last updated.\nAlternatively you could write a timestamp to a database table or use whatever other shared systems you have.","Q_Score":2,"Tags":"python,selenium,web-scraping,monitoring","A_Id":64109028,"CreationDate":"2020-09-27T20:24:00.000","Title":"Is there a python module to indicate if the .py is running?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I want to set up a Node-RED GUI with multiple inputs and outputs etc. and some nodes containing python functions. The partner who will use this wants to input some own custom processors to work with the data and should be able to access my python functions, but I don't want to expose all the code I used. Is that possible?","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":72,"Q_Id":64133760,"Users Score":0,"Answer":"Not really.\nThe best you could do is to create a fork of the Python Function node that loads the python from a file rather than present a edit area in the Node-RED editor.\nThe python code would still be available on the disk of the machine running Node-RED but wouldn't be accessible via the editor. This works if the partner is only accessing Node-RED via the web, but if they have access to the machine it's running on then they will have access to everything.","Q_Score":0,"Tags":"python,node-red","A_Id":64135238,"CreationDate":"2020-09-30T08:29:00.000","Title":"Using Python nodes in Node-RED without exposing code","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I created a virtual environment and tried to import selenium inside one of the python files I have inside that venv and it's throwing me this error.\nBoth my python3 and pip3 have the same source.\nI tried pip3 install selenium and python3 -m pip install -U selenium, both didn't work.\nI can import the selenium package from the command line without any problem. Will this be any problem with my text editor? I'm using vscode.","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":507,"Q_Id":64170612,"Users Score":1,"Answer":"My python interpreter inside vscode was pointing to another version of python. I changed that to pyton3 and it worked. To change the python3 interpreter, open command palette(ctrl+shift+p) and search for,\nPython : Select Interpreter. Now change that to the correct path and you're good to go.","Q_Score":1,"Tags":"python-3.x,selenium,selenium-webdriver,visual-studio-code,pip","A_Id":64180221,"CreationDate":"2020-10-02T11:16:00.000","Title":"Unable to import 'selenium' : pylint(import-error)","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":1,"System Administration and DevOps":0,"Web Development":0},{"Question":"Using python telegram-upload API I was able to upload the file to telegram cloud and also I stored the corresponding file_id of that file. Now this API provides telegram-download method which downloads all files from my telegram cloud, My question is how can I download some selected files only with help of file_id that I am storing. Is it possible using some other method?","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":226,"Q_Id":64184585,"Users Score":0,"Answer":"You can download a file first then you send it using your code. I recommend to use telethon library to upload file faster.\nAlso you can use transload links to upload without downloading which uploads a file faster.","Q_Score":0,"Tags":"python,telegram,telegram-upload","A_Id":64427432,"CreationDate":"2020-10-03T13:01:00.000","Title":"Download selected file from telegram?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I have a question about Apache Kafka. I have a scenario where I want to find information in a large 6GB .xml file, which is the latest dump of wikipedia. As such, it contains logical collections of information, generally spanning 20-25 lines. My goal is to have names of 2 football players as input, and search through this document additional information (which is present there) and compare them. Since I will need to parse through the document every time new names are entered, I am considering a sequential search and regex pattern matching. Later maybe experiment with indexing and inverse indexing. For a start my input file will contain about 100 000 lines.\nI know the general concept of how Kafka works. There are servers which work as storage and other servers that are in charge of exporting and importing information. Then there are clients that can access this data and process it. My question is since this is just a school project, is there a point in me using kafka to process the input 6GB file? I have a simple demo running where I open and read a file in python and send it using a kafka producer and then \"receive\" it using a kafka consumer, but I am confused if it is not simpler to just read lines of the file as I go and look for the information I want.","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":32,"Q_Id":64197687,"Users Score":1,"Answer":"Kafka supports parallell processing by virtue of enabling topics to be split into partitions. In your case, I would consider creating a topic and splitting it into, say 50 partitions to begin with. You can then write records to the topic in quick succession from a single threaded process processing the xml.\nCreate a consumer that reads from the topic and performs the search; consider maybe outputting matching records to a final topic that your client can read from.\nInstead of multi-threading the consumer (using in-process threading), you can just start multiple isolated instances of it, as long as you configure them to all join the same consumer group. Have your producer round robin records (this is done automatically as long as each record has a unique key -- you could use a uuid if you have no natural key). As you bring up more instances of the consumer, the kafka brokers will ensure that each picks up a roughly equal subset of the partitions and so you get a sensible load balancing effect. You should find that performance increase as you increase the number of consumer processes up to the number of cores you have available to perform the search.\nBy having the process which produces records to the input topic in the first place also await a result on the final output topic, you can easily write the result to the same tty as that which you use to launch to initial processing.\nStarting multiple separate processes in the consumer group mimics the pattern of horizontal scaling in a distributed system. For example, in kubernetes, you can increase the number of replicas of a deployment (\/replicaset) and if you are using kafka in the above manner, you will see a similar load balancing behaviour.","Q_Score":0,"Tags":"python,apache-kafka","A_Id":64198535,"CreationDate":"2020-10-04T17:23:00.000","Title":"Using kafka as a framework to search for information","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"Im Trying To Buy A Laptop Automatically But When I Try To Put In The Credit Card Number It Gives The Error selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate Im Using Xpath btw.","AnswerCount":2,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":686,"Q_Id":64211415,"Users Score":0,"Answer":"If u try to locate it by id. It is false. I'd will be keep changing for everytime. Try to identify with name \"addCreditCardaNumber\"\nIt will work for u.","Q_Score":0,"Tags":"python,selenium,selenium-webdriver,webdriver","A_Id":66450741,"CreationDate":"2020-10-05T15:14:00.000","Title":"Im Trying To Automate Buying Something on Amazon With Selenium Python","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I am trying to develop a React Client that consumes the Frappe REST API but on each request I am getting a CORS error. according to the community forum, I have to edit the Nginx conf to allow CORS requests but I am facing this error on my machine (localhost) and the dev server is not Nginx so how can I allow CORS requests on localhost?","AnswerCount":2,"Available Count":2,"Score":0.0,"is_accepted":false,"ViewCount":591,"Q_Id":64229353,"Users Score":0,"Answer":"I don't think you need to configure CORS, especially during local development.\nLet's say your React client is running the port 8080 and Frappe server is on 8000. You can proxy all your requests from localhost:8080 to localhost:8000.\nIn production, you can generate the static assets and put them in the www folder of the Frappe app.","Q_Score":0,"Tags":"python-3.x,erpnext,frappe","A_Id":64578766,"CreationDate":"2020-10-06T15:47:00.000","Title":"Frappe Framework ERPNEXT: How to enable CORS on localhost","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I am trying to develop a React Client that consumes the Frappe REST API but on each request I am getting a CORS error. according to the community forum, I have to edit the Nginx conf to allow CORS requests but I am facing this error on my machine (localhost) and the dev server is not Nginx so how can I allow CORS requests on localhost?","AnswerCount":2,"Available Count":2,"Score":0.0,"is_accepted":false,"ViewCount":591,"Q_Id":64229353,"Users Score":0,"Answer":"Add \"allow_cors\": \"*\" to common_site_config.json and restart bench","Q_Score":0,"Tags":"python-3.x,erpnext,frappe","A_Id":69908944,"CreationDate":"2020-10-06T15:47:00.000","Title":"Frappe Framework ERPNEXT: How to enable CORS on localhost","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I'm using a Jupyter Notebook connected to an AWS EMR Cluster. The subnets and security groups attached to all nodes of the cluster are able to connect to the internet.\nHowever, when I attempt to pip install <package> from the Notebook attached to the EMR cluster, I get the following error:\nWARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7f6b4c6bd710>: Failed to establish a new connection: [Errno 101] Network is unreachable')': .....\nI have also tried to make a GET request to the internet r = requests.get('https:\/\/www.google.com'), which was also unsuccessful. I'm unable to figure out what is preventing my connectivity to the internet from my Jupyter Notebook on AWS EMR.","AnswerCount":2,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":952,"Q_Id":64234516,"Users Score":0,"Answer":"In my case it turned out to be a problem with the sg ElasticMapReduceEditors-Livy & ElasticMapReduceEditors-Editor\nI had created, I just added a rule in both sg that would allow all outbound traffic to test if that solved it, luckily it solved it, then I removed both rules and mysteriously my notebooks kept working as usual,\nI think it was bad behavior from aws.","Q_Score":2,"Tags":"python,amazon-web-services,pip,jupyter-notebook,amazon-emr","A_Id":69194527,"CreationDate":"2020-10-06T22:12:00.000","Title":"Unable to install package through pip on AWS EMR Notebook","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":1,"System Administration and DevOps":0,"Web Development":0},{"Question":"I want to transmit data with a slow web-service; but all the same I want to avoid busy waiting.\nIs there a way that I can do asyncio.sleep in the ETL graph? do other async\/await commands can be used in bonobo?","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":53,"Q_Id":64252120,"Users Score":0,"Answer":"Currently there is no active work on bonobo library, and the async features of it does not actually work. So, there is no way of using asyncio.sleep in current bonobo version.","Q_Score":0,"Tags":"python,bonobo-etl","A_Id":65639623,"CreationDate":"2020-10-07T20:50:00.000","Title":"How can I use asyncio.sleep in python-bonobo","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":1,"System Administration and DevOps":0,"Web Development":0},{"Question":"I want the bot to send an inline button of \"sendPicture\" and after clicking it the user can able to click picture from the camera and send it. In this process users not able to send the picture from their gallery. Is telethon support it or is there any other methods to implement this scenario?","AnswerCount":1,"Available Count":1,"Score":0.3799489623,"is_accepted":false,"ViewCount":188,"Q_Id":64256962,"Users Score":2,"Answer":"No. You can just send them text asking nicely.","Q_Score":2,"Tags":"python,telethon,telegram-api","A_Id":64257102,"CreationDate":"2020-10-08T06:32:00.000","Title":"Can Telegram Bot ask user to take picture from camera?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"Using DynamoDB.\nAn online store where you see a snapshot of the data for 20 items on the first screen, including price, picture, product name.\nWhen you click on the product it reveals more data, such as description, any deals, more images, etc.\nWould you make another call to the database for the second view or just get all the data on the first call?","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":18,"Q_Id":64262361,"Users Score":1,"Answer":"When considering \"best practices\", I wouldn't consider one approach a clear winner over another. Which approach works best for you will depend on the specifics of your application. For example, are you in a high latency\/low bandwidth environment? Are you on a mobile device? How often is this query run? Etc.\nI would recommend starting by pulling back all the data you need in a single request because it's simpler. If you run into performance issues along the way, you can circle back and try fetching additional data on an as-needed basis.\nIf there is a \"best practice\" here, it's to avoid pre-optimizing your code. Now if only I could take my own advice :)","Q_Score":0,"Tags":"python,database,amazon-dynamodb,dynamodb-queries","A_Id":64266772,"CreationDate":"2020-10-08T12:18:00.000","Title":"Read entire item from database to begin with or make a second call when needed","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"Tried installing requests-html through pip and pip3 in many different ways (recommendations from other StackOverflow threads) and I keep on getting the same error ImportError: No module named requests_html.\nChecked pip and pip3 list, both of them have requests and requests-html. Also my python is updated, its 3.7.6.\nAlso weirdly I started writing the code on jupyter notebook, and everything worked, however when I switched to sublime text that's where the problem occurred.\nPlease help.\nP.s. I am on a mac, and I am a beginer.","AnswerCount":3,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":2034,"Q_Id":64270822,"Users Score":0,"Answer":"I am no Mac expert but.. I know Mac has its own python2 and 3 inner versions which is critical for the OS to operate, so you maybe installed pip into those ones instead of the ones you are operating via PATH.","Q_Score":0,"Tags":"python,pip,python-requests,python-requests-html","A_Id":64270932,"CreationDate":"2020-10-08T21:24:00.000","Title":"\"ImportError: No module named requests_html\" after installing requests-html to pip and pip3","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I am making a sneaker bot\nIt's Client based!\nSo I want to open multiple Web drivers in selenium\nwhat I got:\ntasks = input(\"How many Drivers you want to open?\") Then it should open the Number of drivers you have entered!\nLike I enter 4:\nThen it should open 4 drivers\nalso, these drivers should do something with the XPath search. I also want this amount of drivers to do the same thing but the most important thing is to open the amount of entered web drivers.\nThank u!","AnswerCount":1,"Available Count":1,"Score":0.1973753202,"is_accepted":false,"ViewCount":56,"Q_Id":64296110,"Users Score":1,"Answer":"I have no idea where is the question ...\nPlease read how to ask a question ...\nBut I am going to try to do it anyways (will not provide the code, cuz you did not eighter and at a same time i can bet you did not even tried).\nafter you get the number of tasks in integer form, I would recommend you to create an array, where you will put the tasks amount of browser references (for example, if you want 4 browsers, you will just put it in a forloop and append them to the array) and after that you will you can itterate through the array everyime you will be performing any action (its like making it for 1 browser instace, just everything will be wrapped in a forloop that will itterate the array).\nI recommend you to edit the question and provide a code you have tried and describe the specific problem.","Q_Score":0,"Tags":"python,selenium,webdriver,bots","A_Id":64299905,"CreationDate":"2020-10-10T17:07:00.000","Title":"Open mutliple selenium webdrivers (tasks) with a command in python","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I have to extract Folium map to pgn\/img.\nI've tried a lot of path, via selenium, converting to pdf and reconvert to png, pdfkit, imgkit but nothing worked...\nSome help?","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":171,"Q_Id":64303759,"Users Score":0,"Answer":"You can do it like this with Selenium :\n\nYou create your map with folium as an HTML file\nYou open your Selenium browser at the url file:\/\/{your_folder_path}\/{your_HTML_map_file}\nYou wait for a few seconds in order to use the method save_screenshot() from your Selenium browser","Q_Score":0,"Tags":"python,extract,folium,file-conversion,html-to-pdf","A_Id":64338111,"CreationDate":"2020-10-11T11:59:00.000","Title":"Python Folium export map error via pdfkit and selenium","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I was wondering if there was anyway to check how many times admin is used within a certain time period and modify roles based off that. For example if someone started kicking loads of people within a short time span it removes their roles to do that?\nAny help is appreciated.","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":649,"Q_Id":64324907,"Users Score":0,"Answer":"Setup up your bot to record to a file each time an admin command like kick was used (and the time alongside it) and increase a counter by 1, when the counter reaches a certain threshold you read the file and minus the old time from the most recent which gives you your time range. Use this to decide wherever to kick the user.\nThen you can decide. You can also reset the counter every X minutes to ensure they bot doesn't get out of control","Q_Score":0,"Tags":"python,discord,discord.py","A_Id":64324971,"CreationDate":"2020-10-12T20:25:00.000","Title":"Discord.py auto moderation","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"For my current web (reactJS) app, i need to implement a push notification system. Backend APIs are written in Python on Django Framework. So how should i implement socket, should I write in python, or may I write in node or node for the client-side and python for the server-side?","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":321,"Q_Id":64337145,"Users Score":1,"Answer":"Are you looking for Web push or mobile app? The term push is really synonymous with 'push notifications' meaning the client will get them even if the app \/ website is open or not.\nIf that is what you are looking for then you need to be looking at service workers and push notification servers.\nIf you want to simply send alerts to users on your page you can for sure use a simple websocket or long poll done in JS with a node server to emit to.","Q_Score":1,"Tags":"socket.io,serversocket,python-sockets","A_Id":64362885,"CreationDate":"2020-10-13T14:25:00.000","Title":"Socket for push notification","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"So the thing is I have a github repo for my discord bot, and every time a message is sent in the server, the json file along that code is supposed to be updated, but it does not get updated after every refresh... Now is it due to the traffic of the github databases, or does the json file get updated only on the server of the bot(in my case, its heroku) and not in the github?","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":222,"Q_Id":64351250,"Users Score":0,"Answer":"The file that's on your Heroku server is independent of GitHub. Even if your Heroku instance is a Git repository, that Git repository is independent of the one on GitHub. In order to get the JSON file from the repository on Heroku to GitHub, you'd have to add and commit the file, make sure that GitHub was added as a remote, and then push, using a suitable set of credentials.\nHowever, having said that, you probably don't want to do that. Data files, like your JSON file, don't belong in repositories. Code repositories are best suited to storing your code and the data it operates on should be stored elsewhere, such as in a database or in a set of files.","Q_Score":0,"Tags":"python,json,github,heroku,discord","A_Id":64363666,"CreationDate":"2020-10-14T10:20:00.000","Title":"Does the json file in a github repository require time to be updated?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":1,"System Administration and DevOps":0,"Web Development":0},{"Question":"I would like to be able to, from the command line or in a Python script\/program, execute the action that you can do manually in the iTunes application: File->Library->Export Library. So that I can have a copy of an up to date xml file of my iTunes library. Is this possible?","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":43,"Q_Id":64364784,"Users Score":0,"Answer":"I have searched for a solution to this problem for quite a lot of time. And I still have not found anything.\nHowever, as I needed the XML only for the \"played count\" and \"name\" variable of each track I instead created an AppleScript which is then directly called in the python script in which I used to import the XML file.\nNote that the AppleScript directly asks the Music app for up-to-date data but it does need the Music app to be running.\nMaybe this can be adapted to work for your problem, too.\nHope this helps, Alex.","Q_Score":1,"Tags":"python,xml,bash,command-line,ituneslibrary","A_Id":64487115,"CreationDate":"2020-10-15T04:15:00.000","Title":"Mac Terminal Command Line command to perform the iTunes File->Library->Export Library action?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":1,"Web Development":0},{"Question":"I use Selenium in Python and I want to scrape a lot of websites from one company (many hundreds). But that shouldn't burden the system under any circumstances and because this is a very large website anyway, it shouldn't be a problem for them.\nNow my question is if the company can somehow discover that I'm doing web scraping if I'm acting like a human. That means I stay on a website for an extra long time and allow extra time to pass.\nI don't think you can recognize me by my IP, because the period of time is very long while I do this and I think it looks like normal traffic.\nAre there any other ways that websites can see that I am doing webscraping or generally running a script?\nMany Thanks\n(P.S.:I know that a similar question has already been asked, but the answer was simply that he doesn't behave like a human and visits the website too quickly. But it's different for me ...)","AnswerCount":2,"Available Count":2,"Score":0.0996679946,"is_accepted":false,"ViewCount":782,"Q_Id":64366381,"Users Score":1,"Answer":"One way they can tell is from your browser headers","Q_Score":0,"Tags":"python,selenium,selenium-webdriver,web-scraping","A_Id":64407049,"CreationDate":"2020-10-15T07:00:00.000","Title":"Can websites detect web scraping if I act like a human (Selenium, Python)?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I use Selenium in Python and I want to scrape a lot of websites from one company (many hundreds). But that shouldn't burden the system under any circumstances and because this is a very large website anyway, it shouldn't be a problem for them.\nNow my question is if the company can somehow discover that I'm doing web scraping if I'm acting like a human. That means I stay on a website for an extra long time and allow extra time to pass.\nI don't think you can recognize me by my IP, because the period of time is very long while I do this and I think it looks like normal traffic.\nAre there any other ways that websites can see that I am doing webscraping or generally running a script?\nMany Thanks\n(P.S.:I know that a similar question has already been asked, but the answer was simply that he doesn't behave like a human and visits the website too quickly. But it's different for me ...)","AnswerCount":2,"Available Count":2,"Score":1.2,"is_accepted":true,"ViewCount":782,"Q_Id":64366381,"Users Score":1,"Answer":"When you scraping make sure that you respect the robots.txt file which is based at the root of the website. It set the rules of crawling: which parts of the website should not be scraped, how frequently it can be scraped.\nUser navigation patterns are monitored by large companies to detect bots and scraping attempts. There are many anti scraping tools available in market which are using AI to monitor the various patterns to differentiate between a human and a bot.\nSome of the main techniques used to prevent scraping apart from software are\nCaptcha,\nHoney traps,\nUA monitoring,\nIP monitoring,\nJavascript encryption, etc..\nThere are many more, so what i am saying is that yes it can be detected.","Q_Score":0,"Tags":"python,selenium,selenium-webdriver,web-scraping","A_Id":64407609,"CreationDate":"2020-10-15T07:00:00.000","Title":"Can websites detect web scraping if I act like a human (Selenium, Python)?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I have an application that sends some data to a backend on a webserver where that data is being analysed and put through a neural network to come back with some other data that is being displayed on the client front-end.\nNow, when a lot of clients are connecting simultaneously, we have to scale up, and then down again + there is network latency and you always need an Internet connection.\nBecause the backend isn't doing that much, besides just analysing and putting data through a neural network... it would be ideal to put this backend functionality in each client... no more need to scale up (every client has his own backend on board) + no more network latency.\nThe problem is of course that the most valuable thing is the trained neural network data, that in this case should also be present on each client... and of course I don't want it to be stolen as it costed a lot of money to buy training data for our neural network + then we had to train it as well...\nSo my question is:\nIs there any way to run the backend neural network on the client side in a secure way, preventing our valuable trained neural network from being stolen?\nThank you for any advice!","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":28,"Q_Id":64366950,"Users Score":0,"Answer":"In short, there is no absolute way to prevent that, no.\nIf you give give a binary to be executed to a client, there will be ways for your client to copy it in some shape or form and execute it again.","Q_Score":2,"Tags":"python,neural-network,artificial-intelligence,backend","A_Id":64367350,"CreationDate":"2020-10-15T07:41:00.000","Title":"Secure ways of running backend functionality on client side","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I have been working on a python 3 app that downloads data from a Node.js server. The json file is updated regularly (this is a chat app), but my current code compares the already downloaded data to the complete data on the server. Is there a way to only download the text that is not in a variable, so as to not waste bandwidth?","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":50,"Q_Id":64376060,"Users Score":0,"Answer":"With an API that just returns JSON, no, there is no way to only download just the changed data, as that isn't how JSON works. The server you're downloading from has no way of knowing what your local state is. If this is a JSON file of, say, events, and they have an ID or timestamp, the NodeJS server could be written to accept a ?timestamp=last_event_you_have or ?since=id_of_last_message and only return objects since that time or ID, but that requires changes on the NodeJS server side of things.","Q_Score":1,"Tags":"node.js,json,python-3.x,http","A_Id":64380032,"CreationDate":"2020-10-15T16:35:00.000","Title":"Downlad parts of JSON file from Node.js server","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":1,"System Administration and DevOps":0,"Web Development":0},{"Question":"I cannot get_dialogs with Telethon Bot since the method is only available for a client. Therefore when I try to access the entity with its id I receive an error. How do I access an entity(channel) if I cannot parse all channels and neither access them by id? I would like to parse messages from a private group and use Telegram Bot for it.","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":1135,"Q_Id":64397819,"Users Score":0,"Answer":"Adding the bot to the channel while it's running should be enough for the bot to obtain its access hash. Alternatively, sending a message there while the bot is inside should also work. This hash will be saved to the .session file. Regardless, if you use the \"marked\" ID (for channels, prefix -100 to the real ID or use types.PeerChannel(real id)), Telethon will know you mean a channel, which can help make it work.","Q_Score":2,"Tags":"python,telegram-bot,telethon","A_Id":64400566,"CreationDate":"2020-10-16T23:58:00.000","Title":"How to get entity of a channel using Telethon bot API?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"We are doing Salesforce (old instance) to Salesforce (new instance) migration. We have Standard and custom objects in our old which are need to be migrated to the new Salesforce instance. We are doing this in the following steps:-\n1: Using salesforce describe rest api create tables in MySQL with similar mapping of salesforce object. Tables are created with referential integrity using \"referenceTo\" field of describe call. Example: services\/data\/v49.0\/sobjects\/User\/describe\/.\n2: Get data from salesforce using query rest api and put into MySQL tables. Example: \/services\/data\/v39.0\/query\/?q=.\n3: In this part, we have some confusion like which api we should use so that we can push records to destination org.\nWe are using Python for this task.","AnswerCount":3,"Available Count":2,"Score":0.1325487884,"is_accepted":false,"ViewCount":202,"Q_Id":64406180,"Users Score":2,"Answer":"Although your approach is not exactly best practice, to answer your question, you should use the Bulk API to push the record data into new org (assuming you have lots of data). There are also lots of backup\/restore vendors out there that you could have just used like Ownbackup, Odaseva, Backupify, Grax","Q_Score":0,"Tags":"python,migration,salesforce","A_Id":64772134,"CreationDate":"2020-10-17T18:54:00.000","Title":"Salesforce to Salesforce MIgration","Data Science and Machine Learning":0,"Database and SQL":1,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"We are doing Salesforce (old instance) to Salesforce (new instance) migration. We have Standard and custom objects in our old which are need to be migrated to the new Salesforce instance. We are doing this in the following steps:-\n1: Using salesforce describe rest api create tables in MySQL with similar mapping of salesforce object. Tables are created with referential integrity using \"referenceTo\" field of describe call. Example: services\/data\/v49.0\/sobjects\/User\/describe\/.\n2: Get data from salesforce using query rest api and put into MySQL tables. Example: \/services\/data\/v39.0\/query\/?q=.\n3: In this part, we have some confusion like which api we should use so that we can push records to destination org.\nWe are using Python for this task.","AnswerCount":3,"Available Count":2,"Score":0.0665680765,"is_accepted":false,"ViewCount":202,"Q_Id":64406180,"Users Score":1,"Answer":"If it's a one-time process (only migration), then go with the any of the data management tool:\n\nData Import wizard\nData Loader\nDataLoader.io\nworkbench\n\nAll the above-mentioned tool are free and has a variety of features.\nSteps to import:\n\nExport all the data from the old org.\nNow determine the order of insertion based on the object\nrelationship field:\n\n\nif objects have relation in between, start with Importing the parent\nrecords first.\nif there are child records, map the new id of the parent by applying\nv-lookup between the parent success sheet and child record sheet.\nrepeat step 2, until you end up with migration of all records.\n\nFYI: If the records for the objects are less than 50k, then go with the inbuild import wizard it can give you more control on controlling workflows and process triggering which other tools would not.","Q_Score":0,"Tags":"python,migration,salesforce","A_Id":65623302,"CreationDate":"2020-10-17T18:54:00.000","Title":"Salesforce to Salesforce MIgration","Data Science and Machine Learning":0,"Database and SQL":1,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I'm currently making a program which fetches the prices and discounts etc. off the Steam sales search page, however all the prices are in $ and I need them to be in \u00a3. I'm relatively new to Python and Selenium, and have been unable to fix the issue myself, and I haven't seen anyone else with a similar problem.\nI'm using the Selenium webdriver, and don't know how to configure its region, but I'm assuming that it'll probably need to be something inputted into Chrome Options, but I don't know.\nThanks in advance.","AnswerCount":2,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":483,"Q_Id":64413000,"Users Score":0,"Answer":"Add &cc=US or ¤cy=1 to url. I can't tell which will work if you dont give specific url.\nCan you give an example url?","Q_Score":0,"Tags":"python,selenium,discord.py,steam,regional-settings","A_Id":64515957,"CreationDate":"2020-10-18T11:56:00.000","Title":"Setting the region in the Python Selenium Webdriver?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"Problem: Not able to find\/click\/send.keys to any of elements on a specific webpage.\nWebDriver Chrome\nI can interact with that webpage until I click a certain link where a new tab is opened and from that page I'm unable to do anything.\nWhat I've tried so far after extensive research here:\n1- Since a new tab was opened, I've \"forced\" the code looking into the new tab using driver.switch_to.window and to be sure I've printed the current page title and it's the correct one.\n2- I've tried both implicit\/explicit waiting (even time.sleep) in order to wait for the page to load.\n3- Research pointed me to iframe.. I don't have any iframe but nevertheless I've searched for the number of frames (2) and switched between both and tried to find elements using xpath (full xpath) and none were find.\n4- Maximize window.\nSo I tried \"everything\" in the books. Inspecting the elements that I needed to interact I've found that every single one was pointing to a JS with some arguments.\n href=\"JavaScript:SWESubmitForm(document.SWEForm2_0,s_10,\"s_2_1_96_0\",\"VRId-0\")\" tabindex=\"2997\" id=\"s_2_1_96_0\">Track Faults \nLooking further found that s_10 is a variable within the script which have different options from s_0 to s_14.\n\nvar s_0 = {action:\"\/esales_enu\/start.swe\",target:\"_sweview\",SWECmd:\"GotoView\",SWEMethod:\"GotoView\",SWEView:\"L2C Track Channel Reference View BT\",SWEApplet:\"L2C New CZ Home Page Applet BT\",SWEReqRowId:\"0\",SWESP:\"false\",SWENeedContext:\"true\",SWEKeepContext:\"0\",SWEDIC:\"false\"};\n\nvar s_10 = {action:\"\/esales_enu\/start.swe\",target:\"_sweview\",SWECmd:\"GotoView\",SWEMethod:\"GotoView\",SWEView:\"New Portal Fault Search View ORH BT\",SWEApplet:\"L2C New CZ Home Page Applet BT\",SWEReqRowId:\"0\",SWESP:\"false\",SWENeedContext:\"true\",SWEKeepContext:\"0\",SWEDIC:\"false\"};\nSo I started searching and found driver.execute_script() and I've wrote:\ndriver.execute_script(\"SWESubmitForm(document.SWEForm2_0,s_10,'s_2_1_96_0','VRId-0')\")\nBut this occurs:\nselenium.common.exceptions.JavascriptException: Message: javascript error: SWESubmitForm is not defined \nWhat I'm missing here? Syntax? If someone can shed some light on which direction should I go, it will be really appreciated.\nThanks.","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":69,"Q_Id":64415512,"Users Score":0,"Answer":"I was able to fix this and learn a few things. The JS \"theory\" I was taking didn't made any sense since the driver.execute_script() isn't used the way I was thinking but rather injecting code or finding elements as well.\nSecond, the solution was actually frames :)\nI was so focused in switching between the two frames I had found initially that for not one moment I thought that INSIDE those two frames could had been more frames!\nAnd basically that was it, after switching to the main frame, I've switched again to the 2nd frame inside that main one and was able to find through xpath.","Q_Score":0,"Tags":"javascript,python-3.x,selenium,selenium-webdriver","A_Id":64449627,"CreationDate":"2020-10-18T16:13:00.000","Title":"Selenium + Python - Not able to interact\/click with JavaScript","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I have a question; that's as to how to make a discord.py bot send a message by itself upon on_ready(): ?\nIts only in one server and it doesn't matter what channel at the moment, i couldn't find anything in the documentation-\nany help is appreciated ^^!","AnswerCount":3,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":66,"Q_Id":64434044,"Users Score":0,"Answer":"I don't believe the on_ready object has access to send. So I don't think it is possible via the on_ready object. You would have to set it via a different action.","Q_Score":0,"Tags":"python,discord.py,discord.py-rewrite","A_Id":64435136,"CreationDate":"2020-10-19T19:34:00.000","Title":"How to get a bot to send a message automatically?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I made an app that counts active members of my telegram channel using Telethon(telegram api library). my app gets the active rate per hour via api and writes it into a google spreads sheet.\nI had to use 'String session' so I would not have to log in more than once, and keep the app running every day.\nHowever, I'm not really sure when this string session thing expires. The official document does not say anything about it.\nI read on a website 'Treat the session file\/string, your API ID and hash as your password. Anyone got on hold of these 3 will be able to gain full access to your Telegram account until you revoke it.' so If it is until I revoke it, I assume it lasts forever?\nIt would be great if anyone could tell me when it expires.","AnswerCount":1,"Available Count":1,"Score":0.6640367703,"is_accepted":false,"ViewCount":390,"Q_Id":64437962,"Users Score":4,"Answer":"StringSession is just like any other session, it won't expire unless revoke the session.","Q_Score":1,"Tags":"python-3.x,telegram,telethon","A_Id":64438217,"CreationDate":"2020-10-20T03:18:00.000","Title":"When does string session expires?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I'm using python selenium to automate user login for a website. However, this specific website uses a 2-step authentication when logging in: After clicking login, it then sends a code to your email address, which you have to enter into the website in order to authenticate. Any ideas what I should do or how I can access that code using python selenium?","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":317,"Q_Id":64448819,"Users Score":0,"Answer":"You should log in to the website in the traditional way with selenium.\nNow the site will ask for the verification -> So you need to read your mails somehow.\n\nI see 2 good ways to solve this:\n\nLogin to the email service with selenium in other window or tab -> Open emails-> Select the right email -> Read out the verification code with a regex argument from the email content.\n\nOR\n\nYou can read your emails with IMAP -> Select the correct email with IMAP -> Once again: Read out the verification code with a regex argument from the email content.\n\n\nNow you got the verification code simply write it in and log in.","Q_Score":2,"Tags":"python,selenium,authentication","A_Id":64449076,"CreationDate":"2020-10-20T15:51:00.000","Title":"How to automate website login that uses 2-step verification using python selenium?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I am trying build a BOT using put_bot boto3 API operation and processbehaviour is set to BUILD. Lex will build the bot but after 2 or 3 mins it changes its status to NOT_BUILD. When i tried to BUILD the same for second time manually or through calling put_bot api again, then it build successfully. This status change happen intermittently. This happens 1 out of 10 times. Any help in this is highly appreciated.\nThanks","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":104,"Q_Id":64459384,"Users Score":0,"Answer":"I've seen similar using the Java Lex Model building SDK and in my case it was a result of the build failing.\nHave you looked for errors wtih put_bot ?","Q_Score":0,"Tags":"python,amazon-web-services,api,boto3,amazon-lex","A_Id":64643774,"CreationDate":"2020-10-21T08:13:00.000","Title":"Amazon Lex changing status from Ready to NOT_BUILD after 2 or 3 mins","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I am very new to Python, it`s about 6 months, I have a very basic question.\nWhere is the source of \"driver.page_source\", dose it retrieve and read from the local content of the website, I mean whatever server has already sent to the local page, or every time it will send a request or call directly to the website server.\nI want to scrape live data so I need to do \"driver.page_source\" every minute, while I don't want to disturb the website server by sending too many requests to the server.","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":550,"Q_Id":64482921,"Users Score":0,"Answer":"driver.page_source retrieves the HTML of the url you passed on driver.get(url).\nIt gets the same code as seen on your browser when you click Inspect Element.\nIf you want specific elements and not the whole page though, you should loop up Xpaths and use driver.find_elements_by_xpath(xpath_expression).","Q_Score":0,"Tags":"python,selenium,webdriver,scrape","A_Id":64483335,"CreationDate":"2020-10-22T13:06:00.000","Title":"Python Selenium \"driver.page_source\" call from website server or local content","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"C:\\Users\\xxlda\\AppData\\Local\\Programs\\Python\\Python38-32\\python.exe C:\/Users\/xxlda\/PycharmProjects\/python-telegram-bot\/app.py\nCause exception while getting updates.\nTraceback (most recent call last):\n File \"C:\\Users\\xxlda\\AppData\\Local\\Programs\\Python\\Python38-32\\lib\\site-packages\\aiogram\\dispatcher\\dispatcher.py\", line 359, in start_polling\n updates = await self.bot.get_updates(limit=limit, offset=offset, timeout=timeout)\n File \"C:\\Users\\xxlda\\AppData\\Local\\Programs\\Python\\Python38-32\\lib\\site-packages\\aiogram\\bot\\bot.py\", line 95, in get_updates\n result = await self.request(api.Methods.GET_UPDATES, payload)\n File \"C:\\Users\\xxlda\\AppData\\Local\\Programs\\Python\\Python38-32\\lib\\site-packages\\aiogram\\bot\\base.py\", line 200, in request\n return await api.make_request(self.session, self.__token, method, data, files,\n File \"C:\\Users\\xxlda\\AppData\\Local\\Programs\\Python\\Python38-32\\lib\\site-packages\\aiogram\\bot\\api.py\", line 104, in make_request\n return check_result(method, response.content_type, response.status, await response.text())\n File \"C:\\Users\\xxlda\\AppData\\Local\\Programs\\Python\\Python38-32\\lib\\site-packages\\aiogram\\bot\\api.py\", line 82, in check_result\n exceptions.ConflictError.detect(description)\n File \"C:\\Users\\xxlda\\AppData\\Local\\Programs\\Python\\Python38-32\\lib\\site-packages\\aiogram\\utils\\exceptions.py\", line 137, in detect\n raise err(cls.text or description)\naiogram.utils.exceptions.TerminatedByOtherGetUpdates: Terminated by other getupdates request; make sure that only one bot instance is running\nCause exception while getting updates.\nTraceback (most recent call last):\n File \"C:\\Users\\xxlda\\AppData\\Local\\Programs\\Python\\Python38-32\\lib\\site-packages\\aiogram\\dispatcher\\dispatcher.py\", line 359, in start_polling\n updates = await self.bot.get_updates(limit=limit, offset=offset, timeout=timeout)\n File \"C:\\Users\\xxlda\\AppData\\Local\\Programs\\Python\\Python38-32\\lib\\site-packages\\aiogram\\bot\\bot.py\", line 95, in get_updates\n result = await self.request(api.Methods.GET_UPDATES, payload)\n File \"C:\\Users\\xxlda\\AppData\\Local\\Programs\\Python\\Python38-32\\lib\\site-packages\\aiogram\\bot\\base.py\", line 200, in request\n return await api.make_request(self.session, self.__token, method, data, files,\n File \"C:\\Users\\xxlda\\AppData\\Local\\Programs\\Python\\Python38-32\\lib\\site-packages\\aiogram\\bot\\api.py\", line 104, in make_request\n return check_result(method, response.content_type, response.status, await response.text())\n File \"C:\\Users\\xxlda\\AppData\\Local\\Programs\\Python\\Python38-32\\lib\\site-packages\\aiogram\\bot\\api.py\", line 82, in check_result\n exceptions.ConflictError.detect(description)\n File \"C:\\Users\\xxlda\\AppData\\Local\\Programs\\Python\\Python38-32\\lib\\site-packages\\aiogram\\utils\\exceptions.py\", line 137, in detect\n raise err(cls.text or description)\naiogram.utils.exceptions.TerminatedByOtherGetUpdates: Terminated by other getupdates request; make sure that only one bot instance is running\n\nI launch the bot's telegram, there are no errors, I write an SMS to the bot and errors pop up, what's the problem?\nhere is the link not github, there is my bot completely-https:\/\/github.com\/bloodyt3ars\/python-telegram-bot.git","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":349,"Q_Id":64490487,"Users Score":0,"Answer":"make sure that only one bot instance is running , it means you should be only person you use that bot token on the time, make sure you don't run the bot on any server like heroku... or with another console after that you can run the code","Q_Score":0,"Tags":"python-3.x,telegram-bot","A_Id":64866758,"CreationDate":"2020-10-22T21:00:00.000","Title":"I launch the bot's telegram, there are no errors, I write an SMS to the bot and errors pop up, what's the problem?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I cannot seem to figure out how to resize images and gifs using discord.py\nIf anyone could tell me how exactly to resize images and gifs using discord.py it would be greatly appreciated as it powers core features of my bot.","AnswerCount":2,"Available Count":1,"Score":-0.1973753202,"is_accepted":false,"ViewCount":2698,"Q_Id":64492624,"Users Score":-2,"Answer":"Use a module like Pillow (import PIL)","Q_Score":1,"Tags":"python,imagemagick,python-imaging-library,discord.py,gif","A_Id":67894597,"CreationDate":"2020-10-23T01:23:00.000","Title":"Resizing image discord.py","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I am learning how to use python.\nFor the project I am working on, I have hundreds of datasheets containing a City, Species, and Time (speciesname.csv).\nI also have a single datasheet that has all cities in the world with their latitude and longitude point (cities.csv).\nMy goal is to have 2 more columns for latitude and longitude (from cities.csv) in every (speciesname.csv) datasheet, corresponding to the location of each species.\nI am guessing my workflow will look something like this:\nGo into speciesname.csv file and find the location on each line\nGo into cities.csv and search for the location from speciesname.csv\nCopy the corresponding latitude and longitude into new columns in speciesname.csv.\nI have been unsuccessful in my search for a blog post or someone else with a similar question. I don't know where to start so anyone with a starting point would be very helpful.\nThank you.","AnswerCount":2,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":32,"Q_Id":64507660,"Users Score":0,"Answer":"You can achieve it in many ways.\nThe simplest way I can think of to approach this problem is:\n\ncollect all cities.csv data inside a dictionary {\"cityname\":(lat,lon), ...}\nread line by line your speciesname.csv and for each line search by key (key == speciesname_cityname) in the dictionary.\nwhen you find a correspondence add all data from the line and the lat & lon separated by comma to a buffer string that has to end with a \"\\n\" char\nwhen the foreach line is ended your buffer string will contains all the data and can be used as input to the write to file function","Q_Score":0,"Tags":"python,csv,parsing","A_Id":64507904,"CreationDate":"2020-10-23T21:44:00.000","Title":"How do I pull specific data from one file and add it to another file in a specific spot?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"Is there a way to detect if someone has charles proxy or a network tracking app, tracking your program's requests\/apis. And if so is there a way to prevent such a thing.\nJust wanted a bit of insight into this, any information is much appreciated.","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":211,"Q_Id":64510362,"Users Score":0,"Answer":"It's possible to achieve it: Here are two solutions I could think of\n\nAfter the SSL finish, you can get the certificate and check whether or not it's a self-signed certificate. Then, you might reject the Request connection.\n\nUse the SSL-Pinning technique that only accepts the certificate from your own and rejects others. It will prevent the app is intercepted from Charles Proxy or other Proxy tools like Proxyman, Fiddler, and Wireshark.","Q_Score":0,"Tags":"python-requests,charles-proxy","A_Id":64569777,"CreationDate":"2020-10-24T05:42:00.000","Title":"Is there a way to detect if someone has charles proxy or a network tracking app, tracking your program's requests\/apis","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I have a file of around 16mb in size and am using python boto3 upload_file api to upload this file into the S3 bucket. However, I believe the API is internally choosing multipart upload and gives me an \"Anonymous users cannot initiate multipart upload\" error.\nIn some of the runs of the application, the file generated may be smaller (few KBs) in size.\nWhat's the best way to handle this scenario in general or fix the error I mentioned above?\nI currently have a Django application that generates a file when run and uploads this file directly into an S3 bucket.","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":266,"Q_Id":64528274,"Users Score":2,"Answer":"Ok, so unless you've opened your S3 bucket up for the world to upload to (which is very much NOT recommended), it sounds like you need to setup the permissions for access to your S3 bucket correctly.\nHow to do that will vary a little depending on how you're running this application - so let's cover off a few options - in all cases you will need to do two things:\n\nAssociate your script with an IAM Principal (an IAM User or an IAM Role depending on where \/ how this script is being run).\nAdd permissions for that principal to access the bucket (this can be accomplished either through an IAM Policy, or via the S3 Bucket Policy)\n\nLambda Function - You'll need to create an IAM Role for your application and associate it with your Lambda function. Boto3 should be able to assume this role transparently for you once configured.\nEC2 Instance or ECS Task - You'll need to create an IAM Role for your application and associate it with your EC2 instance\/ECS Task. Boto3 will be able to access the credentials for the role via instance metadata and should automatically assume the role.\nLocal Workstation Script - If you're running this script from your local workstation, then boto3 should be able to find and use the credentials you've setup for the AWS CLI. If those aren't the credentials you want to use you'll need to generate an access key and secret access key (be careful how you secure these if you go this route, and definitely follow least privilege).\nNow, once you've got your principal you can either attach an IAM policy that grants Allow permissions to upload to the bucket to the IAM User or Role, or you can add a clause to the Bucket Policy that grants that IAM User or Role access. You only need to do one of these.\nMulti-part uploads are performed via the same S3:PutObject call as single part uploads (though if your files are small I'd be surprised it was using multi-part for them). If you're using KMS one small trick to be aware of is that you need permission to use the KMS key for both Encrypt and Decrypt permissions if encrypting a multi-part upload.","Q_Score":0,"Tags":"python-3.x,amazon-web-services,amazon-s3,boto3","A_Id":64529021,"CreationDate":"2020-10-25T20:10:00.000","Title":"Uploading a file through boto3 upload_file api to AWS S3 bucket gives \"Anonymous users cannot initiate multipart uploads. Please authenticate.\" error","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I want to login to site: 'https:\/\/portal.librus.pl\/rodzina\/synergia\/loguj' with requests in python, unfortunately if you inspect and go to network settings it shows main method without 'form data', in addition it has csrftoken placed in script like this:\nscript type=\"text\/javascript\">\nvar csrfTokenName = \"requestkey\";\nvar csrfTokenValue = \"MC44MDU4NjQwMCAxdjA0MDk5NtjdKdm32NdksMkfmuMTU0NzZjMTNi1WE1ZmJdfmUdfOpNIlhMDQx\";\n<\/script\nSo i don't really know how to get the content out of this regard you got hundreds of script tags in html file\nSo the question is how to login to this site?\n(i've watched all tutorials on internet, and non of them work)","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":50,"Q_Id":64616982,"Users Score":0,"Answer":"I've found the request inside network tab and this is the things that I saw (I might have missed some information cause I got error for not having account).\n\n\nIn network tab if you try to login with non empty values you will have a POST request with name 'action'.\nInside this request you will find the URL you want \"https:\/\/portal.librus.pl\/rodzina\/login\/action\"\nAnd last part is the form data in the bottom","Q_Score":0,"Tags":"authentication,web-scraping,python-requests,csrf-token","A_Id":64630295,"CreationDate":"2020-10-30T23:30:00.000","Title":"Loging with python without form data","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I'm trying to use pytractor.\nWhen I write import statement from pytractor.webdriver import Firefox\nFirefox is not referenced-not found. Neither Chrome.","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":48,"Q_Id":64646287,"Users Score":0,"Answer":"I think this is because, since Python 3.5, async is a protected keyword but is used in the mixins.py script. I renamed the variable and it fixed the problem","Q_Score":0,"Tags":"python-3.x,selenium-webdriver","A_Id":69111208,"CreationDate":"2020-11-02T13:14:00.000","Title":"How to use pytractor with selenium?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I am running a scraper that crawls about 25 domains with in total about 100k pages. The speed in the beginning is about 500 pages per minute, but it slows down considerably afterwards to about one hundred pages per minute. My reasoning was that this could be due to the throttling that I enabled in the settings.py file. Yet, after running the spider for five hours, I noticed that the speed went even down to about 5 pages per minute.\nMy initial understanding was that the spider would probably be stuck at one domain that was slow in responding. However, this understanding didn't make sense anymore when I spotted some huge fluctuations in the scraping speed. After being stuck for hours in the range 0-20 pages per minute, occasionally there is a speed up to 100-500 pages per minute(mosttimes for only one or two minutes). How could the latter speedups be explained considering that a domain would be slowing down in its responses to me?","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":34,"Q_Id":64659107,"Users Score":0,"Answer":"Your initial understanding is probably correct - sound like you are getting temporarily throttled which results in fluctuating speeds.","Q_Score":0,"Tags":"python,web-scraping,scrapy,throttling","A_Id":64659274,"CreationDate":"2020-11-03T08:21:00.000","Title":"How do unexpected speed fluctuations in Scrapy relate to throttling?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"Given the start node and goal node in a graph, I want to find one simple path between these two nodes. I do not want the shortest path, but need any random simple path.\nI tried using all_simple_paths from networkx, but this module seems to calculate all the simple paths before returning anything. This takes a long time to run.\nIs there a way to find just one simple path?\nAlso, I would ideally like to make sure this path does not cross any \"obstacles\". These obstacles are a predefined set of nodes from the same graph. Is there a way to add in this constraint?\nPS: I don't necessarily need to use networkx. The code I am writing is in Python.","AnswerCount":2,"Available Count":2,"Score":0.0,"is_accepted":false,"ViewCount":252,"Q_Id":64669751,"Users Score":0,"Answer":"You could treat this as a min cost network flow problem where your start node wants to send a unit of flow (demand = -1) to your goal node (demand = 1). You can set the edge capacities to 1 and you can set all the edge weights to 0 except for those around \"obstacle\" nodes. For those obstacle nodes you can set all the edges either coming into or going out of them to have a weight of 1. The algorithm will try to find any arbitrary path using only edges with weight 0, but will use weight 1 edges if no path exists with only weight 0 edges.\nSee the nx.min_cost_flow function. This function requires you to make your graph a directed graph nx.DiGraph if it's not already.","Q_Score":1,"Tags":"python-3.x,networkx","A_Id":64683145,"CreationDate":"2020-11-03T19:57:00.000","Title":"Getting a random simple path between two nodes in a graph","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"Given the start node and goal node in a graph, I want to find one simple path between these two nodes. I do not want the shortest path, but need any random simple path.\nI tried using all_simple_paths from networkx, but this module seems to calculate all the simple paths before returning anything. This takes a long time to run.\nIs there a way to find just one simple path?\nAlso, I would ideally like to make sure this path does not cross any \"obstacles\". These obstacles are a predefined set of nodes from the same graph. Is there a way to add in this constraint?\nPS: I don't necessarily need to use networkx. The code I am writing is in Python.","AnswerCount":2,"Available Count":2,"Score":0.0,"is_accepted":false,"ViewCount":252,"Q_Id":64669751,"Users Score":0,"Answer":"I managed to solve this problem by using the RRT algorithm. It gives a random path between the source and destination nodes and also avoids obstacles.","Q_Score":1,"Tags":"python-3.x,networkx","A_Id":65011907,"CreationDate":"2020-11-03T19:57:00.000","Title":"Getting a random simple path between two nodes in a graph","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I am trying to make a business out of my discord.py skills. I have made a discord.py bot and want to export it from pycharm to be able to sell it on a site like fiverr. Does someone know how I could do that? (I am an absolute noob at this, pls help me anyone...)","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":55,"Q_Id":64685987,"Users Score":0,"Answer":"So you're exporting the source code? If so, you could copy & paste, but it isn't the most professional way of doing things, you could send them a link to a private github repo, or else send them the folder with the src files etc. There's plenty of ways of doing it. But find out what works for you. :)","Q_Score":0,"Tags":"python,discord.py","A_Id":64770431,"CreationDate":"2020-11-04T18:47:00.000","Title":"Discord.py - Exporting code from pycharm for selling","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I have used Pycharm to connect my remote server and it worked well. But after I added a second connection to another server. It reports errors: Couldn't upload helpers for remote interpreter: Authentication failed: Exhausted available authentication methods, and the previous connection failed as well.\nHere are some measures that I took but not valid.\n\nDelete the new connection.\nDelete all connection and remote interpreter, then reconnect to the remote server\nRestart my laptop.\n\nThe last thing I did and worked for me is\ngo to Deployment -> connection -> Authentication, choose Key pair openssh or PuTTY, select the private key path. Test Connection and it worked.","AnswerCount":2,"Available Count":2,"Score":0.0996679946,"is_accepted":false,"ViewCount":2030,"Q_Id":64691078,"Users Score":1,"Answer":"go to Deployment -> connection -> Authentication, choose Key pair openssh or PuTTY, select the private key path. Test Connection and it worked.","Q_Score":0,"Tags":"python,pycharm,remote-server","A_Id":64691081,"CreationDate":"2020-11-05T04:08:00.000","Title":"Pycharm: Couldn't upload helpers for remote interpreter: Authentication failed: Exhausted available authentication methods","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I have used Pycharm to connect my remote server and it worked well. But after I added a second connection to another server. It reports errors: Couldn't upload helpers for remote interpreter: Authentication failed: Exhausted available authentication methods, and the previous connection failed as well.\nHere are some measures that I took but not valid.\n\nDelete the new connection.\nDelete all connection and remote interpreter, then reconnect to the remote server\nRestart my laptop.\n\nThe last thing I did and worked for me is\ngo to Deployment -> connection -> Authentication, choose Key pair openssh or PuTTY, select the private key path. Test Connection and it worked.","AnswerCount":2,"Available Count":2,"Score":0.0,"is_accepted":false,"ViewCount":2030,"Q_Id":64691078,"Users Score":0,"Answer":"If you face problem in Pycharm, there are high chances you are not able to SSH using Putty \/ Terminal (Mac). Please try ssh'ing into host via Putty \/ Terminal first. If you fix the issue with your Putty \/ Terminal (Mac), you'll automatically fix Pycharm's Authentication Failed Error too. That's how, I fixed mine.","Q_Score":0,"Tags":"python,pycharm,remote-server","A_Id":67093275,"CreationDate":"2020-11-05T04:08:00.000","Title":"Pycharm: Couldn't upload helpers for remote interpreter: Authentication failed: Exhausted available authentication methods","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I'm really new to web scraping. Is there anyone that could tell me how to search on google.com with Selenium in Python?","AnswerCount":3,"Available Count":1,"Score":0.0665680765,"is_accepted":false,"ViewCount":97,"Q_Id":64715022,"Users Score":1,"Answer":"Selenium probably isn't the best. other libraries\/tools would work better. BeautifulSoup is the first one that comes to mind","Q_Score":1,"Tags":"python,selenium","A_Id":64715127,"CreationDate":"2020-11-06T12:51:00.000","Title":"How to search on Google with Selenium in Python?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I need some advise in one of my usecase regarding Cloudtrail and Python boto3.\nI have some cloudtrail events like configured and i need to send the report of all those events manually by downloading the file of events.\nI am planning to automate this stuff using python boto3. Can you please advise how can i use boto3 to get the cloudtrail events for some specific date i should paas at runtime along with the csv or json files downloaded and sent over the email. As of now i have created a python script which shows the cloudtrail event but not able to download the files. Please advise","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":95,"Q_Id":64719698,"Users Score":0,"Answer":"My suggestions is to simply configure the deliver of those events to an S3 bucket, and you have there the file of events. This configuration is part of your trail configuration and doesn't need boto3.\nYou can then access events files stored on S3 using boto3 (personally the best way to interact with AWS resources), and manipulate those files as you prefer.","Q_Score":0,"Tags":"python,amazon-web-services","A_Id":64722873,"CreationDate":"2020-11-06T18:15:00.000","Title":"Download cloudtrail event","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I am going to use Kafka as a message broker in my application. This application is written entirely using Python. For a part of this application (Login and Authentication), I need to implement a request-reply messaging system. In other words, the producer needs to get the response of the produced message from the consumer, synchronously.\nIs it feasible using Kafka and its Python libraries (kafka-python, ...) ?","AnswerCount":1,"Available Count":1,"Score":0.1973753202,"is_accepted":false,"ViewCount":1322,"Q_Id":64748185,"Users Score":1,"Answer":"I'm facing the same issue (request-reply for an HTTP hit in my case)\nMy first bet was (100% python):\n\nstart a consumer thread,\npublish the request message (including a request_id)\njoin the consumer thread\nget the answer from the consumer thread\nThe consumer thread subscribe to the reply topic (seeked to end) and deals with received messages until finding the request_id (modulus timeout)\n\nIf it works for a basic testing, unfortunatly, creating a KafkaConsumer object is a slow process (~300ms) so it's not an option for a system with massive traffic.\nIn addition, if your system deals with parallel request-reply (for example, multi-threaded like a web server is) you'll need to create a KafkaConsumer dedicated to request_id (basically by using request_id as consumer_group) to avoid to have reply to request published by thread-A consumed (and ignored) by thread-B.\nSo you can't here reclycle your KafkaConsumer and have to pay the creation time for each request (in addition to processing time on backend).\nIf your request-reply processing is not parallelizable you can try to keep the KafkaConsuser object available for threads started to get answer\nThe only solution I can see at this point is to use a DB (relational\/noSQL):\n\nrequestor store request_id in DB (as local as possible) aznd publish request in kafka\nrequestor poll DB until finding answer to request_id\nIn parallel, a consumer process receiving messages from reply topic and storing result in DB\n\nBut I don't like polling..... It wil generate heavy load on DB in a massive traffic system\nMy 2CTS","Q_Score":7,"Tags":"python,apache-kafka,synchronous,kafka-python","A_Id":66068833,"CreationDate":"2020-11-09T08:35:00.000","Title":"How to implement request-reply (synchronous) messaging paradigm in Kafka?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":1,"Web Development":0},{"Question":"I have a python script that runs for a few hours. There is some probability of it failing or hanging. Therefore, I would like to find a way to monitor this script's progress while away from the computer it runs on. I have been looking into multiple options, but none of them are robust and straightforward enough.\nIf someone has some ideas, I'd love to hear them.\nHere are some options I considered:\n\nThe computer running python is connected to my dropbox account. Therefore, I could write an HTML file to my dropbox. This would work on other computers. I could open the HTML file on another computer and refresh the page to get the latest version. However, this would not work on my phone. It would also not push any updates (not necessary, but that would be a great feature).\n\nI looked into building an online dashboard that would visualize data. The idea would be to update the data as the script runs, thereby getting the script's latest state. I looked into plotly and anvil. However, I could not find a way of programmatically updating the data online.","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":169,"Q_Id":64761733,"Users Score":0,"Answer":"As a very simple quick fix you could make a log file and have it update it at intervals. Pretty straightforward.","Q_Score":0,"Tags":"python","A_Id":64762514,"CreationDate":"2020-11-10T01:53:00.000","Title":"Monitoring the progress of a python script online, or on my phone, or another computer","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I've been searching for a long time and still couldn't find an answer...\nI can find the element by literally searching through every single divs in the website\nbut that's obviously very stupid, is there a way to trace back xpath so next time\nI can use the xpath it got earlier when I rerun the program?","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":39,"Q_Id":64763149,"Users Score":0,"Answer":"To locate the full Xpath of an element, you can simply use the Google Chrome console and inspect the element that you want, do\nRight Click -> Copy -> Copy Xpath\/Copy Full Xpath\nor in code, walk up the tree from your element to root and compile a full Xpath.\nIf you are talking a partial Xpath locator such as \/\/input[@id='hello'], there is no implemented feature for that yet. However, you can write your own function that takes in an element and analyzes it to get its attribute, such as its element type and id, and return a complied Xpath.","Q_Score":0,"Tags":"python","A_Id":64763354,"CreationDate":"2020-11-10T05:01:00.000","Title":"After locating an element with selenium, can you trace back\/report back the xpath of the element?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"i'm solliciting you today because i've a problem with selenium.\nmy goal is to make a full automated bot that create an account with parsed details (mail, pass, birth date...) So far, i've managed to almost create the bot (i just need to access to gmail and get the confirmation code).\nMy problem is here, because i've tried a lot of things, i have a Failed to load resource: the server responded with a status of 429 ()\nSo, i guess, instagram is blocking me.\nhow could i bypass this ?","AnswerCount":2,"Available Count":2,"Score":0.0,"is_accepted":false,"ViewCount":8428,"Q_Id":64769138,"Users Score":0,"Answer":"The answer is in the description of the HTTP error code. You are being blocked because you made too many requests in a short time.\nReduce the rate at which your bot makes requests and see if that helps. As far as I know there's no way to \"bypass\" this check by the server.\nCheck if the response header has a Retry-After value to tell you when you can try again.","Q_Score":1,"Tags":"python,selenium,instagram,http-status-code-429","A_Id":64769229,"CreationDate":"2020-11-10T12:41:00.000","Title":"How can I bypass the 429-error from www.instagram.com?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"i'm solliciting you today because i've a problem with selenium.\nmy goal is to make a full automated bot that create an account with parsed details (mail, pass, birth date...) So far, i've managed to almost create the bot (i just need to access to gmail and get the confirmation code).\nMy problem is here, because i've tried a lot of things, i have a Failed to load resource: the server responded with a status of 429 ()\nSo, i guess, instagram is blocking me.\nhow could i bypass this ?","AnswerCount":2,"Available Count":2,"Score":0.0,"is_accepted":false,"ViewCount":8428,"Q_Id":64769138,"Users Score":0,"Answer":"Status code of 429 means that you've bombarded Instagram's server too many times ,and that is why Instagram has blocked your ip.\nThis is done mainly to prevent from DDOS attacks.\nBest thing would be to try after some time ( there might be a Retry-After header in the response).\nAlso, increase the time interval between each request and set the specific count of number of requests made within a specified time (let's say 1 hr).","Q_Score":1,"Tags":"python,selenium,instagram,http-status-code-429","A_Id":64769319,"CreationDate":"2020-11-10T12:41:00.000","Title":"How can I bypass the 429-error from www.instagram.com?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I'm using selenium, and I want to get that 20. Is there any way to do it?\n<div _ngcontent-c12=\"\" class=\"percentage\"><p _ngcontent-c12=\"\">20<\/p><\/div>","AnswerCount":3,"Available Count":1,"Score":0.0665680765,"is_accepted":false,"ViewCount":76,"Q_Id":64777902,"Users Score":1,"Answer":"The getText() applies to any element found with Selenium and retrieves its innerText. You could locate the element via whatever way you want, and then apply the method getText().\nExample: driver.find_element_by_xpath(\"Insert\/Element\/XPath\/Here\").getText()","Q_Score":0,"Tags":"python,python-3.x,python-2.7,selenium,selenium-chromedriver","A_Id":64777968,"CreationDate":"2020-11-10T22:46:00.000","Title":"Get this element attribute via selenium","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I'm currently learning how to use the Tweepy API, and is there a way to filter quoted Tweets and blocked users? I'm trying to stop search from including quoted Tweets and Tweets from blocked users. I have filtered Retweets and replies already.\nHere's what I have:\nfor tweet in api.search(q = 'python -filter:retweets AND -filter:replies', lang = 'en', count = 100):","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":85,"Q_Id":64780218,"Users Score":0,"Answer":"To filter quotes, use '-filter:quote'","Q_Score":0,"Tags":"python,search,filter,tweepy","A_Id":68703363,"CreationDate":"2020-11-11T03:48:00.000","Title":"Tweepy API Search Filter","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"Can you please help me i figuring this out.\nWhile reading a .docx file from python-docx (docx.Document(file_name)), how can I detect if the docx file is correct or corrupt.\nI've got some cases where these input docx files are either empty or corrupt.\nHow can I flag these cases using this library","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":398,"Q_Id":64784422,"Users Score":1,"Answer":"There is no such feature in python-docx. Part of the reason is that while a file could be determined to be valid or invalid according to the schema in the ISO specification, many small discrepancies are permitted by each client. What is permitted varies between clients; some things that LibreOffice will accept produce a repair error in Microsoft Word, for example.\nThe only reliable way to determine this is to attempt to open the file with the target client, perhaps using automation like VBA in the case of Microsoft Word.","Q_Score":0,"Tags":"python,python-docx","A_Id":64792695,"CreationDate":"2020-11-11T10:12:00.000","Title":"Detecting corrupt document files with python-docx","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":1,"System Administration and DevOps":0,"Web Development":0},{"Question":"I have a Python script that uses Selenium Webdriver. I want to run this on a server remotely.\nWhen running my script directly on the server through command window, everything works fine.\nBut when I'm trying to trigger this remotely through a program on my local PC to the server, it doesn't work. It seems to stop at driver = webdriver.Chrome(webdriverlink). It doesn't fail, it just won't continue. So it seems like it doesn't actually opens a browser. Isn't this possible when you're not logged in on the server and not running it directly from there?\n\nI've saved my script in a py-file, and is triggering it in both ways throuh the command line python <myFileName>.py","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":154,"Q_Id":64804603,"Users Score":1,"Answer":"I found a solution from another forum after a lot of searching. So for all of you struggling with the same problem, this worked for me:\nI've added the following code:\noptions = webdriver.ChromeOptions()\noptions.add_experimental_option('excludeSwitches', ['enable-logging'])\nand changed from: driver = webdriver.Chrome(webdriverlink) to driver = webdriver.Chrome(executable_path=webdriverlink,options=options)","Q_Score":0,"Tags":"python,selenium,selenium-webdriver,webdriver","A_Id":64857847,"CreationDate":"2020-11-12T13:14:00.000","Title":"Calling Python script with Selenium Webdriver from PC on a server","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"Hellow, i'm working with network programing and it's been so hard to create a logic that allows to stream a video from a single server to multiple clients with no delay.\nwhich means that i have to implement a parallel execution during the stream to all connected clients in order to display the images at the same time.\nand why is that important for my project it's because i'm intending to have large number of clients (from 200 to approximately 700), now with 10 clients that delay is nothing but with 700 clients could significantly increase the delay to several minutes (not sure but possible).\nfor those who don't know what's the cause of the dely, it's from the for loop that i'm using which contain the send function for each frame, and that is a serial execution.\ni tried threading and multiprocessing and even function schedule but every thing got messy, previously i was using socket & opencv, but for some reason it caused issues during the streaming, now i switched to Netgear & Vidgear but i'm still struggling.\nHope someone can help.\nPS: multicast is just not right for the job, after i tried it i was receiving errors because of the length of the transmitted images, UDP protocol will NOT accept more then 65535 byte.","AnswerCount":2,"Available Count":1,"Score":0.0996679946,"is_accepted":false,"ViewCount":280,"Q_Id":64805512,"Users Score":1,"Answer":"Per your comment, everything is in the same network, and we have multicast for exactly your problem. Rather than sending the same data over and over to multiple hosts, you can send a single stream of traffic to many receivers.\nYou set up the clients to subscribe to a multicast group, normally a group in the 239.0.0.0\/8 Organization-Local scope. Your server then sends its traffic to the same multicast group to which the clients have subscribed. The single traffic stream will be received and processed by every client subscribed to the multicast group.\nBecause multicast sends to multiple clients, you must use a connectionless transport protocol, e.g. UDP. Connection-oriented transport protocols, e.g. TCP, create connections between two hosts, so they cannot be used with multicast, which is one-to-many.\nBy default, multicast only works in the same network. We do have multicast routing to send traffic to other networks, but it is very different than the usual unicast routing. Also, you cannot multicast on the public Internet because the ISPs do not have multicast routing. You can multicast to a different site across the Internet by using a tunnel that supports multicast, e.g. GRE. Both the source and destination routers need to be configured for multicast routing, as well as any routers in the path of the multicast packets (the Internet routers on see the unicast tunnel packets, not the multicast packets, so you can send the multicast across the Internet).","Q_Score":1,"Tags":"python-3.x,sockets,opencv,networking,network-programming","A_Id":64806863,"CreationDate":"2020-11-12T14:10:00.000","Title":"Send data over network to multiple recipient parallely","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I would like to create bot in whatsapp. The Idea is here when a user sends a whatsapp message to my phone number that should reply to them.\nI have researched twillio since it's like a sandbox twillio uses it own whatsapp number and So when a new user comes the user should send the sandbox number. Then I searched for whatsapp API I don't have any company or anything so there I can't use the official whatsapp API.\nIs there some other way of using whatsapp API when I recieve a message it should reply immediately.","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":597,"Q_Id":64815647,"Users Score":2,"Answer":"Unfortunately, since WhatsApp is only opening its APIs to its trusted\/business partners, you can only access it using their services. Some of the examples includes,\n\nVonage\nInfobip\nWati\nTwilio as you mentioned\n\nIf these are not enough for you, you can always write your own bot in Python using selenium to scrape the web app for WhatsApp and listen to new message by checking the page regularly.","Q_Score":0,"Tags":"python,automation,whatsapp","A_Id":64816080,"CreationDate":"2020-11-13T04:47:00.000","Title":"How to create a whatsapp bot?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I am attempting to use python to pull down a zone file. After going through hoards of documentation, I am still stuck on one line of code:\ndns.zone.from_xfr(dns.query.xfr('3.211.54.86','megacorpone.com'))\nI get the following error:\nsocket.error: [Errno 111] Connection refused\nI've hardcoded ns2.megacorpone.com's IP to isolate any problems. For some reason this connection continues to refuse. Is anyone able to shed some light on this problem?\nThanks all","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":392,"Q_Id":64816473,"Users Score":0,"Answer":"Running the same command with the domain name instead of IP worked.","Q_Score":0,"Tags":"python,python-2.7,dns","A_Id":64837670,"CreationDate":"2020-11-13T06:33:00.000","Title":"Issues transferring zone file with dnspython","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"Im using python with the selenium package and the chromedriver to scrape a webpage. The page is automatically reloading itself (probably via java-script) every 30 min, which destroyes my scraping-progress.\nThe webscraping is done with a script and the javascript ist needed for further interaction with the page.\nIs it possible to stop the page from reloading itself, maybe with a flag for the chromedriver?","AnswerCount":2,"Available Count":2,"Score":0.0,"is_accepted":false,"ViewCount":1247,"Q_Id":64817185,"Users Score":0,"Answer":"You can just open the DevTools and pause the Debugger. Works in Firefox and Chrome (tested)!","Q_Score":0,"Tags":"python,selenium,selenium-chromedriver","A_Id":64817219,"CreationDate":"2020-11-13T07:45:00.000","Title":"Stop webpage from auto-reload with selenium\/chromedriver","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"Im using python with the selenium package and the chromedriver to scrape a webpage. The page is automatically reloading itself (probably via java-script) every 30 min, which destroyes my scraping-progress.\nThe webscraping is done with a script and the javascript ist needed for further interaction with the page.\nIs it possible to stop the page from reloading itself, maybe with a flag for the chromedriver?","AnswerCount":2,"Available Count":2,"Score":1.2,"is_accepted":true,"ViewCount":1247,"Q_Id":64817185,"Users Score":0,"Answer":"Found a Solution myself. I had to execute the following dev-tools command with the webdriver:\ndriver.execute_cdp_cmd('Emulation.setScriptExecutionDisabled', {'value': True})","Q_Score":0,"Tags":"python,selenium,selenium-chromedriver","A_Id":64919208,"CreationDate":"2020-11-13T07:45:00.000","Title":"Stop webpage from auto-reload with selenium\/chromedriver","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"Does anybody know what is the IP address of the sender when using Python smtplib to send emails? The sender's email address will be Gmail. Does the email originate from the IP of the computer in which the Python script is executed? Or is it an IP from the Gmail servers?\nI want to make an email marketing program such as Mailchimp for my personal use, but much smaller and with far fewer features than Mailchimp. One key aspect of marketing software is IP. The emails need to be sent from an IP with a good reputation. So, when you send emails with Mailchimp, they are sent from the Mailchimp (well reputable) IPs. I'd like to know what IP will Python send emails from.","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":271,"Q_Id":64823473,"Users Score":1,"Answer":"Does the email originate from the IP of the computer in which the Python script is executed? Or is it an IP from the Gmail servers?\n\nNeither. As shown in the example in the docs, it will be the IP address of the server you're connecting to, to send the email. So you'll probably want to connect to the SMTP server of your domain host.\nIf you run an SMTP server locally, it will be IP addr of that machine. And the connection is likely to be rejected by the recipient's smtp server.\nHowever note that in any case, the SMTP server may add the originating IP address to the email headers.\nTo not have your emails outright rejected by the server, take a look at Sender Policy Framework (SPF). Doesn't ensure your email won't get marked as Spam though.","Q_Score":1,"Tags":"python,ip,smtplib","A_Id":64824225,"CreationDate":"2020-11-13T15:22:00.000","Title":"From what IP address does SMTP lib send an email from?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I am trying to install Bazaar on Debian 10.6, but cloning the central repository (from the course server) to my local machine. I get the following error:\n\nbzr: ERROR: Unsupported protocol for url \"sftp: \/\/<User>@<Ip Server>\/home\/<user>\/public\/trunk\": Unable to import paramiko (required for sftp support): No module named paramiko.","AnswerCount":1,"Available Count":1,"Score":0.1973753202,"is_accepted":false,"ViewCount":24,"Q_Id":64828543,"Users Score":1,"Answer":"Paramiko is a python package that is used for SSH connections.\nI guess clone command is trying to make an SSH connection and because Paramiko is not installed you get an Error.\nTry to make pip install paramiko and then make the clone again (:","Q_Score":0,"Tags":"python","A_Id":64828600,"CreationDate":"2020-11-13T21:57:00.000","Title":"Problems installing Bazzar on Debian 10.6","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I have a flask application that uses flask-socketio and python-socketio to facilitate communication between a socketio server in the cloud and a display device via a hardware device.\nI have a display namespace which exposes the display facing events, and also uses a separate client class which connects and talks to the server in the cloud. This works well as designed, but now I want to trigger the connection method in my client class from a different namespace. So far I have not been able to get this to work.\nWhat I have tried is adding the display namespace class to the flask context, then passing that into the socketio.on_namespace() method. Then from the other namespace I am grabbing it from current_app and trying to trigger the connection to the cloud server. This returns a 'RuntimeError: working outside of application context' error.\nSo at this point I'm still researching how to do this correctly, but I was hoping someone has dealt with something like this before, and knows how to access methods on one namespace from a different one.","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":53,"Q_Id":64829415,"Users Score":0,"Answer":"I found a solution. Instead of instantiating my client class from the display namespace, I instantiate it before I add the namespaces to socketio. Then I pass the client object into both namespaces when I call the socketio.on_namespace() method.","Q_Score":0,"Tags":"python,flask-socketio,python-socketio","A_Id":64836959,"CreationDate":"2020-11-13T23:42:00.000","Title":"access methods on one socketio namespace from a different one","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I'm new to programming, this being my first post, but I was wondering if there was a way to make selenium run while my computer is turned off? I made an instagram bot with selenium webdriver python but it only runs when I run it from my computer and turns off when I turn off my computer. I have seen post saying to host it in PythonAnywhere which I tried but selenium requires a webdriver (im using firefox so gekodriver). Is there a way of fixing this or is there another way of making an insagram bot without selenium webdriver?\nThank you","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":230,"Q_Id":64840650,"Users Score":0,"Answer":"You can use virtual machine and host a server and then through DevOps pipelines (Azure Pipelines\/ Jenkins) can make your scripts run without manual intervention.","Q_Score":0,"Tags":"python,selenium,selenium-webdriver,bots,instagram","A_Id":64840829,"CreationDate":"2020-11-15T02:24:00.000","Title":"How can I have a python selenium webdriver script run 24\/7?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I want to make a script in python that interacts with a webpage that has quite a lot of javascript in it (it's a webpage that computes a bunch of physics stuff).\nI don't want my code to break if the page formatting changes and I want it to run offline so I would prefer my script to run on a local html copy of the page I got (all the JS code is accessible in the HTML source, there is no call to an external server). I wanted to use the requests library to do it, but it only works with URLs. Is there any library to do this? Note that I want to interact with the HTML (input values and look at the outputs etc..), I know that I can parse the file but that's not what I'm asking. I'm also totally new to web bots or anything related.\nRight now I can open my .html version of the page offline with chrome and interact with it, so there has to be a way to automate this somehow. I'm also not against using something else than python if there is a better library for this in another language.","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":320,"Q_Id":64841002,"Users Score":0,"Answer":"interesting question, best way I can think to do that is use a web framework and then just scrape the data using requests. I am familiar with flask and its simple to use but im sure there are other options as well","Q_Score":3,"Tags":"python,html,web,web-scraping,python-requests","A_Id":64841338,"CreationDate":"2020-11-15T03:43:00.000","Title":"Interact with local HTML file with python","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"Hello I am trying to click on a button within python selenium that has no ID. The button has the text Submit an application within it. However there are multiple phrases of \"Submit an application\" on the website. The attributes of the website are:\n<a class= \"button\" href = \"(a link)\" analytics-on=\"click\" analytics-category=\"Body\" analytics-event=\"(a link)\" analytics-label=\"Submit an application||(a link)\">","AnswerCount":2,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":94,"Q_Id":64851471,"Users Score":0,"Answer":"you can try with css_selector.\ndriver.find_element_by_css_selector('[analytics-on=\"click\"]').click()\ndriver.find_element_by_css_selector('[analytics-category=\"Body\"]').click()\nor link_text method is also a good choice in locating an a tag,\ndriver.find_element_by_link_text(\"Submit an application\").click()","Q_Score":0,"Tags":"python,python-3.x,selenium,selenium-webdriver,selenium-chromedriver","A_Id":64852151,"CreationDate":"2020-11-16T01:05:00.000","Title":"clicking on button no ID in Selenium python","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I'm looking to build a test with 4000 browser sessions using a Locust\/Selenium approach. I tried using the same approach using JMeter\/Selenium using headless browsers and for many reasons I've come to the conclusion that it's not possible. I'm familiar with running Browser-based load testing tools and with TrueClient I can reach around 45 Browsers\/users per\/Slave\/agent (loaded servers, 64RAM and 16CPU's). Silk Performer BDLT suggests around 15 or 20 sessions. I've looked at 'realbrowserlocusts' and it's really peaked my interest because I used locust as part of some Covid-19 application testing I ran back in February. Are there any metrics of how many browsers a slave can handle using selenium and any metrics of how many slaves Locust can handle?\nShould I be looking at PhantomJS instead? My org is looking to do away with all commercial tools and going to opensource and need to add a browser-based load testing solution. JMeter\/Gatling\/Locust work great for the protocol level applications\nAny suggestions would be great.","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":462,"Q_Id":64852055,"Users Score":0,"Answer":"The only real answer is \"it depends\".\nI've been able to run some 30 clients & a total of maybe 2-3 clicks\/s on a moderately advanced site (plenty of javascript, but nothing crazy) on 2018 macbook pro (6-core i7, 16GB RAM). The problem is that you can't really know before hand how much resources you will need, and you may start to get intermittent errors as you run out of resources.\nWhen running Selenium, the browser will be the heavy part so expect similar performance with Locust+Selenium as you would get with Jmeter+Selenium.\nPhantomJS is in most cases slower than headless Chrome so I wouldnt bother with that.","Q_Score":0,"Tags":"python,selenium-chromedriver,phantomjs,locust","A_Id":64854425,"CreationDate":"2020-11-16T02:45:00.000","Title":"Guidelines with Locust\/Selenium and the number of browser sessions per\/slave","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"So, I have a problem and 2 hours of research haven't helped.\nI want to write a method that will return me a Boolean if a web-domain(e.g. \"google.com\", without http\/s) has an ssl-certificate.\nI have a big csv of domain names, which I need to process and check if the domains have a ssl-certificate, is there a method maybe in pythons ssl module?\nThank you for your help, TomiiPomii.","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":576,"Q_Id":64856092,"Users Score":1,"Answer":"By using module ssl you can call ssl.get_server_certificate((host, port)) to retrieve SSL of specified host in PEM format. You will then have to parse the PEM file to actually retrieve the certificate values, but if you only care about the website having or not having it, simply check the return value.","Q_Score":1,"Tags":"python,ssl,web,dns,certificate","A_Id":64856235,"CreationDate":"2020-11-16T10:11:00.000","Title":"How to check if a domain has a ssl certificate in python?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I tried everything but i don't find a working code for my discord.py bot to ask for a picture, wait for the user to send it, and then reply. I hope you can help me out.","AnswerCount":3,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":493,"Q_Id":64873399,"Users Score":0,"Answer":"I do not know about discord.py but are there not events for pictures sent by users? If so what about replying when the event is triggered?","Q_Score":0,"Tags":"python,discord,discord.py","A_Id":64873551,"CreationDate":"2020-11-17T10:25:00.000","Title":"How do i make a bot wait for a picture?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I have an embedded system that monitors many sensors (GPS, Motor Speed, Temperature..)\nI set up a client-server using sockets in python and I can send from the client(the embedded system) to the server(a Laptop) a string message like \" Hello from client \".\nBut for me it would be great to send data with predifined categories so the server knows the type of the message. Upon reception the server knows it has recieved for example a GPS coordinate or motor speed.\nIs there a way to do it in TCP or using any library or framework?\nThank you","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":29,"Q_Id":64894636,"Users Score":0,"Answer":"I can think of three ways\n\nPut the message kind in the header (next to the message size, probably)\nUse different sockets for each message kind (ugly and complicated but works)\nUse some sort of pub-sub e.g. Kafka, ZeroMQ or RabbitMQ to convey topics","Q_Score":0,"Tags":"python,sockets,tcp,embedded","A_Id":64913137,"CreationDate":"2020-11-18T14:04:00.000","Title":"send data with categories from client to a server using python","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"How can I automate clicking on non-browser interactive interfaces (eg. Exe software) without image recognization? (For example, Microsoft's onenote, or some niche software)\nI want to learn related technologies, but I don't know which direction I should go, I hope someone can give me some guidance.\n(Now I can only use Python's image recognition and click technology for these non-browser software, but this efficiency is very low. I hope that I can skip the image recognition & click directly, and can directly manipulate the underlying data processing like manipulating HTTP request or CSS selenium )","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":52,"Q_Id":64908810,"Users Score":0,"Answer":"Free tools are - Autoit, AutoHotkey, Winium etc\nFree tools cannot recognise most of the windows so might need to consider paid tools.\nPaid options are - Testcomplete, UFT(QTP), Automation anywhere, UI path, Ranorex, Squish\nMy personal recommendation is Test complete. Because I never found a blocker in windows automation, user friendly and economically viable.","Q_Score":0,"Tags":"python,selenium,automation,exe","A_Id":64915073,"CreationDate":"2020-11-19T09:28:00.000","Title":"How to automate Exe software like automate Chrome in Selenium?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I was wandering if there was a way I could easily use Windows Task Scheduler in order to run multiple scripts at the same time. I am wanting to host multiple discord bots on a spare PC that would each have their own bot key for different discord servers; my current understanding is that you cannot easily run multiple node.js bots like this (currently I have 1 in the scheduler and the other I have to run manually) but I was wandering if this is something that can be done in Python or if I can make it happen with node.","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":163,"Q_Id":64914853,"Users Score":0,"Answer":"I am not really into Python, don't fall for the name hehe\nAnyway, logically speaking this is a threading problem, did you try enabling multiple threading in the program you are using? or dynamically allow the program\/code to create multiple threads to handle multiple tasks related to the Node.js file you are using!?\nBest Regards","Q_Score":0,"Tags":"python,node.js,discord,autostart","A_Id":64914968,"CreationDate":"2020-11-19T15:32:00.000","Title":"Running multiple Node.js or Python scripts","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"The python API code,\nclient.get_buy_price(currency_pair = 'XRP-EUR')\ngives me the XRP amount as only \"0.25\".\nHow can I see more decimal, like 0.25126453?","AnswerCount":2,"Available Count":1,"Score":0.0996679946,"is_accepted":false,"ViewCount":606,"Q_Id":64917923,"Users Score":1,"Answer":"For some reason in the python API, get_buy_price(...) and get_sell_price(...) only give 2 decimals, but if you use get_spot_price(...) you will get more decimals","Q_Score":0,"Tags":"python,decimal,coinbase-api","A_Id":66049444,"CreationDate":"2020-11-19T18:34:00.000","Title":"How can I get more decimal from Coinbase API?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I am trying to code a script in Python that will take a time input from the user and then open a web page every day at that time. For example: I run the program and say, \"open [page] at [time]\" ideally it would store this in a document in the directory so the program could be closed when not needed and maintain times. The program would then wait until the specified time and then open a web page that was specified earlier at that time.\nI know this is probably super easy but I fully acknowledge I'm a noob. Thanks for not rubbing it in.\nAny help would be great! Thanks!\n-Meteor","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":49,"Q_Id":64924016,"Users Score":0,"Answer":"Just write a script to open a webpage using python. Then you can schedule it to run at a specific time using crontab.","Q_Score":0,"Tags":"python,time,automation","A_Id":64924048,"CreationDate":"2020-11-20T04:57:00.000","Title":"Python Script that opens a web window at a specific time","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I am creating calendar events with Python in a shared Google calendar. Up to a few days ago, everything worked fine, but suddendly I am not seeing the events created through Google Calendar API on mobile devices anymore.\nThe events are still showing up in the web interface and I can access the events through API.\nI assumed that there are syncinc issues between the app and, but whenever I create an event manually in the calendar (web interface), it automatically pops up on the mobile app. The issue occurs with two Android phones.","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":246,"Q_Id":64926707,"Users Score":4,"Answer":"If you problem is for allday events, then I just encountered the same problem: via the API I generate allday events, with start date = end date. It was working last fall, and as in your case now they appear on the Web interface, and not on the mobile app.\nI dragged one such event to another date on the Web interface, and then it appeared on the mobile. Then, I examined the event under debugger from my code(php in my case), and I observed that dragging the event to another date resulted in having the end date being start date + 1 day. So I changed my code to generate all day events with end date being start date + 1 day, which fixed the problem: now they appear on both the Web interface and the mobile app.\nI guess something changed on the mobile app during a recent update.","Q_Score":1,"Tags":"python,android,google-calendar-api","A_Id":65671801,"CreationDate":"2020-11-20T09:13:00.000","Title":"Event Created Through API not Visible on Google Calendar App","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I have seen a discord bot existing that could show specific videos from its camera.\nIs there any way I could add a command that would accept a direct video link and display it on a discord bot's camera? At least to show a video. (During streams)","AnswerCount":2,"Available Count":2,"Score":0.0,"is_accepted":false,"ViewCount":6909,"Q_Id":64939089,"Users Score":0,"Answer":"To my knowledge there is no way (yet) for a bot to stream a video in video format on Discord. Bots cannot stream anything other than audio on a voice channel.","Q_Score":1,"Tags":"python,video,discord,bots,discord.py","A_Id":64944981,"CreationDate":"2020-11-21T01:54:00.000","Title":"Is it possible to show a video on discord bot camera?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I have seen a discord bot existing that could show specific videos from its camera.\nIs there any way I could add a command that would accept a direct video link and display it on a discord bot's camera? At least to show a video. (During streams)","AnswerCount":2,"Available Count":2,"Score":1.2,"is_accepted":true,"ViewCount":6909,"Q_Id":64939089,"Users Score":3,"Answer":"I think that it may be possible but not yet available to the general public. I think maybe the bot you are referring to could have been taken down for breaking TOS. You might be able to do it but I do not think its allowed.","Q_Score":1,"Tags":"python,video,discord,bots,discord.py","A_Id":64948022,"CreationDate":"2020-11-21T01:54:00.000","Title":"Is it possible to show a video on discord bot camera?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I am having a use case to get data from a specific site which needs to have requests via session every time. I have created the session in python and also cookies are set which contain my logged in details.\nI am currently hosting my script on a data center but the account is getting blocked. I am thinking of requesting the data via proxy but still feel that if my session is created from a different machine and proxy is used to get data via session then what are the chances that the proxy ip is going to be black-listed?\nWhat are the possible solutions here to cater this kind of problem.","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":112,"Q_Id":64981885,"Users Score":0,"Answer":"Scenarios in Python Requests Session differs according to different regions,\nLike in some countries some headers are not permitted in the request due to country laws.\n\nSince you are already making logged in session that means your user-agent and other headers are being set according to the response from login request.\n\nOne of the solution might be using a proxy of a country which doesnot have strict rules of data extraction from that platfrom.","Q_Score":2,"Tags":"python,session,proxy,web-crawler","A_Id":64985958,"CreationDate":"2020-11-24T07:23:00.000","Title":"python requests via session proxy","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I have dropbox filepicker in my project, once user select file i am receiving URL of the dropbox, now i want a content of that file using that URL in python.\nhere is the link which i received from dropbox picker : https:\/\/www.dropbox.com\/s\/ocissavtfvvdh2g\/images.png?dl=0\nI have checked this link https:\/\/sodocumentation.net\/dropbox-api\/topic\/408\/downloading-a-file, but it will ask for the path, but i don't have path of the file, i just have URL of the file","AnswerCount":3,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":178,"Q_Id":64983177,"Users Score":1,"Answer":"I just have to use 1 in dl querystring.\nhttps:\/\/www.dropbox.com\/s\/ocissavtfvvdh2g\/images.png?dl=1 like this","Q_Score":1,"Tags":"python,dropbox","A_Id":65005272,"CreationDate":"2020-11-24T08:59:00.000","Title":"get dropbox file content using dropbox URL","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I want to validate bearer tokens in an API, which are passed from a browser application. The API will validate the tokens against Keycloak by calling the ..\/userinfo endpoint and respond with the desired content if the check runs ok. Problem is that If I pass a bearer token from the browser, or any other app for that matter (e.g. postman), to the api and try to validate it - I get \"401 Token verification failed\".\nThe following works:\n\nFetch token from Keycloak through Postman by using username and password.\nFetch userinfo from Keycloak through Postman using the resulting bearer token.\n\nOR\n\nFetch token from Keycloak directly from the API by using username and password.\nFetch userinfo from Keycloak from the API using the resulting bearer token.\n\nThe following does NOT work:\n\nFetch token from Keycloak through Postman by using username and password.\nSend request to API, running in a docker container, containing the bearer token.\nAPI tries to fetch userinfo using the token = \"401 Token verification failed\"\n\nUsing same client_id and client_secret in all scenarios.\nIs this a docker networking thing? Or is some Keycloak configuration necessary in order for the API to validate token \"on behalf\" of the client? Quite new to authorization flows.","AnswerCount":1,"Available Count":1,"Score":0.1973753202,"is_accepted":false,"ViewCount":726,"Q_Id":64988971,"Users Score":1,"Answer":"Figured it out: I was calling Keycloak at different urls, localhost:8080 from postman and host.docker.internal:8080 from the api running in Docker. Turns out you have to call Keycloak at same URL from both ends. I switched to using my machines' ip in both cases and voila!","Q_Score":3,"Tags":"python,django,docker,oauth,keycloak","A_Id":65009835,"CreationDate":"2020-11-24T14:52:00.000","Title":"Validating keycloak bearer token on behalf of client","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I want to get info from API via Python, which has infinite info updating (it is live - for example live video or live monitoring). So I want to stop this GET request after interval (for example 1 second), then process these information and then repeat this cycle.\nAny ideas? (now I am using requests module, but I do not know, how to stop receiving data and then get them)","AnswerCount":1,"Available Count":1,"Score":0.1973753202,"is_accepted":false,"ViewCount":350,"Q_Id":64993141,"Users Score":1,"Answer":"I might be off here, but if you hit an endpoint at a specific time, it should return the JSON at that particular moment. You could then store it and use it in whatever process you have created.\nIf you want to hit it again, you would just use requests to hit the endpoint.","Q_Score":1,"Tags":"python,python-3.x,get,python-requests","A_Id":64993189,"CreationDate":"2020-11-24T19:00:00.000","Title":"Python GET request from live\/infinite API endpoint","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I have a python-selenium-bot that takes a screenshot of an image and saves it in an assets folder which my frontend flutter-app then accesses.\nHowever I think there is a better way to do this. Is it possible to not save the picture on the disk but rather in a variable or something like that so that I can send it via my rest api. This would be a lot cleaner and not so error prone.\nIs there any way to save a svg without taking a screenshot of it?","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":68,"Q_Id":65043258,"Users Score":0,"Answer":"Update:\nMy solution now is calling driver.get_screenshot_as_bas64 this allows me to save the screenshot as base64 in a variable and then pass that through my API. My Frontend can then decoded that screenshot again.","Q_Score":1,"Tags":"python,api,selenium,flutter,web-scraping","A_Id":65050437,"CreationDate":"2020-11-27T20:01:00.000","Title":"How to save an svg-element using selenium not locally?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"Let's say I have 10 company names.\nHow to find their \"nearly-exact\" linkedin url?\nex.\nA company is named \"ABC Corp\", its' linkedin url might be \"https:\/\/www.linkedin.com\/company\/abc-corp\"\nbut it's not accurate.\nThere's a way to improve the precision by use google to search linkedin+company name to find the result, but the error rate is still high.\nis there any better solution?","AnswerCount":2,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":557,"Q_Id":65045681,"Users Score":0,"Answer":"The link you'd fetch would be https:\/\/www.linkedin.com\/search\/results\/all\/?keywords=COMPANY_NAME_HERE which searches for the company name.\nThen, you can use beautiful soup to scrape the first result which is usually the company you are looking for","Q_Score":0,"Tags":"javascript,python,google-api,linkedin,linkedin-api","A_Id":65045706,"CreationDate":"2020-11-28T01:50:00.000","Title":"Use company name to find exact linkedin url","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"Im trying to make a script that sent an email with python using smtp.smtplib , almost of examples i found while googling shows how to call this function with only smtpserver and port parameters.\ni want to added other paramaters : domain and binding IP\ni tried this : server = smtplib.SMTP(smtpserver, 25,'mydomain.com',5,'myServerIP')\nI got this as error : TypeError: init() takes at most 5 arguments (6 given)\nCan you suggest a way to do this?","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":58,"Q_Id":65094095,"Users Score":0,"Answer":"This error is likely because the parameters are invalid (there is one too many). Try looking at the smtplib docs to see what parameters are valid","Q_Score":0,"Tags":"python,smtplib","A_Id":65094148,"CreationDate":"2020-12-01T16:05:00.000","Title":"Added more parameters to smtplib.SMTP in python","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I Have converted a Python Selenium file into .exe using pyinstaller. However when I open the file the CMD screen flashes and Thats It, The GUI screen to run the executable file is not observed. My Code uses inputs from An Excel\nPre - Condition which I followed\n\nWeb Driver path is updated to the path where the Exe file is located\nChanged the Excel path to the folder where EXE file is located\n\nCould you help me with this option?\nIntent: Trying to Distribute my automation script to my Peers for Testing purpose","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":87,"Q_Id":65095619,"Users Score":0,"Answer":"You probably cant specify the webdriver path like this, since the path differs from computer to computer. try to input it in the same folder all together and then let the path field for the path of the webdriver free. might or might not work","Q_Score":2,"Tags":"python,selenium,browser-automation","A_Id":65096283,"CreationDate":"2020-12-01T17:39:00.000","Title":"Unable to Open an Python Selenium exe file in Windows 10","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I parse pcap file with scapy python , and there is TCP packet in that pcap that I want to know what is the answer of this pcaket, How can I do that?\nFor example : client and server TCP stream\nclient-> server : \"hi\"\nserver-> client : \"how are you\"\nWhen I get \"hi\" packet (with scapy) how can I get \"how are you\" ?","AnswerCount":1,"Available Count":1,"Score":0.3799489623,"is_accepted":false,"ViewCount":524,"Q_Id":65105876,"Users Score":2,"Answer":"Look at the TCP sequence number of the message from the client. Call this SeqC.\nThen look for the first message from the client whose TCP acknowledgement sequence is higher than SeqC (usually it will be equal to SeqC plus the size of the client's TCP payload). Call this PacketS1.\nStarting with PacketS1, collect the TCP payloads from all packets until you see a packet sent by the server with the TCP PSH (push) flag set. This suggests the end of the application-layer message. Call these payloads PayloadS1 to PayloadSN.\nConcatenate PayloadS1 to PayloadSN. This is the likely application-layer response to the client message.","Q_Score":5,"Tags":"python,tcp,scapy,pcap","A_Id":65165995,"CreationDate":"2020-12-02T10:02:00.000","Title":"Find answer to tcp packet in PCAP with scapy","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I am trying to get the error text from the below using XPath\n<p id=\"username-error\" class=\"error-msg\" role=\"alert\" data-error=\"messages.INVALID_USERNAME\">Sorry, we don't recognize this email.<\/p>\nhere is my code:\nmessage = driver.find_element_by_xpath('\/\/*[@id=\"username-error\"]').text\nThis piece of code is not returning anything. Kindly help me to get the error text such as \"Sorry, we don't recognize this email.\"","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":366,"Q_Id":65110228,"Users Score":0,"Answer":"A minor change in the XPath has fixed the problem.\nmessage = driver.find_element_by_xpath('\/\/*[@data-error=\"messages.INVALID_USERNAME\"]').text","Q_Score":0,"Tags":"python,selenium-webdriver,xpath,gettext,webautomation","A_Id":65125130,"CreationDate":"2020-12-02T14:37:00.000","Title":"How to get Error message text using xpath in selenium python?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"So i have created an automation bot to do some stuff for me on the internet .. Using Selenium Python..After long and grooling coding sessions ..days and nights of working on this project i have finally completed it ...Only to be randomly greeted with a Error 1015 \"You are being rate limited\".\nI understand this is to prevent DDOS attacks. But it is a major blow.\nI have contacted the website to resolve the matter but to no avail ..But the third party security software they use says that they the website can grant my ip exclusion of rate limiting.\nSo i was wondering is there any other way to bypass this ..maybe from a coding perspective ...\nI don't think stuff like clearing cookies will resolve anything ..or will it as it is my specific ip address that they are blocking\nNote:\nThe TofC of the website i am running my bot on doesn't say you cant use automation software on it ..but it doesn't say you cant either.\nI don't mind coding some more to prevent random access denials ..that i think last for 24 hours which can be detrimental as the final stage of this build is to have my program run daily for long periods of times.\nDo you think i could communicate with the third party security to ask them to ask the website to grant me access ..I have already tried resolving the matter with the website. All they said was that A. On there side it says i am fine\nB. The problem is most likely on my side ..\"Maybe some malicious software is trying to access our website\" which .. malicious no but a bot yes. That's what made me think maybe it would be better if i resolved the matter myself.\nDo you think i may have to implement wait times between processes or something. Im stuck.\nThanks for any help. And its a single bot!","AnswerCount":3,"Available Count":1,"Score":0.0665680765,"is_accepted":false,"ViewCount":6688,"Q_Id":65128879,"Users Score":1,"Answer":"I see some possibilities for you here:\n\nIntroduce wait time between requests to the site\nReduce the requests you make\nExtend your bot to detect when it hits the limit and change your ip address (e.g. by restarting you router)\n\nThe last one is the least preferable I would assume and also the most time consuming one.","Q_Score":1,"Tags":"python-3.x,selenium,selenium-webdriver,cloudflare,rate-limiting","A_Id":65128935,"CreationDate":"2020-12-03T15:23:00.000","Title":"How to bypass being rate limited ..HTML Error 1015 using Python","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I am currently working on a project in which I am using a webcam attached to a raspberry pi to then show what the camera is seeing through a website using a client and web server based method through python, However, I need to know how to link the raspberry pi to a website to then output what it sees through the camera while then also outputting it through the python script, but then i don't know where to start\nIf anyone could help me I would really appreciate it.\nMany thanks.","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":111,"Q_Id":65141422,"Users Score":0,"Answer":"So one way to do this with python would be to capture the camera image using opencv in a loop and display it to a website hosted on the Pi using a python frontend like flask (or some other frontend). However as others have pointed out, the latency on this would be so bad any processing you wish to do would be nearly impossible.\nIf you wish to do this without python, take a look at mjpg-streamer, that can pull a video feed from an attached camera and display it on a localhost website. The quality is fairly good on localhost. You can then forward this to the web (if needed) using port forwarding or an application like nginx.\nIf you want to split the recorded stream into 2 (to forward one to python and to broadcast another to a website), ffmpeg is your best bet, but the FPS and quality would likely be terrible.","Q_Score":0,"Tags":"python,raspberry-pi","A_Id":65146963,"CreationDate":"2020-12-04T09:56:00.000","Title":"raspberry pi using a webcam to output to a website to view","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I'd like to modify the Extensions that I send in the client Hello packet with python.\nI've had a read of most of the source code found on GitHub for urllib3 but I still don't know how it determines which TLS extensions to use.\nI am aware that it will be quite low level and the creators of urllib3 may just import another package to do this for them. If this is the case, which package do they use?\nIf not, how is this determined?\nThanks in advance for any assistance.","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":107,"Q_Id":65171598,"Users Score":0,"Answer":"The HTTPS support in urllib3 uses the ssl package which uses the openssl C-library. ssl does not provide any way to directly fiddle with the TLS extension except for setting the hostname in the TLS handshake (i.e. server_name extension aka SNI).","Q_Score":1,"Tags":"python,ssl,httpclient,tls1.2,urllib3","A_Id":65172963,"CreationDate":"2020-12-06T18:29:00.000","Title":"How does urllib3 determine which TLS extensions to use?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I have an embedded system on which I can connect to internet. This embedded system must send sensor data to PC client.\nI put a socket client using python on my PC. I put a socket server ( using C++ language on the embedded system because you can only use C++ ).\nI can succesfully connect from my PC to the embedded system using the sockets and send and recieve whatever I want.\nNow, the problem is I use local IP to connect to the system and both of them must be connected to the same Wifi router.\nIn the real application, I won't know where the embedded system is in the world. I need to get to it through internet, because it will be connectet to internet through 4g.\nMy question is, how can I connect to it through internet, if the embedded system is connected to internet using 4G?\nThank you","AnswerCount":2,"Available Count":2,"Score":0.0996679946,"is_accepted":false,"ViewCount":56,"Q_Id":65202965,"Users Score":1,"Answer":"Realistically in typical situations, neither a PC nor an embedded device hanging off a 4g modem will likely have (or should be allowed) to have externally routable addresses.\nWhat this practically means is that you need to bounce your traffic through a mutually visible relay in the cloud.\nOne very common way of doing that for IoT devices (which is basically to say, connected embedded devices) is to use MQTT. You'll find support in one form or another for most computing platforms with any sort of IP networking capability.\nOf course there are many other schemes, too - you can do something with a RESTful API, or websockets (perhaps as an alternate mode of an MQTT broker), or various proprietary IoT solutions offered by the big cloud platforms.\nIt's also going to be really key that you wrap the traffic in SSL, so you'll need support for that in your embedded device, too. And you'll have to think about which CA certs you package, and what you do about time given its formal requirement as an input to SSL validation.","Q_Score":1,"Tags":"python,sockets,tcp,embedded","A_Id":65211281,"CreationDate":"2020-12-08T16:33:00.000","Title":"Socket over internet Python Embedded","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I have an embedded system on which I can connect to internet. This embedded system must send sensor data to PC client.\nI put a socket client using python on my PC. I put a socket server ( using C++ language on the embedded system because you can only use C++ ).\nI can succesfully connect from my PC to the embedded system using the sockets and send and recieve whatever I want.\nNow, the problem is I use local IP to connect to the system and both of them must be connected to the same Wifi router.\nIn the real application, I won't know where the embedded system is in the world. I need to get to it through internet, because it will be connectet to internet through 4g.\nMy question is, how can I connect to it through internet, if the embedded system is connected to internet using 4G?\nThank you","AnswerCount":2,"Available Count":2,"Score":0.0,"is_accepted":false,"ViewCount":56,"Q_Id":65202965,"Users Score":0,"Answer":"I think your problem is more easily solved if you reverse the roles of your embedded system and PC. If you are communicating to a device using IP protocols across cellular networks, it is much easier to have the device connect to a server on the PC rather than the other way around. Some networks\/cellular modems do not allow server sockets and in any case, the IP address is usually dynamically allocated and therefore difficult to know. By having the device connect to a server, it \"knows\" the domain name (or IP address) and port to which it should make the connection. You just have to make sure that there is indeed a server program running at that host bound to some agreed upon port number. You can wake up the device to form the connection based on a number of criteria, e.g. time or amount of collected data, etc.","Q_Score":1,"Tags":"python,sockets,tcp,embedded","A_Id":65205272,"CreationDate":"2020-12-08T16:33:00.000","Title":"Socket over internet Python Embedded","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I have two python programs. Program 1 displays videos in a grid with multiple controls on it, and Program 2 performs manipulations to the images and sends it back depending on the control pressed in Program 1.\nEach video in the grid is running in its own thread, and each video has a thread in Program 2 for sending results back.\nI'm running this on the same machine though and I was unable to get multiple socket connections working to and from the same address (localhost). If there's a way of doing that - please stop reading and tell me how!\nI currently have one socket sitting independent of all of my video threads in Program 1, and in Program 2 I have multiple threads sending data to the one socket in an array with a flag for which video the data is for. The problem is when I have multiple threads sending data at the same time it seems to scramble things and stop working. Any tips on how I can achieve this?","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":81,"Q_Id":65203271,"Users Score":0,"Answer":"Regarding If there's a way of doing that - please stop reading and tell me how!.\nThere's a way of doing it, assuming you are on Linux or using WSL on Windows, you could use the hostname -I commend which will output an IP that looks like 192.168.X.X.\nYou can use that IP in your python program by binding your server to that IP instead of localhost or 127.0.0.1.","Q_Score":0,"Tags":"python,multithreading","A_Id":65203946,"CreationDate":"2020-12-08T16:51:00.000","Title":"Multiple threads sending over one socket simultaneously?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":1,"System Administration and DevOps":0,"Web Development":0},{"Question":"Good day everybody. I'm still learning parsing data with Python. I'm now trying to familiarize myself with Chrome Developer Tools. My question is when inspecting a directory website like TruePeopleSearch.com, how do I copy or view the variables that holds the data such as Name, Phone, and Address? I tried browsing the tool, but since I'm new with the Developer tool, I'm so lost with all the data. I would appreciate if the experts here points me to the right direction.\nThank you all!","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":129,"Q_Id":65206058,"Users Score":0,"Answer":"Upon further navigating the Developer Console, I learned that these strings are located in these variables, by copying the JS paths.\nNAME & AGE\ndocument.querySelector(\"#personDetails > div:nth-child(1)\").innerText\nADDRESS\ndocument.querySelector(\"#personDetails > div:nth-child(4)\").innerText\nPHONE NUMBERS\ndocument.querySelector(\"#personDetails > div:nth-child(6)\").innerText\nSTEP 1\nFrom the website, highlight are that you need to inspect and click \"Inspect Element\"\nSTEP 2\nUnder elements, right-click the highlighted part and copy the JS path\nSTEP 3\nNavigate to console and paste the JS path and add .innerText and press Enter","Q_Score":0,"Tags":"python,json,parsing,uri","A_Id":65207019,"CreationDate":"2020-12-08T20:00:00.000","Title":"Grabbing values (Name, Address, Phone, etc.) from directory websites like TruePeopleSearch.com with Chrome Developer Tool","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"this is kind of a dumb question but how would I make a discord.py event to automatically react to a message with a bunch of different default discord emojis at once. I am new to discord.py","AnswerCount":2,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":265,"Q_Id":65223410,"Users Score":0,"Answer":"You have to use on_message event. Its a default d.py function. It is an automatic function.","Q_Score":0,"Tags":"python,discord,discord.py","A_Id":65923529,"CreationDate":"2020-12-09T19:13:00.000","Title":"How would I use a bot to send multiple reactions on one message? Discord.py","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I am creating a multiplayer game and I would like the communication between my server program (written in python) and the clients (written in c# - Unity) to happen via UDP sockets.\nI recently came across the concept of UDP Multicast, and it sounds like it could be much better for my use case as opposed to using UDP Unicast , because my server needs to update all of the clients (players) with the same content every interval. So, rather than sending multiple identical packets to all the clients with UDP unicast, I would like to be able to only send one packet to all the clients using multicast, which sounds much more efficient.\nI am new to multicasting and my questions are:\nHow can I get my server to multicast to clients across the internet?\nDo I need my server to have a special public multicast IP address? If so how do I get one?\nIs it even possible to multicast across the internet? or is multicasting available only within my LAN?\nAnd what are the pros and cons with taking the multicast approach?\nThank you all for your help in advance!!","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":134,"Q_Id":65228805,"Users Score":1,"Answer":"You can't multicast on the Internet. Full stop.\nBasically, multicast is only designed to work when there's someone in charge of the whole network to set it up. As you noted, that person needs to assign the multicast IP addresses, for example.","Q_Score":1,"Tags":"python,sockets,unity3d,networking,udp","A_Id":65239846,"CreationDate":"2020-12-10T05:08:00.000","Title":"How can I get my server to UDP multicast to clients across the internet? Do I need a special multicast IP address?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"API Gateway is connected to a lambda function that sleeps for 4 seconds. When I execute the API 20 times in 1 second, the first few calls completes the job in 5 seconds but the other calls take more time eg:12 sec,20 sec, and sometimes runs into timeout errors\n\nAre the APIs dependent on its previous calls (i.e) for the 20th API call to get executed the other previous calls must be completed?\nHow to resolve this problem?","AnswerCount":1,"Available Count":1,"Score":0.1973753202,"is_accepted":false,"ViewCount":338,"Q_Id":65250756,"Users Score":1,"Answer":"Some things to consider when working with AWS API Gateway and AWS Lambda.\nCreating an instance (execution environment) of your Lambda Function takes a while. This is called a cold start. If your Lambda runs in a VPC this might take longer. Shouldn't take less than a couple seconds though.\nOnce that Lamdba runs, API Gateway can reuse this instance. If there isn't traffic for some time the Lamdba goes back to cold storage. The next request through API Gateway will create another instance of the Lambda.\nGiven enough traffic AWS Lambda triggers concurrent executions, which means there will be more Lambda instances. This is called automatic scaling. Again, it takes a cold start. You have no control when, or how many Lambda instances are running. Welcome to serverless.\nAWS Lambda limits its execution time to 15 minutes. Your function cannot exceed this limit. It will be shutdown hard.\nAPI Gateway has an integration timeout of 30 seconds. This means your Lambda needs to finish within 30 seconds. Otherwise API Gateway returns 502 while your function is still running.","Q_Score":0,"Tags":"python,amazon-web-services,amazon-ec2,aws-lambda,aws-api-gateway","A_Id":65254999,"CreationDate":"2020-12-11T11:44:00.000","Title":"Does AWS API gateway calls takes long if executed multiple times in 1 second?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I'm stuck in a rock and a hard place.\nI have been given a very restricted Python 2\/3 environment where both os and subprocess library are banned. I am testing file upload functionality using Python selenium; creating the file is straight forward, however, when I use the method driver.send_keys(xyz), the send_keys is expecting a full path rather than just a file name. Any suggestions on how to work around this?","AnswerCount":2,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":52,"Q_Id":65250817,"Users Score":0,"Answer":"with no way of using the os module it would seem you're SOL unless the placement of your script is static (i.e. you define the current working dirrectory as a constant) and you then handle all the path\/string operations within that directory yourself.\nyou won't be able to explore what's in the directory but you can keep tabs on any files you create and just store the paths yourself, it will be tedious but there's no reason why it shouldn't work.\nyou won't be able to delete files i don't think but you should be able to clear their contents","Q_Score":0,"Tags":"python,python-3.x,selenium-webdriver","A_Id":65250942,"CreationDate":"2020-12-11T11:47:00.000","Title":"Get current directory - 'os' and 'subprocess' library are banned","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I am working on a python project which involves real time communication between a Raspberry Pi device and Cloud server. After receiving data in the cloud server in real time, I have to display the output in a web application in real time as well. What is the correct way to do this ?\nAs web applications use socket.io (web sockets) and communication between Raspberry Pi and cloud can be done through a normal socket, I am confused on whether to proceed with normal socket or web socket.\nAny feedback will be greatly appreciated.","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":31,"Q_Id":65255354,"Users Score":0,"Answer":"It all depends on whether you want to work with the TCP stack or the HTTP stack, while TCP sockets have a smaller footprint, HTTP sockets have better support with high level libraries. If you don't need multiple simultaneous connections or high latency requirements, they are both identical, so go with the one that is easier to work with, ie the web socket.","Q_Score":0,"Tags":"python,sockets,websocket,raspberry-pi","A_Id":65257643,"CreationDate":"2020-12-11T16:48:00.000","Title":"What is the correct method to communicate in real time between Raspberry PI and Cloud Server and then display output in web app?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I want to get source of a webpage using python requests.get(), but sometimes it retrieves the page source before some elements even load. Is there a way to make the requests.get() wait until certain element appear? The page does not load on javascript and I don't need selenium suggestions.","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":121,"Q_Id":65258258,"Users Score":0,"Answer":"When you say \"retrieves page before elements load\" you are implying that some elements are loaded or retrieved via JavaScript. What you get with requests.get() is the initial document and the same thing your browser gets. The difference is that the browser can then read and execute instructions contained in the initial document and fetch more resources, while python can not.\nYou either have to use selenium or go into your browser development tools and see if you can maybe find some endpoints from which the data is getting pulled down.","Q_Score":0,"Tags":"python,python-requests","A_Id":65258350,"CreationDate":"2020-12-11T20:33:00.000","Title":"Make requests.get() wait untill page is fully loaded?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I can't find anyway to setup TTL on a document within AWS Elasticsearch utilizing python elasticsearch library.\nI looked at the code of the library itself, and there are no argument for it, and I yet to see any answers on google.","AnswerCount":1,"Available Count":1,"Score":0.1973753202,"is_accepted":false,"ViewCount":460,"Q_Id":65271287,"Users Score":1,"Answer":"There is none, you can use the index management policy if you like, which will operate at the index level, not at the doc level. You have a bit of wriggle room though in that you can create a pattern data-* and have more than 1 index, data-expiring-2020-..., data-keep-me.\nYou can apply a template to the pattern data-expiring-* and set a transition to delete an index after lets say 20 days. If you roll over to a new index each day you will the oldest day being deleted at the end of the day once it is over 20 days.\nThis method is much more preferable because if you are deleting individual documents that could consume large amounts of your cluster's capacity, as opposed to deleting entire shards. Other NoSQL databases such as DynamoDB operate in a similar fashion, often what you can do is add another field to your docs such as deletionDate and add that to your query to filter out docs which are marked for deletion, but are still alive in your index as a deletion job has not yet cleaned them up. That is how the TTL in DynamoDB behaves as well, data is not deleted the moment the TTL expires it, but rather in batches to improve performance.","Q_Score":2,"Tags":"python,amazon-web-services,elasticsearch","A_Id":65282126,"CreationDate":"2020-12-13T00:30:00.000","Title":"Is there a way to set TTL on a document within AWS Elasticsearch utilizing python library?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"How do I display the user's Name + Discord Tag? As in:\nI know that;\nf\"Hello, <@{ctx.author.id}>\"\nwill return the user, and being pinged.\n(@user)\nAnd that;\nf\"Hello, {ctx.author.name}\"\nwill return the user's nickname, but WITHOUT the #XXXX after it.\n(user)\nBut how do I get it to display the user's full name and tag?\n(user#XXXX)","AnswerCount":2,"Available Count":1,"Score":0.0996679946,"is_accepted":false,"ViewCount":480,"Q_Id":65281822,"Users Score":1,"Answer":"To get user#XXXX you can just do str(ctx.author) (or just put it in your f-string and it will automatically be converted to a string). You can also do ctx.author.discriminator to get their tag (XXXX).","Q_Score":0,"Tags":"python,discord,discord.py","A_Id":65281985,"CreationDate":"2020-12-13T23:30:00.000","Title":"How to get author's Discord Tag shown","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I have made a Scrapy web crawler which can scrape Amazon. It can scrape by searching for items using a list of keywords and scrape the data from the resulting pages.\nHowever, I would like to scrape Amazon for large portion of its product data. I don't have a preferred list of keywords with which to query for items. Rather, I'd like to scrape the website evenly and collect X number of items which is representative of all products listed on Amazon.\nDoes anyone know how scrape a website in this fashion? Thanks.","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":68,"Q_Id":65292028,"Users Score":1,"Answer":"I'm putting my comment as an answer so that others looking for a similar solution can find it easier.\nOne way to achieve this is to going through each category (furniture, clothes, technology, automotive, etc.) and collecting a set number of items there. Amazon has side\/top bars with navigation links to different categories, so you can let it run through there.\nThe process would be as follows:\n\nFollow category urls from initial Amazon.com parse\nUse a different parse function for the callback, one that will scrape however many items from that category\nEnsure that data is writing to a file (it will probably be a lot of data)\n\nHowever, such an approach would not be representative in the proportions of each category in the total Amazon products. Try looking for a \"X number of results\" label for each category to compensate for that. Good luck with your project!","Q_Score":0,"Tags":"python,web-scraping,scrapy","A_Id":65314264,"CreationDate":"2020-12-14T15:50:00.000","Title":"How to scrape data from multiple unrelated sections of a website (using Scrapy)","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"Can someone tell me what the difference between on_raw_message_delete()\/OnRawMessageDelete and on_message_delete()\/OnMessageDelete is?","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":164,"Q_Id":65295903,"Users Score":2,"Answer":"on_raw_message_delete unlike on_message_delete it's called regardless of the state of the internal cache.\nLet's say you send a message, the bot caches it, but the bot suddenly restarts (the message won't be in the cache anymore), if you delete the message, on_message_delete won't be called, but on_raw_message_delete will.","Q_Score":0,"Tags":"python,discord.py","A_Id":65295973,"CreationDate":"2020-12-14T20:19:00.000","Title":"Difference between on_message_delete() and on_raw_message_delete()","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I'm building a bot that logs into zoom at specified times and the links are being obtained from whatsapp. So i was wondering if it is was possible to retrieve those links from whatsapp directly instead of having to copy paste it into python. Google is filled with guides to send messages but is there any way to READ and RETRIEVE those messages and then manipulate it?","AnswerCount":5,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":10184,"Q_Id":65299796,"Users Score":0,"Answer":"You can read all the cookies from whatsapp web and add them to headers and use the requests module or you can also use selenium with that.","Q_Score":0,"Tags":"python,selenium,whatsapp","A_Id":68296269,"CreationDate":"2020-12-15T03:56:00.000","Title":"How do I read whatsapp messages from a contact using python?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I need to match substring in format !some text!https:\/\/test.com\/ where anything between two ! marks is text and after comes url without spaces,\nso far I made a regular expression which can find substring\n(!(.*)!((https|http):\\\/\\\/[^\\s+])) but it doesn't work as desired when similar pattern is duplicated anywhere in the text.\nfor example\n!text goes here!https:\/\/amazon.com and some other text comes here should parse\n!text goes here!https:\/\/amazon.com\nand it does perfectly, but when I duplicate such pattern anywhere in the text it matches whole text between the two texts.\ni.e.\n!text goes here!https:\/\/amazon.com some text here !text goes here!https:\/\/amazon.com and after some other text matches whole text\nshould match two separate !text goes here!https:\/\/amazon.com some but it will select substring till the end of second match\n!text goes here!https:\/\/amazon.com some text here !text goes here!https:\/\/amazon.com\nperhaps it takes whole text between first ! and the last !\nIs there a approach halt matching when space met after the url text","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":36,"Q_Id":65316180,"Users Score":3,"Answer":"!.*! matches the longest possible substring surrounded by exclamation points; in other words, it goes from the first bang to the last one.\nYou want to match from the first bang to the next one, which could be a non-greedy match !.*?! or the more precise match, ![^!]*!.","Q_Score":0,"Tags":"javascript,python,regex,parsing,regex-negation","A_Id":65316215,"CreationDate":"2020-12-16T01:49:00.000","Title":"How to separate matching substring at first met space char regex","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I'm working a program that will utilize Selenium\/Webdriver to open a webpage, enter some data, and open a new page which is a PDF. Ultimately I would like to download that PDF into a folder. I know it is possible to download a PDF into a folder if you have the URL in your script, but I'm struggling to find a way to download it if it is opened within the program.\nA) Is there a way to download a PDF that is opened explicitly in Chrome using a script?\nB) Is there a way to extract the URL from an opened webpage that then be fed back into the program to download from?","AnswerCount":1,"Available Count":1,"Score":0.1973753202,"is_accepted":false,"ViewCount":101,"Q_Id":65317600,"Users Score":1,"Answer":"While I was doing a selenium project, I faced a similar issue.\nI would click on a link that was referring to a PDF file but instead of downloading, the selenium chromedriver would just open it in a new tab.\nWhat solved my problem was that right after I started new chromedriver session, I manually disabled this feature:\n\nIn your Chrome settings, go to Privacy and Securtiy\nSelect Site Settings\nScroll down and click on Additional preferences\nFind a section named 'PDF documents'\nTurn on the option that says \"Download PDF files instead of automatically opening them in Chrome\"\n\nNow, any PDF link you click on will download the file instead of opening them in a new tab. Note that you need to do this every time you start a new chromedriver. Changing this setting in your main Chrome application won't help.","Q_Score":1,"Tags":"python,selenium,selenium-webdriver","A_Id":65317668,"CreationDate":"2020-12-16T05:01:00.000","Title":"Download Opened PDF with Python","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I have 2 python files that do Web scraping using Selenium and Beautifulsoup and store the results in separate CSV files say file1.csv and file2.csv. Now, I want to deploy these files on the Azure cloud, I know Azure function apps will be ideal for this. But, I don't know how Functions app will support Selenium driver on it.\nBasically, I want to time trigger my 2 web scraping files and store the results in two separate files file1.csv and file2.csv that will be stored in blob storage on Azure cloud. Can someone help me with this task?\nHow can I use the selenium driver on Azure functions app?","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":447,"Q_Id":65352652,"Users Score":2,"Answer":"Deploying on virtual machines or EC2 is the only option that one can use to achieve this task.\nAlso, with Heroku, we will be able to run selenium on the cloud by adding buildpacks. But when it comes to storing the files, we will not be able to store files on heroku as heroku does not persist the files. So, VMs or EC2 instances are the only options for this task.","Q_Score":2,"Tags":"python,azure,selenium-webdriver,web-scraping,beautifulsoup","A_Id":65372990,"CreationDate":"2020-12-18T06:28:00.000","Title":"Deploy Python Web Scraping files on Azure cloud(function apps)","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I had some embedded profile pictures in datastudio worked great until a couple of days.\nI used this kind of URL:\n\"https:\/\/twitter.com\/(userName)\/profile_image?size=original\"\nIt suddely doesn't work at all. I couldn't find any alternative either.\nPlease Help","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":18,"Q_Id":65384347,"Users Score":0,"Answer":"Since the legacy twitter.com was shut down on 15th December 2020, these URLs will no longer respond.\nYou will need to use the official Twitter API (either v1.1 or v2) to retrieve the User object, and get the profile image from there.","Q_Score":0,"Tags":"python,twitter","A_Id":65385680,"CreationDate":"2020-12-20T20:11:00.000","Title":"link to Twitter User profile Pictures not Working anymore","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I am trying to make a server info command and I want it to display the server name, boost count, boost members and some other stuff as well.\nOnly problem is I have looked at the docs and searched online and I cant find out how to find the boost information.\nI dont have any code as Ive not found any code to try and use for myself\nIs there any way to get this information?","AnswerCount":2,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":2193,"Q_Id":65385757,"Users Score":4,"Answer":"Guild Name - guild_object.name\nBoost count - guild_object.premium_subscription_count\nBoosters, the people who boosted the server - guild_object.premium_subscribers\nIf your doing this in a command as I assume, use ctx.guild instead of guild_object. For anything further, you can re-read the docs as all of the above information is in it under the discord.Guild","Q_Score":1,"Tags":"python,discord.py","A_Id":65386320,"CreationDate":"2020-12-20T23:15:00.000","Title":"Get the number of boosts in a server discord.py","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"So i have created a telegram bot in python.\nIt is run by its main.py file.\nIt is running on the server.\nBut sometimes it stops Due to internet or some minor issue.\nCan there be code which can automatically restart the the main.py code on server maybe using Deamonize library.\nIf Yes,please suggest how to do\n\nThank YOU","AnswerCount":1,"Available Count":1,"Score":0.1973753202,"is_accepted":false,"ViewCount":130,"Q_Id":65398617,"Users Score":1,"Answer":"If you are using Linux, you can use crontab\ncrontab is used for scheduling event","Q_Score":1,"Tags":"python,server,telegram-bot","A_Id":68671493,"CreationDate":"2020-12-21T19:10:00.000","Title":"Can we Automatically Restart the Telgram bot(Python) once it is stopped on server using Deamonize or something else?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I have bunch of urls to fetch. I wish to know which one would be better strategy for speed and good User experience.\nStrategy:\n\nUsing Javascript (fetch or XHR). All urls will be fetched on client side.\nSend only one fetch\/XHR (calling python in JS) to python and get all the URL's data in python and send back the response to client.\n\nPlease let me know which one would be better strategy.\nThank you,\nNamratha Patil.","AnswerCount":2,"Available Count":2,"Score":0.0996679946,"is_accepted":false,"ViewCount":46,"Q_Id":65403964,"Users Score":1,"Answer":"You can priorities your data loading on may factors like:\n\nWhether the data is required in first fold to render elements and similar factors. if you go with 2nd strategy if will delay the loading and you will have bad user experience.","Q_Score":1,"Tags":"javascript,python","A_Id":65404063,"CreationDate":"2020-12-22T06:07:00.000","Title":"Wish to Know Better Strategy for Fetching Multiple URLS Javascript + Python","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I have bunch of urls to fetch. I wish to know which one would be better strategy for speed and good User experience.\nStrategy:\n\nUsing Javascript (fetch or XHR). All urls will be fetched on client side.\nSend only one fetch\/XHR (calling python in JS) to python and get all the URL's data in python and send back the response to client.\n\nPlease let me know which one would be better strategy.\nThank you,\nNamratha Patil.","AnswerCount":2,"Available Count":2,"Score":1.2,"is_accepted":true,"ViewCount":46,"Q_Id":65403964,"Users Score":1,"Answer":"The network requests are computationally expensive, so batching and sending only one request is almost always preferable.","Q_Score":1,"Tags":"javascript,python","A_Id":65403974,"CreationDate":"2020-12-22T06:07:00.000","Title":"Wish to Know Better Strategy for Fetching Multiple URLS Javascript + Python","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I try to convert this String to only the link: {\"link\":\"https:\/\/i.imgur.com\/zfxsqlk.png\"}\nI'm trying to create a discord bot, which sends random pictures from the API https:\/\/some-random-api.ml\/img\/red_panda.\nWith imageURL = json.loads(requests.get(redpandaurl).content) I get the json String, but what do I have to do that I only get the Link like this https:\/\/i.imgur.com\/zfxsqlk.png\nSorry if my question is confusingly written, I'm new to programming and don't really know how to describe this problem.","AnswerCount":2,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":50,"Q_Id":65405747,"Users Score":0,"Answer":"What you get from json.loads() is a Python dict. You can access values in the dict by specifying their keys.\nIn your case, there is only one key-value pair in the dict: \"link\" is the key and \"https:\/\/i.imgur.com\/zfxsqlk.png\" is the value. You can get the link and store it in the value by appending [\"link\"] to your line of code:\nimageURL = json.loads(requests.get(redpandaurl).content)[\"link\"]","Q_Score":0,"Tags":"python,json","A_Id":65406385,"CreationDate":"2020-12-22T08:56:00.000","Title":"Convert Json format String to Link{\"link\":\"https:\/\/i.imgur.com\/zfxsqlk.png\"}","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":1,"System Administration and DevOps":0,"Web Development":0},{"Question":"Hi i was created a new file\/folder in s3 bucket now i want to change the name of the folder\/file name .\nhow to rename the file\/folder name in s3 bucket by using python boto3","AnswerCount":2,"Available Count":1,"Score":0.0996679946,"is_accepted":false,"ViewCount":1899,"Q_Id":65408114,"Users Score":1,"Answer":"There is no 'rename' function in Amazon S3. Object names are immutable.\nYou would need to:\n\nCopy the object to a new Key (filename)\nDelete the original object\n\nPlease note that folders do not actually exist in Amazon S3. Rather, the full path of the object is stored in its Key. Thus, it is not possible to rename folders, since that would involve renaming all objects within that path. (And objects can't be renamed, as mentioned above.)\nIf you wanted to \"rename a folder\", you could write a Python script that will:\n\nObtain a listing of all objects within the given Prefix\nLoop through each object, then:\nCopy the object to a new Key\nDelete the original object\n\nIf you do not want to code this, then there are some tools (eg Cyberduck) that give a nice user interface and can do many of these operations for you.","Q_Score":0,"Tags":"python,python-3.x,amazon-web-services,amazon-s3,boto3","A_Id":65431512,"CreationDate":"2020-12-22T11:44:00.000","Title":"How to rename the s3 file name by using python boto3","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"Let's say we have an established websocket connection between our server and a remote server and the remote server from time to time sends some data to us.\nSo does our websocket connection spend outbound data traffic when receiving data?\nMy guess is it does not, because the receiving data gets accumulated in the memory. So when you do a .recv(), the websocket just pulls out the data from the memory locally and sends nothing to the other server. Is this correct?","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":135,"Q_Id":65432174,"Users Score":1,"Answer":"At the level of the web socket protocol it is likely that recipient server sends a ping message to the originating client every couple of minutes, which responds with a pong.\nSo, the few-tens-of-bytes pong message flows back every once in a while.\nAt the TCP \/ IP level, the receiver server responds to every second incoming data packet with an ACK packet, comprising 30 bytes. Incoming data packets can carry up to 1460 bytes of payload data plus the 30 bytes.\nSo, there's a minimal amount of reverse data transmission, but it's not zero.\nIf you must have zero reverse transmission you need to use a datagram protocol. But datagram protocols are lossy: there's no way to recover a lost datagram.","Q_Score":1,"Tags":"python,websocket,protocols,packet","A_Id":65432341,"CreationDate":"2020-12-23T23:29:00.000","Title":"Does a websocket spend outbound data traffic when receiving data?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I tried importing websocket, but python returns\nImportError: no module named websocket.\nI checked sys.path, and the directory that the websocket package is in, Library\/Python\/3.7\/lib\/python\/site-packages, is included. I also confirmed that there's __init__.py in the package.\nI tried importing the other modules in Library\/Python\/3.7\/lib\/python\/site-packages: none of them can be imported.\nWhy can't I import any of the packages in that path?","AnswerCount":3,"Available Count":3,"Score":1.2,"is_accepted":true,"ViewCount":3195,"Q_Id":65489160,"Users Score":3,"Answer":"You may have more than one python version installed.\nFor example, the script you are running is defined to run with python 2.7 but your default python version is >=3 so pip will install websocket for python 3 only.\nTry using\npip2 install websocket\nAlso check to see if you have #!\/usr\/bin\/python2.7 in your first line of the script","Q_Score":1,"Tags":"python,macos,websocket,module,package","A_Id":66350542,"CreationDate":"2020-12-29T08:11:00.000","Title":"ImportError: No module named websocket","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I tried importing websocket, but python returns\nImportError: no module named websocket.\nI checked sys.path, and the directory that the websocket package is in, Library\/Python\/3.7\/lib\/python\/site-packages, is included. I also confirmed that there's __init__.py in the package.\nI tried importing the other modules in Library\/Python\/3.7\/lib\/python\/site-packages: none of them can be imported.\nWhy can't I import any of the packages in that path?","AnswerCount":3,"Available Count":3,"Score":0.0,"is_accepted":false,"ViewCount":3195,"Q_Id":65489160,"Users Score":0,"Answer":"Whatever editor you are using to write your code. I use Visual Studio Code. On some libraries, I have to restart the editor when installing a library. I had to restart Atom just adding plugins.","Q_Score":1,"Tags":"python,macos,websocket,module,package","A_Id":65513121,"CreationDate":"2020-12-29T08:11:00.000","Title":"ImportError: No module named websocket","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I tried importing websocket, but python returns\nImportError: no module named websocket.\nI checked sys.path, and the directory that the websocket package is in, Library\/Python\/3.7\/lib\/python\/site-packages, is included. I also confirmed that there's __init__.py in the package.\nI tried importing the other modules in Library\/Python\/3.7\/lib\/python\/site-packages: none of them can be imported.\nWhy can't I import any of the packages in that path?","AnswerCount":3,"Available Count":3,"Score":0.0,"is_accepted":false,"ViewCount":3195,"Q_Id":65489160,"Users Score":0,"Answer":"Sometimes you have to restart your ide when installing some libraries. And in others you may need to reboot the whole system","Q_Score":1,"Tags":"python,macos,websocket,module,package","A_Id":65489407,"CreationDate":"2020-12-29T08:11:00.000","Title":"ImportError: No module named websocket","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I have more than 210000 links which I want to scrape. Is there any way we can print how many links it has completed scraping during the execution and sleep or pause execution for 10 mins after every 10000 page count?","AnswerCount":2,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":170,"Q_Id":65519951,"Users Score":0,"Answer":"great response! I was thinking of using the sleep inside an if statement, for example.\nif parsed_pages == 100*x\ntime.sleep(rand(20,150)\nelse:","Q_Score":1,"Tags":"python-3.x,scrapy","A_Id":65531000,"CreationDate":"2020-12-31T10:55:00.000","Title":"Count scraped items from scrapy during execution and pause or sleep after a certain number of pages","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"If I use GMail and send a message to myself it appears unread in my inbox.\nIf I send a message via the GMail API using a python script, the message is sent up it only appears in \"All Mail\" and \"Sent\" and is marked as read.\nI have checked my filters and there is nothing there that would be doing this (and I would expect filters to apply consistently across both of the above use-cases).\nAny thoughts?","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":170,"Q_Id":65531150,"Users Score":0,"Answer":"Looks like one of those \"Google\" things.\nWaited an hour - no changes to the code but the behaviour is now as expected !","Q_Score":0,"Tags":"python,gmail-api","A_Id":65532786,"CreationDate":"2021-01-01T15:47:00.000","Title":"GMail-API (Python) EMail Sent To Self - By Passes Inbox","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"How do I get the user\/member object in discord.py with only the Name#Discriminator? I searched now for a few hours and didn't find anything. I know how to get the object using the id, but is there a way to convert Name#Discriminator to the id?\nThe user may not be in the Server.","AnswerCount":2,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":1332,"Q_Id":65533874,"Users Score":0,"Answer":"There's no way to do it if you aren't sure they're in the server. If you are, you can search through the servers' members, but otherwise, it wouldn't make sense. Usernames\/Discriminators change all the time, while IDs remain unique, so it would become a huge headache trying to implement that. Try doing what you want by ID, or searching the server.","Q_Score":1,"Tags":"python,discord.py","A_Id":65534406,"CreationDate":"2021-01-01T21:31:00.000","Title":"Discord.py get user with Name#0001","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"How would I error check if there is a previous page for the chromium browser to go to? I have a button which freezes out if the chromium instance has just been launched and there is not a previous page to go to using the command page.goback()","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":85,"Q_Id":65540674,"Users Score":0,"Answer":"This is a bug and is not intended behavior. You should consider reporting it as an issue on the GitHub repo.","Q_Score":1,"Tags":"python,error-handling,chromium,pyppeteer","A_Id":65589496,"CreationDate":"2021-01-02T15:15:00.000","Title":"How to error check pyppeteer page.goBack()","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I'm developing a matchmaking bot for discord. I read through the documentation and looked through stack exchange, I can't find relevant information on this. I'm trying to eliminate clutter and make it easier for a users to accept and decline a matchmaking request. The bot would listen for a \"!match @user\" and handle the event. I only want the @user to see and accept it and the user that sent the message to see it (obviously). Right now, I believe it's not possible unless it assigns a secret role and deletes these roles after they are finished; this wouldn't be ideal though. Any information or forward to a post, that would help tremendously. Thanks in advance.","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":2595,"Q_Id":65541872,"Users Score":0,"Answer":"You can not currently hide messages for any specific user\/role.\nYou can try 2 approaches:\n\nSend Message in User DMs\nMake a channel and set it permissions for a specific role and assign that role to the user","Q_Score":0,"Tags":"python,discord.py","A_Id":65547946,"CreationDate":"2021-01-02T17:13:00.000","Title":"discord.py: Hide messages from all other users, but the @mention user","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I want the person who used the command to be able to delete the result. I have put the user's ID in the footer of the embed, and my question is: how do I get that data from the message where the user reacted to.\nreaction.message.embed.footer doesn't work. I currently don't have code as I was trying to get that ID first.\nThanks in advance!","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":114,"Q_Id":65549860,"Users Score":1,"Answer":"discord.Message object has no attribute embed, but it has embeds. It returns you a list of embeds that the message has. So you can simply do: reaction.message.embeds[0].footer.","Q_Score":0,"Tags":"python,python-3.x,discord.py,discord.py-rewrite","A_Id":65549978,"CreationDate":"2021-01-03T12:30:00.000","Title":"Get embed footer from reaction message","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I'm still working on this cocktail app and I ran into another issue. After some time, the api call will drop after an hour; the api will only stay open for an hour. After that hour, I have to make a new api call. But I don't want to start all over when trying to pull data back. The data I'm using has over 300 thousand records in it to match, which on average takes about 10 hours to complete. Sorry, if this question does not make any sense I'm still new to coding and I'm doing my best!\nFor instance:\n\nMake api call\nUse data from csv file 300k records to get data back on\nRetrieve the data back.\nAPI connection drops--- Will need to make new api connection.\nMake API call\nDetermine what data has been matched from csv file, then pick up from the next record that has not been matched.\n\nIs there a way to make sure that when I have to get a new API key or when the connection drops, whenever the connection is made again it picks up where it left off?","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":34,"Q_Id":65556049,"Users Score":0,"Answer":"Sounds like you want your API to have something like \"sessions\" that the client can disconnect and reconnect to.\nJust some sort of session ID the server can use to reconnect you to whatever long running query it is doing.\nIt probably gets quite complex though. The query may finish when no client is connected and so the results need to be stored somewhere keyed by the session id so that when the API reconnects it can get the results.\nPS. 10 hours to match 300K records? That is a crazy amount of time ! I'd be doing some profiling on that match algorithm.","Q_Score":0,"Tags":"python,api,soap","A_Id":65556139,"CreationDate":"2021-01-03T23:36:00.000","Title":"How do I have an API pickup where it left off after the request times out","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I'm looking for a way to make the bot screen share a website. And the link to be a parameter after the command. (To clarify: The website starting URL doesn't change. Only a token that's given after the starting URL. For example: \"https:\/\/websitehere.com\/branch\/ 'token goes here'","AnswerCount":2,"Available Count":1,"Score":0.0996679946,"is_accepted":false,"ViewCount":2426,"Q_Id":65560530,"Users Score":1,"Answer":"Bots Can't Screenshare! Discord Bot API Do Not Support It Yet!","Q_Score":1,"Tags":"python,discord,discord.py","A_Id":65563382,"CreationDate":"2021-01-04T09:45:00.000","Title":"Is there a way ot make a discord bot screen share a website","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"So I am developing a Bot using discord.py and I want to get all permissions the Bot has in a specific Guild. I already have the Guild Object but I don't know how to get the Permissions the Bot has. I already looked through the documentation but couln't find anything in that direction...","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":111,"Q_Id":65571220,"Users Score":1,"Answer":"From a Member object, like guild.me (a Member object similar to Bot.user, essentially a Member object representing your bot), you can get the permissions that member has from the guild_permissions attribute.","Q_Score":0,"Tags":"python,discord.py","A_Id":65571368,"CreationDate":"2021-01-04T23:27:00.000","Title":"discord.py get all permissions a bot has","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"Hello and sorry for this basic question since I am new to programming.\nI have a Sony FCB IP camera that works on Visca protocol. I tried socket programming in python to send visca commands but had no idea what i was doing.\nI need to access the ptz port of my IP camera and start sending visca commands to it through my python code. How can I establish a connection between local system and the camera and ensure that commands are sent through TCP protocol?","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":97,"Q_Id":65574483,"Users Score":0,"Answer":"You can ask for the .dll\/.so from OEM, in case you are stuck. ;)","Q_Score":1,"Tags":"python","A_Id":65590956,"CreationDate":"2021-01-05T07:08:00.000","Title":"Can I access an IP Camera PTZ port through socket programming in Python and send hex bytes to it?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I am building my website which contains a python(.py) file, html, css and JS file. I want to know that how can I run my python script in siteground from my hosting account so that it can scrape data from a site and output a JSON file to Javascript file which can display it on the webpage.","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":448,"Q_Id":65583183,"Users Score":0,"Answer":"I would use cron jobs to run jobs in the foreground","Q_Score":0,"Tags":"python,selenium","A_Id":65584141,"CreationDate":"2021-01-05T16:52:00.000","Title":"How can I run my Python Script in Siteground Hosting Server","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I'm making a python discord bot, and I don't want the bot to care about the command being in lower\/upper case.\nHow can I do that without aliases=['clear', 'Clear']?","AnswerCount":2,"Available Count":1,"Score":0.2913126125,"is_accepted":false,"ViewCount":314,"Q_Id":65605569,"Users Score":3,"Answer":"Use the lower method. \"Clear\".lower() returns \"clear\".","Q_Score":0,"Tags":"python,discord,discord.py,discord.py-rewrite","A_Id":65605599,"CreationDate":"2021-01-07T01:42:00.000","Title":"How to make discord.py bot case-insensitive?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"Let's say you have some proprietary python + selenium script that needs to run daily. If you host them on AWS, Google cloud, Azure, etc. are they allowed to see your script ? What is the best practice to \"hide\" such script when hosted online ?\nAny way to \"obfuscate\" the logic, such as converting python script to binary ?","AnswerCount":2,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":60,"Q_Id":65607541,"Users Score":4,"Answer":"Can the cloud vendors access your script\/source code\/program\/data?\n\nI am not including government\/legal subpoenas in this answer.\nThey own the infrastructure. They govern access. They control security.\nHowever, in the real world there are numerous firewalls in place with auditing, logging and governance. A cloud vendor employee would risk termination and\/or prison time for bypassing these controls.\nSecrets (or rumors) are never secret for long and the valuation of AWS, Google, etc. would vaporize if they violated customer trust.\nTherefore the answer is yes, it is possible but extremely unlikely. Professionally, I trust the cloud vendors with the same respect I give my bank.","Q_Score":0,"Tags":"python,amazon-web-services,selenium,google-cloud-platform,hosting","A_Id":65607689,"CreationDate":"2021-01-07T06:20:00.000","Title":"How do cloud services have access to your hosted script?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":1,"Web Development":0},{"Question":"I am using the Selenium standalone server for a remote web driver. One thing I am trying to figure out is how to start\/stop it effectively. On their documentation, it says\n\"the caller is expected to terminate each session properly, calling either Selenium#stop() or WebDriver#quit.\"\nWhat I am trying to do is figure out how to programmatically close the server, but is that even necessary? In other words, would it be okay to have the server up and running at all times, but to just close the session after each use with something like driver.quit()? Therefore when I'm not using it the server would be up but there would be no sessions.","AnswerCount":2,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":216,"Q_Id":65614348,"Users Score":0,"Answer":"you were right. Use seleniums driver.quit() as it properly closes all browser windows and ends driver's session\/process. Especially the latter is what you want, because you most certainly run the script headless.\nI have a selenium script running on as Raspberry Pi (hourly cron job) headless. That script calls driver.quit() at the end of each iteration. When i do a -ps A (to list al active processes under unix), no active selenium\/python processes are shown anymore.\nHope that satisfies your question!","Q_Score":0,"Tags":"python,selenium,selenium-webdriver","A_Id":65616980,"CreationDate":"2021-01-07T14:34:00.000","Title":"Properly starting\/stopping Selenium standalone server","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I'm trying to connect to my bitbucket using API token generated in Bitbucket but the connection is returning HTTP 401 error when using Python requests module.\nI need help to facilitate the completion of a task.","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":233,"Q_Id":65616081,"Users Score":0,"Answer":"make sure that you use basic authentication and then set your username and use the personal token as password.\nYour username still required since the token associated with it","Q_Score":0,"Tags":"python,api,python-requests,bitbucket","A_Id":65616976,"CreationDate":"2021-01-07T16:17:00.000","Title":"Bitbucket API connection using token","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"Is there a way to find an inspect element without opening the f12 menu?","AnswerCount":1,"Available Count":1,"Score":-0.1973753202,"is_accepted":false,"ViewCount":186,"Q_Id":65635325,"Users Score":-1,"Answer":"In chrome and a few other browsers you can hit \"Ctrl+Shift+I\"","Q_Score":1,"Tags":"python,selenium,web-inspector","A_Id":65635372,"CreationDate":"2021-01-08T19:31:00.000","Title":"How to search for an inspect element without using the F12 menu","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"My code fails when it tries to import requests, despite it already being installed. I did a pip list and saw the module requested there, I uninstalled it and reinstalled it with both pip install and pip3 install, also added sudo both times. Whenever I try to install it I get the message that the requirement is already satisfied. Is there something else that I could try?\nIf it helps I am using VSCode on a Mac, I also have Jupyter and Spyder installed and have used them before however I\u2019ve never used the requests module on this device.\nUPDATE:\nI created a virtualenv and installed requests there, when running the script in the venv I am not getting the error anymore, however I am still curious why it was being thrown on the base env, anything else I could check?","AnswerCount":3,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":1133,"Q_Id":65651546,"Users Score":1,"Answer":"You probably have multiple installations\/environments.\nBefore the \"import requests\", line put \"import sys; print(sys.executable)\".\nThis prints the python executable being used - verify that it is the same one that you can successfully import requests with .","Q_Score":0,"Tags":"python,python-requests,python-module","A_Id":65651608,"CreationDate":"2021-01-10T08:53:00.000","Title":"ImportError: No module named requests but module already exists","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":1,"System Administration and DevOps":0,"Web Development":0},{"Question":"I'm using django-channels. When an exception is raised within a consumer it will log the error, the websocket connection will be disconnected and then it will re-connect again.\nI would like to send a websocket message before it disconnects. I've tried to catch the error, send the message and then re-raise the exception. But the message still isn't sent.\nWhat's the best way of achieving this?","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":460,"Q_Id":65653802,"Users Score":2,"Answer":"When you raise an error it seems like the the actual raising of the error takes precedence of sending the message which happens later.\nSo the solution I went with in the end was to catch the exception in place, append the exception and check whether there were any exceptions to be raised after a message was sent.\nIf there was an error to raise, raise it. That way errors are raised server side and any errors will get known frontend side as well.\nAlternatively, which might be a better solution. Catch the error and then log the exception. Have a special method that sends the error back to frontend and then return early.\nThat way the server will never disconnect and there is no need for re-connection. Which saves some time and processing.","Q_Score":0,"Tags":"python,django,channels","A_Id":65654479,"CreationDate":"2021-01-10T13:29:00.000","Title":"Send a websocket message when exceptions is raised in django channels","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I've got a bluetooth RGB bulb that I want to use for notifications in discord. It seems that following standard guides I would need to get an admin to add a bot account to each discord I wanted this functionality in.\nIs there a way to get access to messages accessible in my main account without getting another account\/bot added? I only need to be able to read out the messages so I can parse them and trigger RGB stuff.\nI would prefer to do this in python if possible but other solutions are fine if need be.","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":59,"Q_Id":65657004,"Users Score":0,"Answer":"Self bots are against discord's TOS.\nYou can try using on_message(message) and using that to parse the message.content","Q_Score":0,"Tags":"python,discord","A_Id":65657053,"CreationDate":"2021-01-10T18:34:00.000","Title":"How to access discord messages from main account instead of bot account","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I have a scraping project I threw together with scrapy in python. Now I've come across certain data that are only loaded onto the page as a user scrolls down. How do I emulate this in my scrapy spiders?","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":68,"Q_Id":65659131,"Users Score":0,"Answer":"Inspect the page (Press F12) in Google Chrome or similar.\nClick on the Network tab.\nScroll down on the page until you the new data is rendered.\nAt the same time the new data is rendered in your browser you should see a new file pop up in the inspection panel.\nThe file could be anything depending on the site but most of the time it's JSON.\nClick on the file in inspection panel and copy the Request URL.\nBack in scrapy you can send a request to this URL the get the dynamically rendered data.","Q_Score":0,"Tags":"python-3.x,web-scraping,scrapy","A_Id":65664542,"CreationDate":"2021-01-10T22:19:00.000","Title":"Scraping data requiring scrolling down with scrapy in python","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"Hi I'm new to scraping data from twitter and I've been working on a project to collect tweets and their replies. I scraped the tweets using twitter API , but I couldn't scrape their replies. Any suggestions ?","AnswerCount":3,"Available Count":2,"Score":0.0,"is_accepted":false,"ViewCount":980,"Q_Id":65718539,"Users Score":0,"Answer":"You can achieve this by:\nYou want to get replies of @barackobama and Tweet Id is 1234 just example.\nSearch tweets having @barackobama is tweet text. then make filter on InrplytweetId=1234\nby this method you can get replies of this tweet.","Q_Score":0,"Tags":"python,twitter","A_Id":65725531,"CreationDate":"2021-01-14T11:58:00.000","Title":"Ideas for scraping reply for each tweet","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"Hi I'm new to scraping data from twitter and I've been working on a project to collect tweets and their replies. I scraped the tweets using twitter API , but I couldn't scrape their replies. Any suggestions ?","AnswerCount":3,"Available Count":2,"Score":0.0665680765,"is_accepted":false,"ViewCount":980,"Q_Id":65718539,"Users Score":1,"Answer":"Scraping is not using API. API is a comfortable method to get data while scraping is doing it manually. If you want to get replies, but the API doesn't provide them, you can use selenium. It is well-known library used for scraping data.","Q_Score":0,"Tags":"python,twitter","A_Id":65718617,"CreationDate":"2021-01-14T11:58:00.000","Title":"Ideas for scraping reply for each tweet","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I just want to invoke a grpc method and exit from the process, I don't need the response back to the client.\nMy use case is like.\nAWS Lambda will invoke only invoke multiple grpc request and exit without waiting for the response","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":456,"Q_Id":65731855,"Users Score":0,"Answer":"gRPC on-the-wire does not support fire-and-forget RPCs. The client is free to ignore the results and the server can respond immediately before actually processing the request.","Q_Score":1,"Tags":"architecture,grpc,grpc-python","A_Id":65816083,"CreationDate":"2021-01-15T07:11:00.000","Title":"Sending grpc Requests without waiting for response?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I am making a bot that is using cogs and has a couple different commands. I want this bot to only reply in one of the two bot commands channels of the server I\u2019m using it on. I have seen that i can use ctx.channel.id = whatever the Id is, but i would prefer the bot to not be able to respond in the channel at all, including to .help commands. I have seen people do this with on_message, but I\u2019m not sure how I would do that with cogs. Any help would be much appreciated. My intended result is basically to have the bot only respond in two channels, the two bot channels that i specify, to any commands including the .help command. Thanks!","AnswerCount":2,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":2916,"Q_Id":65737898,"Users Score":1,"Answer":"The easiest way to do this is not actually via code but via permissions on the server. On your server you should find a role with the same name as your bot, whose permissions (including send messages) you can change for seperate channels.","Q_Score":0,"Tags":"python,python-3.x,discord,discord.py,discord.py-rewrite","A_Id":65740071,"CreationDate":"2021-01-15T14:28:00.000","Title":"How can I restrict a bot to responding in certain channels with discord.py?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I am requesting an API which sometimes gives a string that contain \"*\" characters.\nI have to post the output on discord, where ** makes the text bold.\nI want to see if a string contains any * and if so put a markdown \\ escape character in front of the *.\nHow can I accomplish this?","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":48,"Q_Id":65742637,"Users Score":1,"Answer":"As @Random Davis rightly pointed out in the comments, you can use str.replace(\"*\",\"\\*\")and it will replace all the * occurrence.","Q_Score":0,"Tags":"python,python-3.x","A_Id":65742846,"CreationDate":"2021-01-15T19:45:00.000","Title":"If character in string put escape character \"\\\" in front of the found character","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":1,"System Administration and DevOps":0,"Web Development":0},{"Question":"I am trying to build a telegram bot. In that I need to know if anyone is registered with a email address. I checked the documentation but didn't found any answer. If it is possible with telegram core api please feel free to answer.","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":104,"Q_Id":65746922,"Users Score":0,"Answer":"There is no possibility to register in Telegram using anything else except phone number.\nYou can NOT use email, social auth or any other kind of tokens.\nOnly phone number.","Q_Score":0,"Tags":"python,python-3.x,api,telegram,telegram-bot","A_Id":65761409,"CreationDate":"2021-01-16T06:02:00.000","Title":"How to programmatically check if a email is registered in the Telegram?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I am using turtle files containing biographical information for historical research. Those files are provided by a major library and most of the information in the files is not explicit. While people's professions, for instance, are sometimes stated alongside links to the library's URIs, I only have URIs in the majority of cases. This is why I will need to retrieve the information behind them at some point in my workflow, and I would appreciate some advice.\nI want to use Python's RDFLib for parsing the .ttl files. What is your recommended workflow? Should I read the prefixes I am interested in first, then store the results in .txt (?) and then write a script to retrieve the actual information from the web, replacing the URIs?\nI have also seen that there are ways to convert RDFs directly to CSV, but although CSV is nice to work with, I would get a lot of unwanted \"background noise\" by simply converting all the data.\nWhat would you recommend?","AnswerCount":1,"Available Count":1,"Score":0.1973753202,"is_accepted":false,"ViewCount":160,"Q_Id":65748870,"Users Score":1,"Answer":"RDFlib's all about working with RDF data. If you have RDF data, my suggestion is to do as much RDF-native stuff that you can and then only export to CSV if you want to do something like print tabular results or load into Pandas DataFrames. Of course there are always more than one way to do things, so you could manipulate data in CSV, but RDF, by design, has far more information in it than a CSV file can so when you're manipulating RDF data, you have more things to get hold of.\n\nmost of the information in the files is not explicit\n\nBetter phrased: most of the information is indicated with objects identified by URIs, not given as literal values.\n\nI want to use Python's RDFLib for parsing the .ttl files. What is your recommended workflow? Should I read the prefixes I am interested in first, then store the results in .txt (?) and then write a script to retrieve the actual information from the web, replacing the URIs?\n\nNo! You should store the ttl files you can get and then you may indeed retrieve all the other data referred to by URI but, presumably, that data is also in RDF form so you should download it into the same graph you loaded the initial ttl files in to and then you can have the full graph with links and literal values it it as your disposal to manipulate with SPARQL queries.","Q_Score":1,"Tags":"python-3.x,rdflib,turtle-rdf","A_Id":66113979,"CreationDate":"2021-01-16T10:56:00.000","Title":"Workflow for interpreting linked data in .ttl files with Python RDFLib","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I am trying to create an AWS lambda function that help user collect data from a website (using selenium and headless-chromium). The website requires sms code verification during login so I need to get it from the user and pass it back to the AWS Lambda function\nthe flow will be:\n\nusername & password send to lambda function\nlambda function start, chromium auto login with username & password\nwaiting for sms code from user\nuser enter sms code, code pass to lambda function\nlambda function continue\n\nis it possible to do so? like the input() function when running python locally\nthanks!!\n*first question in stackoverflow! let me know if anything doesn't make sense","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":772,"Q_Id":65762905,"Users Score":0,"Answer":"We need at least two Lambdas.\nSecond Lambda:\n\nTakes OTP & UserId as input.\nWrites a record to DynamoDb with userId and OTP.\n\nFirst Lambda:\n\nLambda is invoked with user Id and password\nStart a selenium browser session.\nLogin and write a record to Dynamo.\nKeep checking Dynamo every second for an entry for userId with OTP in Dynamo.\nSet a timeout and complete Login Process.\n\nMain disadvantage of this approach is we have first Lambda running for entire time of login. But if we can't break the login and OTP process, I don't see another way.\nMay be having a ECS Fargate task instead of Lambda function may save some cost as we can easily run multiple selenium browser sessions in single ECS Task.","Q_Score":1,"Tags":"python,amazon-web-services,selenium,aws-lambda,headless-browser","A_Id":65763580,"CreationDate":"2021-01-17T16:07:00.000","Title":"AWS Lambda: Python user input possible?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"How can I modify a raw Unix timestamp so that it shows that it is 5 hours behind (as an example). I'm looking to do this with a javascript or python. The more lightweight the better. I'm basically looking for how to manually decode a given unix timestamp and change some of its numbers so that it gives me back a unix timestamp showing a different time. It would be even greater if I could automatically adjust it to a users personal time-zone using javascript\/python.","AnswerCount":2,"Available Count":2,"Score":0.0,"is_accepted":false,"ViewCount":56,"Q_Id":65765799,"Users Score":0,"Answer":"Convert to the number of hours you want to offset by to seconds, and then add or subtract it from the Unix timestamp. As far as getting the user's personal time zone, I'm not sure how you would do that without language specific code.","Q_Score":0,"Tags":"javascript,python,timestamp,unix-timestamp","A_Id":65765871,"CreationDate":"2021-01-17T20:50:00.000","Title":"How to manually modify Unix TimeStamp so that its 5 hours behind UTC?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"How can I modify a raw Unix timestamp so that it shows that it is 5 hours behind (as an example). I'm looking to do this with a javascript or python. The more lightweight the better. I'm basically looking for how to manually decode a given unix timestamp and change some of its numbers so that it gives me back a unix timestamp showing a different time. It would be even greater if I could automatically adjust it to a users personal time-zone using javascript\/python.","AnswerCount":2,"Available Count":2,"Score":0.0,"is_accepted":false,"ViewCount":56,"Q_Id":65765799,"Users Score":0,"Answer":"How to choose a time 5 hours numerically smaller\nThis would be relevant for if, for example, you were testing whether the submission of an exam answer is within 5 hours of the exam's start time.\nFor this, just subtract 5 * 3600 * 1000 from the Unix timestamp's numerical value.\nWhat you are actually proposing to do is extremely unwise\nYou seem to be planning to create a Unix timestamp of a different point in time which, when expressed as UTC but with the annotation \"UTC\" deleted, will match the local time display expected by a user who is 5 hours behind UTC. I can see why you are tempted to do this but it is a very bad idea.\n\nUnix Timestamps do not default to be in UTC, they describe a point in time across all of space simultaneously. If you shift the value of a Unix timestamp, it is no longer a Unix timestamp, just as (mass of car minus 50 kg) is no longer the mass of that car. The value is either the mass of a different car that is 50kg lighter, or an incorrect value for the mass of the original car.\n\nUnix timestamps are unambiguous. Once you know that a variable contains a Unix timestamp, you can stop worrying about any if's, but's or maybe's. It is solid and definite. What you are creating is a horrible thing which looks like a Unix timestamp of an timepoint, but it is not. What variable name are you going to give it to prevent confusion? You might give the physical property a new name, such as the goalieTimeStamp, which is distinguished from Unix timestamps by being displaced by 5 hours.\n\nIf a person is 5 hours behind UTC now (in January), that person will likely be a different number of hours behind UTC in summertime. This is a mess.\n\n\nI think you are doing this so that you can display a local time nicely. Choose a different, better, way to achieve this.\nYou should use the localisation system in the relevant language to obtain and display the local time, which will depend not only on the location of the user, but also the time of year. This will also allow you to deal with languages etc, if you need to.\nAnd throughout your code you will have a clear distinction between the timepoint of your event (invariant across space) and how a local user will express that time in their timezone, time of year and language.\nA good library for this in Javascript is moment.js. It is rather heavyweight, but this is because the task is much more heavyweight that it first seems!","Q_Score":0,"Tags":"javascript,python,timestamp,unix-timestamp","A_Id":65766210,"CreationDate":"2021-01-17T20:50:00.000","Title":"How to manually modify Unix TimeStamp so that its 5 hours behind UTC?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I have a Google Sheet with a list of products for an online store, but no prices. Is there a script that will scrape the website and add the prices for each into my Google Sheet? To add to the complexity, the website is only accessible with a login.\nI'm very new to programming so any help would be appreciated.\nThanks","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":39,"Q_Id":65780501,"Users Score":0,"Answer":"There is no such script, but there are libaries that help you scrape data from websites.\nLet's say you pick Selenium.\nThe script would look something like this.\nOpen webbrowser, navigate to your site, login, and then do stuff you need to do. It does exactly what a normal user could do. Click on a element, input text, select an item, etc...","Q_Score":0,"Tags":"javascript,python,web-scraping","A_Id":65780779,"CreationDate":"2021-01-18T18:56:00.000","Title":"Scraping data off a website that requires a login","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I am looking to delete a photo sent by my bot (reply_photo()), I can't find any specific reference to doing so in the documentation, and have tried delete_message(), but don't know how to delete a photo. Is it currently possible?","AnswerCount":3,"Available Count":2,"Score":0.0,"is_accepted":false,"ViewCount":410,"Q_Id":65803490,"Users Score":0,"Answer":"You need to have the chat_id and the message_id of that message sent by bot, then you can delete using context.bot.delete_message(chat_id, message_id).\nNote: Bot cannot delete a message if it was sent more than 48 hours ago.","Q_Score":0,"Tags":"python,telegram,telegram-bot,python-telegram-bot","A_Id":65804336,"CreationDate":"2021-01-20T05:18:00.000","Title":"Telegram Bot delete sent photos?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I am looking to delete a photo sent by my bot (reply_photo()), I can't find any specific reference to doing so in the documentation, and have tried delete_message(), but don't know how to delete a photo. Is it currently possible?","AnswerCount":3,"Available Count":2,"Score":0.0,"is_accepted":false,"ViewCount":410,"Q_Id":65803490,"Users Score":0,"Answer":"It's currently possible in Telegram API, not the Bot API unfortunately. It's a shame :(","Q_Score":0,"Tags":"python,telegram,telegram-bot,python-telegram-bot","A_Id":65803544,"CreationDate":"2021-01-20T05:18:00.000","Title":"Telegram Bot delete sent photos?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I want to create a recommendation system of sorts, and I'm trying to find playlists with a given song in them to see what other songs people listen alongside that. Is there a way to do that using the Spotify API in Python?","AnswerCount":2,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":163,"Q_Id":65820060,"Users Score":1,"Answer":"Nope, there isn't. You could however use the endpoint https:\/\/api.spotify.com\/v1\/recommendationsto get related songs to a song. You give a song, artist and genre to Spotify and they give a playlist with similar songs back to you. That could allow you to achieve what you're trying to do.","Q_Score":1,"Tags":"python,python-3.x,spotify,spotipy","A_Id":65833490,"CreationDate":"2021-01-21T01:43:00.000","Title":"Is there a way to get a list of playlists containing a particular song in Spotipy (Spotify Python Api)","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I built a selenium script on my computer at home, On my home computer it works great. I took it into work and it launches the web driver and then it starts clicking on the wrong buttons(if it makes a difference my home computer is a iMac and my work computer is a HP pc). I've double checked the id tags, name tags and I've even selected the path's thru a full xpath. Nothing changes it keeps clicking the buttons next to the correct ones. I've never seen anything like this. Does anyone have any idea what is going on??\nI can't post my code so I am sorry about that. Any advice would be greatly appreciated.","AnswerCount":1,"Available Count":1,"Score":0.3799489623,"is_accepted":false,"ViewCount":52,"Q_Id":65853252,"Users Score":2,"Answer":"I faced a similar problem. Make sure the browser's zoom level is at 100%","Q_Score":1,"Tags":"python,selenium","A_Id":71697945,"CreationDate":"2021-01-22T21:53:00.000","Title":"Selenium Clicking on different elements on different computers","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I want to curl top news(title, discription, and image) from any news website by using built in (python\/nlp\/machine learning)api. And i want to use that api in php to get all this data. My problem is i have to fetch data from any news site, so to fetch data from multiple sites which api i use...","AnswerCount":1,"Available Count":1,"Score":0.1973753202,"is_accepted":false,"ViewCount":33,"Q_Id":65855855,"Users Score":1,"Answer":"To fetch data from multiple API's with cURL you can add multiple cURL requests. Your question is to ambiguous to answer. Please provide an example of your code as different websites will have different API's and requirements. To get a better answer you need to be more specific.","Q_Score":0,"Tags":"python,php","A_Id":65855915,"CreationDate":"2021-01-23T04:51:00.000","Title":"How to curl data using Python(NLP\/ML etc) built in api in php..?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I want to code a timed ban or mute, but with that I can restart my bot. Is there a nice libary or anyone has an idea to code it?\nThank you very much!\nI code with discordpy cogs","AnswerCount":2,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":75,"Q_Id":65860119,"Users Score":0,"Answer":"If it involves restarting your bot, then you cannot use your RAM to store your data but you need to use your Hard Disk for that. When your bot is running, it is storing its data inside the RAM and that's why you can re-use them while the bot is online. Once it goes offline or gets restarted, all the data are removed from the RAM because the program is shut down.\nTo store those data within the Hard Disk, you need a database. For such small projects, you can use JSON or SQLite. If the project scales, you can move to another SQL like MySQL that will handle a more complex and heavy database.\nTo make a bot that can do a timed message:\n\nYou need to store the data of when the message is going to be sent on your hard disk (database), then use that data to send that message. For example, you want to send \"hello\" in 1 day. That basically means that you want to send it at 8\/7\/2021 6:19 PM (it's 7\/7\/2021 6:19 PM right now). So, you store 8\/7\/2021 6:19 PM as a piece of data of when the bot is going to send the message.\nThen you make the bot compare the current time with the time that you saved on your database. If it is greater, then it will send the message and delete the data from the database.\nYou can use the same technique with timed ban, role and everything else.\n\nFrom a technical standpoint, you can use Discordpy for all the Discord stuff, datetime to check the time, JSON (or SQlite3) for the database.","Q_Score":1,"Tags":"python,discord,discord.py-rewrite","A_Id":68290575,"CreationDate":"2021-01-23T14:07:00.000","Title":"Discordpy timed message or ban or role, WITH bot restart | discordpy","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"Couldn\u2019t think of a solution yet.\nI have a scraper running currently. In the first 80% of one loop (every 3-4 hours), the entire process can be run in a headless server without any output needed as it uses Selenium and BS4.\nHowever for the rest of the remaining 20%, I could not program some clicking and typing actions with Selenium for that specific website. I am assuming that it is because the single page website has many many frames (I might be wrong.)\nSo, to combat this and to get around it, I basically used PyAutoGui to control my mouse and click and enter things in textfields repeatedly. I did this by specifying coordinates for each button.\nHow can I do this on Ubuntu 18.04(the server) but without a monitor? Is there a way to fake a monitor of a certain resolution so the coordinates I select when the server is plugged into a monitor of resolution xxx,yyy still works exactly without issue when i create a fake output of resolution xxx,yyy\nI have an extra monitor but I don\u2019t want it to be running all night and day and letting snooping eyes see (i live in a shared house).\nThanks\nEDIT: I reread this after posting it and sorry if the text seems messy im very tired.\nWhat i mean is that the places where the mouse is supposed to click are determined by coordinates relative to ur monitor. How can i replicate this if the monitor is unplugged?\nSorry again","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":43,"Q_Id":65872167,"Users Score":0,"Answer":"No software solution, but you could use a \"hdmi dummy\" with your current resolution. It emulates a display, so your computer will work like usually.","Q_Score":0,"Tags":"python,linux,selenium,ui-automation","A_Id":65873363,"CreationDate":"2021-01-24T15:03:00.000","Title":"How to fake output to automate UI actions on headless server?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I am making a discord bot in python. At one point in coding the bot, it stopped updating, it kept using old code. Then I realized the bot never stopped. I kicked it and reinvited it, but the problem did not go away.\nDoes anybody know why?","AnswerCount":2,"Available Count":2,"Score":1.2,"is_accepted":true,"ViewCount":121,"Q_Id":65879004,"Users Score":0,"Answer":"Kicking will not fix it, it's a problem some code compliers may have, or just the concurrent running version is still running.\nEditors like sublime can do this problem if the existing program has not been stopped.\nThis can be easily fixed by going back into Discord's developer portal and \"regenerate token\". After this, update with the same new token in your code with running the bot token. This will force terminate any builds running.","Q_Score":0,"Tags":"python,discord,discord.py","A_Id":65879050,"CreationDate":"2021-01-25T04:30:00.000","Title":"Discord.py bot stays active even after code is stopped","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I am making a discord bot in python. At one point in coding the bot, it stopped updating, it kept using old code. Then I realized the bot never stopped. I kicked it and reinvited it, but the problem did not go away.\nDoes anybody know why?","AnswerCount":2,"Available Count":2,"Score":0.0,"is_accepted":false,"ViewCount":121,"Q_Id":65879004,"Users Score":0,"Answer":"It's normal, after about 30 seconds or so, the bot should stop","Q_Score":0,"Tags":"python,discord,discord.py","A_Id":65879030,"CreationDate":"2021-01-25T04:30:00.000","Title":"Discord.py bot stays active even after code is stopped","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"This is a more general question on how the solution could be designed.\nThe goal is to have an API which would take some a text or a list of texts, process it and return transformed ones. How should I approach it so that whenever a text is requested, all the requests that contains list of texts would wait and one text would be processed first?","AnswerCount":2,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":93,"Q_Id":65884372,"Users Score":0,"Answer":"In general you would have a heap (priority queue) of requests in line. Your API can fetch highest-priority requests from this heap.\nIn this particular case, if all you need is strs first and lists second, define your priority function like lambda x: 2 if isinstance(x, str) else 1.\nNote that this does not take into account of potential timeouts.. if you have lots of str requests, the list ones could have to wait a while.","Q_Score":1,"Tags":"python,nginx,flask,celery,scheduled-tasks","A_Id":65884481,"CreationDate":"2021-01-25T12:02:00.000","Title":"Queuing tasks with prioritization rules","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I am faced with a challenge to move json files from AWS s3 bucket to sharepoint.\nIf anyone can share inputs on if this is doable and what's the simplest of approach to accomplish this(thinking python script in AWS lambda).\nThanks in Advance","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":1549,"Q_Id":65922580,"Users Score":0,"Answer":"Using S3 boto API, you can download the files from any bucket to local drive.\nThen using Office365 API, you can upload that local file to Share point. Make sure you check the disk space in local before doing the download option.","Q_Score":0,"Tags":"python,amazon-s3,sharepoint","A_Id":70940346,"CreationDate":"2021-01-27T15:47:00.000","Title":"Approach to move files from s3 to sharepoint","Data Science and Machine Learning":0,"Database and SQL":1,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I am showing some scraped data on my Django website.. The data is changing a couple times every hour so it needs to be updated. I am using Beautiful Soup to scrape the data, then im sending it to the view and passing it in a context dictionary to present it on a website.\nThe problem is that scraping function takes some time to work and because of it the website is not loading until the function does its work. How can I make it load faster? There is no API on the data webstie.","AnswerCount":2,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":32,"Q_Id":65922805,"Users Score":0,"Answer":"It depends of your algorithm, if it is an algorithm that goes through a lot of elements that can be or not updated you can save some attribute for not enter in the element. It's a first thought if you show the algorithm maybe we can give you a better answer :)","Q_Score":0,"Tags":"python,django,web-scraping,beautifulsoup","A_Id":65923047,"CreationDate":"2021-01-27T16:00:00.000","Title":"How to send scraped data to a page but without waiting for the page to load?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I wrote a program that automatically navigates me threw a website, but how do I copy my current URL.\nContext: I am attempting to code a watch2gether bot that automatically creates a watch2gether room","AnswerCount":2,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":30,"Q_Id":65926560,"Users Score":1,"Answer":"I don't know which programming language you use, but in Python3 its simply driver.current_url","Q_Score":0,"Tags":"python-3.x,selenium,discord","A_Id":65926716,"CreationDate":"2021-01-27T20:03:00.000","Title":"How do i copy a URL from searchbar?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"Really weird problem here. I have a Python Application running inside a Docker Container which makes requests in different threads to a http restapi. When I run the Container, I get the error:\nERROR - host not reachable abc on thread abc. Stopping thread because of HTTPConnectionPool(host='corporate.proxy.com', port=111111): Max retries exceeded with url: http:\/\/abc:8080\/xyz (Caused by ProxyError('Cannot connect to proxy.', RemoteDisconnected('Remote end closed connection without response')))\nWhen I log in onto the docker host and make the request with curl, then it works.\nWhen I execute the request inside the docker container (docker exec ....), then it works.\nWhen I start the python interpreter inside the container and make the request with the requests module (like application does it), then it works.\nThe Container is attached to the host network of the docker host machine\nDid anyone had also an issue like this?","AnswerCount":1,"Available Count":1,"Score":0.1973753202,"is_accepted":false,"ViewCount":312,"Q_Id":65933402,"Users Score":1,"Answer":"Thanks to @Tarique and others I've found the solution:\nI've added a startup delay of 30 seconds to the container to connect to the docker host network correctly. Then startet the requests.session. Additionally I removed the http_proxy and https_proxy env var from the container.","Q_Score":1,"Tags":"python,docker,proxy,python-requests","A_Id":65934707,"CreationDate":"2021-01-28T08:22:00.000","Title":"Python Docker Container gets ProxyError, despite I can connect to Server manually","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":1,"Web Development":0},{"Question":"I created an API, deployed it on the server on 8443 port and setup cloudflare's SSL certificates.\nEverything works perfectly, but I got a problem that urls in api-root still have http scheme. Also I set X-Forwarded-Proto $scheme header in nginx.conf. What could be the problem?","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":176,"Q_Id":65935111,"Users Score":2,"Answer":"I solved it. I had to add SECURE_PROXY_SSL_HEADER = ('HTTP_X_FORWARDED_PROTO', 'https') to my settings.py","Q_Score":0,"Tags":"python,django,django-rest-framework","A_Id":65936226,"CreationDate":"2021-01-28T10:20:00.000","Title":"Change urls' scheme from http to https in DRF api-root","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I have built a Socket TCP \/ IP server that listens on a specific port and then, with that data, makes a rest query to another server, and that response is returned through the port where it received it.\nAll Socket server is made in Python 3.8 and works great.\nI need to know how to implement this code from my Socket server to an Azure Functions, so that it provides permanent service?\nI appreciate the goodwill of anyone who can offer an answer.\nThanks Total.","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":68,"Q_Id":65954851,"Users Score":0,"Answer":"Simple answer: you cannot do that. Azure Functions are Event-based (such as an HTTP call). If you need to provide TCP socket, maybe hosting your python code in a container, e.g. Azure Container Instances, might be a good way to go.","Q_Score":0,"Tags":"python,sockets","A_Id":65956535,"CreationDate":"2021-01-29T12:52:00.000","Title":"How to implement a python code to work as Azure Function?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":1,"System Administration and DevOps":0,"Web Development":0},{"Question":"I am using Selenium with Python and want to access the plain HTML source code before it is parsed and the DOM is modified by the browser. I do not want to use \"driver.page_source\" as it is giving me back the DOM after parsing and for example dynamically created elements are included. I know I could do a second request with for example requests but I am looking for a way to extract it without doing an additional request. Any ideas?","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":55,"Q_Id":65973698,"Users Score":0,"Answer":"You can get the plain HTML source by using driver.get(f\"view-source:{url}\"). Then get the body of the source using driver.find_element_by_tag_name('body').text","Q_Score":0,"Tags":"python,selenium,selenium-chromedriver","A_Id":65974783,"CreationDate":"2021-01-30T22:17:00.000","Title":"How to get unparsed HTML source code with Python and Selenium","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I have python function which receives and store live tick data from a server(By API requestes). And another function which fetch the data to candle bars of 1minute and appends it to pandas Data frame. Then i want to call another function which apply some mathematical computations and manages order execution in live market.\nBut i am confused in which method to use between Multi-threading, Multi-processing or AsyncIO. What i want is uninterrupted flow of tick data which receives data in fractions of milliseconds to my system so that i donot miss any realtime data, And at the same time able to manage orders and perform mathematical computations.\nPlease advise me which option will be better to choose from the above?","AnswerCount":1,"Available Count":1,"Score":0.1973753202,"is_accepted":false,"ViewCount":162,"Q_Id":65989090,"Users Score":1,"Answer":"I think you should use multi-threading and\/or asyncIO since getting processes to work together can be a pain in the neck. Since you need to do computations, you should store the data\/use a queue and use a second process to make the calculations,and you can add more math processes if one couldn't catch up with data inflow.\nOn second thought. You'll still have to carefully pack data fast enough so you don't spend most of your time transferring the data(pickle or process Queues don't work 'cause they're too slow)you'll need some custom way, say structs to quickly pack and unpack the data.\nBut at that point, you might as well use C\/C++ as your second(math processing) process :)\nTL;DR: Use ansycio and\/or threading to receive data, custom structs to quickly pack\/unpack data, and a few other Python\/C\/C++ etc. processes to retrieve and process the data.","Q_Score":0,"Tags":"python-3.x,multithreading,time-series,multiprocessing,python-asyncio","A_Id":65989180,"CreationDate":"2021-02-01T08:05:00.000","Title":"Is multi-threading or asyncio suitable for receiving real time tick with API requests and at the same time perform other tasks","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":1,"System Administration and DevOps":0,"Web Development":0},{"Question":"I want to use Python to monitor the complete network, if the route or link goes up\/down will get the notification.\nI found a few packages like lanscan, but I\u2019m not sure that will work fine or not.\nBasically, I want to use python same as NMS (Network management system). Please suggest to me some good frameworks or packages.","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":63,"Q_Id":66022670,"Users Score":0,"Answer":"Pure python-based monitoring solutions are not really scalable as python at the core is pretty slow compared to something like c and multiprocessing is not native. Your best bet would be to use an opensource solution like Zabbix or cacti and use python API to interact with the data","Q_Score":0,"Tags":"python,python-3.x,network-programming,nms,opennms","A_Id":68525850,"CreationDate":"2021-02-03T07:01:00.000","Title":"How to monitor the complete network through Python?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"Is there anywhere in Firebase Auth where you can access public and private RSA keys for users? This would be really helpful for my project instead of having to generate some and store them securely","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":372,"Q_Id":66043713,"Users Score":1,"Answer":"There is nothing built into Firebase Authentication for storing key pairs for the user. You'll typically want to use a secondary data store (like Firebase's Realtime Database, or Cloud Firestore) for that and associate the keys with the user's UID.","Q_Score":0,"Tags":"python,ios,swift,firebase-authentication,rsa","A_Id":66049160,"CreationDate":"2021-02-04T10:25:00.000","Title":"Getting Public \/ Private Keys from Firebase Auth","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I have a dataset with token and entity columns. In the token column there is a word and also a URL. I want to get the number of URL in token column. but I didn't find suitable source code. What I found is a way to remove the URL. Is there a way to calculate the number of URLs in the dataset? How do I calculate the number of URLs in a dataset?","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":43,"Q_Id":66089460,"Users Score":0,"Answer":"I Think you can use the count() fun in the DataFrame class in the Pandas library.\nFirst try importing the Pandas as pd\ncreate a object for Dataframe class\nUse the count() with object with necessary parameters.","Q_Score":0,"Tags":"python,python-3.x","A_Id":66089562,"CreationDate":"2021-02-07T15:12:00.000","Title":"Counting URL in dataset","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"We created a Bot using Python SDK V4. Now we want to create the bot with REST API in Azure.\nAs per Microsoft site's suggestion, we need to create API service in C# or Node.js.\nQuestions\n\nDo the Bot Framework and the REST API need to be in the same language like in Python?\nIf the Bot Framework is in Python language and the Bot API service is in C# will it work? If yes, how will they connect with each other?","AnswerCount":1,"Available Count":1,"Score":0.1973753202,"is_accepted":false,"ViewCount":164,"Q_Id":66119587,"Users Score":1,"Answer":"A REST API receives and responds to HTTP requests. A Microsoft Bot Framework bot is a web app service and can be considered a REST API. There is also the Bot Framework REST API that exists separately from the individual bots, and Bot Framework bots send requests to the Bot Framework REST API in order to send messages to various channels. Any rest API can communicate with any other REST API regardless of what language they're written in. They all use HTTP so the protocol is the same.","Q_Score":0,"Tags":"python,c#,botframework,rest,azure-bot-service","A_Id":66165122,"CreationDate":"2021-02-09T13:05:00.000","Title":"BOT Framework in Python and Bot service is in C# will it work?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":1,"System Administration and DevOps":0,"Web Development":0},{"Question":"I am trying to import paramiko library to AWS Lambda. I have tried to do so on lambda using Python version 2.7, 3.6, 3.8. I upload the zip file (created on ec2 machine using cmd, containing all dependencies) by creating a layer on Lambda function, however it keeps giving me the error-No module named Paramiko. Could you please suggest me how to successfully import paramiko to establish an sftp connection.","AnswerCount":1,"Available Count":1,"Score":0.1973753202,"is_accepted":false,"ViewCount":358,"Q_Id":66124600,"Users Score":1,"Answer":"I have imported paramiko to Lambda Python 3.8 runtime as a layer. The point is you have to pip install it and pack it into a zip file in amznlinux2 x86 EC2 instance with Python3.8 installed. And make sure all content in the zip file is in a folder named python.","Q_Score":0,"Tags":"python,amazon-web-services,aws-lambda,paramiko,pysftp","A_Id":68801883,"CreationDate":"2021-02-09T18:07:00.000","Title":"Problem in importing Paramiko in AWS Lambda","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"Let me describe it as briefly and clearly as possible:\nI have 10 different copies of a node JS based program running on 10 different desktops. I want to create a Node JS based (or any other technology) web app deployed on a server which will check if these 10 programs are online or not.\nAny suggestions as to how I can implement this?\nNote: The node JS based desktop apps are running on electron.","AnswerCount":2,"Available Count":2,"Score":0.0,"is_accepted":false,"ViewCount":38,"Q_Id":66156309,"Users Score":0,"Answer":"You can use 2 most probable ways.\n\nif you want to know immediately whether out of 10 programs, if any of them goes offline then you should use Socket.io\n\n\nYour server nodejs program will act as server and your 10 desktop program will work as client. Your 10 client socket connection will connect with server socket connection and server socket can check whether socket client is still connected or not based on Ping\/Pong concept of socket.\n\nin brief, Ping\/pong technique in which server sends Ping event on socket connection and client will receive server's ping event and send Pong event back to server.\nif client is not sending Pong event back in predefined time interval on getting Ping event then that client is offline or disconnected.\n\nYou can periodically (say every 1\/5\/10 minutes etc ) call simple HTTP request and check if response status is 200 or not. If any of the 10 desktop program is offline then you will know it by response status whether it is 200 or not.","Q_Score":0,"Tags":"javascript,python,node.js,electron,backend","A_Id":68328337,"CreationDate":"2021-02-11T14:08:00.000","Title":"Node JS detect connectivity to all Node JS programs","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"Let me describe it as briefly and clearly as possible:\nI have 10 different copies of a node JS based program running on 10 different desktops. I want to create a Node JS based (or any other technology) web app deployed on a server which will check if these 10 programs are online or not.\nAny suggestions as to how I can implement this?\nNote: The node JS based desktop apps are running on electron.","AnswerCount":2,"Available Count":2,"Score":0.0,"is_accepted":false,"ViewCount":38,"Q_Id":66156309,"Users Score":0,"Answer":"While you can use socket.io for this there may also be a simpler way and that is to just use a post request \/ cron to check every X minutes if the server is reachable from 'Checking' server (that would just be the server that is doing the check)\nSo why not use socket.io? Well, without knowing how you node servers are setup, its hard to say if socket.io would be a good fit, this is simply because socket.io uses WSS to connect, so unless you are running it from the browser it will need additional configurations \/ modules setup on the server to actually use WSS (if you do go this route, you will need socket.io-client module on each system, this is important because this will allow you to connect to the socket.io server, also make sure the version of socket.io matches the socket.io-client build)\nAll in all, if I was building this out, I would probably just do a simple ping of each server and log it to a DB or what not but your requirements will really dictate the direction you go","Q_Score":0,"Tags":"javascript,python,node.js,electron,backend","A_Id":68328746,"CreationDate":"2021-02-11T14:08:00.000","Title":"Node JS detect connectivity to all Node JS programs","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"Couldn't find any answers other than one that didn't actually zoom out the same way, because when i tried grabbing elements they just came out as ' ' when it was too far zoomed out (Which Doesn't Happen With Manual Zoom) the code I tried then is:\n\ndriver.execute_script(\"document.body.style.zoom='25%'\")\n\nThe reason I need this is there are elements a need to access that can only be seen by scrolling, but by zooming out it shows all of them. If there's another way to do that then that'll be fine","AnswerCount":2,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":33,"Q_Id":66165179,"Users Score":0,"Answer":"Figured it out, basically when I was looping through all elements with the classname of the things in the list, I did the command you suggested and it scrolled down a bit each one, showing more of them, then those scroll it down, etc till it reaches the bottom, thanks, stumped me for a while","Q_Score":0,"Tags":"python,selenium","A_Id":66168175,"CreationDate":"2021-02-12T01:10:00.000","Title":"How Do You Zoom In\/Out Using Chromes Zoom","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I have just started with AWS Serverless and I am having some doubts. Here is my use case and what I have tried and done so far:\nUse Case:\nMake multiple GET and POST requests to an API using HTTP API(not REST API) in AWS using lambda function.\nWhat I have done:\nCreated an HTTP API. Using $default stage currently. Created a POST route. Created a function(in python) with POST request. Attached the function integration with my POST route. I am successfully able to call this route using my frontend code(written in vanilla js). Using the data that I receive from frontend, I call an external API using it's URL in my python lambda function.\nProblem:\nI want to make a GET request to another API using it's URL. Will I have to make another lambda function to do so?\nAny help will be great. Pardon me if I have asked a silly question. It's just that I am new to AWS and HTTP API. Thank You for your time!","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":495,"Q_Id":66167053,"Users Score":1,"Answer":"Based on the comments.\nA single lambda function can be used for both POST and GET requests. For this, you can have either two routes, one for POST and one for GET. Both can be integrated with the same function.\nAlternatively, you can have one ANY route to route everything into a single function.\nThe function can have same file and same handler. However, its logic must be probably modified to handle POST and GET events differently, depending on your use-case.","Q_Score":1,"Tags":"python,amazon-web-services,http,aws-lambda,aws-http-api","A_Id":66167317,"CreationDate":"2021-02-12T06:02:00.000","Title":"How to make multiple HTTP method calls in AWS using HTTP API?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"Is it possible to upload and manipulate a photo in the browser with GitHub-pages? The photo doesn't need to be stored else than just for that session.\nPS. I'm new to this area and I am using python to manipulate the photo.","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":25,"Q_Id":66200998,"Users Score":1,"Answer":"GitHub pages allows users to create static HTML sites. This means you have no control over the server which hosts the HTML files - it is essentially a file server.\nEven if you did have full control over the server (e.g. if you hosted your own website), it would not be possible to allow the client to run Python code in the browser since the browser only interprets JavaScript.\nTherefore the most easy solution is to re-write your code in JavaScript.\nFailing this, you could offer a download link to your Python script, and have users trust you enough to run it on their computer.","Q_Score":1,"Tags":"python,github-pages","A_Id":66201055,"CreationDate":"2021-02-14T23:08:00.000","Title":"Can I manipulate an image in the browser with github pages?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I've created Python script which makes a GET request to infrastructure monitoring tool to fetch a json object of problems that occurred in the last 30 days. All these problems have unique ids.\nAfter that, it makes a POST request to push this json object record by record to another API end-point.\nI want to attach this script to cron job to execute it every 5 minutes, but I could not figure out a way to only get or post previously not sent events. So it always fetches a list of problems in the last 30 days and it pushes all of them.\nI thought about writing this json to local file and comparing it with the latest request, but then the previously fetched records also becomes considered new. So I am stuck and couldn't find a similarly asked question. I am open to all suggestions as long as things do not get too complicated :)","AnswerCount":2,"Available Count":2,"Score":0.0,"is_accepted":false,"ViewCount":51,"Q_Id":66215476,"Users Score":0,"Answer":"You need some sort of persistence.\nYou can save a file with the timestamp of the most recent event.\nThen, the next time the script executes, it only POSTs events with a timestamp later than the one in the saved file, and then updates the saved file.\nIf the API to which you're POSTing has a way to fetch the last event you sent to it (NOT the last event it received from any client), you don't even need the file, the API is your persistence.","Q_Score":0,"Tags":"python,python-3.x,api,http,python-requests","A_Id":66218203,"CreationDate":"2021-02-15T21:14:00.000","Title":"Python API GET\/POST Unique JSON Records","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":1,"System Administration and DevOps":0,"Web Development":0},{"Question":"I've created Python script which makes a GET request to infrastructure monitoring tool to fetch a json object of problems that occurred in the last 30 days. All these problems have unique ids.\nAfter that, it makes a POST request to push this json object record by record to another API end-point.\nI want to attach this script to cron job to execute it every 5 minutes, but I could not figure out a way to only get or post previously not sent events. So it always fetches a list of problems in the last 30 days and it pushes all of them.\nI thought about writing this json to local file and comparing it with the latest request, but then the previously fetched records also becomes considered new. So I am stuck and couldn't find a similarly asked question. I am open to all suggestions as long as things do not get too complicated :)","AnswerCount":2,"Available Count":2,"Score":0.0,"is_accepted":false,"ViewCount":51,"Q_Id":66215476,"Users Score":0,"Answer":"Simplest way would be to keep a local file in your preferred format that tracks the unique IDs of all items you've pushed, ever. Load that in with your cron'd script, and then do your pull. Compare each item in the pull against your cached values, and generate a list of new items to push. Add these items to your cached values variable, push the items, and then write the entirety of your cached values out to file. A potentially faster (in terms of I\/O, if your file gets large) way of doing this would be to keep all the values in the file, one per line, and then just open the file with \"append\" mode and write out only the new values to it, one per line.","Q_Score":0,"Tags":"python,python-3.x,api,http,python-requests","A_Id":66215565,"CreationDate":"2021-02-15T21:14:00.000","Title":"Python API GET\/POST Unique JSON Records","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":1,"System Administration and DevOps":0,"Web Development":0},{"Question":"I am trying to build a simple project where I need to access a public website, input a single string of text, and get the output from the result of \"submitting\" the string of text.\nI know this can be done with Selenium but I was wondering if it's possible to do this silently. The webpage does not have an API, it's just a database where you input a single text of string, query the result and display it.\nIs this possible to do at all with Python?, again hopefully silently where when this runs it won't take over the monitor and can potentially and eventually be done on an arduino?\nThanks in advance,","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":31,"Q_Id":66217790,"Users Score":0,"Answer":"Resolution:\nI was able to do this with Selenium using headless (previously: phantom) options.\nWorks like a charm, no UI being invaded. Querying what I need takes 2.5 seconds to get results.","Q_Score":0,"Tags":"python,python-3.x,web-scraping,automation","A_Id":66220715,"CreationDate":"2021-02-16T02:03:00.000","Title":"Py - Possible to Input Text into Web page and Extract Result Silently?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I can log in to a certain website using Postman. Doing so will set a \"bearer-token\" Cookie. When I export this to Python code in Postman, it generates a request that already explicitly supplies the token. However if I just post the login data with requests without giving that token, I get a 403.\nHow does Postman negotiate that token, can you give me a Python snippet that will?","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":59,"Q_Id":66226855,"Users Score":0,"Answer":"Solved this, the login page was redirecting right after login.\nSo in the requests.post\/get function, supply the allow_redirects=True argument and then look for a Set-Cookie entry in the response headers.","Q_Score":0,"Tags":"python-requests,postman","A_Id":66328943,"CreationDate":"2021-02-16T15:00:00.000","Title":"How does Postman get the bearer Token","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I am running a very simple Locust script that is using the standard requests module and Python 3.7.7\nThe error is:\n'in get_adapter\nraise InvalidSchema(\"No connection adapters were found for {!r}\".format(url))\nrequests.exceptions.InvalidSchema: No connection adapters were found for '=https:\/\/.....'\nNot sure where the '=' sign is coming from in the request url?","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":74,"Q_Id":66246334,"Users Score":0,"Answer":"Sorry, found the answer I had mistyped the cmd line for launching locust and had an extra = in there","Q_Score":0,"Tags":"python-3.x,locust","A_Id":66246806,"CreationDate":"2021-02-17T16:44:00.000","Title":"running a simple locust scritp and getting No connection adapters error","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"Newbie here... 2 days into learning this.\nIn a learning management system, there is an element (a plus mark icon) to click which adds a form field upon each click.\u00a0 The goal is to click the icon, which generates a new field, and then put text into the new field.\u00a0 This field does NOT exist when the page loads... it's added dynamically based on the clicking of the icon.\nWhen I try to use \"driver.find_element_by_*\" (have tried ID, Name and xpath), I get an error that it can't be found. I'm assuming it's because it wasn't there when the page loaded. Any way to resolve this?\nBy the way, I've been successful in scripting the login process and navigating through the site to get to this point. So, I have actually learned how to find other elements that are static.\nLet me know if I need to provide more info or a better description.\nThanks,\nBill","AnswerCount":2,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":45,"Q_Id":66250310,"Users Score":0,"Answer":"Apparently I needed to have patience and let something catch up...\nI added:\nimport time\nand then:\ntime.sleep(3)\nafter the click on the icon to add the field. It's working!","Q_Score":0,"Tags":"python,selenium,selenium-chromedriver","A_Id":66250513,"CreationDate":"2021-02-17T21:38:00.000","Title":"Selenium\/Python - Finding Dynamically Created Fields","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"We are using python 3.5.1 and requests 2.25. I am using request.post to get a token. It fails when we ran the first time, the same thing when we ran 2nd time it is running successfully. Did anyone face the same issue before?","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":48,"Q_Id":66264366,"Users Score":0,"Answer":"I found the solution, it is due to dependency packages. When we downgraded the below packages, it started to work\n\nPrevious Packages which are not working: cachetools==4.2.1 google-auth==1.27.0 pytz==2021.1 rsa==4.5\nWorking packages cachetools==4.2.0 rsa==4.6 pytz==2020.4 google-auth==1.23.0","Q_Score":0,"Tags":"python-3.x,python-requests","A_Id":66279038,"CreationDate":"2021-02-18T16:48:00.000","Title":"python3 request.post is failing intermittently","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I'm trying to create a chatbot using gupshup, but I don't have much experience with JS, and for my implementation, it will be easier to code in python, but I'm not finding any material about it.\nIs it possible to develop a ChatBot with python using GupShup?\nThanks","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":199,"Q_Id":66264967,"Users Score":0,"Answer":"Right Now Gupshup Only Provides JS Language Support, but we are looking to have python as a development language also for chatbots.","Q_Score":1,"Tags":"python,gupshup","A_Id":67054515,"CreationDate":"2021-02-18T17:22:00.000","Title":"Using python to create a ChatBot in GupShup Bot Builder","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I am using the discord.ext.commands Bot class. I need to get user info from id's I have in a dictionary, so that I can send them direct messages. I know there is a method to get user info from id using the client class (the .get_user_info() ) function, however I am not using the client class, only the Bot class. Is there a way to get user info using the bot class?","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":81,"Q_Id":66286627,"Users Score":0,"Answer":"Use user = bot.fetch_user(id). After this, you can access all the user's attributes through user, such as user.display_name.","Q_Score":0,"Tags":"python,discord,discord.py","A_Id":66286810,"CreationDate":"2021-02-20T00:13:00.000","Title":"How to get user info given discord id using the discord Bot class?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I tried everything that I found. Tried to connect with extension but it was unsuccessful (I didn't find a ext config). Tried to internal settings (about:config). Tried connect with JS inside chrome.\nCan I just use proxy for entire process (WebDriver)?","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":62,"Q_Id":66313304,"Users Score":0,"Answer":"Chrome cant work with socks5, but you can use it in Firefox with addon (Proxyfoxy)\n\nCreate WEbDrive\nInstall addon\nSetup proxy and select it\nDone!","Q_Score":1,"Tags":"python,python-3.x,selenium-webdriver,socks","A_Id":66313481,"CreationDate":"2021-02-22T09:47:00.000","Title":"How to work with SOCKS5 in selenium Firefox\/Chrome","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I did pip freeze, and found requests, therefore I have requests, but I am getting an error saying\n\nModuleNotFoundError: No module named 'requests'\n\nI just installed Python 3.9.2, and Python 3.8 is still on my computer. (Stating this in case it's a contributing factor to my problem.)\nPlease help!","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":91,"Q_Id":66344764,"Users Score":0,"Answer":"Are you sure requests was installed in the correct python install, you can do python3.9 -mpip list | grep requests to check if requests is installed for python3.9, if it doesn't show up you can use python3.9 -mpip install requests to install it. (the error might be the pip command is the wrong python instances pip command)","Q_Score":0,"Tags":"python","A_Id":66344793,"CreationDate":"2021-02-24T04:34:00.000","Title":"I have requests installed, but I am still getting the ModuleNotFoundError: No module named 'requests' error","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":1,"System Administration and DevOps":0,"Web Development":0},{"Question":"Is there a minimum value for the WebdriverWait in Python Selenium? By default it is set as 0.5s\ndef __init__(self, driver, timeout, poll_frequency=POLL_FREQUENCY, ignored_exceptions=None):\nAlso, is there any substantial drawback to reducing the polling time?","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":927,"Q_Id":66363499,"Users Score":1,"Answer":"Parameter POLL_FREQUENCY is expected to be float which minimal value can be queried with sys.float_info.min. In my system it returns 2.2250738585072014e-308.\nThere is no \"algorithmic\" drawback since WebDriver is actually a REST service and you use it in synchronous way. However too short period would result in to much useless calls to a driver which could impact the performance of your system, introduce noise to your logs, consume more network traffic, etc.","Q_Score":0,"Tags":"python,selenium,selenium-webdriver,webdriverwait","A_Id":66366686,"CreationDate":"2021-02-25T06:34:00.000","Title":"Python Selenium Webdriver Wait Poll Frequency","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"Can I use the discord API in python to harvest messages from a server (not my own server)? Assuming you have an invite link.\nThanks","AnswerCount":3,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":661,"Q_Id":66372939,"Users Score":0,"Answer":"Well, if your using a discord bot, you need them to invite your bot to their server.\nOther than that you could theoretically listen with a bot on your own account but that would be against the discord TOS.","Q_Score":0,"Tags":"python,discord,discord.py","A_Id":66373023,"CreationDate":"2021-02-25T16:53:00.000","Title":"Is it possible to harvest messages from other peoples discord servers?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"When trying to use requests in my function, it only timeout without additional error. I was trying to use rapidapi for amazon. I already have the host and key, but when I test my function it always timeout. My zip file and my function were on the same directory and I knew that my code was correct.","AnswerCount":1,"Available Count":1,"Score":0.3799489623,"is_accepted":false,"ViewCount":55,"Q_Id":66382131,"Users Score":2,"Answer":"I just figured out that my VPC configuration in Lambda was can only access within the resources of the VPC. I just removed the VPC and it now runs. But when your lambda function will connect to your database, you need to add and configure your VPC.","Q_Score":0,"Tags":"python,amazon-web-services,aws-lambda,python-requests","A_Id":66382132,"CreationDate":"2021-02-26T08:03:00.000","Title":"python requests in AWS Lambda Timeout","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I am streaming certain keywords from Twitter API and I wanted to understand what percentage of the total tweets do I get from using the streaming function in Tweepy? (I am using the free developer account - Is there any benefit in terms of volume if I upgrade to a pro account?)\nThanks","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":42,"Q_Id":66396291,"Users Score":0,"Answer":"Approximately 500,000,000 tweets occur every day. The software you refer to limits you to collect only 500,000 requests per month where each request can contain 120 tweets.","Q_Score":0,"Tags":"twitter,tweepy,twitterapi-python","A_Id":66450332,"CreationDate":"2021-02-27T06:43:00.000","Title":"What percentage of tweets do I get from a free twitter API?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I tried to use python's zmq lib. And now I have two questions:\n\nIs there a way to check socket connection state?\nI'd like to know if connection is established after call connect\n\nI want to one-to-one communication model.\nI tried to use PAIR zmq socket type.\nIn that case if one client is already connected, server will not receive any messages from secondary connected client.\nBut I'd like to get info in the second client that there is another client and server is busy.","AnswerCount":1,"Available Count":1,"Score":-0.1973753202,"is_accepted":false,"ViewCount":284,"Q_Id":66405910,"Users Score":-1,"Answer":"You'd get an error if connect fails.\nBut I guess the real question is how often do you want to check this? once at startup, before each message, or periodically, using some heartbeat?\n\nThat does not make sense, as you can not send info without connecting first.\nHowever, some socket types might give some more info.\n\n\nBut the best way would be to use multiple sockets: one for such status information, and another one for sending data.\nZMQ is made to use multiple sockets.","Q_Score":0,"Tags":"python,zeromq","A_Id":66409386,"CreationDate":"2021-02-28T04:48:00.000","Title":"Python zmq connections","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I made a selenium script that would go to a website and buy a certain product that is currently out of stock. I have an email alert that I would get when the product goes back in stock, however when I receive the email I will probably not be at the computer with the script.\nWhat would be the simplest way to run the program from my phone. Would it be to set up a remote desktop to access my computer from my phone? Also, everything would still run correctly if the monitor was off and if the selenium window was in the background, correct?","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":22,"Q_Id":66415039,"Users Score":0,"Answer":"What you should do to make the entire process automated is link the email alert to your program, when the email comes in it will trigger the program to visit the webpage and automatically buy the item.\nI am assuming everything is already saved payment information and other important info etc., and just keeping the programming running in the background on a computer at all times.\nI would obviously encourage this to be hard wired with an ethernet cord too, if there is an issue or someone disconnects your WiFi and you miss the item.\nHave fun!","Q_Score":1,"Tags":"python,selenium","A_Id":66415380,"CreationDate":"2021-02-28T23:48:00.000","Title":"Is there a way to trigger a selenium program remotely? (Like from my phone or from another laptop","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"import deathbycaptcha as dbc\nclient = dbc.SocketClient(username, password)\nDBC version - 0.1.1","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":231,"Q_Id":66417464,"Users Score":0,"Answer":"just uninstall that library and reinstall it with\npip install git+https:\/\/github.com\/codevance\/python-deathbycaptcha.git\nit works for me","Q_Score":1,"Tags":"python-3.x,api","A_Id":67968838,"CreationDate":"2021-03-01T06:16:00.000","Title":"AttributeError: module 'deathbycaptcha' has no attribute 'SocketClient'","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I am trying to crawl a site[URL : https:\/\/www.khaasfood.com\/shop\/]\nFirst I found I have to get categories with a hierarchy.\n.container has a list of li tags that is the parent category.\nEach parent category may have a children's li tag.\nFirst I have to take parent categories. but how?\n'''\nresponse.css('.container li .cat-item')\n'''\nthis code returns all li tags which means bot parent and child category.","AnswerCount":1,"Available Count":1,"Score":0.1973753202,"is_accepted":false,"ViewCount":72,"Q_Id":66419315,"Users Score":1,"Answer":".container > li.cat-item would only select li tags which are the child of .container element\n.container li.cat-item css selector without > will select all descendant li tags","Q_Score":0,"Tags":"python,html,css,scrapy","A_Id":66420986,"CreationDate":"2021-03-01T09:10:00.000","Title":"Scrapy, how to extract parent li only except child li","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I created a twitter bot using the Tweepy module in Python. I've been looking through the documentation for Tweepy and can not seem to find anything related to this. I just need to get the tweet id of tweets that reply on any of my tweets. I think maybe you could use API.search() but there are no parameters related to replies to your own tweet.\nThanks in Advance","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":219,"Q_Id":66448191,"Users Score":1,"Answer":"Use the api to get the most recent tweets from your account. Then, get your tweet ids. Then get tweets that tag you with api.search(q='to:@yourhandle'). Now, for each tweet you searched, you can see if it has the attribute in_reply_to_status_id_str. If it does, you can get the tweet id from there and match with your current tweets.","Q_Score":0,"Tags":"python,twitter,tweepy","A_Id":66450251,"CreationDate":"2021-03-02T22:41:00.000","Title":"Tweepy get reply","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I've been exploring how async works in Python. So far, I've made a few simple MQTT-based async mini-services (get a message, do something, maybe retrieve something, send a message).\nFor my next mini-project, I'm trying to tie Discord and MQTT together. The goal is to have discord messages appear over MQTT, and mqtt messages on discord. I've got an async discord-client object, and an async mqtt-client object. Both work fine, but connecting them is a bit of an issue.\nMy current approach is to have the Discord object be 'leading', while I put the MQTT object in the Discord object (discord-client.mqtt-client = mqtt-client, meaning I can do things like await self.mqtt-client.publish(). This appears to work, so far.\nMy problem is that this approach feels a bit wrong. Is this a normal approach? Are there other approaches?","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":37,"Q_Id":66461157,"Users Score":0,"Answer":"Answering my own question, with help.\nHaving nested async objects works, but 'smells wrong'. It also makes for odd looking (and therefore difficult to maintain) code, and likely introduces weird bugs and such later on.\nA better approach is to make use of asyncio.Queue, where each of the two client-objects has one queue to write to, and one to read from. Like this:\ndiscord-client -> discord-to-mqtt-queue -> mqtt-client\nmqtt-client -> mqtt-to-discord-queue -> discord-client\n(Not the actual naming of the objects)\nThis works, makes sense, and the resulting code is readable.","Q_Score":1,"Tags":"python,asynchronous,python-asyncio","A_Id":66478637,"CreationDate":"2021-03-03T16:31:00.000","Title":"Two async objects interacting","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":1,"System Administration and DevOps":0,"Web Development":0},{"Question":"I try several ways to handle ElementClickInterceptedException\n\nlike wait the driver\nget the focus and click.\ntry to exec a javascript click\n\nDo you have any ideas how to manage this kind of error ?\nI can put a github repo in attachement","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":51,"Q_Id":66463195,"Users Score":0,"Answer":"I simply delete a blocking element in the dom.","Q_Score":0,"Tags":"python-3.x,selenium,webdriver","A_Id":66600218,"CreationDate":"2021-03-03T18:54:00.000","Title":"How to handle ElementClickInterceptedException selenium python","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I'm trying to create a Twitter bot but when I install the tweepy package it gives me the following error:\nModuleNotFoundError: No module named 'tweepy'\nI've tried uninstalling and installing tweepy and it still doesn't work. i'm running python 3.9.2","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":55,"Q_Id":66492918,"Users Score":0,"Answer":"Found a solution: just type pip install tweepy in to the terminal next to the console","Q_Score":0,"Tags":"python-3.x","A_Id":66495391,"CreationDate":"2021-03-05T12:50:00.000","Title":"No module named 'tweepy'","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I am currently using an offline Windows 10 environment and I need to use pip to install a package.\nThe traditional pip install \"myPackage.tar.gz\" is not working because pip makes a network request and fails because my machine has no internet.\nWith this in mind I tried the following command to ignore dependency checking. pip install myPackage.tar.gz -f.\/ --no-index \u2013-no-deps. The command did install \u201cSuccessfully \u201d but when I tried to use the package I got a ModuleNotFoundError.\nMy question is does pip work offline? If not what would be a work around?\nThank you,\nMarco","AnswerCount":1,"Available Count":1,"Score":0.1973753202,"is_accepted":false,"ViewCount":117,"Q_Id":66496944,"Users Score":1,"Answer":"Do you have those packages already downloaded in your computer?\nIf not, at some point you will need an internet connection.\nThere's pip download command which lets you download packages without installing them:\npip download -r requirements.txt\n(In previous versions of pip, this was spelled pip install --download -r requirements.txt.)\nWhen you have your packages downloaded, use pip install --no-index --find-links \/path\/to\/download\/dir\/ -r requirements.txt to install what you have previously downloaded, without the need to access on internet","Q_Score":1,"Tags":"python,python-3.x,windows,pip,package","A_Id":66497037,"CreationDate":"2021-03-05T17:25:00.000","Title":"Does Python pip work in offline environments?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":1,"System Administration and DevOps":0,"Web Development":0},{"Question":"So I want to get all the messages from the telegram servers I'm in and want to display the messages in the terminal.\nI used the telethon python library but the delay is too much takes like 2 to 3 seconds to fetch the message after it has already appeared in the browser.\nThanks.","AnswerCount":1,"Available Count":1,"Score":0.1973753202,"is_accepted":false,"ViewCount":337,"Q_Id":66505327,"Users Score":1,"Answer":"The best way I found is to use the pyrogram library, it displays messages faster than the Telegram App itself which is what I wanted.","Q_Score":1,"Tags":"python,api,python-requests,telegram","A_Id":71085998,"CreationDate":"2021-03-06T11:50:00.000","Title":"How to get telegram messages without any delay from any server using the API or is there a get request that I can use?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I am looking for a method to get information of a \"trend\" regarding some hashtag\/key word on Twitter. Let`s say I want to measure how often the hashtag\/key word \"Python\" is tweeted in time. For instance, today, \"Python\" is tweeted on average every 1 minute but yesterday it was tweeted on average every 2 minutes.\nI have tried various options but I am always bouncing off the twitter API limitations, i.e. if I try to download all tweets for a hashtag during the last (for example) day, only a certain franction of the tweets is downloaded (via tweepy.cursor).\nDo you have any ideas \/ script examples of achieving similar results? Libraries or guides to recommend? I did not find any help searching on the internet. Thank you.","AnswerCount":2,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":388,"Q_Id":66505663,"Users Score":0,"Answer":"Try a library called:\nGetOldTweets or GetOldTweets3\nTwitter Search, and by extension its API, are not meant to be an exhaustive source of tweets. The Twitter Streaming API places a limit of just one week on how far back tweets can be extracted from that match the input parameters. So in order to extract all historical tweets relevant to a set of search parameters for analysis, the Twitter Official API needs to be bypassed and custom libraries that mimic the Twitter Search Engine need to be used.","Q_Score":0,"Tags":"python,web-scraping,twitter,tweepy","A_Id":66505759,"CreationDate":"2021-03-06T12:33:00.000","Title":"Tweets scraping - how to measure tweeting intensity?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I have a long list with urls and some of them are \"parked-free\" by godaddy\nIs there any technical way to recognize such pages without opening them on browser?\nThe page is technically live and\nrequests.head('url').status_code\nreturns 200 so it doesn't help\nTrying to get the content, I only receive \"Enable Javascript...\" message\nI also tried to use some metatags but they are not visible in beautiful soup\nSelenium could probably help, but I would like to avoid it for this specific problem\nIs there any simplier solution for this?","AnswerCount":1,"Available Count":1,"Score":0.3799489623,"is_accepted":false,"ViewCount":186,"Q_Id":66507046,"Users Score":2,"Answer":"If it's just godaddy, you can try either resolving the domain or trying to make a request with random path (say, \/dkfiwifhe). The few domain I tested all resolve to 34.102.136.180, and return HTTP 200 for any path.\nOf course, this can change anytime and likely won't work on other parking sites, sedo resolve to 91.195.241.137, including all the subdomain (godaddy return nxdomain for random subdomain and canonical naked domain for www), but return 403 for any path.\nDepending on how many unique parking sites in your list, you might as well just look up at list of parking sites provider and craft special script for all of them.\nAnother alternative is, some DNS providers allows filtering parked domain, so you can just attempt to resolve against them. Service recommendation is off topic though, so you can just google them yourself.","Q_Score":2,"Tags":"python,beautifulsoup,python-requests,urllib2","A_Id":66507468,"CreationDate":"2021-03-06T14:59:00.000","Title":"How can I recognize a \"parked free\" website?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I have been trying to do some coding with sockets recently and found out that i often get broken pipe errors, especially when working with bad connections.\nIn an attempt to solve this problem I made sure to sleep after every socket operation. It works but it is slow and ugly.\nIs there any other way of making a socket connection more stable?","AnswerCount":2,"Available Count":1,"Score":0.0996679946,"is_accepted":false,"ViewCount":124,"Q_Id":66517476,"Users Score":1,"Answer":"...server and client getting out of sync\n\nBasically you say that your application is buggy. And the way to make the connection more stable is therefor to fix these bugs, not to work around it with some explicit sleep.\nWhile you don't show any code, a common cause of \"getting out of sync\" is the assumption that a send on one side is matched exactly by a recv on the other side. Another common assumption is that send will actually send all data given and recv(n) will receive exactly n bytes of data.\nAll of these assumptions are wrong. TCP is not a message based protocol but a byte stream. Any message semantics need to be explicitly added on top of this byte stream, for example by prefixing messages with a length or by having a unique message separator or by having a fixed message size. And the result of send and recv need to be checked to be sure that all data have been send or all expected data have been received - and if not more send or recv would need to be done until all data are processed.\nAdding some sleep often seems to \"fix\" some of these problems by basically adding \"time\" as a message separator. But it is not a real fix, i.e. it affects performance but it is also not 100% reliable either.","Q_Score":0,"Tags":"python,python-3.x,sockets","A_Id":66519467,"CreationDate":"2021-03-07T14:16:00.000","Title":"Python sockets really unreliable","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":1,"System Administration and DevOps":0,"Web Development":0},{"Question":"I want to know how to get the most recent message sent in a channel of a discord server with discord webhooks in python? I have not tried anything yet.","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":206,"Q_Id":66523058,"Users Score":0,"Answer":"Webhooks are only meant for sending messages, not reading messages in a channel. The only way to get the last message from the channel is if you have a bot user in the server that can read message history in that channel.","Q_Score":0,"Tags":"python,discord,discord.py,webhooks","A_Id":66523831,"CreationDate":"2021-03-08T00:48:00.000","Title":"How to use dhooks to find the most recent message sent in a channel","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I am looking for a way to print out the raw http query string when I use the request library in python3. It's for troubleshooting purposes. Anyone has an idea how to do this. I tried to use prepared requests, but it is not what I'm looking for.Any suggestion?\nthanks","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":77,"Q_Id":66531827,"Users Score":0,"Answer":"import requests\nresponse = requests.get('https:\/\/api.github.com')\nYou can use:\nresponse.text or response.content or response.raw","Q_Score":0,"Tags":"http,python-requests","A_Id":67973369,"CreationDate":"2021-03-08T14:41:00.000","Title":"output raw query string with python requests library","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"Didn't find an answer regarding my particular question, so I'm sorry if this has been asked already.\nI've created a Python program that scrapes the articles posted on news websites by certain keywords. On average, when running it once in the evening, it would be searching through 2000 articles of the day. Now I obviously want this program to run on loop 24\/7 looking for new articles in realtime (or every 5 minutes). When it hits something based on my keywords, I get notified.\nTherefore, I wanted to know whether you guys have any good recommendations on hosting? I've heard about AWS Lambda but wanted to get a second opinion. Anything that costs below -$250 a month is possible :) Maybe someone has a similar project running or can confirm my idea with AWS.\nThanks in advance!","AnswerCount":2,"Available Count":1,"Score":0.0996679946,"is_accepted":false,"ViewCount":526,"Q_Id":66555271,"Users Score":1,"Answer":"Great question, once your script starts do you ever run new scripts or can you just leave the terminal running?\nIn the latter case, you need Amazon ec2, not Lambda. Lambda is for running functions, an Ec2 is the \"cloud computer\" that you are looking for to \"host\" and run your program.\nLook into Ec2, and use EBS or EFB for storage. S3 is good for storing images, or links, or objects, but if you are using an Ec2 instance (cloud computer) and don't need to store your data as an object and don't need to use a dedicated MYSQL or NOSQL database, just store the info in your EBS or EFB. Remember, EBS and EFB are the hard drive of the computer (your ec2), and Amazon RDS is database, Amazon Aurora is inside RDS and is for MYSQL, PostGRESL, and S3 is like a image \/ object drive. For example, if you had an ebook you were going to distribute, you would store your ebook in S3.\nYou can set up an Ec2 and EBS for free too. Just use the free tier and use the t2.micro for ec2 instance. See how it runs for a few days and then go bigger when necessary.","Q_Score":0,"Tags":"python,amazon-web-services,web-scraping,aws-lambda,hosting","A_Id":66556908,"CreationDate":"2021-03-09T21:48:00.000","Title":"Python Web Scraping - How to scrape a News website 24\/7 for new articles?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"How can you tell in general when a website is rendering content in javascript? I typically use bs4 to scrape and I when I can't find a tag, I'm not sure if its because its javascript rendered (which bs4 cant detect) or if I did something wrong.","AnswerCount":1,"Available Count":1,"Score":0.537049567,"is_accepted":false,"ViewCount":130,"Q_Id":66575569,"Users Score":3,"Answer":"Compare the output of your request with the html returned from a browser request. In Chrome and Firefox, press F12 and the console will appear. Under the network tab you can see all the requests that have been made. If the Network tab is empty, refresh the page. The response from the first request in the Network tab should match the response you recieved from the Python request. If it doesn't match, either your request differs from the browser request, or javascript is doing some post processing.\nSubsequent requests in the Network tab may be from javascript running, from iframes, images, or much more.","Q_Score":3,"Tags":"python-3.x,web-scraping,beautifulsoup","A_Id":66575613,"CreationDate":"2021-03-11T02:16:00.000","Title":"Web scraping: How to tell in general if a page has content rendered in javascript","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"So I'm making a discord bot, and I want it to be able to obtain the guild ID of the guild where a command was issued. How would I go about implementing this? I tried using bot.guilds, but I need a method of nailing down the exact id.","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":43,"Q_Id":66575715,"Users Score":1,"Answer":"Have you tried doing message.guild.id ?\nThat should return the guild id for the guild the command was executed in.","Q_Score":0,"Tags":"python-3.x,discord.py","A_Id":66576105,"CreationDate":"2021-03-11T02:39:00.000","Title":"Getting current guild id for instance","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I want to know how I can send messages in discord, without creating a bot.\nLike I want the program to send messages through my own account. Most of the results I got when I searched this up is to create a bot. But I would like to know if there's a way to do it without creating the bot. Thanks :)","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":90,"Q_Id":66577819,"Users Score":0,"Answer":"To access discord with a bot trough your own account, you can't use a discord bot. What you could do, is to automate \"your input\" in discord. Imagine a google sheet for example and now recording your input to copy the first line, delete it afterwards, then paste it in discord and send the message. now you could repeat this for every line in the file.\n(You can find such program using google)\nBUT this solution restricts you to your input. Any events discord provides like on_member_join for example aren't useable for this approach. It's more a user bot than a discord bot","Q_Score":0,"Tags":"python,python-3.x,list,discord,bots","A_Id":66578384,"CreationDate":"2021-03-11T07:02:00.000","Title":"How to send messages in discord without using bot application","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I made a telegram bot using telepot, one of the issues I have is that groups can still invite and make the bot join their channels, even with \/setjoingroups disabled. Is there a way to list these groups and leave them from the code or from @BotFather ?\nthanks","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":595,"Q_Id":66577829,"Users Score":0,"Answer":"You can look bot settings inline mode and group privacy","Q_Score":0,"Tags":"python,bots,telegram,telepot","A_Id":67948892,"CreationDate":"2021-03-11T07:04:00.000","Title":"Leaving group channels as a bot","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I want to check if a specific key has a specific value in a dynamodb table with\/without retrieving the entire item. Is there a way to do this in Python using boto3?\nNote: I am looking to match a sort key with its value and check if that specific key value pair exists in the table,","AnswerCount":2,"Available Count":2,"Score":1.2,"is_accepted":true,"ViewCount":1886,"Q_Id":66592148,"Users Score":1,"Answer":"It sounds like you want to fetch an item by it's sort key alone. While this is possible with the scan operation, it's not ideal.\nDynamoDB gives us three ways to fetch data: getItem, query and scan.\nThe getItem operation allows you to fetch a single using it's primary key. The query operation can fetch multiple items within the same partition, but requires you to specify the partition key (and optionally the sort key). The scan operation lets you fetch items by specifying any attribute.\nTherefore, if you want to fetch data form DynamoDB without using the full primary key or partition key, you can use the scan operation. However, be careful when using scan. From the docs:\n\nThe Scan operation returns one or more items and item attributes by accessing every item in a table or a secondary index.\n\nThe scan operation can be horribly inefficient if not used carefully. If you find yourself using scans frequently in your application or in a highly trafficked area of your app, you probably want to reorganize your data model.","Q_Score":0,"Tags":"python,amazon-dynamodb","A_Id":66593453,"CreationDate":"2021-03-11T23:41:00.000","Title":"Is there a way to check if a key=value exists in a DynamoDB table?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I want to check if a specific key has a specific value in a dynamodb table with\/without retrieving the entire item. Is there a way to do this in Python using boto3?\nNote: I am looking to match a sort key with its value and check if that specific key value pair exists in the table,","AnswerCount":2,"Available Count":2,"Score":0.0,"is_accepted":false,"ViewCount":1886,"Q_Id":66592148,"Users Score":0,"Answer":"What Seth said is 100% accurate, however, if you can add a GSI you can use the query option on the GSI. You could create a GSI that is just the value of the sort key, allowing you to query for records that match that sort key. You can even use the same field, and if you don't need any of the data you can just project the keys, keeping the cost relatively low.","Q_Score":0,"Tags":"python,amazon-dynamodb","A_Id":66608130,"CreationDate":"2021-03-11T23:41:00.000","Title":"Is there a way to check if a key=value exists in a DynamoDB table?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I am building a web scraper that has to retrieve quickly the text of a web page, from HTML only. I'm using Python, requests and BeautifulSoup.\nI would like to detect if the web page content is pure HTML or if it's rendered from Javascript. In this last case, I would just return an error message saying that this cannot be done.\nI know about headless browsers to render the Javascript but in this case I really just need to detect it the fastest way possible without having to render it.\nIt's not really possible to detect script tag as there are many in every webpage and that doesn't mean the text content is rendered in Javascript necessarily.\nIs there something I could check jn the HTML that tells me accurately that the body content will be rendered from Javascript?\nThank you","AnswerCount":1,"Available Count":1,"Score":0.1973753202,"is_accepted":false,"ViewCount":43,"Q_Id":66598057,"Users Score":1,"Answer":"There is nothing in the initial DOM that shows beforehand that the site is rendered with js. These are some stuff you could try:\n\nAnalyzing several websites and make a guess on where the site\nis rendered with js based on the page's content size.\nYou could also get the html of different pages of the site\nand compare the content length (for a js-rendered site, the contents\nof different pages are likely to be the same\/similar before any code is executed).\nCheck the content size of the scripts or detect the scripts names of\nfamous technologies like react, vue and angular","Q_Score":0,"Tags":"javascript,python,html,web-scraping,python-requests","A_Id":66615160,"CreationDate":"2021-03-12T10:18:00.000","Title":"How do I detect if a web page is rendered dynamically from Javascript in Python?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I am building a Python 3.6 application which distributes specific jobs between available nodes over network. There is one server which builds jobs. Clients connect to the server and are assigned a job, which they return as finished after computation completes.\nA job consists of a dict object with instructions, which can get kind of large (> 65536 bytes, probably < 30 MB).\nIn my first attempt I used the Twisted library to exchange messages via a basic Protocol derived from twisted.internet.protocol. When sending a serialized object using self.transport.write() and receiving it on the other hand over the callback function dataReceived() only 65536 bytes are received. Probably that's the buffer size.\nIs there a \"simple\" protocol which allows me to exchange larger messages between a server and multiple clients in Python 3.6, without adding too much coding overhead?\nThanks a lot in advance!","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":68,"Q_Id":66602702,"Users Score":0,"Answer":"Finally I used websockets. It works like a charm, even for large messages.","Q_Score":0,"Tags":"python,python-3.x,networking,tcp,twisted","A_Id":67690254,"CreationDate":"2021-03-12T15:25:00.000","Title":"Simple network protocol to send large dict\/JSON messages between python nodes","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":1,"System Administration and DevOps":1,"Web Development":0},{"Question":"I'm new to application development and decided to use AWS services for this project. however, I am having difficulty deploying chalice. every time I run \"chalice deploy\", I get an error.\nhere are the steps I followed along with commands for Windows:\n\nupgraded my powershell\n\"virtualenv enve\" : then \".\\venv\\Scripts\\activate\" # install and run virtual environment\n\"pip install aws cli\" : # install aws command line interface\n\"aws configure\" : # configure my AWS_KEY and AWS_SCERET\n\"pip install chalice\" : # install chalice\n\"chalice new-project\": # created a new project\n\"chalice deploy\" # deploy\n\nI get\n\nAn error occurred (InvalidClientTokenId) when calling the GetRole\noperation: The security token included in the request is invalid.\n\nI'm able to use localhost and run my application but not able to deploy to the server. I don't know what i'm doing wrong. someone, please help!\nadditional info:\nmy operating system is windows 10. I upgraded my PowerShell to 7","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":118,"Q_Id":66605637,"Users Score":0,"Answer":"I somehow figured it out!. The error occurred because the command \"\n\nchalice deploy\n\n\" was used in the wrong directory. Make sure you are in the directory where your chalice file is before initializing it to deploy.","Q_Score":0,"Tags":"python,amazon-web-services,amazon-ec2,aws-lambda,serverless","A_Id":67134224,"CreationDate":"2021-03-12T18:51:00.000","Title":"how to solve \"An error occurred (InvalidClientTokenId)\" AWS Chalice deployment error","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"Am new to fastapi. I have worked with multiple web frameworks in other languages and found the common pattern of middlewares for various purposes. e.g. If I have an API route that I want to authenticate then I would use a middleware that does the authentication. If I want to augment the incoming request I would use a middleware. FastAPI does have middlewares (A very small section in docs) but also has dependencies. I was looking to authenticate my API routes and started looking for examples and all the examples I find use dependencies. What (dependency or middleware) would be recommended way to authenticate an API route and why?","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":3124,"Q_Id":66632841,"Users Score":13,"Answer":"The way I see it:\n\nDependency: you use it to run code for preparing variables, authentication and so on.\nMiddleware: you need to check some stuff first and reject or forward the request to your logic.\n\nThe middleware can be seen as a superset of a Dependency, as the latter is a sort of middleware that returns a value which can be used in the request. Though, in the middleware, you can log your requests or cache the results and access the response of the request (or even forward the request, call some other API and so on).\nTL;DR\nA Dependency is a sort of common logic that is required before processing the request (e.g. I need the user id associated to this token), while a Middleware can do that, it can also access the response to that request. Dependencies are the preferred way to create a middleware for authentication","Q_Score":9,"Tags":"python,fastapi","A_Id":66634433,"CreationDate":"2021-03-15T05:34:00.000","Title":"fastapi dependency vs middleware","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"Created a Flash Restful API with various end points for my website. Some endpoints need the users username and password to get user specific data. I\u2019m currently sending these as parameters in the API call but I\u2019m assuming this isn\u2019t secure so how does one do this safely?","AnswerCount":2,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":71,"Q_Id":66639116,"Users Score":0,"Answer":"you can make a seperate api route that acts as a login and returns a sessionID\/token on a successful login that can be used for authenticating to those endpoints you mentioned.","Q_Score":0,"Tags":"python,api,flask-restful","A_Id":66639816,"CreationDate":"2021-03-15T13:40:00.000","Title":"How to safely send a user data in API call","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"After a click, a mini banner (or a container drawer) opens, which I should click on it, but I can't interact with it.\nI was trying the \"driver.switchTo\" command but since there is no Iframe, I don't know how to do it.\nthe body is this:\n <a class=\"option\"> Go <\/a>","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":93,"Q_Id":66645314,"Users Score":1,"Answer":"I've had issues with this in the past, and largely I couldn't figure out what to do either. If you can just leave your computer alone during the process you can always try to use pyautogui to kind of cheat your way through it.","Q_Score":1,"Tags":"python,selenium,switch-statement","A_Id":66645596,"CreationDate":"2021-03-15T20:36:00.000","Title":"Switch to panel - Python Selenium","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"What is the password for the command \"safaridriver --enable\"?\nI tried to do the command \"safaridriver --enable\" in the terminal my MAC asks me the password. I put the password for sudo command but it doesn't seem to be correct.","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":84,"Q_Id":66652103,"Users Score":0,"Answer":"If youre using Safari 10 the command \"safaridriver --enable\" wont work, there is no known workaround, you could upgrade a higher version. Using \"safaridriver --enable\" should work there.","Q_Score":1,"Tags":"python,selenium,safari","A_Id":66652198,"CreationDate":"2021-03-16T09:10:00.000","Title":"What is password for command \"safaridriver --enable\"","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I m trying to find out, how i can run and control different instances of Firefox using selenium in python. I m using the geckodriver, to run Firefox. I know, that i can have multiple Profiles in Firefox and that i can run a new Profile seperatly under about:profiles. This would solve my problem, if i knew how i can switch between the windows coresponding to the different profiles. I d be happy, if someone could tell me, how to do that and even more, if someone knows of a threat, where i can read up on it. Thanks for your time.","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":105,"Q_Id":66658967,"Users Score":0,"Answer":"Thanks for the answere. I ll read into it more. For now i ve found a simpler solution, to my problem. I can say driver1 = webdriver.Firefox(PATH) and driver2 = webdriver.Firefox(PATH) and then control them separately. Sorry that i bothered when it was so easy and thanks for the help.","Q_Score":0,"Tags":"python,selenium","A_Id":66664866,"CreationDate":"2021-03-16T16:05:00.000","Title":"Selenium python, multiple geckodriver instances","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I am building an application that uses two legged authentication. I got an API key and API Secret, but now I am confused.\nI am currently storing my api keys and secrets in a .yml file. But I would like to distribute the .app code, which will end up having the .yml file.\nBut the .app file will contain the .yml, file, which is bad since everyone will be able to see the API key and Secret.\nHow can I store the API key and Secret such that my application can access the key and secret without the users seeing it?","AnswerCount":2,"Available Count":2,"Score":1.2,"is_accepted":true,"ViewCount":578,"Q_Id":66660180,"Users Score":1,"Answer":"The answer depends on a few variables:\n\nIs your source included?\nIs it possible to use a server to call the API for you? If so, can you also apply restrictions to the call that the server makes?\nIs using compiled code for where you store the key an option? If so, is it possible to obfuscate it?\n\nHere are my suggestions for different scenarios from experience:\nThe source is not included and using a server is an option, and restrictions can be applied, however using compiled code is not an argument\nThen use a server to make requests. Let's say you need to make a call to example.com\/api\/v1, and you want to call a specific function with a specific set of arguments, then you can only allow requests to that specific API, with that specific set of arguments, and that specific function. This way, it means nothing to a potential attacker since it only calls to one function and nothing else.\nThe source is not included, using a server is not an option, and compiled code is not an option either\nWell, there's not much you can do, obfuscation is your best shot. The best way to do something like this is to hide it deep within your code, and make it obscure, etc., etc., etc.,\nThe source is included, using a server is not an option, but you can use compiled code\nUse really obfuscated assembly and don't share the source for that if you can. For instance, you can have red herring instructions, and just like before, you should hide it deep in your code.\nThe source is not included, using a server is not an option, but you can use compiled code\nFor this it's the same as above, since the source for the assembly wouldn't be included\nIf I didn't list your scenario here, then feel free to comment and I'll edit my answer","Q_Score":4,"Tags":"python,api,api-key","A_Id":67026216,"CreationDate":"2021-03-16T17:19:00.000","Title":"How to secure API keys in applications that will be distributed to clients","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I am building an application that uses two legged authentication. I got an API key and API Secret, but now I am confused.\nI am currently storing my api keys and secrets in a .yml file. But I would like to distribute the .app code, which will end up having the .yml file.\nBut the .app file will contain the .yml, file, which is bad since everyone will be able to see the API key and Secret.\nHow can I store the API key and Secret such that my application can access the key and secret without the users seeing it?","AnswerCount":2,"Available Count":2,"Score":0.0,"is_accepted":false,"ViewCount":578,"Q_Id":66660180,"Users Score":0,"Answer":"While I consider the existing answer technically correct, it may be wroth pointing out that there are some security issues with hardcoding api keys in distributed software.\nThe nature of an API key is volatile, it is not designed to last forever.\nWhat would happen if the API key is invalidated? Wouldn't that render all distributed software useless then?\nAnd what would happen if the API key has write privileges and is compromised? How could you distinguish between legit and malicious writes?\nEven though I understand the overhead, a scenario were the end user can set dedicated keys, obtained by the end user itself, and a way to replace that initial key, would help with above two questions.\nOne of the API Key features is to be used by a machine that acts on behalf of a user, but if all the users are the same, this feature becomes meaningless.","Q_Score":4,"Tags":"python,api,api-key","A_Id":67091736,"CreationDate":"2021-03-16T17:19:00.000","Title":"How to secure API keys in applications that will be distributed to clients","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I have been playing around with django channels + angular. I have created an app that simply sends notifications to front end with a counter 1,2,3,4. It works fine, except if I open the page in multiple tabs. I am also not able to disconnect from the websocket, I can use unsubscribe but it does not really close the connection but that's more kind of a angular question. Anyways how can I make my socket multithread, So if I make multiple requests from the same computer but from different tabs it will work and 2 different instances of the consumer will be created therefore, If I load 2 pages in the same computer the counter should be different independently increasing counter. Do I need redis for that ?","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":465,"Q_Id":66662639,"Users Score":0,"Answer":"my url router was missing .as_asgi()\nthis worked:\nURLRouter([path('wscrawpredict', CrawPredictConsume.as_asgi(),name=\"wscraw\")])","Q_Score":0,"Tags":"python,django,angular,websocket,django-channels","A_Id":66829614,"CreationDate":"2021-03-16T20:08:00.000","Title":"multiple independent websockets connections","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I'm making a simple project where I will have a downloadable scraper on an HTML website. The scraper is made in Python and is converted to a .exe file for downloading purposes. Inside the python code, however, I included a Google app password to an email account, because the scraper sends an email and I need the server to login with an available Google account. Whilst .exe files are hard to get source code for, I've seen that there are ways to do so, and I'm wondering, how could I make it so that anyone who has downloaded the scraper.exe file cannot see the email login details that I will be using to send them an email when the scraper needs to? If possible, maybe even block them from accessing any of the .exe source code or bytecode altogether? I'm using the Python libraries bs4 and requests.\nAdditionally, this is off-topic, however, as it is my first time developing a downloadable file, even whilst converting the Python file to a .exe file, my antivirus picked it up as a suspicious file. This is like a 50 line web scraper and obviously doesn't have any malicious code within it. How can I make the code be less suspicious to antivirus programs?","AnswerCount":2,"Available Count":2,"Score":0.0,"is_accepted":false,"ViewCount":91,"Q_Id":66672822,"Users Score":0,"Answer":"Firstly, why is it even sending them an email? Since they'll be running the .exe, it can pop up a window and offer to save the file. If an email must be sent, it can be from the user's gmail rather than yours.\n\nSecondly, using your gmail account in this way may be against the terms of service. You could get your account suspended, and it may technically be a felony in the US. Consult a lawyer if this is a concern.\n\nTo your question, there's basically no way to obfuscate the password that will be more than a mild annoyance to anyone with the least interest. At the end of the day, (a) the script runs under the control of the user, potentially in a VM or a container, potentially with network communications captured; and (b) at some point it has to decrypt and send the password. Decoding and following either the script, or the network communications that it makes will be relatively straightforward for anyone who wants to put in quite modest effort.","Q_Score":1,"Tags":"python,html,download,exe,source-code-protection","A_Id":66673330,"CreationDate":"2021-03-17T12:08:00.000","Title":"Python exe - how can I restrict viewing source and byte code?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I'm making a simple project where I will have a downloadable scraper on an HTML website. The scraper is made in Python and is converted to a .exe file for downloading purposes. Inside the python code, however, I included a Google app password to an email account, because the scraper sends an email and I need the server to login with an available Google account. Whilst .exe files are hard to get source code for, I've seen that there are ways to do so, and I'm wondering, how could I make it so that anyone who has downloaded the scraper.exe file cannot see the email login details that I will be using to send them an email when the scraper needs to? If possible, maybe even block them from accessing any of the .exe source code or bytecode altogether? I'm using the Python libraries bs4 and requests.\nAdditionally, this is off-topic, however, as it is my first time developing a downloadable file, even whilst converting the Python file to a .exe file, my antivirus picked it up as a suspicious file. This is like a 50 line web scraper and obviously doesn't have any malicious code within it. How can I make the code be less suspicious to antivirus programs?","AnswerCount":2,"Available Count":2,"Score":1.2,"is_accepted":true,"ViewCount":91,"Q_Id":66672822,"Users Score":1,"Answer":"Sadly even today,there is no perfect solution to this problem.\n\nThe ideal usecase is to provide this secret_password from web application,but in your case seems unlikelly since you are building a rather small desktop app.\nThe best and easiest way is to create a function providing this secret_password in a separate file,and compile this file with Cython,thing that will obcufate your script(and your secret_password) at a very good extend.Will this protect you from lets say Anonymous or a state security agency?No.Here comes the reasonable thinking about how secret and important really your password is and from who you mainly can be harmed.\nFinally before compiling you can 'salt' your script or further obscufate it with bcrypt or other libaries.\n\nAs for your second question antiviruses and specifically windows don't like programms running without installers and unsigned.\nYou can use inno setup to create a real life program installer.\nIf you want to deal with UAC or other issues related to unsigned programms you can sign your programm(will cost money).","Q_Score":1,"Tags":"python,html,download,exe,source-code-protection","A_Id":66673053,"CreationDate":"2021-03-17T12:08:00.000","Title":"Python exe - how can I restrict viewing source and byte code?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I am using a Bing Maps key in the geopy.geocoder.Bing package to get latitude and longitude of addresses in a dataset. It has been working just fine for the last 2 months and then suddenly today I am getting this GeocoderAuthenticationFailure: Non-successful status code 401 error. I am unsure if it is a problem with the geopy function itself or if there is something broken with the API. I just looked at the key in my Bing Maps account and everything looks fine. I created a new key to see if that would work but it gave me the same error. Does anyone know if this is an error with the API or if there is something I can do to fix this?","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":135,"Q_Id":66698879,"Users Score":0,"Answer":"The error code came from a typo when calling the apikey variable from my env file. After fixing the typo it ran just fine.","Q_Score":0,"Tags":"python,api,bing-maps,geopy,geocoder","A_Id":66699490,"CreationDate":"2021-03-18T20:46:00.000","Title":"GeocoderAuthenticationFailure: Non-successful status code 401","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I want to do a connection between a computer simulating being a server and another computer being a user, both with Linux.\nIn the first computer I've created a directory called \"server\" and in that directory I've done the following command:\npython3 -m http.server 8080\nThen I can see that directory going to the localhost. But what I want is to see that localhost from the other computer, I tried with wget, and the gnome system of sharing files but none of them worked, and I'm not seeing any solution online.","AnswerCount":3,"Available Count":2,"Score":0.0,"is_accepted":false,"ViewCount":968,"Q_Id":66708739,"Users Score":0,"Answer":"I'm not sure I fully understand your question but if you want to reach your folder from an other computer with your command python3, you can use the option -b followed by an IP address used on your linux.","Q_Score":0,"Tags":"python,html,linux,networking,server","A_Id":66708913,"CreationDate":"2021-03-19T13:03:00.000","Title":"How to connect to a Localhost in linux","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":1,"Web Development":0},{"Question":"I want to do a connection between a computer simulating being a server and another computer being a user, both with Linux.\nIn the first computer I've created a directory called \"server\" and in that directory I've done the following command:\npython3 -m http.server 8080\nThen I can see that directory going to the localhost. But what I want is to see that localhost from the other computer, I tried with wget, and the gnome system of sharing files but none of them worked, and I'm not seeing any solution online.","AnswerCount":3,"Available Count":2,"Score":0.0,"is_accepted":false,"ViewCount":968,"Q_Id":66708739,"Users Score":0,"Answer":"If I'm understanding your question correctly you want to connect to the server from another machine on the same network.\nIf so you can run hostname -I on the server to output to the local IP address of the server, you can then use this on the other machine to connect to it (provided they are on the same network)","Q_Score":0,"Tags":"python,html,linux,networking,server","A_Id":66709470,"CreationDate":"2021-03-19T13:03:00.000","Title":"How to connect to a Localhost in linux","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":1,"Web Development":0},{"Question":"when I use browser.find_element_by_css_selector(\".LkLjZd ScJHi OzU4dc \").click(),\nMessage: no such element: Unable to locate element: {\"method\":\"css selector\",\"selector\":\".LkLjZd ScJHi OzU4dc \"}\nThis error occurs despite of element is that <button class=\"LkLjZd ScJHi OzU4dc \" jsaction=\"click:TiglPc\" jsname=\"gxjVle\" style=\"font-family: Roboto, "Noto Sans KR", sans-serif;\">total review<\/button>\nhow to click the butten....","AnswerCount":2,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":81,"Q_Id":66721294,"Users Score":0,"Answer":"you can use the xpath of the element.\n\nInstall xpath Helper in your chrome.\nRefresh the page or close it and enter again.\nClick on addon from the toolbar.\nShift + click the element you needed.\nThan browser.find_element_by_xpath()\n\nPS: If it's a long string you can cut the start of it and replace with another \/.","Q_Score":0,"Tags":"python,selenium,selenium-webdriver,css-selectors","A_Id":66721356,"CreationDate":"2021-03-20T11:58:00.000","Title":"Is it possible to click button class in Selenium Python?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"is it possible to call a web service from cooja? May be I can read from border-router then call web service (via python script for example). I can ping border-router but I dont know how to read from node or write to node in cooja.I am new to contiki-ng and cooja. thanks in advance","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":59,"Q_Id":66732962,"Users Score":0,"Answer":"You can try the websense example at the folder ~\/contiki\/examples\/ipv6\/sky-websense","Q_Score":0,"Tags":"python,web-services,serial-communication,cooja,contiki-ng","A_Id":67021154,"CreationDate":"2021-03-21T13:35:00.000","Title":"is it possible to call a web service from cooja?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":1,"Web Development":0},{"Question":"I want to write a Bot that changes the color of a Role on a server every 1 minute. What should I do?\nThe bot is supposed to do this only on one server","AnswerCount":2,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":95,"Q_Id":66763781,"Users Score":0,"Answer":"i think you can do this by background task, but i think it's not good idea to do that role :\/ and maybe your bot will be get ratelimit from discord","Q_Score":0,"Tags":"python-3.x,discord,discord.py","A_Id":66766334,"CreationDate":"2021-03-23T13:07:00.000","Title":"Discord Bot RGB Role Python","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I'm using docker-compose to run a python API and a Localstack instance in 2 separate containers, for local development.\nThe API has an endpoint which generates a presigned AWS S3 URL and redirects the user, in order to load images directly from S3.\nIn local development, the API instantiates a boto3 client using the address of the localstack container as a custom endpoint url (ie: boto3.client(\"s3\",endpoint_url=\"http:\/\/localstack:4566\")) which allows the API to access resources within the localstack container.\nThe problem is that the presigned URL returned by the boto3 client uses the localstack address, and the browser cannot load it, since the localstack resources are exposed to the host machine, at http:\/\/localhost:4566.\nIf I try to set the aws resources endpoint url to localhost in the boto3 client instantiation, then the API, which is running inside of a container, will look for AWS resources within it OWN CONTAINER's localhost, and not the host machine, where the localstack resources are exposed.\nIs there any way to access localstack resources, running in a docker container, from both the host machine's browser AND a different container, using the same address?\n[Edit] I'm using docker on mac, in case that changes anything [\/Edit]","AnswerCount":3,"Available Count":1,"Score":0.0665680765,"is_accepted":false,"ViewCount":4441,"Q_Id":66767975,"Users Score":1,"Answer":"Perhaps you can give host.docker.internal:4566 a try, since most likely the localstack service will be the only one listening to it.","Q_Score":4,"Tags":"python,amazon-web-services,docker,localstack","A_Id":67388365,"CreationDate":"2021-03-23T17:11:00.000","Title":"Docker-compose: How to access Localstack resources both from container and host, using same network address","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":1,"Web Development":1},{"Question":"I have a doubt. I've been looking for information and haven't found anything. I would like to know if it is possible to send a message with a bot which will be eliminated after a certain time. I'm working with c# but I think other examples will work for me. Thanks in advance.","AnswerCount":2,"Available Count":2,"Score":0.0,"is_accepted":false,"ViewCount":595,"Q_Id":66789927,"Users Score":0,"Answer":"I'm not sure if telegram API provides that.\nbut you can implement it in the bot to do that, maybe with \"time\" or \"schedule\" in python","Q_Score":0,"Tags":"python,c#,telegram-bot","A_Id":66790013,"CreationDate":"2021-03-24T21:52:00.000","Title":"how to send auto delete messages whit Telegram bot","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I have a doubt. I've been looking for information and haven't found anything. I would like to know if it is possible to send a message with a bot which will be eliminated after a certain time. I'm working with c# but I think other examples will work for me. Thanks in advance.","AnswerCount":2,"Available Count":2,"Score":0.0996679946,"is_accepted":false,"ViewCount":595,"Q_Id":66789927,"Users Score":1,"Answer":"I thought I would delete my own message after a while but I think it would not be a solution in a well active chat, since there are many messages coming in all the time and I would not know for sure what the id of the message I sent is. I would have to do another algorithm to find out what that id is, and what I was trying to see is if there was some simpler method already implemented in the api for these specific cases. Send messages using the bot with self-destruct after a while.","Q_Score":0,"Tags":"python,c#,telegram-bot","A_Id":66824462,"CreationDate":"2021-03-24T21:52:00.000","Title":"how to send auto delete messages whit Telegram bot","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"My knowledge of networking is limited. So I might state the question wrong:\nSo as I understand that the code app.run(host='0.0.0.0'. Meaning you host on all of your current machine IP address.\nSo I suppose that hosting on app.run(host='10.0.0.0' mean that I am hosting on all IP address from range: 10.0.0.1 ~ 10.255.255.255.\nBut when I try to host on 10.0.0.0. The other machine on the same network cannot detect my flask service. (my machine ip address is 10.0.0.70, so I think 10.0.0.0 cover my machine ip address)\nSo there is flawed in my understanding, what is it?\n1\/ you can't host flask on 10.0.0.0\n2\/ 10.0.0.0 is not really the range of ip from 10.0.0.1 to 10.255.255.255\n3\/ something else.\nAn","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":84,"Q_Id":66824017,"Users Score":0,"Answer":"0.0.0.0 - 0.255.255.255 is reserved for software.\n10.0.0.0 - 10.255.255.255 is reserved for local addresses.\nIf you use 10.?.?.? then you have to enter the address to your address bar.","Q_Score":0,"Tags":"python,flask,networking","A_Id":66824083,"CreationDate":"2021-03-26T20:37:00.000","Title":"what does it mean to host on 10.0.0.0 IP address for flask webserver","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"Sorry for my stupid question, but i dont understand how to add all permissions in my voice channel for one user. I have this await channel.set_permissions(member, some_permission), but i dont know what permission i need to use","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":304,"Q_Id":66824019,"Users Score":0,"Answer":"You should just make a role with those permissions and then give that role to the user. for example, await member.add_roles(var) var being the role and member being the member you want to add the role too.","Q_Score":0,"Tags":"python,python-3.x,discord,discord.py","A_Id":66847972,"CreationDate":"2021-03-26T20:37:00.000","Title":"How to give all permissions in voice channel discord.py","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I trained a chatbot in a file on my computer. I have everything else for my website up and running but I cannot get the python file into the site. Do you have any advice for how I can make an html messenger that uses user inputs and converses with the AI chatbot that way? Thank you!","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":69,"Q_Id":66828255,"Users Score":1,"Answer":"One easy way to do this is by building a server, where you can have your chatbot code. You can use Flask. The instant the HTML asks a question, you send an API call with the required message, which when received by the API runs that message through the chatbot and returns the response in the output. This can continuously go on in the chat.","Q_Score":0,"Tags":"python,html,css,flask","A_Id":66828292,"CreationDate":"2021-03-27T05:21:00.000","Title":"How the heck do I import my python chatbot into my html code?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I 'm using repl.it to run my discord bot but I have encountered a problem. Sometimes, at the end of the day usually, whatever the users did does not save in the json file assigned to store the data. Does anyone know what is happening?","AnswerCount":2,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":376,"Q_Id":66832204,"Users Score":0,"Answer":"Hey i have had the same problem and it is because file changes do not persist if you are not in the editor.\nThe best thing you could do is use a database to store your data.\nTake a look at some databases like mongo db and PostgreSQL.","Q_Score":0,"Tags":"python,json,discord.py,repl.it","A_Id":66833705,"CreationDate":"2021-03-27T14:06:00.000","Title":"Json data doesn't save at the end of the day","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":1,"System Administration and DevOps":0,"Web Development":0},{"Question":"I have two questions about TCP protocol :\n\nHas TCP a limitation on the size of each packet to send them? if yes, what is the largest size of each packet to be sent?\n\nand how does TCP handle files with large size? does it send it at\nonce time(1 packet) or TCP converts it into smaller packets?\n\n\nand I use python and socket library.","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":204,"Q_Id":66857509,"Users Score":0,"Answer":"TCP is a streaming protocol. Which means that you as sender have no control over how many send operations are actually going on in the kernel.\nThere are several mechanisms at play here. A client can call send many times with only one byte. TCP will try to accumulate these bytes until the MTU of the L2 is filled and only then send it, or sooner if a timeout is reached.\nIn one send operation you can send 64K approx.\nTo send a file, you need thus several send operatons until the whole file is sent.\nOn the receiving side of the TCP, there is no relation to the number of send operations and the number of receive operations. If the sender send 5 times 10 bytes consecutively and fast, very likely they will be received as 50 bytes in one receive operation.\nActually for TCP there is no such thing as a packet. It is called a frame.\nPackets are used for UDP, because UDP is not a streaming protocol.\nIn UDP, if you send 5 bytes, the receiver will receive 5 bytes or nothing because UDP is not connection oriented. Whereas TCP is.\nFor UDP the max size is approx 64k. This 64k will get chopped to pieces by IP to send it over the wire, and IP will put it back together upon reception. This is all because of the L2 MTU.","Q_Score":0,"Tags":"python,sockets,websocket,tcp","A_Id":66876468,"CreationDate":"2021-03-29T15:59:00.000","Title":"How does TCP handle a file with large size?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I'm building an app in which I need to communicate data from and to ReactJS and Python. It needs to go both ways - but I'm more concerned about the React part right now. At first I considered JSON, but couldn't find any resources\/ libraries to update JSON files, and people said I should stay away from that. Other than literally creating a text file with data in it, what are my options? The application has to do with getting stock data from a python API, doing calculations on it, and sending the data to ReactJS to render on a webpage. I also need ReactJS to send account data back to Python where we do our MySQL. Any suggestions?","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":296,"Q_Id":66859835,"Users Score":0,"Answer":"Since the ReactJS app is a front-end application, your only real choice to ensure security is exposing an API on the python app which the ReactJS app talks to (via Websockets for example since you're mentioning a bidirectional communication). Maybe take a look at something like Flask with the flask-socketio package.","Q_Score":0,"Tags":"python,json,reactjs","A_Id":66860237,"CreationDate":"2021-03-29T18:44:00.000","Title":"How to communicate data from ReactJS and Python?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":1,"System Administration and DevOps":0,"Web Development":0},{"Question":"The website i scapped blocked me out by showing 406 Not Acceptable on the browser. It might i mistakenly sent too many requests at once on phython code.\nSo i put time.sleep(10) for each loop to not make it look like a DDoS attack, and it seems worked out.\nMy questions are:\n\nHow long would it be reasonable to send between each request? Sleep 10 seconds for each loop makes my code running too slow.\n\nHow to fix the 406 Not Acceptable error on my browsers? They still block me out, only if i chance my ip address but it's not permanent solution.\n\n\nThank you all for your answers and comments. Good day!","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":84,"Q_Id":66867975,"Users Score":1,"Answer":"Any rate-limit errors are all subject to which website you choose to scrape \/ interact with. I could set up a website that only allows you to view it once per day, before throwing HTTP errors at your screen. So to answer your first question, there is no definitive answer. You must test for yourself and see what's the fastest speed you can go, without getting blocked.\nHowever, there is a workaround. If you use proxies, then it's almost impossible to detect and stop the requests from executing, and therefore you will not be hit by any HTTP errors. HOWEVER, JUST BECAUSE YOU CAN, DOESN'T MEAN THAT YOU SHOULD- I am a programmer, not a lawyer. I'm sure there's a rule somewhere that says that spamming a page, even after it tells you to stop, is illegal.\nYour second question isn't exactly related to programming, but I will answer it anyways- try clearing your cookies or refreshing your IP (try using a VPN or such). Other than changing your IP or cookies, there's not many more ways that a page can fingerprint you (in order to block you).","Q_Score":1,"Tags":"python,http-status-code-406","A_Id":66868100,"CreationDate":"2021-03-30T09:11:00.000","Title":"'406 Not Acceptable' after scaping web using python","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I'm trying to use telethon and telebot to send notification from my code in python.\nI have configureted everythings, but the code ask me the following:\nreceiver = InputPeerUser('user_id', 'user_hash')\nI fund user_id, but I don't understand where I could catch my user_hash.","AnswerCount":2,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":1334,"Q_Id":66873563,"Users Score":0,"Answer":"You must have an open dialogue with that user. Or you received a message from him. When all the dialogs are loaded, the information of that user comes with it.","Q_Score":4,"Tags":"python,telegram,telegram-bot,telethon","A_Id":66911440,"CreationDate":"2021-03-30T15:05:00.000","Title":"How to find 'user_hash' on telegram?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"import yfinance as yf\nterminal: ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behavior is the source of the following dependency conflicts.\nyahoo 0.1 requires requests==1.1.0, but you have requests 2.25.1 which is incompatible.\nhow can I address this issue\nRegards! :)","AnswerCount":2,"Available Count":2,"Score":0.0,"is_accepted":false,"ViewCount":1485,"Q_Id":66880867,"Users Score":0,"Answer":"The problem is that yfinance requires a newer version of requests than yahoo. For me it said the same thing as you got when I tried.\nImporting yfinance worked for me.\nI didn't specify anything for requests and just installed the above. Generally, you can specify installation version using =x.x at the end, i.e. pip install requests==1.1.0.","Q_Score":0,"Tags":"python,finance","A_Id":66881009,"CreationDate":"2021-03-31T02:32:00.000","Title":"Addressing a pip\u00b4s dependency issue: requests==2.25.1 versus requests==1.1.0","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":1,"System Administration and DevOps":0,"Web Development":0},{"Question":"import yfinance as yf\nterminal: ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behavior is the source of the following dependency conflicts.\nyahoo 0.1 requires requests==1.1.0, but you have requests 2.25.1 which is incompatible.\nhow can I address this issue\nRegards! :)","AnswerCount":2,"Available Count":2,"Score":0.0,"is_accepted":false,"ViewCount":1485,"Q_Id":66880867,"Users Score":0,"Answer":"It sounds like you have both yahoo and yfinance installed in the same environment\/setup. I suggest creating a virtual environment with virtualenv so you can have the appropriate requests for each libraries.\nAlternatively you can submit a PR for yahoo to use a more recent version of requests","Q_Score":0,"Tags":"python,finance","A_Id":66881232,"CreationDate":"2021-03-31T02:32:00.000","Title":"Addressing a pip\u00b4s dependency issue: requests==2.25.1 versus requests==1.1.0","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":1,"System Administration and DevOps":0,"Web Development":0},{"Question":"I have created project and even i have client_secret.js file, when i try to run the program first time it asks to login, when I tried to login through gmail (which I have used to create project) it says\n\n\"Google hasn\u2019t verified this app\"\n\nMessage, so for that i'm looking how to get out from this step.","AnswerCount":2,"Available Count":1,"Score":-0.0996679946,"is_accepted":false,"ViewCount":71,"Q_Id":66899042,"Users Score":-1,"Answer":"This is occuring because the application you created is not verified by google so you are not able to login in that you need to verify it by google suite and then you are able to login I also get that problem when i tried to create an browser for myself","Q_Score":1,"Tags":"python-3.x,google-sheets,google-drive-api,google-oauth,google-developers-console","A_Id":66899090,"CreationDate":"2021-04-01T05:49:00.000","Title":"How to read files from Google drive through python when it says Google hasn\u2019t verified this app?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"I am a PHP developer, tasked to write Python scripts.\nThe script uses requests, which I found to be the easiest Python equivalent to PHP cURL.\nIf you use cURL in PHP, cURL must be enabled on the server, otherwise, the script won't work.\nIs there likewise any activation or enabling necessary in Python to use requests other than import requests?","AnswerCount":3,"Available Count":2,"Score":0.0,"is_accepted":false,"ViewCount":424,"Q_Id":66902684,"Users Score":0,"Answer":"Install requests on machine you are going to use it: pip install requests.\nNote if you are on linux you might have 2 versions of python installed: python 2.* and python 3.*\nIf this is the case using pip install <package_name> will install it for python 2.*, so you should use pip3 install <package_name> to install it for python 3.\n(I would also suggest using virtualenv, but this is another topic).\nIf package is installed then you can use it in your scripts by simply importing it: import requests\n\nNothing more is needed to enable\/activate libraries(packages). It is the case for all packages in python. (install -> import)","Q_Score":0,"Tags":"python,python-requests","A_Id":66903023,"CreationDate":"2021-04-01T10:18:00.000","Title":"How do I enable Python requests?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I am a PHP developer, tasked to write Python scripts.\nThe script uses requests, which I found to be the easiest Python equivalent to PHP cURL.\nIf you use cURL in PHP, cURL must be enabled on the server, otherwise, the script won't work.\nIs there likewise any activation or enabling necessary in Python to use requests other than import requests?","AnswerCount":3,"Available Count":2,"Score":0.0,"is_accepted":false,"ViewCount":424,"Q_Id":66902684,"Users Score":0,"Answer":"You must also have pip installed on your computer to install the requests module, pip comes installed with python 3.4 and above so you could do something like \"pip install requests\"","Q_Score":0,"Tags":"python,python-requests","A_Id":66902763,"CreationDate":"2021-04-01T10:18:00.000","Title":"How do I enable Python requests?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I am developing a JupyterLab Notebook and I need to embed a website for interaction with a dashboard from within the same notebook. This would be quite straight-forward using an IFrame.\nHowever, to access this website, that I launch on my public server, JWT authentication is required so I need to send an additional header with the token. Which is the best way to do this? I am not familiar with HTML and JS so I am wondering where to start from.\nThanks a lot for your help!","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":127,"Q_Id":66909734,"Users Score":0,"Answer":"Finally, I solved this issue by sending the token in the URL params using SSL traffic.","Q_Score":3,"Tags":"javascript,python,html,iframe,jupyter-lab","A_Id":67059981,"CreationDate":"2021-04-01T18:24:00.000","Title":"Display IFrame IPython with custom auth headers","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":1},{"Question":"If I send out a request to a user for them to reply to a yes or no question, how can I make it such that after a certain amount of time, the question expires and is no longer accepting replies?\nBackground\nI'm currently working on a discord.py bot, and I'm getting to the part where I start scanning the messages for user input. How it works is that somebody can challenge someone to a game of tic tac toe, and then the challenged person has 10 seconds to reply with yes or no. How can I make it such that after 10 seconds, the challenge expires?\nExamples\nOne example I thought of is the bot Dank Memer. If you challenge someone to a tic tac toe game on there, then the challenged person has to respond to you within a time limit, or else the challenge expires. It also works with commands like pls trivia, where after 10 seconds it stops accepting answers to the question.\nI'm pretty sure that people have already asked this question, but I have looked over and over on Google and I can't find a solution. It's probably because of my wording, I think.\nIt's also hard for me to do a minimal-reproducible example, as that would require you guys to have to create a discord account and token for the bot, which is not at all efficient.","AnswerCount":3,"Available Count":1,"Score":0.1325487884,"is_accepted":false,"ViewCount":279,"Q_Id":66926822,"Users Score":2,"Answer":"You can use a await client.wait_for(check=check). Check is a function you must define that needs to return either true or false. Within check, you can add various conditions and actions if you need to. I made a command that does a math equation with it, if you'd like to see an example.","Q_Score":1,"Tags":"python,discord,discord.py","A_Id":66926879,"CreationDate":"2021-04-03T00:54:00.000","Title":"Is there a way to make it such that the user must answer within a time limit on discord.py?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"Doing some stress testing on Janusgraph and required to load est 5 million vertex and edges. Using faker python to generate the fake data and will like to know which is the best method to load such bulk chuck of data? I tried to load the data using the console but getting weird results and timeout.","AnswerCount":1,"Available Count":1,"Score":0.537049567,"is_accepted":false,"ViewCount":75,"Q_Id":66933444,"Users Score":3,"Answer":"It looks like you are using the Gremlin Python client from the tags. A fairly common technique is to use a multi threaded (or multi process) Python application that sends data to the graph in batches (perhaps 50 to 100 addV or addE at a time). It's not clear from your question if you are using Gremlin Server or not. If you are using Gremlin Server you may want to configure the gremlinPool parameter in the YAML file to have sufficient workers for the class of machine you have it running on. It should not take too long to load 5 million vertices and edges in this fashion. If overall time to load the vertices and edges is not a concern, a single thread sending batches of 50 to 100 at a time is still reasonably efficient.","Q_Score":1,"Tags":"gremlin,janusgraph,gremlinpython","A_Id":67140634,"CreationDate":"2021-04-03T16:28:00.000","Title":"Best method to bulk load (5 millions) into Janusgraph","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I have a large (~ 150 MB) CSV output file that I need to translate from Spanish to English via API (e.g. Microsoft, Amazon, etc.) on an as-needed basis.\nThere's 25 columns, and many 100k rows.\nOf those 25 columns, only about half need to be translated, as other half are numeric data.\nMicrosoft & Amazon charge for translation a per-character basis, so rather than translating the entire file, I thought about just picking the 12 columns that actually need to be translated, and translating them on an element-by-element basis\nThe problem is that this leads to several hundred thousand network API calls to Microsoft's translation API -- one for each element of each column. While there's nothing inherently \"wrong\", I can't help but feel there's something inefficient about it.\nWhat are the negative implications of a script that does several hundred thousand network trips to call an API?\nThanks","AnswerCount":1,"Available Count":1,"Score":1.2,"is_accepted":true,"ViewCount":19,"Q_Id":66942077,"Users Score":1,"Answer":"What are the negative implications of a script that does several\nhundred thousand network trips to call an API?\n\nBeyond the possible pricing implications, the most straightforward implication is increased latency. Calls over the network are one of the slowest things a program can do.\nWhile the time to execute a cpu instruction can be measured in nanoseconds, network calls are typically measured in tens or hundreds of milliseconds. That's a difference of x100,000,000.","Q_Score":0,"Tags":"python,pandas,dataframe,networking,network-programming","A_Id":66954216,"CreationDate":"2021-04-04T14:09:00.000","Title":"Negative implications of network trips?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0},{"Question":"I'm just picking up some async Python and trying to write a simple TCP port scanner using the Asyncio module.\nI can open a full-fledged TCP connection with 3-way handshakes via asyncio.open_connection. However, I want to create an SYN-ACK half-open connection\u2014similar to what nmap uses\u2014using asyncio. I was rummaging through the streams API but couldn't find anything. Is there a high-level method to do this? If not, how do I do this?","AnswerCount":1,"Available Count":1,"Score":0.1973753202,"is_accepted":false,"ViewCount":203,"Q_Id":66976054,"Users Score":1,"Answer":"asyncio doesn't give you such control on TCP\/IP stack layers and even hides some complex tasks such as callbacks, low-level protocols, transports.\nYou can do it using a raw socket.\nModules that can be useful\n\npython-nmap\nscapy","Q_Score":1,"Tags":"python,asynchronous,tcp,async-await,python-asyncio","A_Id":67002079,"CreationDate":"2021-04-06T20:30:00.000","Title":"Is there an easy way to do half open TCP connection via Python Asyncio?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":0,"Python Basics and Environment":0,"System Administration and DevOps":1,"Web Development":0},{"Question":"Exactly what the title says. I have someone abusing a certain command that I would like to restrict a role that they have from using said command. Is there a ay to do that? and if so, can someone please write the example code?","AnswerCount":1,"Available Count":1,"Score":0.0,"is_accepted":false,"ViewCount":178,"Q_Id":66978526,"Users Score":0,"Answer":"under the decorator for your command (@bot.command()) use @commands.has_role(\"roleName\") to limit the command to only that role.","Q_Score":0,"Tags":"python,discord.py","A_Id":66983699,"CreationDate":"2021-04-07T01:47:00.000","Title":"Is there a way for me to disable a command for a certain role in discord.py?","Data Science and Machine Learning":0,"Database and SQL":0,"GUI and Desktop Applications":0,"Networking and APIs":1,"Other":1,"Python Basics and Environment":0,"System Administration and DevOps":0,"Web Development":0}]