recurrent neural network
Dotaz
Zobrazit nápovědu
Recently a new so-called energy complexity measure has been introduced and studied for feedforward perceptron networks. This measure is inspired by the fact that biological neurons require more energy to transmit a spike than not to fire, and the activity of neurons in the brain is quite sparse, with only about 1% of neurons firing. In this letter, we investigate the energy complexity of recurrent networks, which counts the number of active neurons at any time instant of a computation. We prove that any deterministic finite automaton with m states can be simulated by a neural network of optimal size [Formula: see text] with the time overhead of [Formula: see text] per one input bit, using the energy O(e), for any e such that [Formula: see text] and e=O(s), which shows the time-energy trade-off in recurrent networks. In addition, for the time overhead [Formula: see text] satisfying [Formula: see text], we obtain the lower bound of [Formula: see text] on the energy of such a simulation for some constant c>0 and for infinitely many s.
It has been known for discrete-time recurrent neural networks (NNs) that binary-state models using the Heaviside activation function (with Boolean outputs 0 or 1) are equivalent to finite automata (level 3 in the Chomsky hierarchy), while analog-state NNs with rational weights, employing the saturated-linear function (with real-number outputs in the interval [0,1]), are Turing complete (Chomsky level 0) even for three analog units. However, it is as yet unknown whether there exist subrecursive (i.e. sub-Turing) NN models which occur on Chomsky levels 1 or 2. In this paper, we provide such a model which is a binary-state NN extended with one extra analog unit (1ANN). We achieve a syntactic characterization of languages that are accepted online by 1ANNs in terms of so-called cut languages which are combined in a certain way by usual operations. We employ this characterization for proving that languages accepted by 1ANNs with rational weights are context-sensitive (Chomsky level 1) and we present explicit examples of such languages that are not context-free (i.e. are above Chomsky level 2). In addition, we formulate a sufficient condition when a 1ANN recognizes a regular language (Chomsky level 3) in terms of quasi-periodicity of parameters derived from its real weights, which is satisfied e.g. for rational weights provided that the inverse of the real self-loop weight of the analog unit is a Pisot number.
- MeSH
- jazyk (prostředek komunikace) * MeSH
- neuronové sítě * MeSH
- teoretické modely * MeSH
- Publikační typ
- časopisecké články MeSH
Shared input to a population of neurons induces noise correlations, which can decrease the information carried by a population activity. Inhibitory feedback in recurrent neural networks can reduce the noise correlations and thus increase the information carried by the population activity. However, the activity of inhibitory neurons is costly. This inhibitory feedback decreases the gain of the population. Thus, depolarization of its neurons requires stronger excitatory synaptic input, which is associated with higher ATP consumption. Given that the goal of neural populations is to transmit as much information as possible at minimal metabolic costs, it is unclear whether the increased information transmission reliability provided by inhibitory feedback compensates for the additional costs. We analyze this problem in a network of leaky integrate-and-fire neurons receiving correlated input. By maximizing mutual information with metabolic cost constraints, we show that there is an optimal strength of recurrent connections in the network, which maximizes the value of mutual information-per-cost. For higher values of input correlation, the mutual information-per-cost is higher for recurrent networks with inhibitory feedback compared to feedforward networks without any inhibitory neurons. Our results, therefore, show that the optimal synaptic strength of a recurrent network can be inferred from metabolically efficient coding arguments and that decorrelation of the input by inhibitory feedback compensates for the associated increased metabolic costs.
BACKGROUND: The recent big data revolution in Genomics, coupled with the emergence of Deep Learning as a set of powerful machine learning methods, has shifted the standard practices of machine learning for Genomics. Even though Deep Learning methods such as Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs) are becoming widespread in Genomics, developing and training such models is outside the ability of most researchers in the field. RESULTS: Here we present ENNGene-Easy Neural Network model building tool for Genomics. This tool simplifies training of custom CNN or hybrid CNN-RNN models on genomic data via an easy-to-use Graphical User Interface. ENNGene allows multiple input branches, including sequence, evolutionary conservation, and secondary structure, and performs all the necessary preprocessing steps, allowing simple input such as genomic coordinates. The network architecture is selected and fully customized by the user, from the number and types of the layers to each layer's precise set-up. ENNGene then deals with all steps of training and evaluation of the model, exporting valuable metrics such as multi-class ROC and precision-recall curve plots or TensorBoard log files. To facilitate interpretation of the predicted results, we deploy Integrated Gradients, providing the user with a graphical representation of an attribution level of each input position. To showcase the usage of ENNGene, we train multiple models on the RBP24 dataset, quickly reaching the state of the art while improving the performance on more than half of the proteins by including the evolutionary conservation score and tuning the network per protein. CONCLUSIONS: As the role of DL in big data analysis in the near future is indisputable, it is important to make it available for a broader range of researchers. We believe that an easy-to-use tool such as ENNGene can allow Genomics researchers without a background in Computational Sciences to harness the power of DL to gain better insights into and extract important information from the large amounts of data available in the field.
- MeSH
- genomika MeSH
- neuronové sítě * MeSH
- sekundární struktura proteinů MeSH
- strojové učení * MeSH
- Publikační typ
- časopisecké články MeSH
In neural computation, the essential information is generally encoded into the neurons via their spiking configurations, activation values or (attractor) dynamics. The synapses and their associated plasticity mechanisms are, by contrast, mainly used to process this information and implement the crucial learning features. Here, we propose a novel Turing complete paradigm of neural computation where the essential information is encoded into discrete synaptic states, and the updating of this information achieved via synaptic plasticity mechanisms. More specifically, we prove that any 2-counter machine-and hence any Turing machine-can be simulated by a rational-weighted recurrent neural network employing spike-timing-dependent plasticity (STDP) rules. The computational states and counter values of the machine are encoded into discrete synaptic strengths. The transitions between those synaptic weights are then achieved via STDP. These considerations show that a Turing complete synaptic-based paradigm of neural computation is theoretically possible and potentially exploitable. They support the idea that synapses are not only crucially involved in information processing and learning features, but also in the encoding of essential information. This approach represents a paradigm shift in the field of neural computation.
Deep learning has recently been utilized with great success in a large number of diverse application domains, such as visual and face recognition, natural language processing, speech recognition, and handwriting identification. Convolutional neural networks, that belong to the deep learning models, are a subtype of artificial neural networks, which are inspired by the complex structure of the human brain and are often used for image classification tasks. One of the biggest challenges in all deep neural networks is the overfitting issue, which happens when the model performs well on the training data, but fails to make accurate predictions for the new data that is fed into the model. Several regularization methods have been introduced to prevent the overfitting problem. In the research presented in this manuscript, the overfitting challenge was tackled by selecting a proper value for the regularization parameter dropout by utilizing a swarm intelligence approach. Notwithstanding that the swarm algorithms have already been successfully applied to this domain, according to the available literature survey, their potential is still not fully investigated. Finding the optimal value of dropout is a challenging and time-consuming task if it is performed manually. Therefore, this research proposes an automated framework based on the hybridized sine cosine algorithm for tackling this major deep learning issue. The first experiment was conducted over four benchmark datasets: MNIST, CIFAR10, Semeion, and UPS, while the second experiment was performed on the brain tumor magnetic resonance imaging classification task. The obtained experimental results are compared to those generated by several similar approaches. The overall experimental results indicate that the proposed method outperforms other state-of-the-art methods included in the comparative analysis in terms of classification error and accuracy.
- MeSH
- algoritmy MeSH
- lidé MeSH
- magnetická rezonanční tomografie MeSH
- nádory mozku * MeSH
- neuronové sítě * MeSH
- psaní rukou MeSH
- Check Tag
- lidé MeSH
- Publikační typ
- časopisecké články MeSH
- práce podpořená grantem MeSH
Alcohol Use Disorder (AUD) adversely affects the lives of millions of people, but still lacks effective treatment options. Recent advancements in psychedelic research suggest psilocybin to be potentially efficacious for AUD. However, major knowledge gaps remain regarding (1) psilocybin's general mode of action and (2) AUD-specific alterations of responsivity to psilocybin treatment in the brain that are crucial for treatment development. Here, we conducted a randomized, placebo-controlled crossover pharmaco-fMRI study on psilocybin effects using a translational approach with healthy rats and a rat model of alcohol relapse. Psilocybin effects were quantified with resting-state functional connectivity using data-driven whole-brain global brain connectivity, network-based statistics, graph theory, hypothesis-driven Default Mode Network (DMN)-specific connectivity, and entropy analyses. Results demonstrate that psilocybin induced an acute wide-spread decrease in different functional connectivity domains together with a distinct increase of connectivity between serotonergic core regions and cortical areas. We could further provide translational evidence for psilocybin-induced DMN hypoconnectivity reported in humans. Psilocybin showed an AUD-specific blunting of DMN hypoconnectivity, which strongly correlated to the alcohol relapse intensity and was mainly driven by medial prefrontal regions. In conclusion, our results provide translational validity for acute psilocybin-induced neural effects in the rodent brain. Furthermore, alcohol relapse severity was negatively correlated with neural responsivity to psilocybin treatment. Our data suggest that a clinical standard dose of psilocybin may not be sufficient to treat severe AUD cases; a finding that should be considered for future clinical trials.
- MeSH
- alkoholismus * diagnostické zobrazování farmakoterapie MeSH
- default mode network MeSH
- ethanol MeSH
- halucinogeny * farmakologie MeSH
- krysa rodu rattus MeSH
- lidé MeSH
- magnetická rezonanční tomografie metody MeSH
- mozek diagnostické zobrazování MeSH
- psilocybin farmakologie MeSH
- recidiva MeSH
- zvířata MeSH
- Check Tag
- krysa rodu rattus MeSH
- lidé MeSH
- zvířata MeSH
- Publikační typ
- časopisecké články MeSH
- randomizované kontrolované studie MeSH
In order to refine the analysis of the computational power of discrete-time recurrent neural networks (NNs) between the binary-state NNs which are equivalent to finite automata (level 3 in the Chomsky hierarchy), and the analog-state NNs with rational weights which are Turing-complete (Chomsky level 0), we study an intermediate model αANN of a binary-state NN that is extended with α≥0 extra analog-state neurons. For rational weights, we establish an analog neuron hierarchy 0ANNs ⊂ 1ANNs ⊂ 2ANNs ⊆ 3ANNs and separate its first two levels. In particular, 0ANNs coincide with the binary-state NNs (Chomsky level 3) being a proper subset of 1ANNs which accept at most context-sensitive languages (Chomsky level 1) including some non-context-free ones (above Chomsky level 2). We prove that the deterministic (context-free) language L#={0n1n∣n≥1} cannot be recognized by any 1ANN even with real weights. In contrast, we show that deterministic pushdown automata accepting deterministic languages can be simulated by 2ANNs with rational weights, which thus constitute a proper superset of 1ANNs. Finally, we prove that the analog neuron hierarchy collapses to 3ANNs by showing that any Turing machine can be simulated by a 3ANN having rational weights, with linear-time overhead.
- MeSH
- jazyk (prostředek komunikace) * MeSH
- neuronové sítě * MeSH
- neurony * fyziologie MeSH
- Publikační typ
- časopisecké články MeSH
In this paper, we propose an integrated biologically inspired visual collision avoidance approach that is deployed on a real hexapod walking robot. The proposed approach is based on the Lobula giant movement detector (LGMD), a neural network for looming stimuli detection that can be found in visual pathways of insects, such as locusts. Although a superior performance of the LGMD in the detection of intercepting objects has been shown in many collision avoiding scenarios, its direct integration with motion control is an unexplored topic. In our work, we propose to utilize the LGMD neural network for visual interception detection with a central pattern generator (CPG) for locomotion control of a hexapod walking robot that are combined in the controller based on the long short-term memory (LSTM) recurrent neural network. Moreover, we propose self-supervised learning of the integrated controller to autonomously find a suitable setting of the system using a realistic robotic simulator. Thus, individual neural networks are trained in a simulation to enhance the performance of the controller that is then experimentally verified with a real hexapod walking robot in both collision and interception avoidance scenario and navigation in a cluttered environment.
- MeSH
- chování zvířat fyziologie MeSH
- chůze fyziologie MeSH
- kobylky fyziologie MeSH
- neuronové sítě MeSH
- řízené strojové učení MeSH
- robotika přístrojové vybavení MeSH
- učení vyhýbat se fyziologie MeSH
- zvířata MeSH
- Check Tag
- zvířata MeSH
- Publikační typ
- časopisecké články MeSH
- práce podpořená grantem MeSH
