Jul 26 15:37:55.429302 ip-10-0-138-163 systemd[1]: Starting Kubernetes Kubelet... Jul 26 15:37:55.875428 ip-10-0-138-163 kubenswrapper[2577]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 26 15:37:55.875428 ip-10-0-138-163 kubenswrapper[2577]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 26 15:37:55.875428 ip-10-0-138-163 kubenswrapper[2577]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 26 15:37:55.875428 ip-10-0-138-163 kubenswrapper[2577]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 26 15:37:55.875428 ip-10-0-138-163 kubenswrapper[2577]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 26 15:37:55.877548 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.877464 2577 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 26 15:37:55.883692 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.883676 2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 26 15:37:55.883692 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.883691 2577 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 26 15:37:55.883760 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.883695 2577 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 26 15:37:55.883760 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.883699 2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 26 15:37:55.883760 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.883702 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 26 15:37:55.883760 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.883705 2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 26 15:37:55.883760 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.883708 2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 26 15:37:55.883760 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.883711 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 26 15:37:55.883760 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.883714 2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 26 15:37:55.883760 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.883717 2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 26 15:37:55.883760 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.883719 2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 26 15:37:55.883760 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.883722 2577 feature_gate.go:328] unrecognized feature gate: Example Jul 26 15:37:55.883760 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.883724 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 26 15:37:55.883760 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.883727 2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 26 15:37:55.883760 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.883732 2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 26 15:37:55.883760 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.883736 2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 26 15:37:55.883760 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.883740 2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 26 15:37:55.883760 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.883743 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 26 15:37:55.883760 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.883747 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 26 15:37:55.883760 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.883750 2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 26 15:37:55.883760 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.883753 2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 26 15:37:55.884222 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.883755 2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 26 15:37:55.884222 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.883758 2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 26 15:37:55.884222 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.883761 2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 26 15:37:55.884222 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.883763 2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 26 15:37:55.884222 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.883767 2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 26 15:37:55.884222 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.883769 2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 26 15:37:55.884222 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.883772 2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 26 15:37:55.884222 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.883775 2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 26 15:37:55.884222 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.883778 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 26 15:37:55.884222 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.883781 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 26 15:37:55.884222 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.883784 2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 26 15:37:55.884222 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.883787 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 26 15:37:55.884222 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.883789 2577 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 26 15:37:55.884222 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.883792 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 26 15:37:55.884222 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.883794 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 26 15:37:55.884222 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.883797 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 26 15:37:55.884222 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.883799 2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 26 15:37:55.884222 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.883802 2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 26 15:37:55.884222 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.883804 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 26 15:37:55.884222 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.883807 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 26 15:37:55.884697 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.883811 2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 26 15:37:55.884697 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.883813 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 26 15:37:55.884697 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.883815 2577 feature_gate.go:328] unrecognized feature gate: Example2 Jul 26 15:37:55.884697 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.883819 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 26 15:37:55.884697 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.883822 2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 26 15:37:55.884697 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.883825 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 26 15:37:55.884697 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.883827 2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 26 15:37:55.884697 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.883830 2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 26 15:37:55.884697 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.883833 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 26 15:37:55.884697 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.883835 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 26 15:37:55.884697 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.883838 2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 26 15:37:55.884697 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.883840 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 26 15:37:55.884697 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.883844 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 26 15:37:55.884697 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.883846 2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 26 15:37:55.884697 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.883849 2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 26 15:37:55.884697 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.883851 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 26 15:37:55.884697 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.883854 2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 26 15:37:55.884697 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.883858 2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 26 15:37:55.884697 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.883861 2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 26 15:37:55.884697 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.883864 2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 26 15:37:55.885566 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.883866 2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 26 15:37:55.885566 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.883869 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 26 15:37:55.885566 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.883872 2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 26 15:37:55.885566 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.883875 2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 26 15:37:55.885566 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.883878 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 26 15:37:55.885566 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.883880 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 26 15:37:55.885566 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.883884 2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 26 15:37:55.885566 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.883886 2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 26 15:37:55.885566 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.883889 2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 26 15:37:55.885566 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.883891 2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 26 15:37:55.885566 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.883894 2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 26 15:37:55.885566 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.883897 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 26 15:37:55.885566 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.883899 2577 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 26 15:37:55.885566 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.883902 2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 26 15:37:55.885566 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.883904 2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 26 15:37:55.885566 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.883907 2577 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 26 15:37:55.885566 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.883910 2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 26 15:37:55.885566 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.883913 2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 26 15:37:55.885566 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.883915 2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 26 15:37:55.885566 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.883918 2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 26 15:37:55.886209 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.883920 2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 26 15:37:55.886209 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.883923 2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 26 15:37:55.886209 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.883925 2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 26 15:37:55.886209 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.883928 2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 26 15:37:55.886209 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.883931 2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 26 15:37:55.886209 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.883934 2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 26 15:37:55.886209 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.883937 2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 26 15:37:55.886209 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.883939 2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 26 15:37:55.886209 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.883942 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 26 15:37:55.886209 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.883945 2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 26 15:37:55.886209 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.883947 2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 26 15:37:55.886209 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.883950 2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 26 15:37:55.886209 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.883952 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 26 15:37:55.886209 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.883955 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 26 15:37:55.886209 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.883957 2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 26 15:37:55.886209 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.883960 2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 26 15:37:55.886209 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.883962 2577 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 26 15:37:55.886209 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.883965 2577 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 26 15:37:55.886209 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.885228 2577 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 26 15:37:55.886209 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.885243 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 26 15:37:55.886695 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.885251 2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 26 15:37:55.886695 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.885257 2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 26 15:37:55.886695 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.885262 2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 26 15:37:55.886695 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.885266 2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 26 15:37:55.886695 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.885271 2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 26 15:37:55.886695 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.885275 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 26 15:37:55.886695 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.885280 2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 26 15:37:55.886695 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.885285 2577 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 26 15:37:55.886695 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.885290 2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 26 15:37:55.886695 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.885295 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 26 15:37:55.886695 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.885304 2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 26 15:37:55.886695 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.885308 2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 26 15:37:55.886695 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.885312 2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 26 15:37:55.886695 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.885316 2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 26 15:37:55.886695 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.885321 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 26 15:37:55.886695 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.885325 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 26 15:37:55.886695 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.885330 2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 26 15:37:55.886695 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.885334 2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 26 15:37:55.886695 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.885339 2577 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 26 15:37:55.886695 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.885343 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 26 15:37:55.887222 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.885351 2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 26 15:37:55.887222 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.885362 2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 26 15:37:55.887222 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.885367 2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 26 15:37:55.887222 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.885373 2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 26 15:37:55.887222 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.885377 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 26 15:37:55.887222 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.885383 2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 26 15:37:55.887222 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.885390 2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 26 15:37:55.887222 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.885394 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 26 15:37:55.887222 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.885429 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 26 15:37:55.887222 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.885434 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 26 15:37:55.887222 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.885438 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 26 15:37:55.887222 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.885443 2577 feature_gate.go:328] unrecognized feature gate: Example Jul 26 15:37:55.887222 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.885448 2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 26 15:37:55.887222 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.885457 2577 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 26 15:37:55.887222 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.885461 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 26 15:37:55.887222 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.885465 2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 26 15:37:55.887222 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.885470 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 26 15:37:55.887222 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.885474 2577 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 26 15:37:55.887222 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.885478 2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 26 15:37:55.887675 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.885483 2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 26 15:37:55.887675 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.885487 2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 26 15:37:55.887675 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.885491 2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 26 15:37:55.887675 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.885496 2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 26 15:37:55.887675 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.885500 2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 26 15:37:55.887675 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.885504 2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 26 15:37:55.887675 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.885508 2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 26 15:37:55.887675 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.885517 2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 26 15:37:55.887675 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.885521 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 26 15:37:55.887675 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.885526 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 26 15:37:55.887675 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.885530 2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 26 15:37:55.887675 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.885534 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 26 15:37:55.887675 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.885539 2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 26 15:37:55.887675 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.885543 2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 26 15:37:55.887675 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.885547 2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 26 15:37:55.887675 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.885551 2577 feature_gate.go:328] unrecognized feature gate: Example2 Jul 26 15:37:55.887675 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.885555 2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 26 15:37:55.887675 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.885559 2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 26 15:37:55.887675 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.885564 2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 26 15:37:55.887675 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.885568 2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 26 15:37:55.888180 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.885580 2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 26 15:37:55.888180 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.885584 2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 26 15:37:55.888180 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.885614 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 26 15:37:55.888180 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.885618 2577 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 26 15:37:55.888180 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.885622 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 26 15:37:55.888180 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.885627 2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 26 15:37:55.888180 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.885631 2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 26 15:37:55.888180 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.885635 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 26 15:37:55.888180 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.885639 2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 26 15:37:55.888180 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.885643 2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 26 15:37:55.888180 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.885647 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 26 15:37:55.888180 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.885652 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 26 15:37:55.888180 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.885662 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 26 15:37:55.888180 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.885667 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 26 15:37:55.888180 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.885671 2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 26 15:37:55.888180 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.885676 2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 26 15:37:55.888180 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.885680 2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 26 15:37:55.888180 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.885685 2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 26 15:37:55.888180 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.885689 2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 26 15:37:55.888180 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.885693 2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 26 15:37:55.888671 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.885748 2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 26 15:37:55.888671 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.885934 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 26 15:37:55.888671 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.885940 2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 26 15:37:55.888671 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.885943 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 26 15:37:55.888671 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.885947 2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 26 15:37:55.888671 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.885950 2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 26 15:37:55.888671 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.885953 2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 26 15:37:55.888671 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.885956 2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 26 15:37:55.888671 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.885959 2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 26 15:37:55.888671 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.885962 2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 26 15:37:55.888671 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.885965 2577 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 26 15:37:55.888671 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.885968 2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 26 15:37:55.888671 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.885971 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 26 15:37:55.888671 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.885973 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 26 15:37:55.888671 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.885976 2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 26 15:37:55.888671 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.885979 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 26 15:37:55.888671 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.885983 2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 26 15:37:55.888671 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.885986 2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 26 15:37:55.888671 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.886423 2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 26 15:37:55.889141 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.886429 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 26 15:37:55.889141 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.886432 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 26 15:37:55.889141 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.886434 2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 26 15:37:55.889141 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.886437 2577 feature_gate.go:328] unrecognized feature gate: Example Jul 26 15:37:55.889141 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.886440 2577 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 26 15:37:55.889141 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.886443 2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 26 15:37:55.889141 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.886445 2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 26 15:37:55.889141 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.886448 2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 26 15:37:55.889141 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.886451 2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 26 15:37:55.889141 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.886454 2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 26 15:37:55.889141 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.886457 2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 26 15:37:55.889141 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.886459 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 26 15:37:55.889141 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.886462 2577 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 26 15:37:55.889141 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.886464 2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 26 15:37:55.889141 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.886468 2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 26 15:37:55.889141 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.886471 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 26 15:37:55.889141 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.886473 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 26 15:37:55.889141 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.886476 2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 26 15:37:55.889141 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.886479 2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 26 15:37:55.889141 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.886481 2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 26 15:37:55.889645 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.886484 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 26 15:37:55.889645 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.886486 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 26 15:37:55.889645 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.886489 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 26 15:37:55.889645 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.886492 2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 26 15:37:55.889645 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.886494 2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 26 15:37:55.889645 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.886497 2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 26 15:37:55.889645 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.886499 2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 26 15:37:55.889645 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.886502 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 26 15:37:55.889645 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.886505 2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 26 15:37:55.889645 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.886507 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 26 15:37:55.889645 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.886510 2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 26 15:37:55.889645 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.886513 2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 26 15:37:55.889645 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.886515 2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 26 15:37:55.889645 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.886518 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 26 15:37:55.889645 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.886520 2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 26 15:37:55.889645 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.886523 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 26 15:37:55.889645 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.886525 2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 26 15:37:55.889645 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.886528 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 26 15:37:55.889645 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.886530 2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 26 15:37:55.889645 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.886533 2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 26 15:37:55.890170 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.886535 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 26 15:37:55.890170 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.886538 2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 26 15:37:55.890170 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.886542 2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 26 15:37:55.890170 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.886546 2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 26 15:37:55.890170 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.886549 2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 26 15:37:55.890170 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.886552 2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 26 15:37:55.890170 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.886559 2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 26 15:37:55.890170 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.886563 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 26 15:37:55.890170 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.886565 2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 26 15:37:55.890170 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.886568 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 26 15:37:55.890170 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.886572 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 26 15:37:55.890170 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.886574 2577 feature_gate.go:328] unrecognized feature gate: Example2 Jul 26 15:37:55.890170 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.886577 2577 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 26 15:37:55.890170 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.886580 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 26 15:37:55.890170 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.886582 2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 26 15:37:55.890170 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.886585 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 26 15:37:55.890170 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.886588 2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 26 15:37:55.890170 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.886591 2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 26 15:37:55.890170 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.886594 2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 26 15:37:55.890640 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.886597 2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 26 15:37:55.890640 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.886600 2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 26 15:37:55.890640 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.886603 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 26 15:37:55.890640 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.886605 2577 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 26 15:37:55.890640 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.886608 2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 26 15:37:55.890640 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.886610 2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 26 15:37:55.890640 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.886613 2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 26 15:37:55.890640 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.886616 2577 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 26 15:37:55.890640 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.886619 2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 26 15:37:55.890640 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.886621 2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 26 15:37:55.890640 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.886624 2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 26 15:37:55.890640 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.886626 2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 26 15:37:55.890640 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.886629 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 26 15:37:55.890640 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.886631 2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 26 15:37:55.890640 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.886634 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 26 15:37:55.890640 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.886636 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 26 15:37:55.890640 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.886639 2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 26 15:37:55.890640 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.886642 2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 26 15:37:55.890640 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.886644 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 26 15:37:55.890640 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.886650 2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 26 15:37:55.891125 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.886653 2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 26 15:37:55.891125 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.886656 2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 26 15:37:55.891125 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.886658 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 26 15:37:55.891125 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.886662 2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 26 15:37:55.891125 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.886666 2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 26 15:37:55.891125 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.886668 2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 26 15:37:55.891125 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.886671 2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 26 15:37:55.891125 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.886675 2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 26 15:37:55.891125 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.886678 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 26 15:37:55.891125 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.886681 2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 26 15:37:55.891125 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.886684 2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 26 15:37:55.891125 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.886686 2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 26 15:37:55.891125 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.886689 2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 26 15:37:55.891125 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.886692 2577 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 26 15:37:55.891125 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.886695 2577 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 26 15:37:55.891125 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.886697 2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 26 15:37:55.891125 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.886700 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 26 15:37:55.891125 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.886703 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 26 15:37:55.891125 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.886705 2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 26 15:37:55.891577 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.886780 2577 flags.go:64] FLAG: --address="0.0.0.0" Jul 26 15:37:55.891577 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.886787 2577 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 26 15:37:55.891577 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.886802 2577 flags.go:64] FLAG: --anonymous-auth="true" Jul 26 15:37:55.891577 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.886806 2577 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 26 15:37:55.891577 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.886810 2577 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 26 15:37:55.891577 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.886813 2577 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 26 15:37:55.891577 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.886817 2577 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 26 15:37:55.891577 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.886822 2577 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 26 15:37:55.891577 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.886825 2577 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 26 15:37:55.891577 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.886828 2577 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 26 15:37:55.891577 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.886831 2577 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 26 15:37:55.891577 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.886835 2577 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 26 15:37:55.891577 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.886840 2577 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 26 15:37:55.891577 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.886843 2577 flags.go:64] FLAG: --cgroup-root="" Jul 26 15:37:55.891577 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.886846 2577 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 26 15:37:55.891577 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.886849 2577 flags.go:64] FLAG: --client-ca-file="" Jul 26 15:37:55.891577 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.886858 2577 flags.go:64] FLAG: --cloud-provider="external" Jul 26 15:37:55.891577 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.886861 2577 flags.go:64] FLAG: --cluster-dns="[]" Jul 26 15:37:55.891577 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.886868 2577 flags.go:64] FLAG: --cluster-domain="" Jul 26 15:37:55.891577 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.886870 2577 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 26 15:37:55.891577 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.886873 2577 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 26 15:37:55.891577 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.886876 2577 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 26 15:37:55.891577 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.886880 2577 flags.go:64] FLAG: --container-log-max-files="5" Jul 26 15:37:55.891577 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.886883 2577 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 26 15:37:55.892166 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.886887 2577 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 26 15:37:55.892166 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.886890 2577 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 26 15:37:55.892166 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.886893 2577 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 26 15:37:55.892166 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.886896 2577 flags.go:64] FLAG: --contention-profiling="false" Jul 26 15:37:55.892166 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.886899 2577 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 26 15:37:55.892166 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.886902 2577 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 26 15:37:55.892166 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.886905 2577 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 26 15:37:55.892166 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.886908 2577 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 26 15:37:55.892166 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.886918 2577 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 26 15:37:55.892166 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.886921 2577 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 26 15:37:55.892166 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.886924 2577 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 26 15:37:55.892166 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.886927 2577 flags.go:64] FLAG: --enable-load-reader="false" Jul 26 15:37:55.892166 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.886930 2577 flags.go:64] FLAG: --enable-server="true" Jul 26 15:37:55.892166 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.886933 2577 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 26 15:37:55.892166 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.886941 2577 flags.go:64] FLAG: --event-burst="100" Jul 26 15:37:55.892166 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.886944 2577 flags.go:64] FLAG: --event-qps="50" Jul 26 15:37:55.892166 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.886946 2577 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 26 15:37:55.892166 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.886949 2577 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 26 15:37:55.892166 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.886952 2577 flags.go:64] FLAG: --eviction-hard="" Jul 26 15:37:55.892166 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.886956 2577 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 26 15:37:55.892166 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.886961 2577 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 26 15:37:55.892166 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.886963 2577 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 26 15:37:55.892166 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.886966 2577 flags.go:64] FLAG: --eviction-soft="" Jul 26 15:37:55.892166 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.886969 2577 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 26 15:37:55.892166 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.886972 2577 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 26 15:37:55.892781 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.886981 2577 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 26 15:37:55.892781 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.886984 2577 flags.go:64] FLAG: --experimental-mounter-path="" Jul 26 15:37:55.892781 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.886987 2577 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 26 15:37:55.892781 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.886989 2577 flags.go:64] FLAG: --fail-swap-on="true" Jul 26 15:37:55.892781 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.886992 2577 flags.go:64] FLAG: --feature-gates="" Jul 26 15:37:55.892781 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.886996 2577 flags.go:64] FLAG: --file-check-frequency="20s" Jul 26 15:37:55.892781 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.886999 2577 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 26 15:37:55.892781 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.887002 2577 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 26 15:37:55.892781 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.887005 2577 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 26 15:37:55.892781 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.887008 2577 flags.go:64] FLAG: --healthz-port="10248" Jul 26 15:37:55.892781 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.887012 2577 flags.go:64] FLAG: --help="false" Jul 26 15:37:55.892781 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.887015 2577 flags.go:64] FLAG: --hostname-override="ip-10-0-138-163.ec2.internal" Jul 26 15:37:55.892781 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.887035 2577 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 26 15:37:55.892781 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.887039 2577 flags.go:64] FLAG: --http-check-frequency="20s" Jul 26 15:37:55.892781 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.887042 2577 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 26 15:37:55.892781 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.887046 2577 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 26 15:37:55.892781 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.887049 2577 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 26 15:37:55.892781 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.887052 2577 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 26 15:37:55.892781 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.887055 2577 flags.go:64] FLAG: --image-service-endpoint="" Jul 26 15:37:55.892781 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.887058 2577 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 26 15:37:55.892781 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.887061 2577 flags.go:64] FLAG: --kube-api-burst="100" Jul 26 15:37:55.892781 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.887063 2577 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 26 15:37:55.892781 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.887066 2577 flags.go:64] FLAG: --kube-api-qps="50" Jul 26 15:37:55.892781 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.887069 2577 flags.go:64] FLAG: --kube-reserved="" Jul 26 15:37:55.893383 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.887072 2577 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 26 15:37:55.893383 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.887075 2577 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 26 15:37:55.893383 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.887078 2577 flags.go:64] FLAG: --kubelet-cgroups="" Jul 26 15:37:55.893383 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.887082 2577 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 26 15:37:55.893383 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.887085 2577 flags.go:64] FLAG: --lock-file="" Jul 26 15:37:55.893383 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.887087 2577 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 26 15:37:55.893383 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.887090 2577 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 26 15:37:55.893383 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.887094 2577 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 26 15:37:55.893383 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.887099 2577 flags.go:64] FLAG: --log-json-split-stream="false" Jul 26 15:37:55.893383 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.887108 2577 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 26 15:37:55.893383 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.887111 2577 flags.go:64] FLAG: --log-text-split-stream="false" Jul 26 15:37:55.893383 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.887114 2577 flags.go:64] FLAG: --logging-format="text" Jul 26 15:37:55.893383 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.887117 2577 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 26 15:37:55.893383 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.887120 2577 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 26 15:37:55.893383 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.887123 2577 flags.go:64] FLAG: --manifest-url="" Jul 26 15:37:55.893383 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.887125 2577 flags.go:64] FLAG: --manifest-url-header="" Jul 26 15:37:55.893383 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.887132 2577 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 26 15:37:55.893383 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.887135 2577 flags.go:64] FLAG: --max-open-files="1000000" Jul 26 15:37:55.893383 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.887140 2577 flags.go:64] FLAG: --max-pods="110" Jul 26 15:37:55.893383 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.887143 2577 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 26 15:37:55.893383 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.887146 2577 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 26 15:37:55.893383 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.887149 2577 flags.go:64] FLAG: --memory-manager-policy="None" Jul 26 15:37:55.893383 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.887152 2577 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 26 15:37:55.893383 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.887155 2577 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 26 15:37:55.893383 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.887158 2577 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 26 15:37:55.894074 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.887161 2577 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 26 15:37:55.894074 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.887166 2577 flags.go:64] FLAG: --node-status-max-images="50" Jul 26 15:37:55.894074 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.887169 2577 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 26 15:37:55.894074 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.887172 2577 flags.go:64] FLAG: --oom-score-adj="-999" Jul 26 15:37:55.894074 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.887175 2577 flags.go:64] FLAG: --pod-cidr="" Jul 26 15:37:55.894074 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.887178 2577 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb" Jul 26 15:37:55.894074 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.887185 2577 flags.go:64] FLAG: --pod-manifest-path="" Jul 26 15:37:55.894074 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.887187 2577 flags.go:64] FLAG: --pod-max-pids="-1" Jul 26 15:37:55.894074 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.887190 2577 flags.go:64] FLAG: --pods-per-core="0" Jul 26 15:37:55.894074 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.887193 2577 flags.go:64] FLAG: --port="10250" Jul 26 15:37:55.894074 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.887196 2577 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 26 15:37:55.894074 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.887200 2577 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0a259c8057e27d5c5" Jul 26 15:37:55.894074 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.887204 2577 flags.go:64] FLAG: --qos-reserved="" Jul 26 15:37:55.894074 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.887206 2577 flags.go:64] FLAG: --read-only-port="10255" Jul 26 15:37:55.894074 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.887210 2577 flags.go:64] FLAG: --register-node="true" Jul 26 15:37:55.894074 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.887212 2577 flags.go:64] FLAG: --register-with-taints="" Jul 26 15:37:55.894074 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.887216 2577 flags.go:64] FLAG: --registry-burst="10" Jul 26 15:37:55.894074 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.887224 2577 flags.go:64] FLAG: --registry-qps="5" Jul 26 15:37:55.894074 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.887227 2577 flags.go:64] FLAG: --reserved-cpus="" Jul 26 15:37:55.894074 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.887231 2577 flags.go:64] FLAG: --reserved-memory="" Jul 26 15:37:55.894074 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.887234 2577 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 26 15:37:55.894074 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.887237 2577 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 26 15:37:55.894074 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.887241 2577 flags.go:64] FLAG: --rotate-certificates="false" Jul 26 15:37:55.894074 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.887243 2577 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 26 15:37:55.894074 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.887246 2577 flags.go:64] FLAG: --runonce="false" Jul 26 15:37:55.894682 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.887251 2577 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 26 15:37:55.894682 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.887254 2577 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 26 15:37:55.894682 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.887257 2577 flags.go:64] FLAG: --seccomp-default="false" Jul 26 15:37:55.894682 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.887260 2577 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 26 15:37:55.894682 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.887263 2577 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 26 15:37:55.894682 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.887266 2577 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 26 15:37:55.894682 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.887269 2577 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 26 15:37:55.894682 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.887273 2577 flags.go:64] FLAG: --storage-driver-password="root" Jul 26 15:37:55.894682 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.887275 2577 flags.go:64] FLAG: --storage-driver-secure="false" Jul 26 15:37:55.894682 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.887278 2577 flags.go:64] FLAG: --storage-driver-table="stats" Jul 26 15:37:55.894682 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.887281 2577 flags.go:64] FLAG: --storage-driver-user="root" Jul 26 15:37:55.894682 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.887284 2577 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 26 15:37:55.894682 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.887287 2577 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 26 15:37:55.894682 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.887290 2577 flags.go:64] FLAG: --system-cgroups="" Jul 26 15:37:55.894682 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.887292 2577 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 26 15:37:55.894682 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.887298 2577 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 26 15:37:55.894682 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.887314 2577 flags.go:64] FLAG: --tls-cert-file="" Jul 26 15:37:55.894682 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.887318 2577 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 26 15:37:55.894682 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.887327 2577 flags.go:64] FLAG: --tls-min-version="" Jul 26 15:37:55.894682 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.887330 2577 flags.go:64] FLAG: --tls-private-key-file="" Jul 26 15:37:55.894682 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.887333 2577 flags.go:64] FLAG: --topology-manager-policy="none" Jul 26 15:37:55.894682 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.887336 2577 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 26 15:37:55.894682 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.887339 2577 flags.go:64] FLAG: --topology-manager-scope="container" Jul 26 15:37:55.894682 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.887342 2577 flags.go:64] FLAG: --v="2" Jul 26 15:37:55.894682 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.887346 2577 flags.go:64] FLAG: --version="false" Jul 26 15:37:55.895304 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.887356 2577 flags.go:64] FLAG: --vmodule="" Jul 26 15:37:55.895304 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.887360 2577 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 26 15:37:55.895304 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.887363 2577 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 26 15:37:55.895304 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.887493 2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 26 15:37:55.895304 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.887497 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 26 15:37:55.895304 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.887500 2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 26 15:37:55.895304 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.887503 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 26 15:37:55.895304 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.887507 2577 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 26 15:37:55.895304 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.887510 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 26 15:37:55.895304 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.887513 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 26 15:37:55.895304 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.887516 2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 26 15:37:55.895304 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.887518 2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 26 15:37:55.895304 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.887521 2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 26 15:37:55.895304 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.887523 2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 26 15:37:55.895304 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.887526 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 26 15:37:55.895304 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.887529 2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 26 15:37:55.895304 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.887532 2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 26 15:37:55.895304 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.887534 2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 26 15:37:55.895304 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.887537 2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 26 15:37:55.895304 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.887539 2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 26 15:37:55.895790 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.887542 2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 26 15:37:55.895790 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.887544 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 26 15:37:55.895790 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.887547 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 26 15:37:55.895790 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.887549 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 26 15:37:55.895790 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.887552 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 26 15:37:55.895790 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.887556 2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 26 15:37:55.895790 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.887558 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 26 15:37:55.895790 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.887561 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 26 15:37:55.895790 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.887564 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 26 15:37:55.895790 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.887566 2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 26 15:37:55.895790 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.887569 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 26 15:37:55.895790 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.887571 2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 26 15:37:55.895790 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.887574 2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 26 15:37:55.895790 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.887582 2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 26 15:37:55.895790 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.887585 2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 26 15:37:55.895790 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.887587 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 26 15:37:55.895790 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.887590 2577 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 26 15:37:55.895790 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.887592 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 26 15:37:55.895790 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.887595 2577 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 26 15:37:55.895790 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.887599 2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 26 15:37:55.896288 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.887602 2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 26 15:37:55.896288 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.887604 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 26 15:37:55.896288 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.887607 2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 26 15:37:55.896288 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.887609 2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 26 15:37:55.896288 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.887612 2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 26 15:37:55.896288 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.887615 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 26 15:37:55.896288 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.887618 2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 26 15:37:55.896288 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.887620 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 26 15:37:55.896288 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.887623 2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 26 15:37:55.896288 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.887626 2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 26 15:37:55.896288 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.887628 2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 26 15:37:55.896288 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.887631 2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 26 15:37:55.896288 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.887633 2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 26 15:37:55.896288 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.887636 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 26 15:37:55.896288 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.887638 2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 26 15:37:55.896288 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.887641 2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 26 15:37:55.896288 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.887644 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 26 15:37:55.896288 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.887647 2577 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 26 15:37:55.896288 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.887650 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 26 15:37:55.896288 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.887653 2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 26 15:37:55.896783 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.887655 2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 26 15:37:55.896783 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.887658 2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 26 15:37:55.896783 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.887661 2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 26 15:37:55.896783 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.887663 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 26 15:37:55.896783 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.887666 2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 26 15:37:55.896783 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.887668 2577 feature_gate.go:328] unrecognized feature gate: Example Jul 26 15:37:55.896783 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.887677 2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 26 15:37:55.896783 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.887680 2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 26 15:37:55.896783 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.887682 2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 26 15:37:55.896783 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.887685 2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 26 15:37:55.896783 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.887689 2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 26 15:37:55.896783 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.887694 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 26 15:37:55.896783 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.887696 2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 26 15:37:55.896783 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.887699 2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 26 15:37:55.896783 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.887701 2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 26 15:37:55.896783 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.887704 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 26 15:37:55.896783 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.887707 2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 26 15:37:55.896783 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.887709 2577 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 26 15:37:55.896783 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.887712 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 26 15:37:55.897256 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.887714 2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 26 15:37:55.897256 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.887717 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 26 15:37:55.897256 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.887719 2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 26 15:37:55.897256 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.887721 2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 26 15:37:55.897256 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.887724 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 26 15:37:55.897256 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.887726 2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 26 15:37:55.897256 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.887729 2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 26 15:37:55.897256 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.887731 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 26 15:37:55.897256 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.887734 2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 26 15:37:55.897256 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.887738 2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 26 15:37:55.897256 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.887743 2577 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 26 15:37:55.897256 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.887747 2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 26 15:37:55.897256 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.887750 2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 26 15:37:55.897256 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.887753 2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 26 15:37:55.897256 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.887755 2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 26 15:37:55.897256 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.887758 2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 26 15:37:55.897256 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.887761 2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 26 15:37:55.897256 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.887764 2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 26 15:37:55.897256 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.887767 2577 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 26 15:37:55.897711 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.887770 2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 26 15:37:55.897711 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.887778 2577 feature_gate.go:328] unrecognized feature gate: Example2 Jul 26 15:37:55.897711 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.887781 2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 26 15:37:55.897711 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.887784 2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 26 15:37:55.897711 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.888521 2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 26 15:37:55.897711 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.895611 2577 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 26 15:37:55.897711 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.895624 2577 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 26 15:37:55.897711 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.895641 2577 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 26 15:37:55.897711 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.895647 2577 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 26 15:37:55.897711 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.895711 2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 26 15:37:55.897711 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.895717 2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 26 15:37:55.897711 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.895722 2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 26 15:37:55.897711 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.895725 2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 26 15:37:55.897711 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.895729 2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 26 15:37:55.898073 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.895731 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 26 15:37:55.898073 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.895734 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 26 15:37:55.898073 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.895737 2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 26 15:37:55.898073 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.895739 2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 26 15:37:55.898073 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.895742 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 26 15:37:55.898073 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.895744 2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 26 15:37:55.898073 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.895747 2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 26 15:37:55.898073 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.895750 2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 26 15:37:55.898073 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.895752 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 26 15:37:55.898073 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.895755 2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 26 15:37:55.898073 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.895757 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 26 15:37:55.898073 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.895760 2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 26 15:37:55.898073 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.895762 2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 26 15:37:55.898073 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.895765 2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 26 15:37:55.898073 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.895768 2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 26 15:37:55.898073 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.895770 2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 26 15:37:55.898073 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.895773 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 26 15:37:55.898073 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.895776 2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 26 15:37:55.898073 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.895778 2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 26 15:37:55.898073 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.895783 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 26 15:37:55.898552 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.895786 2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 26 15:37:55.898552 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.895789 2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 26 15:37:55.898552 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.895791 2577 feature_gate.go:328] unrecognized feature gate: Example2 Jul 26 15:37:55.898552 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.895794 2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 26 15:37:55.898552 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.895798 2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 26 15:37:55.898552 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.895801 2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 26 15:37:55.898552 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.895804 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 26 15:37:55.898552 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.895806 2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 26 15:37:55.898552 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.895809 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 26 15:37:55.898552 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.895811 2577 feature_gate.go:328] unrecognized feature gate: Example Jul 26 15:37:55.898552 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.895814 2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 26 15:37:55.898552 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.895817 2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 26 15:37:55.898552 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.895819 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 26 15:37:55.898552 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.895822 2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 26 15:37:55.898552 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.895825 2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 26 15:37:55.898552 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.895828 2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 26 15:37:55.898552 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.895830 2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 26 15:37:55.898552 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.895833 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 26 15:37:55.898552 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.895836 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 26 15:37:55.898552 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.895838 2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 26 15:37:55.899090 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.895842 2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 26 15:37:55.899090 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.895846 2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 26 15:37:55.899090 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.895848 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 26 15:37:55.899090 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.895851 2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 26 15:37:55.899090 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.895854 2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 26 15:37:55.899090 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.895857 2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 26 15:37:55.899090 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.895860 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 26 15:37:55.899090 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.895862 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 26 15:37:55.899090 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.895865 2577 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 26 15:37:55.899090 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.895868 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 26 15:37:55.899090 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.895870 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 26 15:37:55.899090 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.895873 2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 26 15:37:55.899090 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.895876 2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 26 15:37:55.899090 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.895879 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 26 15:37:55.899090 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.895881 2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 26 15:37:55.899090 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.895884 2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 26 15:37:55.899090 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.895886 2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 26 15:37:55.899090 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.895889 2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 26 15:37:55.899090 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.895891 2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 26 15:37:55.899592 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.895894 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 26 15:37:55.899592 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.895896 2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 26 15:37:55.899592 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.895899 2577 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 26 15:37:55.899592 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.895901 2577 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 26 15:37:55.899592 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.895904 2577 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 26 15:37:55.899592 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.895906 2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 26 15:37:55.899592 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.895909 2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 26 15:37:55.899592 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.895912 2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 26 15:37:55.899592 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.895914 2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 26 15:37:55.899592 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.895917 2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 26 15:37:55.899592 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.895920 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 26 15:37:55.899592 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.895922 2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 26 15:37:55.899592 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.895925 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 26 15:37:55.899592 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.895927 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 26 15:37:55.899592 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.895929 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 26 15:37:55.899592 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.895932 2577 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 26 15:37:55.899592 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.895934 2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 26 15:37:55.899592 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.895937 2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 26 15:37:55.899592 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.895939 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 26 15:37:55.899592 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.895942 2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 26 15:37:55.900101 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.895945 2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 26 15:37:55.900101 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.895948 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 26 15:37:55.900101 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.895950 2577 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 26 15:37:55.900101 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.895953 2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 26 15:37:55.900101 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.895955 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 26 15:37:55.900101 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.895958 2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 26 15:37:55.900101 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.895961 2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 26 15:37:55.900101 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.895964 2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 26 15:37:55.900101 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.895966 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 26 15:37:55.900101 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.895969 2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 26 15:37:55.900101 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.895971 2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 26 15:37:55.900101 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.895974 2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 26 15:37:55.900101 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.895976 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 26 15:37:55.900101 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.895979 2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 26 15:37:55.900101 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.895982 2577 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 26 15:37:55.900488 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.895987 2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 26 15:37:55.900488 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.896131 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 26 15:37:55.900488 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.896137 2577 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 26 15:37:55.900488 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.896141 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 26 15:37:55.900488 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.896144 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 26 15:37:55.900488 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.896146 2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 26 15:37:55.900488 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.896149 2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 26 15:37:55.900488 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.896152 2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 26 15:37:55.900488 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.896155 2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 26 15:37:55.900488 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.896158 2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 26 15:37:55.900488 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.896160 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 26 15:37:55.900488 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.896163 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 26 15:37:55.900488 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.896166 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 26 15:37:55.900488 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.896168 2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 26 15:37:55.900488 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.896171 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 26 15:37:55.900860 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.896173 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 26 15:37:55.900860 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.896176 2577 feature_gate.go:328] unrecognized feature gate: Example2 Jul 26 15:37:55.900860 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.896179 2577 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 26 15:37:55.900860 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.896182 2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 26 15:37:55.900860 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.896184 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 26 15:37:55.900860 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.896187 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 26 15:37:55.900860 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.896189 2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 26 15:37:55.900860 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.896192 2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 26 15:37:55.900860 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.896195 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 26 15:37:55.900860 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.896198 2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 26 15:37:55.900860 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.896200 2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 26 15:37:55.900860 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.896203 2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 26 15:37:55.900860 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.896205 2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 26 15:37:55.900860 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.896208 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 26 15:37:55.900860 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.896210 2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 26 15:37:55.900860 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.896212 2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 26 15:37:55.900860 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.896215 2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 26 15:37:55.900860 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.896217 2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 26 15:37:55.900860 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.896220 2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 26 15:37:55.900860 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.896222 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 26 15:37:55.901371 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.896225 2577 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 26 15:37:55.901371 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.896227 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 26 15:37:55.901371 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.896230 2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 26 15:37:55.901371 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.896233 2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 26 15:37:55.901371 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.896235 2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 26 15:37:55.901371 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.896238 2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 26 15:37:55.901371 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.896241 2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 26 15:37:55.901371 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.896243 2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 26 15:37:55.901371 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.896246 2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 26 15:37:55.901371 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.896248 2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 26 15:37:55.901371 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.896251 2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 26 15:37:55.901371 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.896253 2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 26 15:37:55.901371 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.896256 2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 26 15:37:55.901371 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.896259 2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 26 15:37:55.901371 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.896261 2577 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 26 15:37:55.901371 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.896264 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 26 15:37:55.901371 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.896267 2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 26 15:37:55.901371 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.896269 2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 26 15:37:55.901371 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.896272 2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 26 15:37:55.901371 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.896274 2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 26 15:37:55.901847 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.896277 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 26 15:37:55.901847 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.896281 2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 26 15:37:55.901847 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.896285 2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 26 15:37:55.901847 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.896289 2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 26 15:37:55.901847 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.896292 2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 26 15:37:55.901847 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.896295 2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 26 15:37:55.901847 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.896306 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 26 15:37:55.901847 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.896310 2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 26 15:37:55.901847 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.896313 2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 26 15:37:55.901847 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.896317 2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 26 15:37:55.901847 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.896320 2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 26 15:37:55.901847 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.896322 2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 26 15:37:55.901847 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.896325 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 26 15:37:55.901847 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.896328 2577 feature_gate.go:328] unrecognized feature gate: Example Jul 26 15:37:55.901847 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.896330 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 26 15:37:55.901847 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.896333 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 26 15:37:55.901847 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.896335 2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 26 15:37:55.901847 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.896338 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 26 15:37:55.902298 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.896341 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 26 15:37:55.902298 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.896343 2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 26 15:37:55.902298 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.896346 2577 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 26 15:37:55.902298 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.896349 2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 26 15:37:55.902298 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.896351 2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 26 15:37:55.902298 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.896354 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 26 15:37:55.902298 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.896356 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 26 15:37:55.902298 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.896359 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 26 15:37:55.902298 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.896362 2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 26 15:37:55.902298 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.896364 2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 26 15:37:55.902298 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.896367 2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 26 15:37:55.902298 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.896370 2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 26 15:37:55.902298 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.896373 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 26 15:37:55.902298 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.896376 2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 26 15:37:55.902298 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.896379 2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 26 15:37:55.902298 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.896382 2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 26 15:37:55.902298 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.896384 2577 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 26 15:37:55.902298 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.896393 2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 26 15:37:55.902298 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.896396 2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 26 15:37:55.902298 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.896399 2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 26 15:37:55.902807 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.896402 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 26 15:37:55.902807 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.896405 2577 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 26 15:37:55.902807 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.896408 2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 26 15:37:55.902807 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.896410 2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 26 15:37:55.902807 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.896413 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 26 15:37:55.902807 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.896415 2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 26 15:37:55.902807 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:55.896418 2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 26 15:37:55.902807 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.896423 2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 26 15:37:55.902807 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.896521 2577 server.go:962] "Client rotation is on, will bootstrap in background" Jul 26 15:37:55.902807 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.898782 2577 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 26 15:37:55.902807 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.900364 2577 server.go:1019] "Starting client certificate rotation" Jul 26 15:37:55.902807 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.900462 2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 26 15:37:55.902807 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.900493 2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 26 15:37:55.903345 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.903331 2577 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 26 15:37:55.908187 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.908143 2577 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 26 15:37:55.917867 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.917843 2577 log.go:25] "Validated CRI v1 runtime API" Jul 26 15:37:55.920481 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.920465 2577 log.go:25] "Validated CRI v1 image API" Jul 26 15:37:55.923676 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.923660 2577 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 26 15:37:55.927741 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.927718 2577 fs.go:135] Filesystem UUIDs: map[1258cc67-6b0e-472b-8f93-1b65d005e17e:/dev/nvme0n1p3 278c1845-785c-46ad-b50d-4343f5021641:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2] Jul 26 15:37:55.927799 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.927741 2577 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 26 15:37:55.934223 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.934100 2577 manager.go:216] Machine: {Timestamp:2026-07-26 15:37:55.932325613 +0000 UTC m=+0.373054865 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099711 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2093a1bc989b96d8af3407f083f5e4 SystemUUID:ec2093a1-bc98-9b96-d8af-3407f083f5e4 BootID:51fd39a9-ac3c-4a32-92b7-6325449a7e6b Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ab:e4:2b:93:19 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ab:e4:2b:93:19 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:d2:ec:c6:2e:43:4b Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 26 15:37:55.934223 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.934211 2577 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 26 15:37:55.934367 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.934294 2577 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 26 15:37:55.934874 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.934856 2577 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 26 15:37:55.938106 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.938078 2577 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 26 15:37:55.938260 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.938110 2577 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-163.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 26 15:37:55.938934 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.938923 2577 topology_manager.go:138] "Creating topology manager with none policy" Jul 26 15:37:55.938966 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.938937 2577 container_manager_linux.go:309] "Creating device plugin manager" Jul 26 15:37:55.938966 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.938945 2577 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 26 15:37:55.939675 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.939664 2577 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 26 15:37:55.940815 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.940804 2577 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 26 15:37:55.943730 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.943715 2577 state_mem.go:36] "Initialized new in-memory state store" Jul 26 15:37:55.943873 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.943863 2577 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 26 15:37:55.943910 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.943898 2577 kubelet.go:486] "Attempting to sync node with API server" Jul 26 15:37:55.943938 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.943911 2577 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 26 15:37:55.943938 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.943926 2577 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 26 15:37:55.943938 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.943936 2577 kubelet.go:398] "Adding apiserver pod source" Jul 26 15:37:55.944050 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.943945 2577 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 26 15:37:55.944387 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.944374 2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 26 15:37:55.944429 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.944392 2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 26 15:37:55.947715 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.947691 2577 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 26 15:37:55.950228 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.950207 2577 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 26 15:37:55.950352 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.950249 2577 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 26 15:37:55.952201 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.952186 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 26 15:37:55.952287 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.952207 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 26 15:37:55.952287 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.952216 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 26 15:37:55.952287 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.952224 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 26 15:37:55.952287 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.952236 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 26 15:37:55.952287 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.952244 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 26 15:37:55.952287 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.952256 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 26 15:37:55.952287 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.952264 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 26 15:37:55.952287 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.952282 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 26 15:37:55.952538 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.952291 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 26 15:37:55.952538 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.952316 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 26 15:37:55.952538 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.952324 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 26 15:37:55.954073 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.954061 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 26 15:37:55.954145 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.954076 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 26 15:37:55.955819 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:37:55.955792 2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 26 15:37:55.955819 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:37:55.955800 2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-138-163.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 26 15:37:55.956009 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.955974 2577 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-163.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 26 15:37:55.957377 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.957362 2577 server.go:1268] "Started kubelet" Jul 26 15:37:55.957520 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.957475 2577 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 26 15:37:55.957598 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.957546 2577 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 26 15:37:55.957638 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.957621 2577 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 26 15:37:55.958340 ip-10-0-138-163 systemd[1]: Started Kubernetes Kubelet. Jul 26 15:37:55.962451 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.962255 2577 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 26 15:37:55.963793 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.963776 2577 server.go:310] "Adding debug handlers to kubelet server" Jul 26 15:37:55.964696 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:37:55.964651 2577 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 26 15:37:55.973751 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.973730 2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-v7xc9" Jul 26 15:37:55.974127 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.974107 2577 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 26 15:37:55.974127 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.974116 2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 26 15:37:55.974275 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:37:55.974186 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-163.ec2.internal\" not found" Jul 26 15:37:55.974935 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.974918 2577 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 26 15:37:55.974935 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.974922 2577 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 26 15:37:55.974935 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.974945 2577 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 26 15:37:55.975231 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.974996 2577 factory.go:55] Registering systemd factory Jul 26 15:37:55.975231 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.975012 2577 factory.go:223] Registration of the systemd container factory successfully Jul 26 15:37:55.975231 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.975105 2577 reconstruct.go:97] "Volume reconstruction finished" Jul 26 15:37:55.975231 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.975115 2577 reconciler.go:29] "Reconciler: start to sync state" Jul 26 15:37:55.975440 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.975338 2577 factory.go:153] Registering CRI-O factory Jul 26 15:37:55.975440 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.975353 2577 factory.go:223] Registration of the crio container factory successfully Jul 26 15:37:55.975440 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.975401 2577 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 26 15:37:55.975440 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.975426 2577 factory.go:103] Registering Raw factory Jul 26 15:37:55.975440 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.975440 2577 manager.go:1196] Started watching for new ooms in manager Jul 26 15:37:55.975960 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.975900 2577 manager.go:349] Starting recovery of all containers Jul 26 15:37:55.976745 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:37:55.976716 2577 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-138-163.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 26 15:37:55.976869 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:37:55.976850 2577 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 26 15:37:55.977947 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:37:55.976789 2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-163.ec2.internal.18c5e10d9ffe437a default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-163.ec2.internal,UID:ip-10-0-138-163.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-138-163.ec2.internal,},FirstTimestamp:2026-07-26 15:37:55.957334906 +0000 UTC m=+0.398064208,LastTimestamp:2026-07-26 15:37:55.957334906 +0000 UTC m=+0.398064208,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-163.ec2.internal,}" Jul 26 15:37:55.983737 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.983716 2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-v7xc9" Jul 26 15:37:55.985856 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.985844 2577 manager.go:354] Recovery completed Jul 26 15:37:55.987254 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:37:55.987230 2577 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory Jul 26 15:37:55.990961 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.990947 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 26 15:37:55.994519 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.994502 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-163.ec2.internal" event="NodeHasSufficientMemory" Jul 26 15:37:55.994580 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.994534 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-163.ec2.internal" event="NodeHasNoDiskPressure" Jul 26 15:37:55.994580 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.994547 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-163.ec2.internal" event="NodeHasSufficientPID" Jul 26 15:37:55.995166 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.995150 2577 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 26 15:37:55.995166 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.995167 2577 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 26 15:37:55.995304 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.995180 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 26 15:37:55.995304 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.995184 2577 state_mem.go:36] "Initialized new in-memory state store" Jul 26 15:37:55.995966 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.995948 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-163.ec2.internal" event="NodeHasSufficientMemory" Jul 26 15:37:55.996101 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.995977 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-163.ec2.internal" event="NodeHasNoDiskPressure" Jul 26 15:37:55.996101 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.996002 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-163.ec2.internal" event="NodeHasSufficientPID" Jul 26 15:37:55.998316 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.998302 2577 policy_none.go:49] "None policy: Start" Jul 26 15:37:55.998316 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.998318 2577 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 26 15:37:55.998430 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:55.998328 2577 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 26 15:37:56.000328 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:56.000314 2577 policy_none.go:47] "Start" Jul 26 15:37:56.057341 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:56.043623 2577 manager.go:337] "Starting Device Plugin manager" Jul 26 15:37:56.057341 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:37:56.043656 2577 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 26 15:37:56.057341 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:56.043665 2577 server.go:87] "Starting device plugin registration server" Jul 26 15:37:56.057341 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:56.043941 2577 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 26 15:37:56.057341 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:56.043953 2577 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 26 15:37:56.057341 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:56.044158 2577 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 26 15:37:56.057341 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:56.044247 2577 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 26 15:37:56.057341 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:56.044256 2577 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 26 15:37:56.057341 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:37:56.044813 2577 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 26 15:37:56.057341 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:37:56.044864 2577 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-163.ec2.internal\" not found" Jul 26 15:37:56.078045 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:56.078002 2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 26 15:37:56.079186 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:56.079161 2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 26 15:37:56.079314 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:56.079211 2577 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 26 15:37:56.079314 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:56.079239 2577 kubelet.go:2448] "Starting kubelet main sync loop" Jul 26 15:37:56.079314 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:37:56.079283 2577 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 26 15:37:56.081882 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:56.081862 2577 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 26 15:37:56.145083 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:56.144943 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 26 15:37:56.146100 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:56.146078 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-163.ec2.internal" event="NodeHasSufficientMemory" Jul 26 15:37:56.146221 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:56.146117 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-163.ec2.internal" event="NodeHasNoDiskPressure" Jul 26 15:37:56.146221 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:56.146132 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-163.ec2.internal" event="NodeHasSufficientPID" Jul 26 15:37:56.146221 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:56.146163 2577 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-138-163.ec2.internal" Jul 26 15:37:56.157864 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:56.157841 2577 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-138-163.ec2.internal" Jul 26 15:37:56.157967 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:37:56.157867 2577 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-138-163.ec2.internal\": node \"ip-10-0-138-163.ec2.internal\" not found" Jul 26 15:37:56.179994 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:56.179818 2577 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-138-163.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-163.ec2.internal"] Jul 26 15:37:56.180144 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:56.180044 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 26 15:37:56.181079 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:56.181053 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-163.ec2.internal" event="NodeHasSufficientMemory" Jul 26 15:37:56.181208 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:56.181090 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-163.ec2.internal" event="NodeHasNoDiskPressure" Jul 26 15:37:56.181208 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:56.181105 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-163.ec2.internal" event="NodeHasSufficientPID" Jul 26 15:37:56.182000 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:37:56.181983 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-163.ec2.internal\" not found" Jul 26 15:37:56.183551 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:56.183534 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 26 15:37:56.183652 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:56.183635 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-163.ec2.internal" Jul 26 15:37:56.183701 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:56.183666 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 26 15:37:56.184356 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:56.184340 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-163.ec2.internal" event="NodeHasSufficientMemory" Jul 26 15:37:56.184428 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:56.184367 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-163.ec2.internal" event="NodeHasNoDiskPressure" Jul 26 15:37:56.184428 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:56.184377 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-163.ec2.internal" event="NodeHasSufficientPID" Jul 26 15:37:56.184428 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:56.184340 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-163.ec2.internal" event="NodeHasSufficientMemory" Jul 26 15:37:56.184517 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:56.184436 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-163.ec2.internal" event="NodeHasNoDiskPressure" Jul 26 15:37:56.184517 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:56.184449 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-163.ec2.internal" event="NodeHasSufficientPID" Jul 26 15:37:56.186780 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:56.186765 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-163.ec2.internal" Jul 26 15:37:56.186848 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:56.186792 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 26 15:37:56.187549 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:56.187532 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-163.ec2.internal" event="NodeHasSufficientMemory" Jul 26 15:37:56.187648 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:56.187560 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-163.ec2.internal" event="NodeHasNoDiskPressure" Jul 26 15:37:56.187648 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:56.187573 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-163.ec2.internal" event="NodeHasSufficientPID" Jul 26 15:37:56.210925 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:37:56.210899 2577 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-163.ec2.internal\" not found" node="ip-10-0-138-163.ec2.internal" Jul 26 15:37:56.215382 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:37:56.215365 2577 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-163.ec2.internal\" not found" node="ip-10-0-138-163.ec2.internal" Jul 26 15:37:56.277649 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:56.277622 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/34c5f0125abc8dd4d5a5ee6814e83607-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-163.ec2.internal\" (UID: \"34c5f0125abc8dd4d5a5ee6814e83607\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-163.ec2.internal" Jul 26 15:37:56.277749 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:56.277652 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/34c5f0125abc8dd4d5a5ee6814e83607-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-163.ec2.internal\" (UID: \"34c5f0125abc8dd4d5a5ee6814e83607\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-163.ec2.internal" Jul 26 15:37:56.277749 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:56.277672 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0d03d973abb0a97c200617d39e2704ee-config\") pod \"kube-apiserver-proxy-ip-10-0-138-163.ec2.internal\" (UID: \"0d03d973abb0a97c200617d39e2704ee\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-163.ec2.internal" Jul 26 15:37:56.282721 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:37:56.282703 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-163.ec2.internal\" not found" Jul 26 15:37:56.378198 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:56.378169 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0d03d973abb0a97c200617d39e2704ee-config\") pod \"kube-apiserver-proxy-ip-10-0-138-163.ec2.internal\" (UID: \"0d03d973abb0a97c200617d39e2704ee\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-163.ec2.internal" Jul 26 15:37:56.378272 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:56.378205 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/34c5f0125abc8dd4d5a5ee6814e83607-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-163.ec2.internal\" (UID: \"34c5f0125abc8dd4d5a5ee6814e83607\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-163.ec2.internal" Jul 26 15:37:56.378272 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:56.378226 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/34c5f0125abc8dd4d5a5ee6814e83607-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-163.ec2.internal\" (UID: \"34c5f0125abc8dd4d5a5ee6814e83607\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-163.ec2.internal" Jul 26 15:37:56.378332 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:56.378271 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0d03d973abb0a97c200617d39e2704ee-config\") pod \"kube-apiserver-proxy-ip-10-0-138-163.ec2.internal\" (UID: \"0d03d973abb0a97c200617d39e2704ee\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-163.ec2.internal" Jul 26 15:37:56.378332 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:56.378274 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/34c5f0125abc8dd4d5a5ee6814e83607-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-163.ec2.internal\" (UID: \"34c5f0125abc8dd4d5a5ee6814e83607\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-163.ec2.internal" Jul 26 15:37:56.378332 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:56.378276 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/34c5f0125abc8dd4d5a5ee6814e83607-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-163.ec2.internal\" (UID: \"34c5f0125abc8dd4d5a5ee6814e83607\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-163.ec2.internal" Jul 26 15:37:56.383258 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:37:56.383241 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-163.ec2.internal\" not found" Jul 26 15:37:56.484136 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:37:56.484048 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-163.ec2.internal\" not found" Jul 26 15:37:56.513264 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:56.513242 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-163.ec2.internal" Jul 26 15:37:56.519120 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:56.519091 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-163.ec2.internal" Jul 26 15:37:58.286021 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:37:58.286001 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-163.ec2.internal\" not found" Jul 26 15:37:58.386607 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:37:58.386581 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-163.ec2.internal\" not found" Jul 26 15:37:58.487170 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:37:58.487067 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-163.ec2.internal\" not found" Jul 26 15:37:58.495017 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.494997 2577 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 26 15:37:58.576282 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.576242 2577 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-163.ec2.internal" Jul 26 15:37:58.583938 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.583920 2577 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 26 15:37:58.585595 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.585575 2577 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-163.ec2.internal" Jul 26 15:37:58.596456 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.596440 2577 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 26 15:37:58.600581 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.600562 2577 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 26 15:37:58.600706 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.600679 2577 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 26 15:37:58.646345 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.646316 2577 apiserver.go:52] "Watching apiserver" Jul 26 15:37:58.653441 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.653418 2577 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 26 15:37:58.655951 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.655473 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-v46ns","openshift-ovn-kubernetes/ovnkube-node-ztlp6","kube-system/global-pull-secret-syncer-l56b4","kube-system/konnectivity-agent-jbpx6","kube-system/kube-apiserver-proxy-ip-10-0-138-163.ec2.internal","openshift-cluster-node-tuning-operator/tuned-65dgw","openshift-dns/node-resolver-v4mgw","openshift-image-registry/node-ca-mfwqb","openshift-multus/multus-additional-cni-plugins-dzc7t","openshift-multus/multus-x4dkt","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qnht4","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-163.ec2.internal","openshift-multus/network-metrics-daemon-kgk6w","openshift-network-diagnostics/network-check-target-d2nnz"] Jul 26 15:37:58.658102 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.658079 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-65dgw" Jul 26 15:37:58.660251 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.660226 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 26 15:37:58.660349 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.660298 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-7k9gr\"" Jul 26 15:37:58.660349 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.660228 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 26 15:37:58.660825 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.660808 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d2nnz" Jul 26 15:37:58.660912 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:37:58.660890 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d2nnz" podUID="f3e728c9-4710-4f6c-977f-caf52a2bfcaf" Jul 26 15:37:58.662811 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.662790 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-x4dkt" Jul 26 15:37:58.664885 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.664865 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 26 15:37:58.664983 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.664924 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 26 15:37:58.664983 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.664974 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 26 15:37:58.665091 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.664999 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-v46ns" Jul 26 15:37:58.665201 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.665185 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-wp755\"" Jul 26 15:37:58.665356 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.665345 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 26 15:37:58.666990 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.666969 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-cksct\"" Jul 26 15:37:58.667077 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.666980 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 26 15:37:58.667168 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.667157 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 26 15:37:58.667224 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.667211 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-jbpx6" Jul 26 15:37:58.667475 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.667460 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 26 15:37:58.669114 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.669093 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-28cnb\"" Jul 26 15:37:58.669208 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.669133 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 26 15:37:58.669366 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.669352 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 26 15:37:58.669882 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.669867 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-v4mgw" Jul 26 15:37:58.671865 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.671849 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 26 15:37:58.671865 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.671864 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 26 15:37:58.671991 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.671881 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-kdph2\"" Jul 26 15:37:58.672139 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.672122 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-mfwqb" Jul 26 15:37:58.674934 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.674918 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-dzc7t" Jul 26 15:37:58.675177 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.675041 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 26 15:37:58.675383 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.675362 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 26 15:37:58.675518 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.675503 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 26 15:37:58.675570 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.675526 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-w6m7w\"" Jul 26 15:37:58.675625 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.675613 2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 26 15:37:58.677010 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.676993 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 26 15:37:58.677110 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.677092 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 26 15:37:58.677168 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.677123 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-tf5pw\"" Jul 26 15:37:58.677336 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.677318 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l56b4" Jul 26 15:37:58.677473 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:37:58.677410 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l56b4" podUID="f6bd9013-635a-4af8-b0ee-621a280a459a" Jul 26 15:37:58.679679 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.679658 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qnht4" Jul 26 15:37:58.681758 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.681742 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 26 15:37:58.681855 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.681813 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 26 15:37:58.682052 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.682036 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-tfj84\"" Jul 26 15:37:58.682133 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.682068 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 26 15:37:58.682364 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.682348 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kgk6w" Jul 26 15:37:58.682466 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:37:58.682443 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kgk6w" podUID="b5f442b1-a918-43a5-9870-f2183fe5085a" Jul 26 15:37:58.682635 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.682615 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f6bd9013-635a-4af8-b0ee-621a280a459a-kubelet-config\") pod \"global-pull-secret-syncer-l56b4\" (UID: \"f6bd9013-635a-4af8-b0ee-621a280a459a\") " pod="kube-system/global-pull-secret-syncer-l56b4" Jul 26 15:37:58.682709 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.682685 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/098dd2e5-9540-425f-ba4e-71ea0b064be7-serviceca\") pod \"node-ca-mfwqb\" (UID: \"098dd2e5-9540-425f-ba4e-71ea0b064be7\") " pod="openshift-image-registry/node-ca-mfwqb" Jul 26 15:37:58.682747 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.682711 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bd02936a-a782-4e31-89bb-1292065d0abc-cnibin\") pod \"multus-x4dkt\" (UID: \"bd02936a-a782-4e31-89bb-1292065d0abc\") " pod="openshift-multus/multus-x4dkt" Jul 26 15:37:58.682747 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.682734 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/bd02936a-a782-4e31-89bb-1292065d0abc-os-release\") pod \"multus-x4dkt\" (UID: \"bd02936a-a782-4e31-89bb-1292065d0abc\") " pod="openshift-multus/multus-x4dkt" Jul 26 15:37:58.682824 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.682800 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/acf5fc6d-b7e4-4581-8878-f5776e505746-host-slash\") pod \"iptables-alerter-v46ns\" (UID: \"acf5fc6d-b7e4-4581-8878-f5776e505746\") " pod="openshift-network-operator/iptables-alerter-v46ns" Jul 26 15:37:58.682864 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.682834 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f6bd9013-635a-4af8-b0ee-621a280a459a-dbus\") pod \"global-pull-secret-syncer-l56b4\" (UID: \"f6bd9013-635a-4af8-b0ee-621a280a459a\") " pod="kube-system/global-pull-secret-syncer-l56b4" Jul 26 15:37:58.682864 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.682852 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7c179a4b-e953-4d22-9f2d-7cf682e833fd-run\") pod \"tuned-65dgw\" (UID: \"7c179a4b-e953-4d22-9f2d-7cf682e833fd\") " pod="openshift-cluster-node-tuning-operator/tuned-65dgw" Jul 26 15:37:58.682931 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.682899 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/acf5fc6d-b7e4-4581-8878-f5776e505746-iptables-alerter-script\") pod \"iptables-alerter-v46ns\" (UID: \"acf5fc6d-b7e4-4581-8878-f5776e505746\") " pod="openshift-network-operator/iptables-alerter-v46ns" Jul 26 15:37:58.682966 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.682933 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c2653055-9d7a-4aeb-9090-fb495a2b650b-cnibin\") pod \"multus-additional-cni-plugins-dzc7t\" (UID: \"c2653055-9d7a-4aeb-9090-fb495a2b650b\") " pod="openshift-multus/multus-additional-cni-plugins-dzc7t" Jul 26 15:37:58.683016 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.682997 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7c179a4b-e953-4d22-9f2d-7cf682e833fd-etc-sysctl-conf\") pod \"tuned-65dgw\" (UID: \"7c179a4b-e953-4d22-9f2d-7cf682e833fd\") " pod="openshift-cluster-node-tuning-operator/tuned-65dgw" Jul 26 15:37:58.683072 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.683031 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgl7v\" (UniqueName: \"kubernetes.io/projected/7c179a4b-e953-4d22-9f2d-7cf682e833fd-kube-api-access-fgl7v\") pod \"tuned-65dgw\" (UID: \"7c179a4b-e953-4d22-9f2d-7cf682e833fd\") " pod="openshift-cluster-node-tuning-operator/tuned-65dgw" Jul 26 15:37:58.683072 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.683054 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bd02936a-a782-4e31-89bb-1292065d0abc-multus-daemon-config\") pod \"multus-x4dkt\" (UID: \"bd02936a-a782-4e31-89bb-1292065d0abc\") " pod="openshift-multus/multus-x4dkt" Jul 26 15:37:58.683166 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.683086 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bd02936a-a782-4e31-89bb-1292065d0abc-host-run-multus-certs\") pod \"multus-x4dkt\" (UID: \"bd02936a-a782-4e31-89bb-1292065d0abc\") " pod="openshift-multus/multus-x4dkt" Jul 26 15:37:58.683166 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.683121 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-657xn\" (UniqueName: \"kubernetes.io/projected/acf5fc6d-b7e4-4581-8878-f5776e505746-kube-api-access-657xn\") pod \"iptables-alerter-v46ns\" (UID: \"acf5fc6d-b7e4-4581-8878-f5776e505746\") " pod="openshift-network-operator/iptables-alerter-v46ns" Jul 26 15:37:58.683166 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.683146 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7c179a4b-e953-4d22-9f2d-7cf682e833fd-etc-modprobe-d\") pod \"tuned-65dgw\" (UID: \"7c179a4b-e953-4d22-9f2d-7cf682e833fd\") " pod="openshift-cluster-node-tuning-operator/tuned-65dgw" Jul 26 15:37:58.683305 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.683168 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bd02936a-a782-4e31-89bb-1292065d0abc-system-cni-dir\") pod \"multus-x4dkt\" (UID: \"bd02936a-a782-4e31-89bb-1292065d0abc\") " pod="openshift-multus/multus-x4dkt" Jul 26 15:37:58.683305 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.683194 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bd02936a-a782-4e31-89bb-1292065d0abc-multus-cni-dir\") pod \"multus-x4dkt\" (UID: \"bd02936a-a782-4e31-89bb-1292065d0abc\") " pod="openshift-multus/multus-x4dkt" Jul 26 15:37:58.683305 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.683218 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bd02936a-a782-4e31-89bb-1292065d0abc-hostroot\") pod \"multus-x4dkt\" (UID: \"bd02936a-a782-4e31-89bb-1292065d0abc\") " pod="openshift-multus/multus-x4dkt" Jul 26 15:37:58.683305 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.683238 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bd02936a-a782-4e31-89bb-1292065d0abc-etc-kubernetes\") pod \"multus-x4dkt\" (UID: \"bd02936a-a782-4e31-89bb-1292065d0abc\") " pod="openshift-multus/multus-x4dkt" Jul 26 15:37:58.683515 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.683316 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7c179a4b-e953-4d22-9f2d-7cf682e833fd-sys\") pod \"tuned-65dgw\" (UID: \"7c179a4b-e953-4d22-9f2d-7cf682e833fd\") " pod="openshift-cluster-node-tuning-operator/tuned-65dgw" Jul 26 15:37:58.683515 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.683386 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bd02936a-a782-4e31-89bb-1292065d0abc-host-run-netns\") pod \"multus-x4dkt\" (UID: \"bd02936a-a782-4e31-89bb-1292065d0abc\") " pod="openshift-multus/multus-x4dkt" Jul 26 15:37:58.683515 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.683439 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c2653055-9d7a-4aeb-9090-fb495a2b650b-system-cni-dir\") pod \"multus-additional-cni-plugins-dzc7t\" (UID: \"c2653055-9d7a-4aeb-9090-fb495a2b650b\") " pod="openshift-multus/multus-additional-cni-plugins-dzc7t" Jul 26 15:37:58.683515 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.683476 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c2653055-9d7a-4aeb-9090-fb495a2b650b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dzc7t\" (UID: \"c2653055-9d7a-4aeb-9090-fb495a2b650b\") " pod="openshift-multus/multus-additional-cni-plugins-dzc7t" Jul 26 15:37:58.683696 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.683520 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7c179a4b-e953-4d22-9f2d-7cf682e833fd-etc-sysconfig\") pod \"tuned-65dgw\" (UID: \"7c179a4b-e953-4d22-9f2d-7cf682e833fd\") " pod="openshift-cluster-node-tuning-operator/tuned-65dgw" Jul 26 15:37:58.683696 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.683549 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7c179a4b-e953-4d22-9f2d-7cf682e833fd-lib-modules\") pod \"tuned-65dgw\" (UID: \"7c179a4b-e953-4d22-9f2d-7cf682e833fd\") " pod="openshift-cluster-node-tuning-operator/tuned-65dgw" Jul 26 15:37:58.683696 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.683600 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7c179a4b-e953-4d22-9f2d-7cf682e833fd-var-lib-kubelet\") pod \"tuned-65dgw\" (UID: \"7c179a4b-e953-4d22-9f2d-7cf682e833fd\") " pod="openshift-cluster-node-tuning-operator/tuned-65dgw" Jul 26 15:37:58.683696 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.683634 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/356a056a-d4d5-48ed-a10f-928b36becc81-hosts-file\") pod \"node-resolver-v4mgw\" (UID: \"356a056a-d4d5-48ed-a10f-928b36becc81\") " pod="openshift-dns/node-resolver-v4mgw" Jul 26 15:37:58.683696 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.683663 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bd02936a-a782-4e31-89bb-1292065d0abc-host-var-lib-cni-multus\") pod \"multus-x4dkt\" (UID: \"bd02936a-a782-4e31-89bb-1292065d0abc\") " pod="openshift-multus/multus-x4dkt" Jul 26 15:37:58.683696 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.683691 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7c179a4b-e953-4d22-9f2d-7cf682e833fd-etc-kubernetes\") pod \"tuned-65dgw\" (UID: \"7c179a4b-e953-4d22-9f2d-7cf682e833fd\") " pod="openshift-cluster-node-tuning-operator/tuned-65dgw" Jul 26 15:37:58.683975 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.683713 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7c179a4b-e953-4d22-9f2d-7cf682e833fd-etc-systemd\") pod \"tuned-65dgw\" (UID: \"7c179a4b-e953-4d22-9f2d-7cf682e833fd\") " pod="openshift-cluster-node-tuning-operator/tuned-65dgw" Jul 26 15:37:58.683975 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.683738 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fztmw\" (UniqueName: \"kubernetes.io/projected/f3e728c9-4710-4f6c-977f-caf52a2bfcaf-kube-api-access-fztmw\") pod \"network-check-target-d2nnz\" (UID: \"f3e728c9-4710-4f6c-977f-caf52a2bfcaf\") " pod="openshift-network-diagnostics/network-check-target-d2nnz" Jul 26 15:37:58.683975 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.683769 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2j8s\" (UniqueName: \"kubernetes.io/projected/356a056a-d4d5-48ed-a10f-928b36becc81-kube-api-access-d2j8s\") pod \"node-resolver-v4mgw\" (UID: \"356a056a-d4d5-48ed-a10f-928b36becc81\") " pod="openshift-dns/node-resolver-v4mgw" Jul 26 15:37:58.683975 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.683791 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bd02936a-a782-4e31-89bb-1292065d0abc-multus-socket-dir-parent\") pod \"multus-x4dkt\" (UID: \"bd02936a-a782-4e31-89bb-1292065d0abc\") " pod="openshift-multus/multus-x4dkt" Jul 26 15:37:58.683975 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.683813 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvf4l\" (UniqueName: \"kubernetes.io/projected/bd02936a-a782-4e31-89bb-1292065d0abc-kube-api-access-qvf4l\") pod \"multus-x4dkt\" (UID: \"bd02936a-a782-4e31-89bb-1292065d0abc\") " pod="openshift-multus/multus-x4dkt" Jul 26 15:37:58.683975 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.683835 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7c179a4b-e953-4d22-9f2d-7cf682e833fd-tmp\") pod \"tuned-65dgw\" (UID: \"7c179a4b-e953-4d22-9f2d-7cf682e833fd\") " pod="openshift-cluster-node-tuning-operator/tuned-65dgw" Jul 26 15:37:58.683975 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.683860 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bd02936a-a782-4e31-89bb-1292065d0abc-cni-binary-copy\") pod \"multus-x4dkt\" (UID: \"bd02936a-a782-4e31-89bb-1292065d0abc\") " pod="openshift-multus/multus-x4dkt" Jul 26 15:37:58.683975 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.683880 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bd02936a-a782-4e31-89bb-1292065d0abc-multus-conf-dir\") pod \"multus-x4dkt\" (UID: \"bd02936a-a782-4e31-89bb-1292065d0abc\") " pod="openshift-multus/multus-x4dkt" Jul 26 15:37:58.683975 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.683894 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7c179a4b-e953-4d22-9f2d-7cf682e833fd-etc-sysctl-d\") pod \"tuned-65dgw\" (UID: \"7c179a4b-e953-4d22-9f2d-7cf682e833fd\") " pod="openshift-cluster-node-tuning-operator/tuned-65dgw" Jul 26 15:37:58.683975 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.683920 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f6bd9013-635a-4af8-b0ee-621a280a459a-original-pull-secret\") pod \"global-pull-secret-syncer-l56b4\" (UID: \"f6bd9013-635a-4af8-b0ee-621a280a459a\") " pod="kube-system/global-pull-secret-syncer-l56b4" Jul 26 15:37:58.683975 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.683940 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7c179a4b-e953-4d22-9f2d-7cf682e833fd-host\") pod \"tuned-65dgw\" (UID: \"7c179a4b-e953-4d22-9f2d-7cf682e833fd\") " pod="openshift-cluster-node-tuning-operator/tuned-65dgw" Jul 26 15:37:58.683975 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.683953 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmpnp\" (UniqueName: \"kubernetes.io/projected/098dd2e5-9540-425f-ba4e-71ea0b064be7-kube-api-access-jmpnp\") pod \"node-ca-mfwqb\" (UID: \"098dd2e5-9540-425f-ba4e-71ea0b064be7\") " pod="openshift-image-registry/node-ca-mfwqb" Jul 26 15:37:58.683975 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.683968 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bd02936a-a782-4e31-89bb-1292065d0abc-host-var-lib-kubelet\") pod \"multus-x4dkt\" (UID: \"bd02936a-a782-4e31-89bb-1292065d0abc\") " pod="openshift-multus/multus-x4dkt" Jul 26 15:37:58.684560 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.683991 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c2653055-9d7a-4aeb-9090-fb495a2b650b-os-release\") pod \"multus-additional-cni-plugins-dzc7t\" (UID: \"c2653055-9d7a-4aeb-9090-fb495a2b650b\") " pod="openshift-multus/multus-additional-cni-plugins-dzc7t" Jul 26 15:37:58.684560 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.684021 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/356a056a-d4d5-48ed-a10f-928b36becc81-tmp-dir\") pod \"node-resolver-v4mgw\" (UID: \"356a056a-d4d5-48ed-a10f-928b36becc81\") " pod="openshift-dns/node-resolver-v4mgw" Jul 26 15:37:58.684560 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.684146 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/806292e6-a776-4195-a595-263edb35e68b-agent-certs\") pod \"konnectivity-agent-jbpx6\" (UID: \"806292e6-a776-4195-a595-263edb35e68b\") " pod="kube-system/konnectivity-agent-jbpx6" Jul 26 15:37:58.684560 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.684177 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c2653055-9d7a-4aeb-9090-fb495a2b650b-cni-binary-copy\") pod \"multus-additional-cni-plugins-dzc7t\" (UID: \"c2653055-9d7a-4aeb-9090-fb495a2b650b\") " pod="openshift-multus/multus-additional-cni-plugins-dzc7t" Jul 26 15:37:58.684560 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.684222 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c2653055-9d7a-4aeb-9090-fb495a2b650b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dzc7t\" (UID: \"c2653055-9d7a-4aeb-9090-fb495a2b650b\") " pod="openshift-multus/multus-additional-cni-plugins-dzc7t" Jul 26 15:37:58.684560 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.684254 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c2653055-9d7a-4aeb-9090-fb495a2b650b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dzc7t\" (UID: \"c2653055-9d7a-4aeb-9090-fb495a2b650b\") " pod="openshift-multus/multus-additional-cni-plugins-dzc7t" Jul 26 15:37:58.684560 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.684279 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bd02936a-a782-4e31-89bb-1292065d0abc-host-var-lib-cni-bin\") pod \"multus-x4dkt\" (UID: \"bd02936a-a782-4e31-89bb-1292065d0abc\") " pod="openshift-multus/multus-x4dkt" Jul 26 15:37:58.684560 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.684308 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/806292e6-a776-4195-a595-263edb35e68b-konnectivity-ca\") pod \"konnectivity-agent-jbpx6\" (UID: \"806292e6-a776-4195-a595-263edb35e68b\") " pod="kube-system/konnectivity-agent-jbpx6" Jul 26 15:37:58.684560 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.684329 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7c179a4b-e953-4d22-9f2d-7cf682e833fd-etc-tuned\") pod \"tuned-65dgw\" (UID: \"7c179a4b-e953-4d22-9f2d-7cf682e833fd\") " pod="openshift-cluster-node-tuning-operator/tuned-65dgw" Jul 26 15:37:58.684560 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.684348 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/098dd2e5-9540-425f-ba4e-71ea0b064be7-host\") pod \"node-ca-mfwqb\" (UID: \"098dd2e5-9540-425f-ba4e-71ea0b064be7\") " pod="openshift-image-registry/node-ca-mfwqb" Jul 26 15:37:58.684560 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.684376 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zljx\" (UniqueName: \"kubernetes.io/projected/c2653055-9d7a-4aeb-9090-fb495a2b650b-kube-api-access-9zljx\") pod \"multus-additional-cni-plugins-dzc7t\" (UID: \"c2653055-9d7a-4aeb-9090-fb495a2b650b\") " pod="openshift-multus/multus-additional-cni-plugins-dzc7t" Jul 26 15:37:58.684560 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.684424 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bd02936a-a782-4e31-89bb-1292065d0abc-host-run-k8s-cni-cncf-io\") pod \"multus-x4dkt\" (UID: \"bd02936a-a782-4e31-89bb-1292065d0abc\") " pod="openshift-multus/multus-x4dkt" Jul 26 15:37:58.685042 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.684637 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-ztlp6" Jul 26 15:37:58.688718 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.686615 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 26 15:37:58.688718 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.688622 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 26 15:37:58.688718 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.688646 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 26 15:37:58.689065 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.688989 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 26 15:37:58.689065 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.689004 2577 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 26 15:37:58.689065 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.689037 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 26 15:37:58.690033 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.690014 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-prt8g\"" Jul 26 15:37:58.690126 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.690036 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 26 15:37:58.694728 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.694693 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-25 15:32:57 +0000 UTC" deadline="2028-02-19 20:59:04.585466564 +0000 UTC" Jul 26 15:37:58.694728 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.694727 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13757h21m5.890742546s" Jul 26 15:37:58.708533 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.708510 2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-bpl5n" Jul 26 15:37:58.713991 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.713972 2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-bpl5n" Jul 26 15:37:58.735509 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:58.735472 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d03d973abb0a97c200617d39e2704ee.slice/crio-db49eda72935c50f6b454aa5b493d91c8786e8c25a592c37f8633beadd4a16b9 WatchSource:0}: Error finding container db49eda72935c50f6b454aa5b493d91c8786e8c25a592c37f8633beadd4a16b9: Status 404 returned error can't find the container with id db49eda72935c50f6b454aa5b493d91c8786e8c25a592c37f8633beadd4a16b9 Jul 26 15:37:58.735716 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:58.735700 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34c5f0125abc8dd4d5a5ee6814e83607.slice/crio-df6a397dadf62d77409dc9e6848cd97417159f18b021fbd15c7933771f6f5a99 WatchSource:0}: Error finding container df6a397dadf62d77409dc9e6848cd97417159f18b021fbd15c7933771f6f5a99: Status 404 returned error can't find the container with id df6a397dadf62d77409dc9e6848cd97417159f18b021fbd15c7933771f6f5a99 Jul 26 15:37:58.741994 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.741978 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 26 15:37:58.777252 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.777227 2577 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 26 15:37:58.784140 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.784100 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-163.ec2.internal" event={"ID":"0d03d973abb0a97c200617d39e2704ee","Type":"ContainerStarted","Data":"db49eda72935c50f6b454aa5b493d91c8786e8c25a592c37f8633beadd4a16b9"} Jul 26 15:37:58.784580 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.784554 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c2653055-9d7a-4aeb-9090-fb495a2b650b-system-cni-dir\") pod \"multus-additional-cni-plugins-dzc7t\" (UID: \"c2653055-9d7a-4aeb-9090-fb495a2b650b\") " pod="openshift-multus/multus-additional-cni-plugins-dzc7t" Jul 26 15:37:58.784691 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.784583 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c2653055-9d7a-4aeb-9090-fb495a2b650b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dzc7t\" (UID: \"c2653055-9d7a-4aeb-9090-fb495a2b650b\") " pod="openshift-multus/multus-additional-cni-plugins-dzc7t" Jul 26 15:37:58.784691 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.784602 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7c179a4b-e953-4d22-9f2d-7cf682e833fd-etc-sysconfig\") pod \"tuned-65dgw\" (UID: \"7c179a4b-e953-4d22-9f2d-7cf682e833fd\") " pod="openshift-cluster-node-tuning-operator/tuned-65dgw" Jul 26 15:37:58.784691 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.784619 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7c179a4b-e953-4d22-9f2d-7cf682e833fd-lib-modules\") pod \"tuned-65dgw\" (UID: \"7c179a4b-e953-4d22-9f2d-7cf682e833fd\") " pod="openshift-cluster-node-tuning-operator/tuned-65dgw" Jul 26 15:37:58.784691 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.784639 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7c179a4b-e953-4d22-9f2d-7cf682e833fd-var-lib-kubelet\") pod \"tuned-65dgw\" (UID: \"7c179a4b-e953-4d22-9f2d-7cf682e833fd\") " pod="openshift-cluster-node-tuning-operator/tuned-65dgw" Jul 26 15:37:58.784691 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.784661 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/356a056a-d4d5-48ed-a10f-928b36becc81-hosts-file\") pod \"node-resolver-v4mgw\" (UID: \"356a056a-d4d5-48ed-a10f-928b36becc81\") " pod="openshift-dns/node-resolver-v4mgw" Jul 26 15:37:58.784691 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.784661 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c2653055-9d7a-4aeb-9090-fb495a2b650b-system-cni-dir\") pod \"multus-additional-cni-plugins-dzc7t\" (UID: \"c2653055-9d7a-4aeb-9090-fb495a2b650b\") " pod="openshift-multus/multus-additional-cni-plugins-dzc7t" Jul 26 15:37:58.784691 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.784681 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bd02936a-a782-4e31-89bb-1292065d0abc-host-var-lib-cni-multus\") pod \"multus-x4dkt\" (UID: \"bd02936a-a782-4e31-89bb-1292065d0abc\") " pod="openshift-multus/multus-x4dkt" Jul 26 15:37:58.785038 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.784694 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7c179a4b-e953-4d22-9f2d-7cf682e833fd-etc-sysconfig\") pod \"tuned-65dgw\" (UID: \"7c179a4b-e953-4d22-9f2d-7cf682e833fd\") " pod="openshift-cluster-node-tuning-operator/tuned-65dgw" Jul 26 15:37:58.785038 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.784702 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7c179a4b-e953-4d22-9f2d-7cf682e833fd-etc-kubernetes\") pod \"tuned-65dgw\" (UID: \"7c179a4b-e953-4d22-9f2d-7cf682e833fd\") " pod="openshift-cluster-node-tuning-operator/tuned-65dgw" Jul 26 15:37:58.785038 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.784736 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/356a056a-d4d5-48ed-a10f-928b36becc81-hosts-file\") pod \"node-resolver-v4mgw\" (UID: \"356a056a-d4d5-48ed-a10f-928b36becc81\") " pod="openshift-dns/node-resolver-v4mgw" Jul 26 15:37:58.785038 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.784745 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7c179a4b-e953-4d22-9f2d-7cf682e833fd-etc-kubernetes\") pod \"tuned-65dgw\" (UID: \"7c179a4b-e953-4d22-9f2d-7cf682e833fd\") " pod="openshift-cluster-node-tuning-operator/tuned-65dgw" Jul 26 15:37:58.785038 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.784737 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7c179a4b-e953-4d22-9f2d-7cf682e833fd-etc-systemd\") pod \"tuned-65dgw\" (UID: \"7c179a4b-e953-4d22-9f2d-7cf682e833fd\") " pod="openshift-cluster-node-tuning-operator/tuned-65dgw" Jul 26 15:37:58.785038 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.784755 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7c179a4b-e953-4d22-9f2d-7cf682e833fd-var-lib-kubelet\") pod \"tuned-65dgw\" (UID: \"7c179a4b-e953-4d22-9f2d-7cf682e833fd\") " pod="openshift-cluster-node-tuning-operator/tuned-65dgw" Jul 26 15:37:58.785038 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.784773 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7c179a4b-e953-4d22-9f2d-7cf682e833fd-etc-systemd\") pod \"tuned-65dgw\" (UID: \"7c179a4b-e953-4d22-9f2d-7cf682e833fd\") " pod="openshift-cluster-node-tuning-operator/tuned-65dgw" Jul 26 15:37:58.785038 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.784779 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fztmw\" (UniqueName: \"kubernetes.io/projected/f3e728c9-4710-4f6c-977f-caf52a2bfcaf-kube-api-access-fztmw\") pod \"network-check-target-d2nnz\" (UID: \"f3e728c9-4710-4f6c-977f-caf52a2bfcaf\") " pod="openshift-network-diagnostics/network-check-target-d2nnz" Jul 26 15:37:58.785038 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.784807 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c2653055-9d7a-4aeb-9090-fb495a2b650b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dzc7t\" (UID: \"c2653055-9d7a-4aeb-9090-fb495a2b650b\") " pod="openshift-multus/multus-additional-cni-plugins-dzc7t" Jul 26 15:37:58.785038 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.784818 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7c179a4b-e953-4d22-9f2d-7cf682e833fd-lib-modules\") pod \"tuned-65dgw\" (UID: \"7c179a4b-e953-4d22-9f2d-7cf682e833fd\") " pod="openshift-cluster-node-tuning-operator/tuned-65dgw" Jul 26 15:37:58.785038 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.784809 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2j8s\" (UniqueName: \"kubernetes.io/projected/356a056a-d4d5-48ed-a10f-928b36becc81-kube-api-access-d2j8s\") pod \"node-resolver-v4mgw\" (UID: \"356a056a-d4d5-48ed-a10f-928b36becc81\") " pod="openshift-dns/node-resolver-v4mgw" Jul 26 15:37:58.785038 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.784812 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bd02936a-a782-4e31-89bb-1292065d0abc-host-var-lib-cni-multus\") pod \"multus-x4dkt\" (UID: \"bd02936a-a782-4e31-89bb-1292065d0abc\") " pod="openshift-multus/multus-x4dkt" Jul 26 15:37:58.785038 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.784861 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bd02936a-a782-4e31-89bb-1292065d0abc-multus-socket-dir-parent\") pod \"multus-x4dkt\" (UID: \"bd02936a-a782-4e31-89bb-1292065d0abc\") " pod="openshift-multus/multus-x4dkt" Jul 26 15:37:58.785038 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.784888 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvf4l\" (UniqueName: \"kubernetes.io/projected/bd02936a-a782-4e31-89bb-1292065d0abc-kube-api-access-qvf4l\") pod \"multus-x4dkt\" (UID: \"bd02936a-a782-4e31-89bb-1292065d0abc\") " pod="openshift-multus/multus-x4dkt" Jul 26 15:37:58.785038 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.784933 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfn9p\" (UniqueName: \"kubernetes.io/projected/b5f442b1-a918-43a5-9870-f2183fe5085a-kube-api-access-vfn9p\") pod \"network-metrics-daemon-kgk6w\" (UID: \"b5f442b1-a918-43a5-9870-f2183fe5085a\") " pod="openshift-multus/network-metrics-daemon-kgk6w" Jul 26 15:37:58.785038 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.784962 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/cbe2cd41-6ebc-4722-a813-57d83fa1d242-node-log\") pod \"ovnkube-node-ztlp6\" (UID: \"cbe2cd41-6ebc-4722-a813-57d83fa1d242\") " pod="openshift-ovn-kubernetes/ovnkube-node-ztlp6" Jul 26 15:37:58.785038 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.785008 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7c179a4b-e953-4d22-9f2d-7cf682e833fd-tmp\") pod \"tuned-65dgw\" (UID: \"7c179a4b-e953-4d22-9f2d-7cf682e833fd\") " pod="openshift-cluster-node-tuning-operator/tuned-65dgw" Jul 26 15:37:58.785630 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.785019 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bd02936a-a782-4e31-89bb-1292065d0abc-multus-socket-dir-parent\") pod \"multus-x4dkt\" (UID: \"bd02936a-a782-4e31-89bb-1292065d0abc\") " pod="openshift-multus/multus-x4dkt" Jul 26 15:37:58.785630 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.785036 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bd02936a-a782-4e31-89bb-1292065d0abc-cni-binary-copy\") pod \"multus-x4dkt\" (UID: \"bd02936a-a782-4e31-89bb-1292065d0abc\") " pod="openshift-multus/multus-x4dkt" Jul 26 15:37:58.785630 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.785058 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bd02936a-a782-4e31-89bb-1292065d0abc-multus-conf-dir\") pod \"multus-x4dkt\" (UID: \"bd02936a-a782-4e31-89bb-1292065d0abc\") " pod="openshift-multus/multus-x4dkt" Jul 26 15:37:58.785630 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.785083 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/cbe2cd41-6ebc-4722-a813-57d83fa1d242-systemd-units\") pod \"ovnkube-node-ztlp6\" (UID: \"cbe2cd41-6ebc-4722-a813-57d83fa1d242\") " pod="openshift-ovn-kubernetes/ovnkube-node-ztlp6" Jul 26 15:37:58.785630 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.785126 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-163.ec2.internal" event={"ID":"34c5f0125abc8dd4d5a5ee6814e83607","Type":"ContainerStarted","Data":"df6a397dadf62d77409dc9e6848cd97417159f18b021fbd15c7933771f6f5a99"} Jul 26 15:37:58.785630 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.785127 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cbe2cd41-6ebc-4722-a813-57d83fa1d242-var-lib-openvswitch\") pod \"ovnkube-node-ztlp6\" (UID: \"cbe2cd41-6ebc-4722-a813-57d83fa1d242\") " pod="openshift-ovn-kubernetes/ovnkube-node-ztlp6" Jul 26 15:37:58.785630 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.785194 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cbe2cd41-6ebc-4722-a813-57d83fa1d242-host-cni-bin\") pod \"ovnkube-node-ztlp6\" (UID: \"cbe2cd41-6ebc-4722-a813-57d83fa1d242\") " pod="openshift-ovn-kubernetes/ovnkube-node-ztlp6" Jul 26 15:37:58.785630 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.785217 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/cbe2cd41-6ebc-4722-a813-57d83fa1d242-ovnkube-config\") pod \"ovnkube-node-ztlp6\" (UID: \"cbe2cd41-6ebc-4722-a813-57d83fa1d242\") " pod="openshift-ovn-kubernetes/ovnkube-node-ztlp6" Jul 26 15:37:58.785630 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.785219 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bd02936a-a782-4e31-89bb-1292065d0abc-multus-conf-dir\") pod \"multus-x4dkt\" (UID: \"bd02936a-a782-4e31-89bb-1292065d0abc\") " pod="openshift-multus/multus-x4dkt" Jul 26 15:37:58.785630 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.785350 2577 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 26 15:37:58.785941 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.785737 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/cbe2cd41-6ebc-4722-a813-57d83fa1d242-env-overrides\") pod \"ovnkube-node-ztlp6\" (UID: \"cbe2cd41-6ebc-4722-a813-57d83fa1d242\") " pod="openshift-ovn-kubernetes/ovnkube-node-ztlp6" Jul 26 15:37:58.785941 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.785766 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5f442b1-a918-43a5-9870-f2183fe5085a-metrics-certs\") pod \"network-metrics-daemon-kgk6w\" (UID: \"b5f442b1-a918-43a5-9870-f2183fe5085a\") " pod="openshift-multus/network-metrics-daemon-kgk6w" Jul 26 15:37:58.785941 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.785769 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bd02936a-a782-4e31-89bb-1292065d0abc-cni-binary-copy\") pod \"multus-x4dkt\" (UID: \"bd02936a-a782-4e31-89bb-1292065d0abc\") " pod="openshift-multus/multus-x4dkt" Jul 26 15:37:58.785941 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.785791 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/cbe2cd41-6ebc-4722-a813-57d83fa1d242-run-ovn\") pod \"ovnkube-node-ztlp6\" (UID: \"cbe2cd41-6ebc-4722-a813-57d83fa1d242\") " pod="openshift-ovn-kubernetes/ovnkube-node-ztlp6" Jul 26 15:37:58.785941 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.785815 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cbe2cd41-6ebc-4722-a813-57d83fa1d242-host-run-ovn-kubernetes\") pod \"ovnkube-node-ztlp6\" (UID: \"cbe2cd41-6ebc-4722-a813-57d83fa1d242\") " pod="openshift-ovn-kubernetes/ovnkube-node-ztlp6" Jul 26 15:37:58.785941 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.785835 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/cbe2cd41-6ebc-4722-a813-57d83fa1d242-ovn-node-metrics-cert\") pod \"ovnkube-node-ztlp6\" (UID: \"cbe2cd41-6ebc-4722-a813-57d83fa1d242\") " pod="openshift-ovn-kubernetes/ovnkube-node-ztlp6" Jul 26 15:37:58.785941 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.785852 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7c179a4b-e953-4d22-9f2d-7cf682e833fd-etc-sysctl-d\") pod \"tuned-65dgw\" (UID: \"7c179a4b-e953-4d22-9f2d-7cf682e833fd\") " pod="openshift-cluster-node-tuning-operator/tuned-65dgw" Jul 26 15:37:58.785941 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.785866 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cbe2cd41-6ebc-4722-a813-57d83fa1d242-host-slash\") pod \"ovnkube-node-ztlp6\" (UID: \"cbe2cd41-6ebc-4722-a813-57d83fa1d242\") " pod="openshift-ovn-kubernetes/ovnkube-node-ztlp6" Jul 26 15:37:58.785941 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.785917 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f6bd9013-635a-4af8-b0ee-621a280a459a-original-pull-secret\") pod \"global-pull-secret-syncer-l56b4\" (UID: \"f6bd9013-635a-4af8-b0ee-621a280a459a\") " pod="kube-system/global-pull-secret-syncer-l56b4" Jul 26 15:37:58.786327 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.785951 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7c179a4b-e953-4d22-9f2d-7cf682e833fd-host\") pod \"tuned-65dgw\" (UID: \"7c179a4b-e953-4d22-9f2d-7cf682e833fd\") " pod="openshift-cluster-node-tuning-operator/tuned-65dgw" Jul 26 15:37:58.786327 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.785973 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmpnp\" (UniqueName: \"kubernetes.io/projected/098dd2e5-9540-425f-ba4e-71ea0b064be7-kube-api-access-jmpnp\") pod \"node-ca-mfwqb\" (UID: \"098dd2e5-9540-425f-ba4e-71ea0b064be7\") " pod="openshift-image-registry/node-ca-mfwqb" Jul 26 15:37:58.786327 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.785996 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bd02936a-a782-4e31-89bb-1292065d0abc-host-var-lib-kubelet\") pod \"multus-x4dkt\" (UID: \"bd02936a-a782-4e31-89bb-1292065d0abc\") " pod="openshift-multus/multus-x4dkt" Jul 26 15:37:58.786327 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.786007 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7c179a4b-e953-4d22-9f2d-7cf682e833fd-host\") pod \"tuned-65dgw\" (UID: \"7c179a4b-e953-4d22-9f2d-7cf682e833fd\") " pod="openshift-cluster-node-tuning-operator/tuned-65dgw" Jul 26 15:37:58.786327 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.786020 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/350d34fc-ceeb-477e-8486-0a712e354c67-socket-dir\") pod \"aws-ebs-csi-driver-node-qnht4\" (UID: \"350d34fc-ceeb-477e-8486-0a712e354c67\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qnht4" Jul 26 15:37:58.786327 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:37:58.786043 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 26 15:37:58.786327 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.786049 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bd02936a-a782-4e31-89bb-1292065d0abc-host-var-lib-kubelet\") pod \"multus-x4dkt\" (UID: \"bd02936a-a782-4e31-89bb-1292065d0abc\") " pod="openshift-multus/multus-x4dkt" Jul 26 15:37:58.786327 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.785973 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7c179a4b-e953-4d22-9f2d-7cf682e833fd-etc-sysctl-d\") pod \"tuned-65dgw\" (UID: \"7c179a4b-e953-4d22-9f2d-7cf682e833fd\") " pod="openshift-cluster-node-tuning-operator/tuned-65dgw" Jul 26 15:37:58.786327 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.786042 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/350d34fc-ceeb-477e-8486-0a712e354c67-registration-dir\") pod \"aws-ebs-csi-driver-node-qnht4\" (UID: \"350d34fc-ceeb-477e-8486-0a712e354c67\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qnht4" Jul 26 15:37:58.786327 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:37:58.786116 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f6bd9013-635a-4af8-b0ee-621a280a459a-original-pull-secret podName:f6bd9013-635a-4af8-b0ee-621a280a459a nodeName:}" failed. No retries permitted until 2026-07-26 15:37:59.286084872 +0000 UTC m=+2.025438161 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f6bd9013-635a-4af8-b0ee-621a280a459a-original-pull-secret") pod "global-pull-secret-syncer-l56b4" (UID: "f6bd9013-635a-4af8-b0ee-621a280a459a") : object "kube-system"/"original-pull-secret" not registered Jul 26 15:37:58.786327 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.786140 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c2653055-9d7a-4aeb-9090-fb495a2b650b-os-release\") pod \"multus-additional-cni-plugins-dzc7t\" (UID: \"c2653055-9d7a-4aeb-9090-fb495a2b650b\") " pod="openshift-multus/multus-additional-cni-plugins-dzc7t" Jul 26 15:37:58.786327 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.786180 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/356a056a-d4d5-48ed-a10f-928b36becc81-tmp-dir\") pod \"node-resolver-v4mgw\" (UID: \"356a056a-d4d5-48ed-a10f-928b36becc81\") " pod="openshift-dns/node-resolver-v4mgw" Jul 26 15:37:58.786327 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.786205 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c2653055-9d7a-4aeb-9090-fb495a2b650b-os-release\") pod \"multus-additional-cni-plugins-dzc7t\" (UID: \"c2653055-9d7a-4aeb-9090-fb495a2b650b\") " pod="openshift-multus/multus-additional-cni-plugins-dzc7t" Jul 26 15:37:58.786327 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.786208 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/806292e6-a776-4195-a595-263edb35e68b-agent-certs\") pod \"konnectivity-agent-jbpx6\" (UID: \"806292e6-a776-4195-a595-263edb35e68b\") " pod="kube-system/konnectivity-agent-jbpx6" Jul 26 15:37:58.786327 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.786254 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c2653055-9d7a-4aeb-9090-fb495a2b650b-cni-binary-copy\") pod \"multus-additional-cni-plugins-dzc7t\" (UID: \"c2653055-9d7a-4aeb-9090-fb495a2b650b\") " pod="openshift-multus/multus-additional-cni-plugins-dzc7t" Jul 26 15:37:58.786327 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.786276 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c2653055-9d7a-4aeb-9090-fb495a2b650b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dzc7t\" (UID: \"c2653055-9d7a-4aeb-9090-fb495a2b650b\") " pod="openshift-multus/multus-additional-cni-plugins-dzc7t" Jul 26 15:37:58.786327 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.786293 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c2653055-9d7a-4aeb-9090-fb495a2b650b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dzc7t\" (UID: \"c2653055-9d7a-4aeb-9090-fb495a2b650b\") " pod="openshift-multus/multus-additional-cni-plugins-dzc7t" Jul 26 15:37:58.787149 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.786313 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bd02936a-a782-4e31-89bb-1292065d0abc-host-var-lib-cni-bin\") pod \"multus-x4dkt\" (UID: \"bd02936a-a782-4e31-89bb-1292065d0abc\") " pod="openshift-multus/multus-x4dkt" Jul 26 15:37:58.787149 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.786338 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/350d34fc-ceeb-477e-8486-0a712e354c67-device-dir\") pod \"aws-ebs-csi-driver-node-qnht4\" (UID: \"350d34fc-ceeb-477e-8486-0a712e354c67\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qnht4" Jul 26 15:37:58.787149 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.786360 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/350d34fc-ceeb-477e-8486-0a712e354c67-sys-fs\") pod \"aws-ebs-csi-driver-node-qnht4\" (UID: \"350d34fc-ceeb-477e-8486-0a712e354c67\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qnht4" Jul 26 15:37:58.787149 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.786386 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/cbe2cd41-6ebc-4722-a813-57d83fa1d242-host-kubelet\") pod \"ovnkube-node-ztlp6\" (UID: \"cbe2cd41-6ebc-4722-a813-57d83fa1d242\") " pod="openshift-ovn-kubernetes/ovnkube-node-ztlp6" Jul 26 15:37:58.787149 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.786421 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cbe2cd41-6ebc-4722-a813-57d83fa1d242-run-openvswitch\") pod \"ovnkube-node-ztlp6\" (UID: \"cbe2cd41-6ebc-4722-a813-57d83fa1d242\") " pod="openshift-ovn-kubernetes/ovnkube-node-ztlp6" Jul 26 15:37:58.787149 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.786442 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/806292e6-a776-4195-a595-263edb35e68b-konnectivity-ca\") pod \"konnectivity-agent-jbpx6\" (UID: \"806292e6-a776-4195-a595-263edb35e68b\") " pod="kube-system/konnectivity-agent-jbpx6" Jul 26 15:37:58.787149 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.786466 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7c179a4b-e953-4d22-9f2d-7cf682e833fd-etc-tuned\") pod \"tuned-65dgw\" (UID: \"7c179a4b-e953-4d22-9f2d-7cf682e833fd\") " pod="openshift-cluster-node-tuning-operator/tuned-65dgw" Jul 26 15:37:58.787149 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.786488 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/098dd2e5-9540-425f-ba4e-71ea0b064be7-host\") pod \"node-ca-mfwqb\" (UID: \"098dd2e5-9540-425f-ba4e-71ea0b064be7\") " pod="openshift-image-registry/node-ca-mfwqb" Jul 26 15:37:58.787149 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.786516 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cbe2cd41-6ebc-4722-a813-57d83fa1d242-host-run-netns\") pod \"ovnkube-node-ztlp6\" (UID: \"cbe2cd41-6ebc-4722-a813-57d83fa1d242\") " pod="openshift-ovn-kubernetes/ovnkube-node-ztlp6" Jul 26 15:37:58.787149 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.786519 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/356a056a-d4d5-48ed-a10f-928b36becc81-tmp-dir\") pod \"node-resolver-v4mgw\" (UID: \"356a056a-d4d5-48ed-a10f-928b36becc81\") " pod="openshift-dns/node-resolver-v4mgw" Jul 26 15:37:58.787149 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.786538 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrgzd\" (UniqueName: \"kubernetes.io/projected/cbe2cd41-6ebc-4722-a813-57d83fa1d242-kube-api-access-jrgzd\") pod \"ovnkube-node-ztlp6\" (UID: \"cbe2cd41-6ebc-4722-a813-57d83fa1d242\") " pod="openshift-ovn-kubernetes/ovnkube-node-ztlp6" Jul 26 15:37:58.787149 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.786563 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zljx\" (UniqueName: \"kubernetes.io/projected/c2653055-9d7a-4aeb-9090-fb495a2b650b-kube-api-access-9zljx\") pod \"multus-additional-cni-plugins-dzc7t\" (UID: \"c2653055-9d7a-4aeb-9090-fb495a2b650b\") " pod="openshift-multus/multus-additional-cni-plugins-dzc7t" Jul 26 15:37:58.787149 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.786580 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bd02936a-a782-4e31-89bb-1292065d0abc-host-var-lib-cni-bin\") pod \"multus-x4dkt\" (UID: \"bd02936a-a782-4e31-89bb-1292065d0abc\") " pod="openshift-multus/multus-x4dkt" Jul 26 15:37:58.787149 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.786587 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bd02936a-a782-4e31-89bb-1292065d0abc-host-run-k8s-cni-cncf-io\") pod \"multus-x4dkt\" (UID: \"bd02936a-a782-4e31-89bb-1292065d0abc\") " pod="openshift-multus/multus-x4dkt" Jul 26 15:37:58.787149 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.786615 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cbe2cd41-6ebc-4722-a813-57d83fa1d242-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ztlp6\" (UID: \"cbe2cd41-6ebc-4722-a813-57d83fa1d242\") " pod="openshift-ovn-kubernetes/ovnkube-node-ztlp6" Jul 26 15:37:58.787149 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.786640 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f6bd9013-635a-4af8-b0ee-621a280a459a-kubelet-config\") pod \"global-pull-secret-syncer-l56b4\" (UID: \"f6bd9013-635a-4af8-b0ee-621a280a459a\") " pod="kube-system/global-pull-secret-syncer-l56b4" Jul 26 15:37:58.787149 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.786662 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/098dd2e5-9540-425f-ba4e-71ea0b064be7-serviceca\") pod \"node-ca-mfwqb\" (UID: \"098dd2e5-9540-425f-ba4e-71ea0b064be7\") " pod="openshift-image-registry/node-ca-mfwqb" Jul 26 15:37:58.787981 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.786682 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bd02936a-a782-4e31-89bb-1292065d0abc-cnibin\") pod \"multus-x4dkt\" (UID: \"bd02936a-a782-4e31-89bb-1292065d0abc\") " pod="openshift-multus/multus-x4dkt" Jul 26 15:37:58.787981 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.786705 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/bd02936a-a782-4e31-89bb-1292065d0abc-os-release\") pod \"multus-x4dkt\" (UID: \"bd02936a-a782-4e31-89bb-1292065d0abc\") " pod="openshift-multus/multus-x4dkt" Jul 26 15:37:58.787981 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.786726 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/acf5fc6d-b7e4-4581-8878-f5776e505746-host-slash\") pod \"iptables-alerter-v46ns\" (UID: \"acf5fc6d-b7e4-4581-8878-f5776e505746\") " pod="openshift-network-operator/iptables-alerter-v46ns" Jul 26 15:37:58.787981 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.786879 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c2653055-9d7a-4aeb-9090-fb495a2b650b-cni-binary-copy\") pod \"multus-additional-cni-plugins-dzc7t\" (UID: \"c2653055-9d7a-4aeb-9090-fb495a2b650b\") " pod="openshift-multus/multus-additional-cni-plugins-dzc7t" Jul 26 15:37:58.787981 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.787001 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/350d34fc-ceeb-477e-8486-0a712e354c67-etc-selinux\") pod \"aws-ebs-csi-driver-node-qnht4\" (UID: \"350d34fc-ceeb-477e-8486-0a712e354c67\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qnht4" Jul 26 15:37:58.787981 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.787031 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cbe2cd41-6ebc-4722-a813-57d83fa1d242-etc-openvswitch\") pod \"ovnkube-node-ztlp6\" (UID: \"cbe2cd41-6ebc-4722-a813-57d83fa1d242\") " pod="openshift-ovn-kubernetes/ovnkube-node-ztlp6" Jul 26 15:37:58.787981 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.787067 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/cbe2cd41-6ebc-4722-a813-57d83fa1d242-log-socket\") pod \"ovnkube-node-ztlp6\" (UID: \"cbe2cd41-6ebc-4722-a813-57d83fa1d242\") " pod="openshift-ovn-kubernetes/ovnkube-node-ztlp6" Jul 26 15:37:58.787981 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.787096 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f6bd9013-635a-4af8-b0ee-621a280a459a-dbus\") pod \"global-pull-secret-syncer-l56b4\" (UID: \"f6bd9013-635a-4af8-b0ee-621a280a459a\") " pod="kube-system/global-pull-secret-syncer-l56b4" Jul 26 15:37:58.787981 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.787109 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/806292e6-a776-4195-a595-263edb35e68b-konnectivity-ca\") pod \"konnectivity-agent-jbpx6\" (UID: \"806292e6-a776-4195-a595-263edb35e68b\") " pod="kube-system/konnectivity-agent-jbpx6" Jul 26 15:37:58.787981 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.787112 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/acf5fc6d-b7e4-4581-8878-f5776e505746-host-slash\") pod \"iptables-alerter-v46ns\" (UID: \"acf5fc6d-b7e4-4581-8878-f5776e505746\") " pod="openshift-network-operator/iptables-alerter-v46ns" Jul 26 15:37:58.787981 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.787124 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7c179a4b-e953-4d22-9f2d-7cf682e833fd-run\") pod \"tuned-65dgw\" (UID: \"7c179a4b-e953-4d22-9f2d-7cf682e833fd\") " pod="openshift-cluster-node-tuning-operator/tuned-65dgw" Jul 26 15:37:58.787981 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.787110 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c2653055-9d7a-4aeb-9090-fb495a2b650b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dzc7t\" (UID: \"c2653055-9d7a-4aeb-9090-fb495a2b650b\") " pod="openshift-multus/multus-additional-cni-plugins-dzc7t" Jul 26 15:37:58.787981 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.787064 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c2653055-9d7a-4aeb-9090-fb495a2b650b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dzc7t\" (UID: \"c2653055-9d7a-4aeb-9090-fb495a2b650b\") " pod="openshift-multus/multus-additional-cni-plugins-dzc7t" Jul 26 15:37:58.787981 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.787166 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/acf5fc6d-b7e4-4581-8878-f5776e505746-iptables-alerter-script\") pod \"iptables-alerter-v46ns\" (UID: \"acf5fc6d-b7e4-4581-8878-f5776e505746\") " pod="openshift-network-operator/iptables-alerter-v46ns" Jul 26 15:37:58.787981 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.787193 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7c179a4b-e953-4d22-9f2d-7cf682e833fd-run\") pod \"tuned-65dgw\" (UID: \"7c179a4b-e953-4d22-9f2d-7cf682e833fd\") " pod="openshift-cluster-node-tuning-operator/tuned-65dgw" Jul 26 15:37:58.787981 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.787194 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/cbe2cd41-6ebc-4722-a813-57d83fa1d242-ovnkube-script-lib\") pod \"ovnkube-node-ztlp6\" (UID: \"cbe2cd41-6ebc-4722-a813-57d83fa1d242\") " pod="openshift-ovn-kubernetes/ovnkube-node-ztlp6" Jul 26 15:37:58.787981 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.787213 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f6bd9013-635a-4af8-b0ee-621a280a459a-dbus\") pod \"global-pull-secret-syncer-l56b4\" (UID: \"f6bd9013-635a-4af8-b0ee-621a280a459a\") " pod="kube-system/global-pull-secret-syncer-l56b4" Jul 26 15:37:58.788753 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.787225 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c2653055-9d7a-4aeb-9090-fb495a2b650b-cnibin\") pod \"multus-additional-cni-plugins-dzc7t\" (UID: \"c2653055-9d7a-4aeb-9090-fb495a2b650b\") " pod="openshift-multus/multus-additional-cni-plugins-dzc7t" Jul 26 15:37:58.788753 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.787247 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f6bd9013-635a-4af8-b0ee-621a280a459a-kubelet-config\") pod \"global-pull-secret-syncer-l56b4\" (UID: \"f6bd9013-635a-4af8-b0ee-621a280a459a\") " pod="kube-system/global-pull-secret-syncer-l56b4" Jul 26 15:37:58.788753 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.787282 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c2653055-9d7a-4aeb-9090-fb495a2b650b-cnibin\") pod \"multus-additional-cni-plugins-dzc7t\" (UID: \"c2653055-9d7a-4aeb-9090-fb495a2b650b\") " pod="openshift-multus/multus-additional-cni-plugins-dzc7t" Jul 26 15:37:58.788753 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.787323 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bd02936a-a782-4e31-89bb-1292065d0abc-host-run-k8s-cni-cncf-io\") pod \"multus-x4dkt\" (UID: \"bd02936a-a782-4e31-89bb-1292065d0abc\") " pod="openshift-multus/multus-x4dkt" Jul 26 15:37:58.788753 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.787357 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7c179a4b-e953-4d22-9f2d-7cf682e833fd-etc-sysctl-conf\") pod \"tuned-65dgw\" (UID: \"7c179a4b-e953-4d22-9f2d-7cf682e833fd\") " pod="openshift-cluster-node-tuning-operator/tuned-65dgw" Jul 26 15:37:58.788753 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.787375 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/bd02936a-a782-4e31-89bb-1292065d0abc-os-release\") pod \"multus-x4dkt\" (UID: \"bd02936a-a782-4e31-89bb-1292065d0abc\") " pod="openshift-multus/multus-x4dkt" Jul 26 15:37:58.788753 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.787432 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/098dd2e5-9540-425f-ba4e-71ea0b064be7-host\") pod \"node-ca-mfwqb\" (UID: \"098dd2e5-9540-425f-ba4e-71ea0b064be7\") " pod="openshift-image-registry/node-ca-mfwqb" Jul 26 15:37:58.788753 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.787433 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bd02936a-a782-4e31-89bb-1292065d0abc-cnibin\") pod \"multus-x4dkt\" (UID: \"bd02936a-a782-4e31-89bb-1292065d0abc\") " pod="openshift-multus/multus-x4dkt" Jul 26 15:37:58.788753 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.787249 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7c179a4b-e953-4d22-9f2d-7cf682e833fd-etc-sysctl-conf\") pod \"tuned-65dgw\" (UID: \"7c179a4b-e953-4d22-9f2d-7cf682e833fd\") " pod="openshift-cluster-node-tuning-operator/tuned-65dgw" Jul 26 15:37:58.788753 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.787483 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgl7v\" (UniqueName: \"kubernetes.io/projected/7c179a4b-e953-4d22-9f2d-7cf682e833fd-kube-api-access-fgl7v\") pod \"tuned-65dgw\" (UID: \"7c179a4b-e953-4d22-9f2d-7cf682e833fd\") " pod="openshift-cluster-node-tuning-operator/tuned-65dgw" Jul 26 15:37:58.788753 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.787508 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bd02936a-a782-4e31-89bb-1292065d0abc-multus-daemon-config\") pod \"multus-x4dkt\" (UID: \"bd02936a-a782-4e31-89bb-1292065d0abc\") " pod="openshift-multus/multus-x4dkt" Jul 26 15:37:58.788753 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.787532 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bd02936a-a782-4e31-89bb-1292065d0abc-host-run-multus-certs\") pod \"multus-x4dkt\" (UID: \"bd02936a-a782-4e31-89bb-1292065d0abc\") " pod="openshift-multus/multus-x4dkt" Jul 26 15:37:58.788753 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.787558 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-657xn\" (UniqueName: \"kubernetes.io/projected/acf5fc6d-b7e4-4581-8878-f5776e505746-kube-api-access-657xn\") pod \"iptables-alerter-v46ns\" (UID: \"acf5fc6d-b7e4-4581-8878-f5776e505746\") " pod="openshift-network-operator/iptables-alerter-v46ns" Jul 26 15:37:58.788753 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.787609 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cz2t\" (UniqueName: \"kubernetes.io/projected/350d34fc-ceeb-477e-8486-0a712e354c67-kube-api-access-6cz2t\") pod \"aws-ebs-csi-driver-node-qnht4\" (UID: \"350d34fc-ceeb-477e-8486-0a712e354c67\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qnht4" Jul 26 15:37:58.788753 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.787636 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/cbe2cd41-6ebc-4722-a813-57d83fa1d242-run-systemd\") pod \"ovnkube-node-ztlp6\" (UID: \"cbe2cd41-6ebc-4722-a813-57d83fa1d242\") " pod="openshift-ovn-kubernetes/ovnkube-node-ztlp6" Jul 26 15:37:58.788753 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.787662 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7c179a4b-e953-4d22-9f2d-7cf682e833fd-etc-modprobe-d\") pod \"tuned-65dgw\" (UID: \"7c179a4b-e953-4d22-9f2d-7cf682e833fd\") " pod="openshift-cluster-node-tuning-operator/tuned-65dgw" Jul 26 15:37:58.788753 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.787685 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bd02936a-a782-4e31-89bb-1292065d0abc-system-cni-dir\") pod \"multus-x4dkt\" (UID: \"bd02936a-a782-4e31-89bb-1292065d0abc\") " pod="openshift-multus/multus-x4dkt" Jul 26 15:37:58.789288 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.787706 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bd02936a-a782-4e31-89bb-1292065d0abc-multus-cni-dir\") pod \"multus-x4dkt\" (UID: \"bd02936a-a782-4e31-89bb-1292065d0abc\") " pod="openshift-multus/multus-x4dkt" Jul 26 15:37:58.789288 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.787728 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bd02936a-a782-4e31-89bb-1292065d0abc-hostroot\") pod \"multus-x4dkt\" (UID: \"bd02936a-a782-4e31-89bb-1292065d0abc\") " pod="openshift-multus/multus-x4dkt" Jul 26 15:37:58.789288 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.787730 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/acf5fc6d-b7e4-4581-8878-f5776e505746-iptables-alerter-script\") pod \"iptables-alerter-v46ns\" (UID: \"acf5fc6d-b7e4-4581-8878-f5776e505746\") " pod="openshift-network-operator/iptables-alerter-v46ns" Jul 26 15:37:58.789288 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.787747 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/098dd2e5-9540-425f-ba4e-71ea0b064be7-serviceca\") pod \"node-ca-mfwqb\" (UID: \"098dd2e5-9540-425f-ba4e-71ea0b064be7\") " pod="openshift-image-registry/node-ca-mfwqb" Jul 26 15:37:58.789288 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.787750 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bd02936a-a782-4e31-89bb-1292065d0abc-etc-kubernetes\") pod \"multus-x4dkt\" (UID: \"bd02936a-a782-4e31-89bb-1292065d0abc\") " pod="openshift-multus/multus-x4dkt" Jul 26 15:37:58.789288 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.787792 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bd02936a-a782-4e31-89bb-1292065d0abc-hostroot\") pod \"multus-x4dkt\" (UID: \"bd02936a-a782-4e31-89bb-1292065d0abc\") " pod="openshift-multus/multus-x4dkt" Jul 26 15:37:58.789288 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.787798 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/cbe2cd41-6ebc-4722-a813-57d83fa1d242-host-cni-netd\") pod \"ovnkube-node-ztlp6\" (UID: \"cbe2cd41-6ebc-4722-a813-57d83fa1d242\") " pod="openshift-ovn-kubernetes/ovnkube-node-ztlp6" Jul 26 15:37:58.789288 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.787842 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7c179a4b-e953-4d22-9f2d-7cf682e833fd-sys\") pod \"tuned-65dgw\" (UID: \"7c179a4b-e953-4d22-9f2d-7cf682e833fd\") " pod="openshift-cluster-node-tuning-operator/tuned-65dgw" Jul 26 15:37:58.789288 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.787843 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bd02936a-a782-4e31-89bb-1292065d0abc-system-cni-dir\") pod \"multus-x4dkt\" (UID: \"bd02936a-a782-4e31-89bb-1292065d0abc\") " pod="openshift-multus/multus-x4dkt" Jul 26 15:37:58.789288 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.787855 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7c179a4b-e953-4d22-9f2d-7cf682e833fd-etc-modprobe-d\") pod \"tuned-65dgw\" (UID: \"7c179a4b-e953-4d22-9f2d-7cf682e833fd\") " pod="openshift-cluster-node-tuning-operator/tuned-65dgw" Jul 26 15:37:58.789288 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.787869 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bd02936a-a782-4e31-89bb-1292065d0abc-host-run-netns\") pod \"multus-x4dkt\" (UID: \"bd02936a-a782-4e31-89bb-1292065d0abc\") " pod="openshift-multus/multus-x4dkt" Jul 26 15:37:58.789288 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.787885 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bd02936a-a782-4e31-89bb-1292065d0abc-host-run-multus-certs\") pod \"multus-x4dkt\" (UID: \"bd02936a-a782-4e31-89bb-1292065d0abc\") " pod="openshift-multus/multus-x4dkt" Jul 26 15:37:58.789288 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.787902 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bd02936a-a782-4e31-89bb-1292065d0abc-multus-cni-dir\") pod \"multus-x4dkt\" (UID: \"bd02936a-a782-4e31-89bb-1292065d0abc\") " pod="openshift-multus/multus-x4dkt" Jul 26 15:37:58.789288 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.787791 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bd02936a-a782-4e31-89bb-1292065d0abc-etc-kubernetes\") pod \"multus-x4dkt\" (UID: \"bd02936a-a782-4e31-89bb-1292065d0abc\") " pod="openshift-multus/multus-x4dkt" Jul 26 15:37:58.789288 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.787915 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bd02936a-a782-4e31-89bb-1292065d0abc-host-run-netns\") pod \"multus-x4dkt\" (UID: \"bd02936a-a782-4e31-89bb-1292065d0abc\") " pod="openshift-multus/multus-x4dkt" Jul 26 15:37:58.789288 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.787944 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/350d34fc-ceeb-477e-8486-0a712e354c67-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qnht4\" (UID: \"350d34fc-ceeb-477e-8486-0a712e354c67\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qnht4" Jul 26 15:37:58.789288 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.787961 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7c179a4b-e953-4d22-9f2d-7cf682e833fd-sys\") pod \"tuned-65dgw\" (UID: \"7c179a4b-e953-4d22-9f2d-7cf682e833fd\") " pod="openshift-cluster-node-tuning-operator/tuned-65dgw" Jul 26 15:37:58.789288 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.788183 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bd02936a-a782-4e31-89bb-1292065d0abc-multus-daemon-config\") pod \"multus-x4dkt\" (UID: \"bd02936a-a782-4e31-89bb-1292065d0abc\") " pod="openshift-multus/multus-x4dkt" Jul 26 15:37:58.789825 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.788824 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7c179a4b-e953-4d22-9f2d-7cf682e833fd-tmp\") pod \"tuned-65dgw\" (UID: \"7c179a4b-e953-4d22-9f2d-7cf682e833fd\") " pod="openshift-cluster-node-tuning-operator/tuned-65dgw" Jul 26 15:37:58.789825 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.789153 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/806292e6-a776-4195-a595-263edb35e68b-agent-certs\") pod \"konnectivity-agent-jbpx6\" (UID: \"806292e6-a776-4195-a595-263edb35e68b\") " pod="kube-system/konnectivity-agent-jbpx6" Jul 26 15:37:58.789825 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.789550 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7c179a4b-e953-4d22-9f2d-7cf682e833fd-etc-tuned\") pod \"tuned-65dgw\" (UID: \"7c179a4b-e953-4d22-9f2d-7cf682e833fd\") " pod="openshift-cluster-node-tuning-operator/tuned-65dgw" Jul 26 15:37:58.789914 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:37:58.789877 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 26 15:37:58.789914 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:37:58.789893 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 26 15:37:58.789914 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:37:58.789905 2577 projected.go:196] Error preparing data for projected volume kube-api-access-fztmw for pod openshift-network-diagnostics/network-check-target-d2nnz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 26 15:37:58.790007 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:37:58.789955 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f3e728c9-4710-4f6c-977f-caf52a2bfcaf-kube-api-access-fztmw podName:f3e728c9-4710-4f6c-977f-caf52a2bfcaf nodeName:}" failed. No retries permitted until 2026-07-26 15:37:59.289940941 +0000 UTC m=+2.029294253 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-fztmw" (UniqueName: "kubernetes.io/projected/f3e728c9-4710-4f6c-977f-caf52a2bfcaf-kube-api-access-fztmw") pod "network-check-target-d2nnz" (UID: "f3e728c9-4710-4f6c-977f-caf52a2bfcaf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 26 15:37:58.791752 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.791732 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2j8s\" (UniqueName: \"kubernetes.io/projected/356a056a-d4d5-48ed-a10f-928b36becc81-kube-api-access-d2j8s\") pod \"node-resolver-v4mgw\" (UID: \"356a056a-d4d5-48ed-a10f-928b36becc81\") " pod="openshift-dns/node-resolver-v4mgw" Jul 26 15:37:58.791886 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.791867 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvf4l\" (UniqueName: \"kubernetes.io/projected/bd02936a-a782-4e31-89bb-1292065d0abc-kube-api-access-qvf4l\") pod \"multus-x4dkt\" (UID: \"bd02936a-a782-4e31-89bb-1292065d0abc\") " pod="openshift-multus/multus-x4dkt" Jul 26 15:37:58.795175 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.795151 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmpnp\" (UniqueName: \"kubernetes.io/projected/098dd2e5-9540-425f-ba4e-71ea0b064be7-kube-api-access-jmpnp\") pod \"node-ca-mfwqb\" (UID: \"098dd2e5-9540-425f-ba4e-71ea0b064be7\") " pod="openshift-image-registry/node-ca-mfwqb" Jul 26 15:37:58.800170 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.800139 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zljx\" (UniqueName: \"kubernetes.io/projected/c2653055-9d7a-4aeb-9090-fb495a2b650b-kube-api-access-9zljx\") pod \"multus-additional-cni-plugins-dzc7t\" (UID: \"c2653055-9d7a-4aeb-9090-fb495a2b650b\") " pod="openshift-multus/multus-additional-cni-plugins-dzc7t" Jul 26 15:37:58.800969 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.800951 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-657xn\" (UniqueName: \"kubernetes.io/projected/acf5fc6d-b7e4-4581-8878-f5776e505746-kube-api-access-657xn\") pod \"iptables-alerter-v46ns\" (UID: \"acf5fc6d-b7e4-4581-8878-f5776e505746\") " pod="openshift-network-operator/iptables-alerter-v46ns" Jul 26 15:37:58.801043 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.800999 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgl7v\" (UniqueName: \"kubernetes.io/projected/7c179a4b-e953-4d22-9f2d-7cf682e833fd-kube-api-access-fgl7v\") pod \"tuned-65dgw\" (UID: \"7c179a4b-e953-4d22-9f2d-7cf682e833fd\") " pod="openshift-cluster-node-tuning-operator/tuned-65dgw" Jul 26 15:37:58.862474 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.862440 2577 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 26 15:37:58.888860 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.888826 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfn9p\" (UniqueName: \"kubernetes.io/projected/b5f442b1-a918-43a5-9870-f2183fe5085a-kube-api-access-vfn9p\") pod \"network-metrics-daemon-kgk6w\" (UID: \"b5f442b1-a918-43a5-9870-f2183fe5085a\") " pod="openshift-multus/network-metrics-daemon-kgk6w" Jul 26 15:37:58.888860 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.888860 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/cbe2cd41-6ebc-4722-a813-57d83fa1d242-node-log\") pod \"ovnkube-node-ztlp6\" (UID: \"cbe2cd41-6ebc-4722-a813-57d83fa1d242\") " pod="openshift-ovn-kubernetes/ovnkube-node-ztlp6" Jul 26 15:37:58.889023 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.888878 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/cbe2cd41-6ebc-4722-a813-57d83fa1d242-systemd-units\") pod \"ovnkube-node-ztlp6\" (UID: \"cbe2cd41-6ebc-4722-a813-57d83fa1d242\") " pod="openshift-ovn-kubernetes/ovnkube-node-ztlp6" Jul 26 15:37:58.889023 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.888893 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cbe2cd41-6ebc-4722-a813-57d83fa1d242-var-lib-openvswitch\") pod \"ovnkube-node-ztlp6\" (UID: \"cbe2cd41-6ebc-4722-a813-57d83fa1d242\") " pod="openshift-ovn-kubernetes/ovnkube-node-ztlp6" Jul 26 15:37:58.889023 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.888931 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/cbe2cd41-6ebc-4722-a813-57d83fa1d242-systemd-units\") pod \"ovnkube-node-ztlp6\" (UID: \"cbe2cd41-6ebc-4722-a813-57d83fa1d242\") " pod="openshift-ovn-kubernetes/ovnkube-node-ztlp6" Jul 26 15:37:58.889023 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.888944 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/cbe2cd41-6ebc-4722-a813-57d83fa1d242-node-log\") pod \"ovnkube-node-ztlp6\" (UID: \"cbe2cd41-6ebc-4722-a813-57d83fa1d242\") " pod="openshift-ovn-kubernetes/ovnkube-node-ztlp6" Jul 26 15:37:58.889023 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.888995 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cbe2cd41-6ebc-4722-a813-57d83fa1d242-host-cni-bin\") pod \"ovnkube-node-ztlp6\" (UID: \"cbe2cd41-6ebc-4722-a813-57d83fa1d242\") " pod="openshift-ovn-kubernetes/ovnkube-node-ztlp6" Jul 26 15:37:58.889023 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.889012 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cbe2cd41-6ebc-4722-a813-57d83fa1d242-var-lib-openvswitch\") pod \"ovnkube-node-ztlp6\" (UID: \"cbe2cd41-6ebc-4722-a813-57d83fa1d242\") " pod="openshift-ovn-kubernetes/ovnkube-node-ztlp6" Jul 26 15:37:58.889023 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.889022 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/cbe2cd41-6ebc-4722-a813-57d83fa1d242-ovnkube-config\") pod \"ovnkube-node-ztlp6\" (UID: \"cbe2cd41-6ebc-4722-a813-57d83fa1d242\") " pod="openshift-ovn-kubernetes/ovnkube-node-ztlp6" Jul 26 15:37:58.889222 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.889044 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/cbe2cd41-6ebc-4722-a813-57d83fa1d242-env-overrides\") pod \"ovnkube-node-ztlp6\" (UID: \"cbe2cd41-6ebc-4722-a813-57d83fa1d242\") " pod="openshift-ovn-kubernetes/ovnkube-node-ztlp6" Jul 26 15:37:58.889222 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.889052 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cbe2cd41-6ebc-4722-a813-57d83fa1d242-host-cni-bin\") pod \"ovnkube-node-ztlp6\" (UID: \"cbe2cd41-6ebc-4722-a813-57d83fa1d242\") " pod="openshift-ovn-kubernetes/ovnkube-node-ztlp6" Jul 26 15:37:58.889222 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.889065 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5f442b1-a918-43a5-9870-f2183fe5085a-metrics-certs\") pod \"network-metrics-daemon-kgk6w\" (UID: \"b5f442b1-a918-43a5-9870-f2183fe5085a\") " pod="openshift-multus/network-metrics-daemon-kgk6w" Jul 26 15:37:58.889222 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.889088 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/cbe2cd41-6ebc-4722-a813-57d83fa1d242-run-ovn\") pod \"ovnkube-node-ztlp6\" (UID: \"cbe2cd41-6ebc-4722-a813-57d83fa1d242\") " pod="openshift-ovn-kubernetes/ovnkube-node-ztlp6" Jul 26 15:37:58.889222 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.889134 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/cbe2cd41-6ebc-4722-a813-57d83fa1d242-run-ovn\") pod \"ovnkube-node-ztlp6\" (UID: \"cbe2cd41-6ebc-4722-a813-57d83fa1d242\") " pod="openshift-ovn-kubernetes/ovnkube-node-ztlp6" Jul 26 15:37:58.889222 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.889159 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cbe2cd41-6ebc-4722-a813-57d83fa1d242-host-run-ovn-kubernetes\") pod \"ovnkube-node-ztlp6\" (UID: \"cbe2cd41-6ebc-4722-a813-57d83fa1d242\") " pod="openshift-ovn-kubernetes/ovnkube-node-ztlp6" Jul 26 15:37:58.889222 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.889184 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/cbe2cd41-6ebc-4722-a813-57d83fa1d242-ovn-node-metrics-cert\") pod \"ovnkube-node-ztlp6\" (UID: \"cbe2cd41-6ebc-4722-a813-57d83fa1d242\") " pod="openshift-ovn-kubernetes/ovnkube-node-ztlp6" Jul 26 15:37:58.889222 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:37:58.889190 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 26 15:37:58.889222 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.889200 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cbe2cd41-6ebc-4722-a813-57d83fa1d242-host-slash\") pod \"ovnkube-node-ztlp6\" (UID: \"cbe2cd41-6ebc-4722-a813-57d83fa1d242\") " pod="openshift-ovn-kubernetes/ovnkube-node-ztlp6" Jul 26 15:37:58.889586 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.889255 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cbe2cd41-6ebc-4722-a813-57d83fa1d242-host-slash\") pod \"ovnkube-node-ztlp6\" (UID: \"cbe2cd41-6ebc-4722-a813-57d83fa1d242\") " pod="openshift-ovn-kubernetes/ovnkube-node-ztlp6" Jul 26 15:37:58.889586 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:37:58.889261 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5f442b1-a918-43a5-9870-f2183fe5085a-metrics-certs podName:b5f442b1-a918-43a5-9870-f2183fe5085a nodeName:}" failed. No retries permitted until 2026-07-26 15:37:59.3892419 +0000 UTC m=+2.128595193 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b5f442b1-a918-43a5-9870-f2183fe5085a-metrics-certs") pod "network-metrics-daemon-kgk6w" (UID: "b5f442b1-a918-43a5-9870-f2183fe5085a") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 26 15:37:58.889586 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.889282 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/350d34fc-ceeb-477e-8486-0a712e354c67-socket-dir\") pod \"aws-ebs-csi-driver-node-qnht4\" (UID: \"350d34fc-ceeb-477e-8486-0a712e354c67\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qnht4" Jul 26 15:37:58.889586 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.889296 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cbe2cd41-6ebc-4722-a813-57d83fa1d242-host-run-ovn-kubernetes\") pod \"ovnkube-node-ztlp6\" (UID: \"cbe2cd41-6ebc-4722-a813-57d83fa1d242\") " pod="openshift-ovn-kubernetes/ovnkube-node-ztlp6" Jul 26 15:37:58.889586 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.889343 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/350d34fc-ceeb-477e-8486-0a712e354c67-registration-dir\") pod \"aws-ebs-csi-driver-node-qnht4\" (UID: \"350d34fc-ceeb-477e-8486-0a712e354c67\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qnht4" Jul 26 15:37:58.889586 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.889304 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/350d34fc-ceeb-477e-8486-0a712e354c67-registration-dir\") pod \"aws-ebs-csi-driver-node-qnht4\" (UID: \"350d34fc-ceeb-477e-8486-0a712e354c67\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qnht4" Jul 26 15:37:58.889586 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.889379 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/350d34fc-ceeb-477e-8486-0a712e354c67-device-dir\") pod \"aws-ebs-csi-driver-node-qnht4\" (UID: \"350d34fc-ceeb-477e-8486-0a712e354c67\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qnht4" Jul 26 15:37:58.889586 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.889415 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/350d34fc-ceeb-477e-8486-0a712e354c67-sys-fs\") pod \"aws-ebs-csi-driver-node-qnht4\" (UID: \"350d34fc-ceeb-477e-8486-0a712e354c67\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qnht4" Jul 26 15:37:58.889586 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.889440 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/cbe2cd41-6ebc-4722-a813-57d83fa1d242-host-kubelet\") pod \"ovnkube-node-ztlp6\" (UID: \"cbe2cd41-6ebc-4722-a813-57d83fa1d242\") " pod="openshift-ovn-kubernetes/ovnkube-node-ztlp6" Jul 26 15:37:58.889586 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.889448 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/350d34fc-ceeb-477e-8486-0a712e354c67-device-dir\") pod \"aws-ebs-csi-driver-node-qnht4\" (UID: \"350d34fc-ceeb-477e-8486-0a712e354c67\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qnht4" Jul 26 15:37:58.889586 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.889454 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/350d34fc-ceeb-477e-8486-0a712e354c67-socket-dir\") pod \"aws-ebs-csi-driver-node-qnht4\" (UID: \"350d34fc-ceeb-477e-8486-0a712e354c67\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qnht4" Jul 26 15:37:58.889586 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.889463 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cbe2cd41-6ebc-4722-a813-57d83fa1d242-run-openvswitch\") pod \"ovnkube-node-ztlp6\" (UID: \"cbe2cd41-6ebc-4722-a813-57d83fa1d242\") " pod="openshift-ovn-kubernetes/ovnkube-node-ztlp6" Jul 26 15:37:58.889586 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.889499 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/cbe2cd41-6ebc-4722-a813-57d83fa1d242-host-kubelet\") pod \"ovnkube-node-ztlp6\" (UID: \"cbe2cd41-6ebc-4722-a813-57d83fa1d242\") " pod="openshift-ovn-kubernetes/ovnkube-node-ztlp6" Jul 26 15:37:58.889586 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.889512 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cbe2cd41-6ebc-4722-a813-57d83fa1d242-host-run-netns\") pod \"ovnkube-node-ztlp6\" (UID: \"cbe2cd41-6ebc-4722-a813-57d83fa1d242\") " pod="openshift-ovn-kubernetes/ovnkube-node-ztlp6" Jul 26 15:37:58.889586 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.889517 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/350d34fc-ceeb-477e-8486-0a712e354c67-sys-fs\") pod \"aws-ebs-csi-driver-node-qnht4\" (UID: \"350d34fc-ceeb-477e-8486-0a712e354c67\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qnht4" Jul 26 15:37:58.889586 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.889539 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrgzd\" (UniqueName: \"kubernetes.io/projected/cbe2cd41-6ebc-4722-a813-57d83fa1d242-kube-api-access-jrgzd\") pod \"ovnkube-node-ztlp6\" (UID: \"cbe2cd41-6ebc-4722-a813-57d83fa1d242\") " pod="openshift-ovn-kubernetes/ovnkube-node-ztlp6" Jul 26 15:37:58.890167 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.889561 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cbe2cd41-6ebc-4722-a813-57d83fa1d242-host-run-netns\") pod \"ovnkube-node-ztlp6\" (UID: \"cbe2cd41-6ebc-4722-a813-57d83fa1d242\") " pod="openshift-ovn-kubernetes/ovnkube-node-ztlp6" Jul 26 15:37:58.890167 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.889569 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cbe2cd41-6ebc-4722-a813-57d83fa1d242-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ztlp6\" (UID: \"cbe2cd41-6ebc-4722-a813-57d83fa1d242\") " pod="openshift-ovn-kubernetes/ovnkube-node-ztlp6" Jul 26 15:37:58.890167 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.889603 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/350d34fc-ceeb-477e-8486-0a712e354c67-etc-selinux\") pod \"aws-ebs-csi-driver-node-qnht4\" (UID: \"350d34fc-ceeb-477e-8486-0a712e354c67\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qnht4" Jul 26 15:37:58.890167 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.889604 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cbe2cd41-6ebc-4722-a813-57d83fa1d242-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ztlp6\" (UID: \"cbe2cd41-6ebc-4722-a813-57d83fa1d242\") " pod="openshift-ovn-kubernetes/ovnkube-node-ztlp6" Jul 26 15:37:58.890167 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.889608 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/cbe2cd41-6ebc-4722-a813-57d83fa1d242-env-overrides\") pod \"ovnkube-node-ztlp6\" (UID: \"cbe2cd41-6ebc-4722-a813-57d83fa1d242\") " pod="openshift-ovn-kubernetes/ovnkube-node-ztlp6" Jul 26 15:37:58.890167 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.889627 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cbe2cd41-6ebc-4722-a813-57d83fa1d242-etc-openvswitch\") pod \"ovnkube-node-ztlp6\" (UID: \"cbe2cd41-6ebc-4722-a813-57d83fa1d242\") " pod="openshift-ovn-kubernetes/ovnkube-node-ztlp6" Jul 26 15:37:58.890167 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.889562 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cbe2cd41-6ebc-4722-a813-57d83fa1d242-run-openvswitch\") pod \"ovnkube-node-ztlp6\" (UID: \"cbe2cd41-6ebc-4722-a813-57d83fa1d242\") " pod="openshift-ovn-kubernetes/ovnkube-node-ztlp6" Jul 26 15:37:58.890167 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.889651 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/cbe2cd41-6ebc-4722-a813-57d83fa1d242-log-socket\") pod \"ovnkube-node-ztlp6\" (UID: \"cbe2cd41-6ebc-4722-a813-57d83fa1d242\") " pod="openshift-ovn-kubernetes/ovnkube-node-ztlp6" Jul 26 15:37:58.890167 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.889677 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/cbe2cd41-6ebc-4722-a813-57d83fa1d242-ovnkube-script-lib\") pod \"ovnkube-node-ztlp6\" (UID: \"cbe2cd41-6ebc-4722-a813-57d83fa1d242\") " pod="openshift-ovn-kubernetes/ovnkube-node-ztlp6" Jul 26 15:37:58.890167 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.889683 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cbe2cd41-6ebc-4722-a813-57d83fa1d242-etc-openvswitch\") pod \"ovnkube-node-ztlp6\" (UID: \"cbe2cd41-6ebc-4722-a813-57d83fa1d242\") " pod="openshift-ovn-kubernetes/ovnkube-node-ztlp6" Jul 26 15:37:58.890167 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.889702 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cz2t\" (UniqueName: \"kubernetes.io/projected/350d34fc-ceeb-477e-8486-0a712e354c67-kube-api-access-6cz2t\") pod \"aws-ebs-csi-driver-node-qnht4\" (UID: \"350d34fc-ceeb-477e-8486-0a712e354c67\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qnht4" Jul 26 15:37:58.890167 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.889710 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/cbe2cd41-6ebc-4722-a813-57d83fa1d242-log-socket\") pod \"ovnkube-node-ztlp6\" (UID: \"cbe2cd41-6ebc-4722-a813-57d83fa1d242\") " pod="openshift-ovn-kubernetes/ovnkube-node-ztlp6" Jul 26 15:37:58.890167 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.889724 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/cbe2cd41-6ebc-4722-a813-57d83fa1d242-ovnkube-config\") pod \"ovnkube-node-ztlp6\" (UID: \"cbe2cd41-6ebc-4722-a813-57d83fa1d242\") " pod="openshift-ovn-kubernetes/ovnkube-node-ztlp6" Jul 26 15:37:58.890167 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.889740 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/350d34fc-ceeb-477e-8486-0a712e354c67-etc-selinux\") pod \"aws-ebs-csi-driver-node-qnht4\" (UID: \"350d34fc-ceeb-477e-8486-0a712e354c67\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qnht4" Jul 26 15:37:58.890167 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.889759 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/cbe2cd41-6ebc-4722-a813-57d83fa1d242-run-systemd\") pod \"ovnkube-node-ztlp6\" (UID: \"cbe2cd41-6ebc-4722-a813-57d83fa1d242\") " pod="openshift-ovn-kubernetes/ovnkube-node-ztlp6" Jul 26 15:37:58.890167 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.889792 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/cbe2cd41-6ebc-4722-a813-57d83fa1d242-host-cni-netd\") pod \"ovnkube-node-ztlp6\" (UID: \"cbe2cd41-6ebc-4722-a813-57d83fa1d242\") " pod="openshift-ovn-kubernetes/ovnkube-node-ztlp6" Jul 26 15:37:58.890167 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.889829 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/350d34fc-ceeb-477e-8486-0a712e354c67-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qnht4\" (UID: \"350d34fc-ceeb-477e-8486-0a712e354c67\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qnht4" Jul 26 15:37:58.890716 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.889833 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/cbe2cd41-6ebc-4722-a813-57d83fa1d242-run-systemd\") pod \"ovnkube-node-ztlp6\" (UID: \"cbe2cd41-6ebc-4722-a813-57d83fa1d242\") " pod="openshift-ovn-kubernetes/ovnkube-node-ztlp6" Jul 26 15:37:58.890716 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.889865 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/cbe2cd41-6ebc-4722-a813-57d83fa1d242-host-cni-netd\") pod \"ovnkube-node-ztlp6\" (UID: \"cbe2cd41-6ebc-4722-a813-57d83fa1d242\") " pod="openshift-ovn-kubernetes/ovnkube-node-ztlp6" Jul 26 15:37:58.890716 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.889872 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/350d34fc-ceeb-477e-8486-0a712e354c67-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qnht4\" (UID: \"350d34fc-ceeb-477e-8486-0a712e354c67\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qnht4" Jul 26 15:37:58.890716 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.890078 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/cbe2cd41-6ebc-4722-a813-57d83fa1d242-ovnkube-script-lib\") pod \"ovnkube-node-ztlp6\" (UID: \"cbe2cd41-6ebc-4722-a813-57d83fa1d242\") " pod="openshift-ovn-kubernetes/ovnkube-node-ztlp6" Jul 26 15:37:58.891873 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.891857 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/cbe2cd41-6ebc-4722-a813-57d83fa1d242-ovn-node-metrics-cert\") pod \"ovnkube-node-ztlp6\" (UID: \"cbe2cd41-6ebc-4722-a813-57d83fa1d242\") " pod="openshift-ovn-kubernetes/ovnkube-node-ztlp6" Jul 26 15:37:58.898065 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.898040 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrgzd\" (UniqueName: \"kubernetes.io/projected/cbe2cd41-6ebc-4722-a813-57d83fa1d242-kube-api-access-jrgzd\") pod \"ovnkube-node-ztlp6\" (UID: \"cbe2cd41-6ebc-4722-a813-57d83fa1d242\") " pod="openshift-ovn-kubernetes/ovnkube-node-ztlp6" Jul 26 15:37:58.898251 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.898232 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cz2t\" (UniqueName: \"kubernetes.io/projected/350d34fc-ceeb-477e-8486-0a712e354c67-kube-api-access-6cz2t\") pod \"aws-ebs-csi-driver-node-qnht4\" (UID: \"350d34fc-ceeb-477e-8486-0a712e354c67\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qnht4" Jul 26 15:37:58.898308 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.898270 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfn9p\" (UniqueName: \"kubernetes.io/projected/b5f442b1-a918-43a5-9870-f2183fe5085a-kube-api-access-vfn9p\") pod \"network-metrics-daemon-kgk6w\" (UID: \"b5f442b1-a918-43a5-9870-f2183fe5085a\") " pod="openshift-multus/network-metrics-daemon-kgk6w" Jul 26 15:37:58.984384 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:58.984356 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-65dgw" Jul 26 15:37:58.994148 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:58.994111 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c179a4b_e953_4d22_9f2d_7cf682e833fd.slice/crio-d46c4c156da8c351ebf2871d6fc2d105d16c534ae103d782d4a639dad36c672e WatchSource:0}: Error finding container d46c4c156da8c351ebf2871d6fc2d105d16c534ae103d782d4a639dad36c672e: Status 404 returned error can't find the container with id d46c4c156da8c351ebf2871d6fc2d105d16c534ae103d782d4a639dad36c672e Jul 26 15:37:59.000297 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:59.000280 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-x4dkt" Jul 26 15:37:59.008621 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:59.008593 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd02936a_a782_4e31_89bb_1292065d0abc.slice/crio-0e2b244a60df3a36249c274e4c90df6efb99714d5a9d8676649b43cb454b9cc3 WatchSource:0}: Error finding container 0e2b244a60df3a36249c274e4c90df6efb99714d5a9d8676649b43cb454b9cc3: Status 404 returned error can't find the container with id 0e2b244a60df3a36249c274e4c90df6efb99714d5a9d8676649b43cb454b9cc3 Jul 26 15:37:59.018059 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:59.018041 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-v46ns" Jul 26 15:37:59.027847 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:59.027821 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacf5fc6d_b7e4_4581_8878_f5776e505746.slice/crio-e52460df71020e2370837c2cb7bf64e49e3a0a69cf9e0b108f57d78edea47536 WatchSource:0}: Error finding container e52460df71020e2370837c2cb7bf64e49e3a0a69cf9e0b108f57d78edea47536: Status 404 returned error can't find the container with id e52460df71020e2370837c2cb7bf64e49e3a0a69cf9e0b108f57d78edea47536 Jul 26 15:37:59.040954 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:59.040934 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-jbpx6" Jul 26 15:37:59.046780 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:59.046765 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-v4mgw" Jul 26 15:37:59.048589 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:59.048569 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod806292e6_a776_4195_a595_263edb35e68b.slice/crio-d875f6ce7fc93c2e85bcb0abc51539e8826e0ff6e8cb3148b4f8387857e7d89b WatchSource:0}: Error finding container d875f6ce7fc93c2e85bcb0abc51539e8826e0ff6e8cb3148b4f8387857e7d89b: Status 404 returned error can't find the container with id d875f6ce7fc93c2e85bcb0abc51539e8826e0ff6e8cb3148b4f8387857e7d89b Jul 26 15:37:59.052566 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:59.052547 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-mfwqb" Jul 26 15:37:59.054644 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:59.054624 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod356a056a_d4d5_48ed_a10f_928b36becc81.slice/crio-08d3ec019be3d75834c837368e5126d3b1d048ae965e6447d9f284569b02fe84 WatchSource:0}: Error finding container 08d3ec019be3d75834c837368e5126d3b1d048ae965e6447d9f284569b02fe84: Status 404 returned error can't find the container with id 08d3ec019be3d75834c837368e5126d3b1d048ae965e6447d9f284569b02fe84 Jul 26 15:37:59.057230 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:59.057212 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-dzc7t" Jul 26 15:37:59.060307 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:59.060288 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod098dd2e5_9540_425f_ba4e_71ea0b064be7.slice/crio-c66e91cd24f9c486bd4cb3c20fa6d2b701d57f4e09a4ff28f4d928ae860faabd WatchSource:0}: Error finding container c66e91cd24f9c486bd4cb3c20fa6d2b701d57f4e09a4ff28f4d928ae860faabd: Status 404 returned error can't find the container with id c66e91cd24f9c486bd4cb3c20fa6d2b701d57f4e09a4ff28f4d928ae860faabd Jul 26 15:37:59.062864 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:59.062843 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qnht4" Jul 26 15:37:59.065530 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:59.065508 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2653055_9d7a_4aeb_9090_fb495a2b650b.slice/crio-ac6796aee3eab2b87f5084c2e92d8fe48f65034dfa5609e3c6fd77766ac28edf WatchSource:0}: Error finding container ac6796aee3eab2b87f5084c2e92d8fe48f65034dfa5609e3c6fd77766ac28edf: Status 404 returned error can't find the container with id ac6796aee3eab2b87f5084c2e92d8fe48f65034dfa5609e3c6fd77766ac28edf Jul 26 15:37:59.067320 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:59.067294 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-ztlp6" Jul 26 15:37:59.073261 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:59.073244 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod350d34fc_ceeb_477e_8486_0a712e354c67.slice/crio-3c51ea05b83febd31cbfd380a5c9cf3307ea68bd84c08a7ec31614c89ad7e415 WatchSource:0}: Error finding container 3c51ea05b83febd31cbfd380a5c9cf3307ea68bd84c08a7ec31614c89ad7e415: Status 404 returned error can't find the container with id 3c51ea05b83febd31cbfd380a5c9cf3307ea68bd84c08a7ec31614c89ad7e415 Jul 26 15:37:59.076250 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:37:59.076231 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbe2cd41_6ebc_4722_a813_57d83fa1d242.slice/crio-1120e9c2389d0971d310b59c3be648b9ed3c2efff4389363b0811c5608e08db3 WatchSource:0}: Error finding container 1120e9c2389d0971d310b59c3be648b9ed3c2efff4389363b0811c5608e08db3: Status 404 returned error can't find the container with id 1120e9c2389d0971d310b59c3be648b9ed3c2efff4389363b0811c5608e08db3 Jul 26 15:37:59.123343 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:59.123314 2577 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 26 15:37:59.293297 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:59.293205 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fztmw\" (UniqueName: \"kubernetes.io/projected/f3e728c9-4710-4f6c-977f-caf52a2bfcaf-kube-api-access-fztmw\") pod \"network-check-target-d2nnz\" (UID: \"f3e728c9-4710-4f6c-977f-caf52a2bfcaf\") " pod="openshift-network-diagnostics/network-check-target-d2nnz" Jul 26 15:37:59.293297 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:59.293272 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f6bd9013-635a-4af8-b0ee-621a280a459a-original-pull-secret\") pod \"global-pull-secret-syncer-l56b4\" (UID: \"f6bd9013-635a-4af8-b0ee-621a280a459a\") " pod="kube-system/global-pull-secret-syncer-l56b4" Jul 26 15:37:59.293493 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:37:59.293429 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 26 15:37:59.293493 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:37:59.293476 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f6bd9013-635a-4af8-b0ee-621a280a459a-original-pull-secret podName:f6bd9013-635a-4af8-b0ee-621a280a459a nodeName:}" failed. No retries permitted until 2026-07-26 15:38:00.293463183 +0000 UTC m=+3.032816470 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f6bd9013-635a-4af8-b0ee-621a280a459a-original-pull-secret") pod "global-pull-secret-syncer-l56b4" (UID: "f6bd9013-635a-4af8-b0ee-621a280a459a") : object "kube-system"/"original-pull-secret" not registered Jul 26 15:37:59.293733 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:37:59.293707 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 26 15:37:59.293733 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:37:59.293734 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 26 15:37:59.293906 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:37:59.293747 2577 projected.go:196] Error preparing data for projected volume kube-api-access-fztmw for pod openshift-network-diagnostics/network-check-target-d2nnz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 26 15:37:59.293906 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:37:59.293802 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f3e728c9-4710-4f6c-977f-caf52a2bfcaf-kube-api-access-fztmw podName:f3e728c9-4710-4f6c-977f-caf52a2bfcaf nodeName:}" failed. No retries permitted until 2026-07-26 15:38:00.293785308 +0000 UTC m=+3.033138612 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-fztmw" (UniqueName: "kubernetes.io/projected/f3e728c9-4710-4f6c-977f-caf52a2bfcaf-kube-api-access-fztmw") pod "network-check-target-d2nnz" (UID: "f3e728c9-4710-4f6c-977f-caf52a2bfcaf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 26 15:37:59.394012 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:59.393917 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5f442b1-a918-43a5-9870-f2183fe5085a-metrics-certs\") pod \"network-metrics-daemon-kgk6w\" (UID: \"b5f442b1-a918-43a5-9870-f2183fe5085a\") " pod="openshift-multus/network-metrics-daemon-kgk6w" Jul 26 15:37:59.394196 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:37:59.394074 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 26 15:37:59.394196 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:37:59.394138 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5f442b1-a918-43a5-9870-f2183fe5085a-metrics-certs podName:b5f442b1-a918-43a5-9870-f2183fe5085a nodeName:}" failed. No retries permitted until 2026-07-26 15:38:00.394119845 +0000 UTC m=+3.133473133 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b5f442b1-a918-43a5-9870-f2183fe5085a-metrics-certs") pod "network-metrics-daemon-kgk6w" (UID: "b5f442b1-a918-43a5-9870-f2183fe5085a") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 26 15:37:59.717579 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:59.717489 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-25 15:32:58 +0000 UTC" deadline="2028-03-09 16:41:26.235071862 +0000 UTC" Jul 26 15:37:59.717579 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:59.717532 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14209h3m26.517545069s" Jul 26 15:37:59.801556 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:59.801521 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-mfwqb" event={"ID":"098dd2e5-9540-425f-ba4e-71ea0b064be7","Type":"ContainerStarted","Data":"c66e91cd24f9c486bd4cb3c20fa6d2b701d57f4e09a4ff28f4d928ae860faabd"} Jul 26 15:37:59.814818 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:59.814776 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-v46ns" event={"ID":"acf5fc6d-b7e4-4581-8878-f5776e505746","Type":"ContainerStarted","Data":"e52460df71020e2370837c2cb7bf64e49e3a0a69cf9e0b108f57d78edea47536"} Jul 26 15:37:59.834673 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:59.834634 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qnht4" event={"ID":"350d34fc-ceeb-477e-8486-0a712e354c67","Type":"ContainerStarted","Data":"3c51ea05b83febd31cbfd380a5c9cf3307ea68bd84c08a7ec31614c89ad7e415"} Jul 26 15:37:59.850297 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:59.849978 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dzc7t" event={"ID":"c2653055-9d7a-4aeb-9090-fb495a2b650b","Type":"ContainerStarted","Data":"ac6796aee3eab2b87f5084c2e92d8fe48f65034dfa5609e3c6fd77766ac28edf"} Jul 26 15:37:59.862806 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:59.862769 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-v4mgw" event={"ID":"356a056a-d4d5-48ed-a10f-928b36becc81","Type":"ContainerStarted","Data":"08d3ec019be3d75834c837368e5126d3b1d048ae965e6447d9f284569b02fe84"} Jul 26 15:37:59.876001 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:59.875938 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-jbpx6" event={"ID":"806292e6-a776-4195-a595-263edb35e68b","Type":"ContainerStarted","Data":"d875f6ce7fc93c2e85bcb0abc51539e8826e0ff6e8cb3148b4f8387857e7d89b"} Jul 26 15:37:59.882071 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:59.882046 2577 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 26 15:37:59.898366 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:59.898332 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-x4dkt" event={"ID":"bd02936a-a782-4e31-89bb-1292065d0abc","Type":"ContainerStarted","Data":"0e2b244a60df3a36249c274e4c90df6efb99714d5a9d8676649b43cb454b9cc3"} Jul 26 15:37:59.909363 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:59.909330 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-65dgw" event={"ID":"7c179a4b-e953-4d22-9f2d-7cf682e833fd","Type":"ContainerStarted","Data":"d46c4c156da8c351ebf2871d6fc2d105d16c534ae103d782d4a639dad36c672e"} Jul 26 15:37:59.923584 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:37:59.923469 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ztlp6" event={"ID":"cbe2cd41-6ebc-4722-a813-57d83fa1d242","Type":"ContainerStarted","Data":"1120e9c2389d0971d310b59c3be648b9ed3c2efff4389363b0811c5608e08db3"} Jul 26 15:38:00.301887 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:00.301844 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fztmw\" (UniqueName: \"kubernetes.io/projected/f3e728c9-4710-4f6c-977f-caf52a2bfcaf-kube-api-access-fztmw\") pod \"network-check-target-d2nnz\" (UID: \"f3e728c9-4710-4f6c-977f-caf52a2bfcaf\") " pod="openshift-network-diagnostics/network-check-target-d2nnz" Jul 26 15:38:00.302069 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:00.301912 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f6bd9013-635a-4af8-b0ee-621a280a459a-original-pull-secret\") pod \"global-pull-secret-syncer-l56b4\" (UID: \"f6bd9013-635a-4af8-b0ee-621a280a459a\") " pod="kube-system/global-pull-secret-syncer-l56b4" Jul 26 15:38:00.302135 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:00.302076 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 26 15:38:00.302195 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:00.302136 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f6bd9013-635a-4af8-b0ee-621a280a459a-original-pull-secret podName:f6bd9013-635a-4af8-b0ee-621a280a459a nodeName:}" failed. No retries permitted until 2026-07-26 15:38:02.302117329 +0000 UTC m=+5.041470638 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f6bd9013-635a-4af8-b0ee-621a280a459a-original-pull-secret") pod "global-pull-secret-syncer-l56b4" (UID: "f6bd9013-635a-4af8-b0ee-621a280a459a") : object "kube-system"/"original-pull-secret" not registered Jul 26 15:38:00.302697 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:00.302675 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 26 15:38:00.302796 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:00.302706 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 26 15:38:00.302796 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:00.302719 2577 projected.go:196] Error preparing data for projected volume kube-api-access-fztmw for pod openshift-network-diagnostics/network-check-target-d2nnz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 26 15:38:00.302796 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:00.302767 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f3e728c9-4710-4f6c-977f-caf52a2bfcaf-kube-api-access-fztmw podName:f3e728c9-4710-4f6c-977f-caf52a2bfcaf nodeName:}" failed. No retries permitted until 2026-07-26 15:38:02.302751232 +0000 UTC m=+5.042104551 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-fztmw" (UniqueName: "kubernetes.io/projected/f3e728c9-4710-4f6c-977f-caf52a2bfcaf-kube-api-access-fztmw") pod "network-check-target-d2nnz" (UID: "f3e728c9-4710-4f6c-977f-caf52a2bfcaf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 26 15:38:00.402311 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:00.402274 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5f442b1-a918-43a5-9870-f2183fe5085a-metrics-certs\") pod \"network-metrics-daemon-kgk6w\" (UID: \"b5f442b1-a918-43a5-9870-f2183fe5085a\") " pod="openshift-multus/network-metrics-daemon-kgk6w" Jul 26 15:38:00.402514 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:00.402484 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 26 15:38:00.402576 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:00.402548 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5f442b1-a918-43a5-9870-f2183fe5085a-metrics-certs podName:b5f442b1-a918-43a5-9870-f2183fe5085a nodeName:}" failed. No retries permitted until 2026-07-26 15:38:02.402530062 +0000 UTC m=+5.141883356 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b5f442b1-a918-43a5-9870-f2183fe5085a-metrics-certs") pod "network-metrics-daemon-kgk6w" (UID: "b5f442b1-a918-43a5-9870-f2183fe5085a") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 26 15:38:00.718617 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:00.718497 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-25 15:32:58 +0000 UTC" deadline="2028-01-20 06:01:16.607294265 +0000 UTC" Jul 26 15:38:00.718617 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:00.718563 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13022h23m15.888735316s" Jul 26 15:38:00.780967 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:00.780933 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kgk6w" Jul 26 15:38:00.781136 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:00.780933 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l56b4" Jul 26 15:38:00.781136 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:00.781091 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kgk6w" podUID="b5f442b1-a918-43a5-9870-f2183fe5085a" Jul 26 15:38:00.781136 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:00.780934 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d2nnz" Jul 26 15:38:00.781309 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:00.781161 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l56b4" podUID="f6bd9013-635a-4af8-b0ee-621a280a459a" Jul 26 15:38:00.781309 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:00.781257 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d2nnz" podUID="f3e728c9-4710-4f6c-977f-caf52a2bfcaf" Jul 26 15:38:02.319677 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:02.319642 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fztmw\" (UniqueName: \"kubernetes.io/projected/f3e728c9-4710-4f6c-977f-caf52a2bfcaf-kube-api-access-fztmw\") pod \"network-check-target-d2nnz\" (UID: \"f3e728c9-4710-4f6c-977f-caf52a2bfcaf\") " pod="openshift-network-diagnostics/network-check-target-d2nnz" Jul 26 15:38:02.320121 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:02.319710 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f6bd9013-635a-4af8-b0ee-621a280a459a-original-pull-secret\") pod \"global-pull-secret-syncer-l56b4\" (UID: \"f6bd9013-635a-4af8-b0ee-621a280a459a\") " pod="kube-system/global-pull-secret-syncer-l56b4" Jul 26 15:38:02.320121 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:02.319822 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 26 15:38:02.320121 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:02.319877 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f6bd9013-635a-4af8-b0ee-621a280a459a-original-pull-secret podName:f6bd9013-635a-4af8-b0ee-621a280a459a nodeName:}" failed. No retries permitted until 2026-07-26 15:38:06.319859524 +0000 UTC m=+9.059212824 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f6bd9013-635a-4af8-b0ee-621a280a459a-original-pull-secret") pod "global-pull-secret-syncer-l56b4" (UID: "f6bd9013-635a-4af8-b0ee-621a280a459a") : object "kube-system"/"original-pull-secret" not registered Jul 26 15:38:02.320284 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:02.320264 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 26 15:38:02.320284 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:02.320283 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 26 15:38:02.320358 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:02.320295 2577 projected.go:196] Error preparing data for projected volume kube-api-access-fztmw for pod openshift-network-diagnostics/network-check-target-d2nnz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 26 15:38:02.320358 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:02.320347 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f3e728c9-4710-4f6c-977f-caf52a2bfcaf-kube-api-access-fztmw podName:f3e728c9-4710-4f6c-977f-caf52a2bfcaf nodeName:}" failed. No retries permitted until 2026-07-26 15:38:06.320332632 +0000 UTC m=+9.059685923 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-fztmw" (UniqueName: "kubernetes.io/projected/f3e728c9-4710-4f6c-977f-caf52a2bfcaf-kube-api-access-fztmw") pod "network-check-target-d2nnz" (UID: "f3e728c9-4710-4f6c-977f-caf52a2bfcaf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 26 15:38:02.420053 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:02.420020 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5f442b1-a918-43a5-9870-f2183fe5085a-metrics-certs\") pod \"network-metrics-daemon-kgk6w\" (UID: \"b5f442b1-a918-43a5-9870-f2183fe5085a\") " pod="openshift-multus/network-metrics-daemon-kgk6w" Jul 26 15:38:02.420224 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:02.420173 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 26 15:38:02.420224 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:02.420228 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5f442b1-a918-43a5-9870-f2183fe5085a-metrics-certs podName:b5f442b1-a918-43a5-9870-f2183fe5085a nodeName:}" failed. No retries permitted until 2026-07-26 15:38:06.42021127 +0000 UTC m=+9.159564562 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b5f442b1-a918-43a5-9870-f2183fe5085a-metrics-certs") pod "network-metrics-daemon-kgk6w" (UID: "b5f442b1-a918-43a5-9870-f2183fe5085a") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 26 15:38:02.781246 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:02.781213 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l56b4" Jul 26 15:38:02.781246 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:02.781237 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kgk6w" Jul 26 15:38:02.781510 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:02.781237 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d2nnz" Jul 26 15:38:02.781510 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:02.781344 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l56b4" podUID="f6bd9013-635a-4af8-b0ee-621a280a459a" Jul 26 15:38:02.781510 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:02.781494 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kgk6w" podUID="b5f442b1-a918-43a5-9870-f2183fe5085a" Jul 26 15:38:02.781690 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:02.781550 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d2nnz" podUID="f3e728c9-4710-4f6c-977f-caf52a2bfcaf" Jul 26 15:38:04.781351 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:04.781293 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kgk6w" Jul 26 15:38:04.781966 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:04.781426 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kgk6w" podUID="b5f442b1-a918-43a5-9870-f2183fe5085a" Jul 26 15:38:04.781966 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:04.781698 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d2nnz" Jul 26 15:38:04.781966 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:04.781725 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l56b4" Jul 26 15:38:04.781966 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:04.781807 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d2nnz" podUID="f3e728c9-4710-4f6c-977f-caf52a2bfcaf" Jul 26 15:38:04.781966 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:04.781868 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l56b4" podUID="f6bd9013-635a-4af8-b0ee-621a280a459a" Jul 26 15:38:06.356359 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:06.356273 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fztmw\" (UniqueName: \"kubernetes.io/projected/f3e728c9-4710-4f6c-977f-caf52a2bfcaf-kube-api-access-fztmw\") pod \"network-check-target-d2nnz\" (UID: \"f3e728c9-4710-4f6c-977f-caf52a2bfcaf\") " pod="openshift-network-diagnostics/network-check-target-d2nnz" Jul 26 15:38:06.356801 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:06.356441 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f6bd9013-635a-4af8-b0ee-621a280a459a-original-pull-secret\") pod \"global-pull-secret-syncer-l56b4\" (UID: \"f6bd9013-635a-4af8-b0ee-621a280a459a\") " pod="kube-system/global-pull-secret-syncer-l56b4" Jul 26 15:38:06.356801 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:06.356492 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 26 15:38:06.356801 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:06.356521 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 26 15:38:06.356801 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:06.356535 2577 projected.go:196] Error preparing data for projected volume kube-api-access-fztmw for pod openshift-network-diagnostics/network-check-target-d2nnz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 26 15:38:06.356801 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:06.356598 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f3e728c9-4710-4f6c-977f-caf52a2bfcaf-kube-api-access-fztmw podName:f3e728c9-4710-4f6c-977f-caf52a2bfcaf nodeName:}" failed. No retries permitted until 2026-07-26 15:38:14.356578113 +0000 UTC m=+17.095931409 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-fztmw" (UniqueName: "kubernetes.io/projected/f3e728c9-4710-4f6c-977f-caf52a2bfcaf-kube-api-access-fztmw") pod "network-check-target-d2nnz" (UID: "f3e728c9-4710-4f6c-977f-caf52a2bfcaf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 26 15:38:06.356801 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:06.356634 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 26 15:38:06.356801 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:06.356703 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f6bd9013-635a-4af8-b0ee-621a280a459a-original-pull-secret podName:f6bd9013-635a-4af8-b0ee-621a280a459a nodeName:}" failed. No retries permitted until 2026-07-26 15:38:14.356685078 +0000 UTC m=+17.096038369 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f6bd9013-635a-4af8-b0ee-621a280a459a-original-pull-secret") pod "global-pull-secret-syncer-l56b4" (UID: "f6bd9013-635a-4af8-b0ee-621a280a459a") : object "kube-system"/"original-pull-secret" not registered Jul 26 15:38:06.457820 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:06.457783 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5f442b1-a918-43a5-9870-f2183fe5085a-metrics-certs\") pod \"network-metrics-daemon-kgk6w\" (UID: \"b5f442b1-a918-43a5-9870-f2183fe5085a\") " pod="openshift-multus/network-metrics-daemon-kgk6w" Jul 26 15:38:06.457988 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:06.457959 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 26 15:38:06.458064 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:06.458037 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5f442b1-a918-43a5-9870-f2183fe5085a-metrics-certs podName:b5f442b1-a918-43a5-9870-f2183fe5085a nodeName:}" failed. No retries permitted until 2026-07-26 15:38:14.458017102 +0000 UTC m=+17.197370404 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b5f442b1-a918-43a5-9870-f2183fe5085a-metrics-certs") pod "network-metrics-daemon-kgk6w" (UID: "b5f442b1-a918-43a5-9870-f2183fe5085a") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 26 15:38:06.780903 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:06.780866 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kgk6w" Jul 26 15:38:06.781099 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:06.780910 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d2nnz" Jul 26 15:38:06.781099 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:06.780866 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l56b4" Jul 26 15:38:06.781099 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:06.781010 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kgk6w" podUID="b5f442b1-a918-43a5-9870-f2183fe5085a" Jul 26 15:38:06.781099 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:06.781064 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l56b4" podUID="f6bd9013-635a-4af8-b0ee-621a280a459a" Jul 26 15:38:06.781381 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:06.781121 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d2nnz" podUID="f3e728c9-4710-4f6c-977f-caf52a2bfcaf" Jul 26 15:38:08.780832 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:08.780790 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kgk6w" Jul 26 15:38:08.781303 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:08.780926 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l56b4" Jul 26 15:38:08.781303 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:08.780937 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kgk6w" podUID="b5f442b1-a918-43a5-9870-f2183fe5085a" Jul 26 15:38:08.781303 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:08.781074 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d2nnz" Jul 26 15:38:08.781303 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:08.781071 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l56b4" podUID="f6bd9013-635a-4af8-b0ee-621a280a459a" Jul 26 15:38:08.781303 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:08.781157 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d2nnz" podUID="f3e728c9-4710-4f6c-977f-caf52a2bfcaf" Jul 26 15:38:10.780964 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:10.780930 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kgk6w" Jul 26 15:38:10.781389 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:10.780932 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d2nnz" Jul 26 15:38:10.781389 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:10.781055 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kgk6w" podUID="b5f442b1-a918-43a5-9870-f2183fe5085a" Jul 26 15:38:10.781389 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:10.781076 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l56b4" Jul 26 15:38:10.781389 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:10.781147 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d2nnz" podUID="f3e728c9-4710-4f6c-977f-caf52a2bfcaf" Jul 26 15:38:10.781389 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:10.781222 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l56b4" podUID="f6bd9013-635a-4af8-b0ee-621a280a459a" Jul 26 15:38:12.781747 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:12.781714 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l56b4" Jul 26 15:38:12.781747 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:12.781735 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d2nnz" Jul 26 15:38:12.782256 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:12.781837 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l56b4" podUID="f6bd9013-635a-4af8-b0ee-621a280a459a" Jul 26 15:38:12.782256 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:12.781890 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kgk6w" Jul 26 15:38:12.782256 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:12.781974 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d2nnz" podUID="f3e728c9-4710-4f6c-977f-caf52a2bfcaf" Jul 26 15:38:12.782256 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:12.782058 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kgk6w" podUID="b5f442b1-a918-43a5-9870-f2183fe5085a" Jul 26 15:38:14.416664 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:14.416569 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fztmw\" (UniqueName: \"kubernetes.io/projected/f3e728c9-4710-4f6c-977f-caf52a2bfcaf-kube-api-access-fztmw\") pod \"network-check-target-d2nnz\" (UID: \"f3e728c9-4710-4f6c-977f-caf52a2bfcaf\") " pod="openshift-network-diagnostics/network-check-target-d2nnz" Jul 26 15:38:14.416664 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:14.416647 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f6bd9013-635a-4af8-b0ee-621a280a459a-original-pull-secret\") pod \"global-pull-secret-syncer-l56b4\" (UID: \"f6bd9013-635a-4af8-b0ee-621a280a459a\") " pod="kube-system/global-pull-secret-syncer-l56b4" Jul 26 15:38:14.417199 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:14.416748 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 26 15:38:14.417199 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:14.416775 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 26 15:38:14.417199 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:14.416786 2577 projected.go:196] Error preparing data for projected volume kube-api-access-fztmw for pod openshift-network-diagnostics/network-check-target-d2nnz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 26 15:38:14.417199 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:14.416755 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 26 15:38:14.417199 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:14.416835 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f3e728c9-4710-4f6c-977f-caf52a2bfcaf-kube-api-access-fztmw podName:f3e728c9-4710-4f6c-977f-caf52a2bfcaf nodeName:}" failed. No retries permitted until 2026-07-26 15:38:30.416816965 +0000 UTC m=+33.156170267 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-fztmw" (UniqueName: "kubernetes.io/projected/f3e728c9-4710-4f6c-977f-caf52a2bfcaf-kube-api-access-fztmw") pod "network-check-target-d2nnz" (UID: "f3e728c9-4710-4f6c-977f-caf52a2bfcaf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 26 15:38:14.417199 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:14.416876 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f6bd9013-635a-4af8-b0ee-621a280a459a-original-pull-secret podName:f6bd9013-635a-4af8-b0ee-621a280a459a nodeName:}" failed. No retries permitted until 2026-07-26 15:38:30.416862888 +0000 UTC m=+33.156216178 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f6bd9013-635a-4af8-b0ee-621a280a459a-original-pull-secret") pod "global-pull-secret-syncer-l56b4" (UID: "f6bd9013-635a-4af8-b0ee-621a280a459a") : object "kube-system"/"original-pull-secret" not registered Jul 26 15:38:14.517244 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:14.517202 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5f442b1-a918-43a5-9870-f2183fe5085a-metrics-certs\") pod \"network-metrics-daemon-kgk6w\" (UID: \"b5f442b1-a918-43a5-9870-f2183fe5085a\") " pod="openshift-multus/network-metrics-daemon-kgk6w" Jul 26 15:38:14.517449 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:14.517373 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 26 15:38:14.517526 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:14.517467 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5f442b1-a918-43a5-9870-f2183fe5085a-metrics-certs podName:b5f442b1-a918-43a5-9870-f2183fe5085a nodeName:}" failed. No retries permitted until 2026-07-26 15:38:30.517446491 +0000 UTC m=+33.256799779 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b5f442b1-a918-43a5-9870-f2183fe5085a-metrics-certs") pod "network-metrics-daemon-kgk6w" (UID: "b5f442b1-a918-43a5-9870-f2183fe5085a") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 26 15:38:14.781448 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:14.781350 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d2nnz" Jul 26 15:38:14.781611 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:14.781350 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l56b4" Jul 26 15:38:14.781611 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:14.781492 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d2nnz" podUID="f3e728c9-4710-4f6c-977f-caf52a2bfcaf" Jul 26 15:38:14.781611 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:14.781350 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kgk6w" Jul 26 15:38:14.781611 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:14.781590 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l56b4" podUID="f6bd9013-635a-4af8-b0ee-621a280a459a" Jul 26 15:38:14.781782 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:14.781661 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kgk6w" podUID="b5f442b1-a918-43a5-9870-f2183fe5085a" Jul 26 15:38:16.781480 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:16.781445 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kgk6w" Jul 26 15:38:16.781947 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:16.781445 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d2nnz" Jul 26 15:38:16.781947 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:16.781587 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kgk6w" podUID="b5f442b1-a918-43a5-9870-f2183fe5085a" Jul 26 15:38:16.781947 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:16.781445 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l56b4" Jul 26 15:38:16.781947 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:16.781649 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d2nnz" podUID="f3e728c9-4710-4f6c-977f-caf52a2bfcaf" Jul 26 15:38:16.781947 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:16.781724 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l56b4" podUID="f6bd9013-635a-4af8-b0ee-621a280a459a" Jul 26 15:38:18.781321 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:18.781027 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d2nnz" Jul 26 15:38:18.781824 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:18.781022 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l56b4" Jul 26 15:38:18.781824 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:18.781423 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d2nnz" podUID="f3e728c9-4710-4f6c-977f-caf52a2bfcaf" Jul 26 15:38:18.781824 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:18.781065 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kgk6w" Jul 26 15:38:18.781824 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:18.781523 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l56b4" podUID="f6bd9013-635a-4af8-b0ee-621a280a459a" Jul 26 15:38:18.781824 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:18.781601 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kgk6w" podUID="b5f442b1-a918-43a5-9870-f2183fe5085a" Jul 26 15:38:18.965412 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:18.965186 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-x4dkt" event={"ID":"bd02936a-a782-4e31-89bb-1292065d0abc","Type":"ContainerStarted","Data":"f5aefadacb05eeb2dc95ad6d520eec96c7e75e24aa27893ab17b92452c074f04"} Jul 26 15:38:18.966655 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:18.966630 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-65dgw" event={"ID":"7c179a4b-e953-4d22-9f2d-7cf682e833fd","Type":"ContainerStarted","Data":"922b86595738498a5895d5505f42bb90d21e1ade89f99c8b9d19b97dd725fc85"} Jul 26 15:38:18.970809 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:18.970779 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-163.ec2.internal" event={"ID":"0d03d973abb0a97c200617d39e2704ee","Type":"ContainerStarted","Data":"fc4cb4d6f149b4ab678596e06640095e961eeb97e16ede0b9ddb96df071e411a"} Jul 26 15:38:18.972335 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:18.972319 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ztlp6_cbe2cd41-6ebc-4722-a813-57d83fa1d242/ovn-acl-logging/0.log" Jul 26 15:38:18.972676 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:18.972656 2577 generic.go:356] "Generic (PLEG): container finished" podID="cbe2cd41-6ebc-4722-a813-57d83fa1d242" containerID="042ad2cd438a3305f7d5c6b2d633ef7616adeb66a847b61c59d025dad5a4f4ff" exitCode=1 Jul 26 15:38:18.972751 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:18.972686 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ztlp6" event={"ID":"cbe2cd41-6ebc-4722-a813-57d83fa1d242","Type":"ContainerDied","Data":"042ad2cd438a3305f7d5c6b2d633ef7616adeb66a847b61c59d025dad5a4f4ff"} Jul 26 15:38:18.972751 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:18.972734 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ztlp6" event={"ID":"cbe2cd41-6ebc-4722-a813-57d83fa1d242","Type":"ContainerStarted","Data":"6d8f76ebf8b1378b0db78c876f570b195ca9e37aba9908a0ff92aa4a4daf18ad"} Jul 26 15:38:18.982284 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:18.982223 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-x4dkt" podStartSLOduration=2.927777777 podStartE2EDuration="21.982203969s" totalImagesPullingTime="19.054426192s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-26 15:37:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:38:18.981662556 +0000 UTC m=+21.721015893" watchObservedRunningTime="2026-07-26 15:38:18.982203969 +0000 UTC m=+21.721557280" Jul 26 15:38:18.995958 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:18.995834 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-163.ec2.internal" podStartSLOduration=20.995813629 podStartE2EDuration="20.995813629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-26 15:37:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:38:18.995354459 +0000 UTC m=+21.734707771" watchObservedRunningTime="2026-07-26 15:38:18.995813629 +0000 UTC m=+21.735166942" Jul 26 15:38:19.010106 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:19.009859 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-65dgw" podStartSLOduration=2.973016179 podStartE2EDuration="22.009839615s" totalImagesPullingTime="19.036823436s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-26 15:37:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:38:19.009761212 +0000 UTC m=+21.749114546" watchObservedRunningTime="2026-07-26 15:38:19.009839615 +0000 UTC m=+21.749192926" Jul 26 15:38:19.976803 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:19.976637 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ztlp6_cbe2cd41-6ebc-4722-a813-57d83fa1d242/ovn-acl-logging/0.log" Jul 26 15:38:19.977466 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:19.977156 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ztlp6" event={"ID":"cbe2cd41-6ebc-4722-a813-57d83fa1d242","Type":"ContainerStarted","Data":"01f92ebe3c97f341c43ed159ef9646cc8c22152336bcf4c2e1cf8cf68433cae5"} Jul 26 15:38:19.977466 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:19.977232 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ztlp6" event={"ID":"cbe2cd41-6ebc-4722-a813-57d83fa1d242","Type":"ContainerStarted","Data":"f167bcdcf74b12bec62d080301d374b3ae76b6849c7639a01c33f3cd38e6a1f5"} Jul 26 15:38:19.977466 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:19.977276 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ztlp6" event={"ID":"cbe2cd41-6ebc-4722-a813-57d83fa1d242","Type":"ContainerStarted","Data":"28430ece301aca473ca12fa7c29216819406b2e870af17803870ec4577f1dde1"} Jul 26 15:38:19.977466 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:19.977313 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ztlp6" event={"ID":"cbe2cd41-6ebc-4722-a813-57d83fa1d242","Type":"ContainerStarted","Data":"caeee26b660c6d6f2f6cea0b4f0b2c5222e14d5ad66f7ff6f10e887d8703ecf3"} Jul 26 15:38:19.978459 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:19.978429 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-mfwqb" event={"ID":"098dd2e5-9540-425f-ba4e-71ea0b064be7","Type":"ContainerStarted","Data":"d8e080891dc33c65b8909eacdd264c150402658a9459ebc73e5b6fc21961244b"} Jul 26 15:38:19.979705 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:19.979686 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-v46ns" event={"ID":"acf5fc6d-b7e4-4581-8878-f5776e505746","Type":"ContainerStarted","Data":"c1b27aac93ab11b1fa1dcf2ea3de4729312a1a8976b7d627943b1b764fa6aa39"} Jul 26 15:38:19.980948 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:19.980929 2577 generic.go:356] "Generic (PLEG): container finished" podID="34c5f0125abc8dd4d5a5ee6814e83607" containerID="9091636736e3b1f758f8521cdc9cfb82b3bbd3933ce18fb268259a5e16933c32" exitCode=0 Jul 26 15:38:19.981022 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:19.980992 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-163.ec2.internal" event={"ID":"34c5f0125abc8dd4d5a5ee6814e83607","Type":"ContainerDied","Data":"9091636736e3b1f758f8521cdc9cfb82b3bbd3933ce18fb268259a5e16933c32"} Jul 26 15:38:19.982445 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:19.982423 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qnht4" event={"ID":"350d34fc-ceeb-477e-8486-0a712e354c67","Type":"ContainerStarted","Data":"fd8b49c4b367d849cb88b67139f5b7cf2d7365184c185dacd9e0f25ece9782ca"} Jul 26 15:38:19.983855 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:19.983834 2577 generic.go:356] "Generic (PLEG): container finished" podID="c2653055-9d7a-4aeb-9090-fb495a2b650b" containerID="2c6ba20a02fb420f6cd930270394f63af3cc9f021ecb8fc1f8fe2253f706d409" exitCode=0 Jul 26 15:38:19.983939 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:19.983910 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dzc7t" event={"ID":"c2653055-9d7a-4aeb-9090-fb495a2b650b","Type":"ContainerDied","Data":"2c6ba20a02fb420f6cd930270394f63af3cc9f021ecb8fc1f8fe2253f706d409"} Jul 26 15:38:19.985288 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:19.985271 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-v4mgw" event={"ID":"356a056a-d4d5-48ed-a10f-928b36becc81","Type":"ContainerStarted","Data":"62cfc56d4b247f068eed7396b3db3739accff06d05cae59b7e33178c4efab9e9"} Jul 26 15:38:19.986630 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:19.986608 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-jbpx6" event={"ID":"806292e6-a776-4195-a595-263edb35e68b","Type":"ContainerStarted","Data":"e0e5cec7f282d8ce85acc3316631d8d04630d2e867d7e70c30332ca0e2c1c9a7"} Jul 26 15:38:19.991666 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:19.991631 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-mfwqb" podStartSLOduration=3.031663225 podStartE2EDuration="21.991616207s" totalImagesPullingTime="18.959952982s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-26 15:37:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:38:19.991237085 +0000 UTC m=+22.730590392" watchObservedRunningTime="2026-07-26 15:38:19.991616207 +0000 UTC m=+22.730969518" Jul 26 15:38:20.021951 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:20.021909 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-v46ns" podStartSLOduration=4.028619223 podStartE2EDuration="23.02189475s" totalImagesPullingTime="18.993275527s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-26 15:37:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:38:20.02157125 +0000 UTC m=+22.760924593" watchObservedRunningTime="2026-07-26 15:38:20.02189475 +0000 UTC m=+22.761248060" Jul 26 15:38:20.049731 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:20.049690 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-jbpx6" podStartSLOduration=3.076799864 podStartE2EDuration="22.049675943s" totalImagesPullingTime="18.972876079s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:37:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:38:20.035325686 +0000 UTC m=+22.774679027" watchObservedRunningTime="2026-07-26 15:38:20.049675943 +0000 UTC m=+22.789029295" Jul 26 15:38:20.049851 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:20.049834 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-v4mgw" podStartSLOduration=3.083107814 podStartE2EDuration="22.049828834s" totalImagesPullingTime="18.96672102s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-26 15:37:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:38:20.049371833 +0000 UTC m=+22.788725155" watchObservedRunningTime="2026-07-26 15:38:20.049828834 +0000 UTC m=+22.789182144" Jul 26 15:38:20.285671 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:20.285617 2577 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 26 15:38:20.522486 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:20.522409 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-jbpx6" Jul 26 15:38:20.523013 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:20.522989 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-jbpx6" Jul 26 15:38:20.759300 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:20.759189 2577 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-26T15:38:20.285633844Z","UUID":"68cc7837-9ffc-469d-8395-0d340193a45c","Handler":null,"Name":"","Endpoint":""} Jul 26 15:38:20.761532 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:20.761510 2577 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 26 15:38:20.761681 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:20.761556 2577 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 26 15:38:20.781362 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:20.781284 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l56b4" Jul 26 15:38:20.781362 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:20.781310 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kgk6w" Jul 26 15:38:20.781652 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:20.781284 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d2nnz" Jul 26 15:38:20.781652 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:20.781422 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l56b4" podUID="f6bd9013-635a-4af8-b0ee-621a280a459a" Jul 26 15:38:20.781652 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:20.781515 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d2nnz" podUID="f3e728c9-4710-4f6c-977f-caf52a2bfcaf" Jul 26 15:38:20.781652 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:20.781600 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kgk6w" podUID="b5f442b1-a918-43a5-9870-f2183fe5085a" Jul 26 15:38:20.991505 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:20.991465 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-163.ec2.internal" event={"ID":"34c5f0125abc8dd4d5a5ee6814e83607","Type":"ContainerStarted","Data":"ae06ec73d3cf0ea64f7267a3538a6361793f2ede8fda040f57a6158e3e3b1a8e"} Jul 26 15:38:20.994217 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:20.994186 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qnht4" event={"ID":"350d34fc-ceeb-477e-8486-0a712e354c67","Type":"ContainerStarted","Data":"100778b2c63e1d8a4a7e3a991ad9819d576b41cffe820cf49d9474b1867b5c30"} Jul 26 15:38:20.994356 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:20.994244 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-jbpx6" Jul 26 15:38:20.994670 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:20.994651 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-jbpx6" Jul 26 15:38:21.006815 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:21.006774 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-163.ec2.internal" podStartSLOduration=23.006756788 podStartE2EDuration="23.006756788s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-26 15:37:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:38:21.006019526 +0000 UTC m=+23.745372837" watchObservedRunningTime="2026-07-26 15:38:21.006756788 +0000 UTC m=+23.746110099" Jul 26 15:38:21.997835 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:21.997583 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qnht4" event={"ID":"350d34fc-ceeb-477e-8486-0a712e354c67","Type":"ContainerStarted","Data":"e008fd446ee8e7fe4e22b03078ef952105b3fed17158e05a701aaac244a0fbae"} Jul 26 15:38:22.001687 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:22.001661 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ztlp6_cbe2cd41-6ebc-4722-a813-57d83fa1d242/ovn-acl-logging/0.log" Jul 26 15:38:22.002139 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:22.002112 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ztlp6" event={"ID":"cbe2cd41-6ebc-4722-a813-57d83fa1d242","Type":"ContainerStarted","Data":"bd1f2a7766c80b5c14895dcbb11a95c02be97565e5726e7c60915f4cd9092e41"} Jul 26 15:38:22.014569 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:22.014520 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qnht4" podStartSLOduration=2.71414949 podStartE2EDuration="24.014504447s" totalImagesPullingTime="21.300354957s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-26 15:37:58 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:38:22.014061671 +0000 UTC m=+24.753415000" watchObservedRunningTime="2026-07-26 15:38:22.014504447 +0000 UTC m=+24.753857756" Jul 26 15:38:22.781344 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:22.781307 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l56b4" Jul 26 15:38:22.781525 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:22.781307 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d2nnz" Jul 26 15:38:22.781525 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:22.781450 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l56b4" podUID="f6bd9013-635a-4af8-b0ee-621a280a459a" Jul 26 15:38:22.781525 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:22.781499 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kgk6w" Jul 26 15:38:22.781660 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:22.781592 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d2nnz" podUID="f3e728c9-4710-4f6c-977f-caf52a2bfcaf" Jul 26 15:38:22.781696 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:22.781683 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kgk6w" podUID="b5f442b1-a918-43a5-9870-f2183fe5085a" Jul 26 15:38:24.781038 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:24.781010 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d2nnz" Jul 26 15:38:24.781759 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:24.781010 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l56b4" Jul 26 15:38:24.781759 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:24.781120 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d2nnz" podUID="f3e728c9-4710-4f6c-977f-caf52a2bfcaf" Jul 26 15:38:24.781759 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:24.781010 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kgk6w" Jul 26 15:38:24.781759 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:24.781194 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l56b4" podUID="f6bd9013-635a-4af8-b0ee-621a280a459a" Jul 26 15:38:24.781759 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:24.781268 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kgk6w" podUID="b5f442b1-a918-43a5-9870-f2183fe5085a" Jul 26 15:38:25.011687 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:25.011529 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ztlp6_cbe2cd41-6ebc-4722-a813-57d83fa1d242/ovn-acl-logging/0.log" Jul 26 15:38:25.012014 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:25.011984 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ztlp6" event={"ID":"cbe2cd41-6ebc-4722-a813-57d83fa1d242","Type":"ContainerStarted","Data":"00f5be4eb0ab9bd0f440e8390a0f3255da07164c2d2b124192967b91c83b95ec"} Jul 26 15:38:25.012304 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:25.012285 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ztlp6" Jul 26 15:38:25.012411 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:25.012357 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ztlp6" Jul 26 15:38:25.012468 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:25.012450 2577 scope.go:117] "RemoveContainer" containerID="042ad2cd438a3305f7d5c6b2d633ef7616adeb66a847b61c59d025dad5a4f4ff" Jul 26 15:38:25.014494 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:25.014469 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dzc7t" event={"ID":"c2653055-9d7a-4aeb-9090-fb495a2b650b","Type":"ContainerStarted","Data":"abee735462eeaa566e9df6c4a2e04d7fa2639091e86038d762a996cbf6f721b0"} Jul 26 15:38:25.041856 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:25.041828 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ztlp6" Jul 26 15:38:26.021332 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:26.021254 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ztlp6_cbe2cd41-6ebc-4722-a813-57d83fa1d242/ovn-acl-logging/0.log" Jul 26 15:38:26.022369 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:26.022340 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ztlp6" event={"ID":"cbe2cd41-6ebc-4722-a813-57d83fa1d242","Type":"ContainerStarted","Data":"bfee75fdbe64aab52ddfbce7536b21a02162dc18f364bd93e708cdef757fbf6d"} Jul 26 15:38:26.023719 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:26.023702 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ztlp6" Jul 26 15:38:26.027092 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:26.026510 2577 generic.go:356] "Generic (PLEG): container finished" podID="c2653055-9d7a-4aeb-9090-fb495a2b650b" containerID="abee735462eeaa566e9df6c4a2e04d7fa2639091e86038d762a996cbf6f721b0" exitCode=0 Jul 26 15:38:26.027092 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:26.026545 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dzc7t" event={"ID":"c2653055-9d7a-4aeb-9090-fb495a2b650b","Type":"ContainerDied","Data":"abee735462eeaa566e9df6c4a2e04d7fa2639091e86038d762a996cbf6f721b0"} Jul 26 15:38:26.045227 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:26.045201 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ztlp6" Jul 26 15:38:26.048701 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:26.048658 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-ztlp6" podStartSLOduration=8.442068444 podStartE2EDuration="28.048644069s" totalImagesPullingTime="19.606575625s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-26 15:37:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:38:26.047501857 +0000 UTC m=+28.786855186" watchObservedRunningTime="2026-07-26 15:38:26.048644069 +0000 UTC m=+28.787997381" Jul 26 15:38:26.781433 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:26.781381 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l56b4" Jul 26 15:38:26.781631 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:26.781436 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kgk6w" Jul 26 15:38:26.781631 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:26.781436 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d2nnz" Jul 26 15:38:26.781631 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:26.781526 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l56b4" podUID="f6bd9013-635a-4af8-b0ee-621a280a459a" Jul 26 15:38:26.781827 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:26.781648 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kgk6w" podUID="b5f442b1-a918-43a5-9870-f2183fe5085a" Jul 26 15:38:26.781827 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:26.781777 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d2nnz" podUID="f3e728c9-4710-4f6c-977f-caf52a2bfcaf" Jul 26 15:38:26.949976 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:26.949950 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-d2nnz"] Jul 26 15:38:26.953185 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:26.953141 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-l56b4"] Jul 26 15:38:26.954024 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:26.953937 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-kgk6w"] Jul 26 15:38:27.030814 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:27.030788 2577 generic.go:356] "Generic (PLEG): container finished" podID="c2653055-9d7a-4aeb-9090-fb495a2b650b" containerID="f9441daa420632d610e74bd7b82d390ee62d8840d6e94ea84cccf6093fd32c8f" exitCode=0 Jul 26 15:38:27.031193 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:27.030900 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dzc7t" event={"ID":"c2653055-9d7a-4aeb-9090-fb495a2b650b","Type":"ContainerDied","Data":"f9441daa420632d610e74bd7b82d390ee62d8840d6e94ea84cccf6093fd32c8f"} Jul 26 15:38:27.031193 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:27.030980 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l56b4" Jul 26 15:38:27.031193 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:27.031100 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l56b4" podUID="f6bd9013-635a-4af8-b0ee-621a280a459a" Jul 26 15:38:27.031526 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:27.031434 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kgk6w" Jul 26 15:38:27.031526 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:27.031511 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kgk6w" podUID="b5f442b1-a918-43a5-9870-f2183fe5085a" Jul 26 15:38:27.032041 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:27.031541 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d2nnz" Jul 26 15:38:27.032041 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:27.031636 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d2nnz" podUID="f3e728c9-4710-4f6c-977f-caf52a2bfcaf" Jul 26 15:38:28.035816 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:28.035781 2577 generic.go:356] "Generic (PLEG): container finished" podID="c2653055-9d7a-4aeb-9090-fb495a2b650b" containerID="1a78656330fe4c7a86fd15ce2712f2044f6e1feca63eb18b6edeb008d043e3e8" exitCode=0 Jul 26 15:38:28.036238 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:28.035864 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dzc7t" event={"ID":"c2653055-9d7a-4aeb-9090-fb495a2b650b","Type":"ContainerDied","Data":"1a78656330fe4c7a86fd15ce2712f2044f6e1feca63eb18b6edeb008d043e3e8"} Jul 26 15:38:28.781842 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:28.781809 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l56b4" Jul 26 15:38:28.781984 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:28.781944 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l56b4" podUID="f6bd9013-635a-4af8-b0ee-621a280a459a" Jul 26 15:38:28.781984 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:28.781813 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kgk6w" Jul 26 15:38:28.782122 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:28.781813 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d2nnz" Jul 26 15:38:28.782176 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:28.782109 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kgk6w" podUID="b5f442b1-a918-43a5-9870-f2183fe5085a" Jul 26 15:38:28.782223 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:28.782188 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d2nnz" podUID="f3e728c9-4710-4f6c-977f-caf52a2bfcaf" Jul 26 15:38:30.429237 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:30.429153 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fztmw\" (UniqueName: \"kubernetes.io/projected/f3e728c9-4710-4f6c-977f-caf52a2bfcaf-kube-api-access-fztmw\") pod \"network-check-target-d2nnz\" (UID: \"f3e728c9-4710-4f6c-977f-caf52a2bfcaf\") " pod="openshift-network-diagnostics/network-check-target-d2nnz" Jul 26 15:38:30.429237 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:30.429209 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f6bd9013-635a-4af8-b0ee-621a280a459a-original-pull-secret\") pod \"global-pull-secret-syncer-l56b4\" (UID: \"f6bd9013-635a-4af8-b0ee-621a280a459a\") " pod="kube-system/global-pull-secret-syncer-l56b4" Jul 26 15:38:30.429720 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:30.429316 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 26 15:38:30.429720 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:30.429346 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 26 15:38:30.429720 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:30.429376 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 26 15:38:30.429720 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:30.429392 2577 projected.go:196] Error preparing data for projected volume kube-api-access-fztmw for pod openshift-network-diagnostics/network-check-target-d2nnz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 26 15:38:30.429720 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:30.429428 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f6bd9013-635a-4af8-b0ee-621a280a459a-original-pull-secret podName:f6bd9013-635a-4af8-b0ee-621a280a459a nodeName:}" failed. No retries permitted until 2026-07-26 15:39:02.429360284 +0000 UTC m=+65.168713572 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f6bd9013-635a-4af8-b0ee-621a280a459a-original-pull-secret") pod "global-pull-secret-syncer-l56b4" (UID: "f6bd9013-635a-4af8-b0ee-621a280a459a") : object "kube-system"/"original-pull-secret" not registered Jul 26 15:38:30.429720 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:30.429458 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f3e728c9-4710-4f6c-977f-caf52a2bfcaf-kube-api-access-fztmw podName:f3e728c9-4710-4f6c-977f-caf52a2bfcaf nodeName:}" failed. No retries permitted until 2026-07-26 15:39:02.429442012 +0000 UTC m=+65.168795314 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-fztmw" (UniqueName: "kubernetes.io/projected/f3e728c9-4710-4f6c-977f-caf52a2bfcaf-kube-api-access-fztmw") pod "network-check-target-d2nnz" (UID: "f3e728c9-4710-4f6c-977f-caf52a2bfcaf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 26 15:38:30.530065 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:30.530029 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5f442b1-a918-43a5-9870-f2183fe5085a-metrics-certs\") pod \"network-metrics-daemon-kgk6w\" (UID: \"b5f442b1-a918-43a5-9870-f2183fe5085a\") " pod="openshift-multus/network-metrics-daemon-kgk6w" Jul 26 15:38:30.530215 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:30.530164 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 26 15:38:30.530253 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:30.530222 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5f442b1-a918-43a5-9870-f2183fe5085a-metrics-certs podName:b5f442b1-a918-43a5-9870-f2183fe5085a nodeName:}" failed. No retries permitted until 2026-07-26 15:39:02.530209291 +0000 UTC m=+65.269562584 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b5f442b1-a918-43a5-9870-f2183fe5085a-metrics-certs") pod "network-metrics-daemon-kgk6w" (UID: "b5f442b1-a918-43a5-9870-f2183fe5085a") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 26 15:38:30.781767 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:30.781739 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kgk6w" Jul 26 15:38:30.781767 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:30.781755 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d2nnz" Jul 26 15:38:30.782003 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:30.781734 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l56b4" Jul 26 15:38:30.782003 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:30.781863 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kgk6w" podUID="b5f442b1-a918-43a5-9870-f2183fe5085a" Jul 26 15:38:30.782003 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:30.781936 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l56b4" podUID="f6bd9013-635a-4af8-b0ee-621a280a459a" Jul 26 15:38:30.782121 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:30.782027 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d2nnz" podUID="f3e728c9-4710-4f6c-977f-caf52a2bfcaf" Jul 26 15:38:32.094909 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.094831 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-163.ec2.internal" event="NodeReady" Jul 26 15:38:32.095364 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.094995 2577 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 26 15:38:32.126062 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.126020 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-6s9pz"] Jul 26 15:38:32.153446 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.153415 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-2q2tq"] Jul 26 15:38:32.153654 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.153497 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-6s9pz" Jul 26 15:38:32.155869 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.155844 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 26 15:38:32.155986 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.155896 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 26 15:38:32.155986 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.155901 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-hh98k\"" Jul 26 15:38:32.169240 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.169218 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5885fdb759-7qj5q"] Jul 26 15:38:32.169384 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.169318 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-2q2tq" Jul 26 15:38:32.172279 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.172072 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 26 15:38:32.172279 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.172129 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-x7hq5\"" Jul 26 15:38:32.172469 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.172319 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 26 15:38:32.172469 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.172370 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 26 15:38:32.172793 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.172650 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 26 15:38:32.185920 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.185860 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5ccd8b79b5-rnb96"] Jul 26 15:38:32.186087 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.185939 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5885fdb759-7qj5q" Jul 26 15:38:32.189767 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.189083 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 26 15:38:32.189767 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.189151 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 26 15:38:32.189767 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.189090 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 26 15:38:32.189767 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.189379 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 26 15:38:32.191082 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.190050 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-t5mzt\"" Jul 26 15:38:32.195144 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.195010 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 26 15:38:32.209927 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.209906 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-rrqbw"] Jul 26 15:38:32.210058 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.210036 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5ccd8b79b5-rnb96" Jul 26 15:38:32.212327 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.212307 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 26 15:38:32.212453 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.212377 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 26 15:38:32.212533 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.212311 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-wk5h4\"" Jul 26 15:38:32.212782 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.212764 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 26 15:38:32.217243 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.217200 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 26 15:38:32.230021 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.229999 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-5b79d74577-h8slk"] Jul 26 15:38:32.230138 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.230128 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-rrqbw" Jul 26 15:38:32.232360 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.232337 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 26 15:38:32.232484 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.232457 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 26 15:38:32.232550 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.232493 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-fmt2r\"" Jul 26 15:38:32.232602 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.232555 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 26 15:38:32.242883 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.242850 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gz7z5\" (UniqueName: \"kubernetes.io/projected/cb333fda-51e9-4488-acb3-85b027240581-kube-api-access-gz7z5\") pod \"volume-data-source-validator-549c6d54cf-6s9pz\" (UID: \"cb333fda-51e9-4488-acb3-85b027240581\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-6s9pz" Jul 26 15:38:32.243094 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.242897 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/60d326a6-77da-4f72-965f-880f5c94f2b7-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-2q2tq\" (UID: \"60d326a6-77da-4f72-965f-880f5c94f2b7\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-2q2tq" Jul 26 15:38:32.243094 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.242963 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nfsl\" (UniqueName: \"kubernetes.io/projected/60d326a6-77da-4f72-965f-880f5c94f2b7-kube-api-access-6nfsl\") pod \"cluster-monitoring-operator-548b489cb8-2q2tq\" (UID: \"60d326a6-77da-4f72-965f-880f5c94f2b7\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-2q2tq" Jul 26 15:38:32.243094 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.243007 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/60d326a6-77da-4f72-965f-880f5c94f2b7-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-2q2tq\" (UID: \"60d326a6-77da-4f72-965f-880f5c94f2b7\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-2q2tq" Jul 26 15:38:32.255431 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.255391 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-2xmhd"] Jul 26 15:38:32.255569 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.255501 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5b79d74577-h8slk" Jul 26 15:38:32.257964 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.257917 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 26 15:38:32.258086 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.258037 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 26 15:38:32.258086 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.258038 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 26 15:38:32.258086 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.258071 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 26 15:38:32.258303 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.258284 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 26 15:38:32.258369 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.258316 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 26 15:38:32.258369 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.258319 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-lbb85\"" Jul 26 15:38:32.273411 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.273382 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-6s9pz"] Jul 26 15:38:32.273524 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.273433 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-2q2tq"] Jul 26 15:38:32.273524 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.273450 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5885fdb759-7qj5q"] Jul 26 15:38:32.273524 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.273467 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-rrqbw"] Jul 26 15:38:32.273524 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.273495 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-65h7n"] Jul 26 15:38:32.273524 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.273508 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-2xmhd" Jul 26 15:38:32.275998 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.275977 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 26 15:38:32.275998 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.275998 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 26 15:38:32.276157 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.276001 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 26 15:38:32.276157 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.276014 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-wvqlw\"" Jul 26 15:38:32.276157 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.276033 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 26 15:38:32.287473 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.287451 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-5xf7l"] Jul 26 15:38:32.287610 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.287584 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-65h7n" Jul 26 15:38:32.290054 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.290034 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 26 15:38:32.290160 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.290113 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-t6bwt\"" Jul 26 15:38:32.290342 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.290326 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 26 15:38:32.307320 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.307264 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:38:32.308237 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.308205 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-5zpk2"] Jul 26 15:38:32.308351 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.308335 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-5xf7l" Jul 26 15:38:32.310656 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.310634 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 26 15:38:32.310764 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.310668 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 26 15:38:32.310764 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.310688 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 26 15:38:32.310764 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.310670 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-b8h2r\"" Jul 26 15:38:32.332307 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.332284 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5ccd8b79b5-rnb96"] Jul 26 15:38:32.332423 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.332317 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-8wjqp"] Jul 26 15:38:32.332423 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.332414 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-5zpk2" Jul 26 15:38:32.334978 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.334957 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 26 15:38:32.335086 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.335009 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 26 15:38:32.335086 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.335009 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-55hlk\"" Jul 26 15:38:32.343914 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.343889 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c2a7cc81-246f-4d1b-985d-22c611d6b339-serving-cert\") pod \"service-ca-operator-5d6c75f84f-2xmhd\" (UID: \"c2a7cc81-246f-4d1b-985d-22c611d6b339\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-2xmhd" Jul 26 15:38:32.344023 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.343921 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c2a7cc81-246f-4d1b-985d-22c611d6b339-config\") pod \"service-ca-operator-5d6c75f84f-2xmhd\" (UID: \"c2a7cc81-246f-4d1b-985d-22c611d6b339\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-2xmhd" Jul 26 15:38:32.344023 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.343947 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bf408cf7-787a-4839-ab2f-8220f1696bd6-image-registry-private-configuration\") pod \"image-registry-5ccd8b79b5-rnb96\" (UID: \"bf408cf7-787a-4839-ab2f-8220f1696bd6\") " pod="openshift-image-registry/image-registry-5ccd8b79b5-rnb96" Jul 26 15:38:32.344109 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.344035 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbxm4\" (UniqueName: \"kubernetes.io/projected/c2a7cc81-246f-4d1b-985d-22c611d6b339-kube-api-access-wbxm4\") pod \"service-ca-operator-5d6c75f84f-2xmhd\" (UID: \"c2a7cc81-246f-4d1b-985d-22c611d6b339\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-2xmhd" Jul 26 15:38:32.344109 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.344084 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nfsl\" (UniqueName: \"kubernetes.io/projected/60d326a6-77da-4f72-965f-880f5c94f2b7-kube-api-access-6nfsl\") pod \"cluster-monitoring-operator-548b489cb8-2q2tq\" (UID: \"60d326a6-77da-4f72-965f-880f5c94f2b7\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-2q2tq" Jul 26 15:38:32.344204 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.344118 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9af43da1-c25b-4687-820f-6a95afe88786-service-ca-bundle\") pod \"router-default-5b79d74577-h8slk\" (UID: \"9af43da1-c25b-4687-820f-6a95afe88786\") " pod="openshift-ingress/router-default-5b79d74577-h8slk" Jul 26 15:38:32.344204 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.344149 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8z4js\" (UniqueName: \"kubernetes.io/projected/d48d98c5-9029-4b14-ab01-2234e4ecc1f7-kube-api-access-8z4js\") pod \"insights-operator-5885fdb759-7qj5q\" (UID: \"d48d98c5-9029-4b14-ab01-2234e4ecc1f7\") " pod="openshift-insights/insights-operator-5885fdb759-7qj5q" Jul 26 15:38:32.344204 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.344184 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/9af43da1-c25b-4687-820f-6a95afe88786-default-certificate\") pod \"router-default-5b79d74577-h8slk\" (UID: \"9af43da1-c25b-4687-820f-6a95afe88786\") " pod="openshift-ingress/router-default-5b79d74577-h8slk" Jul 26 15:38:32.344336 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.344216 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9af43da1-c25b-4687-820f-6a95afe88786-metrics-certs\") pod \"router-default-5b79d74577-h8slk\" (UID: \"9af43da1-c25b-4687-820f-6a95afe88786\") " pod="openshift-ingress/router-default-5b79d74577-h8slk" Jul 26 15:38:32.344336 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.344248 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vvnn\" (UniqueName: \"kubernetes.io/projected/b9353822-c6c4-47f1-8a41-ecdc92669f5f-kube-api-access-4vvnn\") pod \"cluster-samples-operator-6748489656-rrqbw\" (UID: \"b9353822-c6c4-47f1-8a41-ecdc92669f5f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-rrqbw" Jul 26 15:38:32.344336 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.344284 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bf408cf7-787a-4839-ab2f-8220f1696bd6-registry-certificates\") pod \"image-registry-5ccd8b79b5-rnb96\" (UID: \"bf408cf7-787a-4839-ab2f-8220f1696bd6\") " pod="openshift-image-registry/image-registry-5ccd8b79b5-rnb96" Jul 26 15:38:32.344336 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.344313 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d48d98c5-9029-4b14-ab01-2234e4ecc1f7-tmp\") pod \"insights-operator-5885fdb759-7qj5q\" (UID: \"d48d98c5-9029-4b14-ab01-2234e4ecc1f7\") " pod="openshift-insights/insights-operator-5885fdb759-7qj5q" Jul 26 15:38:32.344528 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.344340 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d48d98c5-9029-4b14-ab01-2234e4ecc1f7-serving-cert\") pod \"insights-operator-5885fdb759-7qj5q\" (UID: \"d48d98c5-9029-4b14-ab01-2234e4ecc1f7\") " pod="openshift-insights/insights-operator-5885fdb759-7qj5q" Jul 26 15:38:32.344528 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.344371 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/9af43da1-c25b-4687-820f-6a95afe88786-stats-auth\") pod \"router-default-5b79d74577-h8slk\" (UID: \"9af43da1-c25b-4687-820f-6a95afe88786\") " pod="openshift-ingress/router-default-5b79d74577-h8slk" Jul 26 15:38:32.344528 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.344445 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b9353822-c6c4-47f1-8a41-ecdc92669f5f-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-rrqbw\" (UID: \"b9353822-c6c4-47f1-8a41-ecdc92669f5f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-rrqbw" Jul 26 15:38:32.344528 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.344482 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gz7z5\" (UniqueName: \"kubernetes.io/projected/cb333fda-51e9-4488-acb3-85b027240581-kube-api-access-gz7z5\") pod \"volume-data-source-validator-549c6d54cf-6s9pz\" (UID: \"cb333fda-51e9-4488-acb3-85b027240581\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-6s9pz" Jul 26 15:38:32.344683 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.344525 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgcrz\" (UniqueName: \"kubernetes.io/projected/9af43da1-c25b-4687-820f-6a95afe88786-kube-api-access-wgcrz\") pod \"router-default-5b79d74577-h8slk\" (UID: \"9af43da1-c25b-4687-820f-6a95afe88786\") " pod="openshift-ingress/router-default-5b79d74577-h8slk" Jul 26 15:38:32.344683 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.344559 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/60d326a6-77da-4f72-965f-880f5c94f2b7-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-2q2tq\" (UID: \"60d326a6-77da-4f72-965f-880f5c94f2b7\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-2q2tq" Jul 26 15:38:32.344683 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.344588 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/d48d98c5-9029-4b14-ab01-2234e4ecc1f7-snapshots\") pod \"insights-operator-5885fdb759-7qj5q\" (UID: \"d48d98c5-9029-4b14-ab01-2234e4ecc1f7\") " pod="openshift-insights/insights-operator-5885fdb759-7qj5q" Jul 26 15:38:32.344683 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.344612 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54z42\" (UniqueName: \"kubernetes.io/projected/bf408cf7-787a-4839-ab2f-8220f1696bd6-kube-api-access-54z42\") pod \"image-registry-5ccd8b79b5-rnb96\" (UID: \"bf408cf7-787a-4839-ab2f-8220f1696bd6\") " pod="openshift-image-registry/image-registry-5ccd8b79b5-rnb96" Jul 26 15:38:32.344683 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.344648 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bf408cf7-787a-4839-ab2f-8220f1696bd6-installation-pull-secrets\") pod \"image-registry-5ccd8b79b5-rnb96\" (UID: \"bf408cf7-787a-4839-ab2f-8220f1696bd6\") " pod="openshift-image-registry/image-registry-5ccd8b79b5-rnb96" Jul 26 15:38:32.344683 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.344676 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d48d98c5-9029-4b14-ab01-2234e4ecc1f7-service-ca-bundle\") pod \"insights-operator-5885fdb759-7qj5q\" (UID: \"d48d98c5-9029-4b14-ab01-2234e4ecc1f7\") " pod="openshift-insights/insights-operator-5885fdb759-7qj5q" Jul 26 15:38:32.344979 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:32.344691 2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 26 15:38:32.344979 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.344716 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bf408cf7-787a-4839-ab2f-8220f1696bd6-trusted-ca\") pod \"image-registry-5ccd8b79b5-rnb96\" (UID: \"bf408cf7-787a-4839-ab2f-8220f1696bd6\") " pod="openshift-image-registry/image-registry-5ccd8b79b5-rnb96" Jul 26 15:38:32.344979 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.344739 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/60d326a6-77da-4f72-965f-880f5c94f2b7-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-2q2tq\" (UID: \"60d326a6-77da-4f72-965f-880f5c94f2b7\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-2q2tq" Jul 26 15:38:32.344979 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:32.344780 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60d326a6-77da-4f72-965f-880f5c94f2b7-cluster-monitoring-operator-tls podName:60d326a6-77da-4f72-965f-880f5c94f2b7 nodeName:}" failed. No retries permitted until 2026-07-26 15:38:32.84475511 +0000 UTC m=+35.584108402 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/60d326a6-77da-4f72-965f-880f5c94f2b7-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-2q2tq" (UID: "60d326a6-77da-4f72-965f-880f5c94f2b7") : secret "cluster-monitoring-operator-tls" not found Jul 26 15:38:32.344979 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.344804 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bf408cf7-787a-4839-ab2f-8220f1696bd6-ca-trust-extracted\") pod \"image-registry-5ccd8b79b5-rnb96\" (UID: \"bf408cf7-787a-4839-ab2f-8220f1696bd6\") " pod="openshift-image-registry/image-registry-5ccd8b79b5-rnb96" Jul 26 15:38:32.344979 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.344834 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bf408cf7-787a-4839-ab2f-8220f1696bd6-bound-sa-token\") pod \"image-registry-5ccd8b79b5-rnb96\" (UID: \"bf408cf7-787a-4839-ab2f-8220f1696bd6\") " pod="openshift-image-registry/image-registry-5ccd8b79b5-rnb96" Jul 26 15:38:32.344979 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.344863 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d48d98c5-9029-4b14-ab01-2234e4ecc1f7-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-7qj5q\" (UID: \"d48d98c5-9029-4b14-ab01-2234e4ecc1f7\") " pod="openshift-insights/insights-operator-5885fdb759-7qj5q" Jul 26 15:38:32.344979 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.344889 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bf408cf7-787a-4839-ab2f-8220f1696bd6-registry-tls\") pod \"image-registry-5ccd8b79b5-rnb96\" (UID: \"bf408cf7-787a-4839-ab2f-8220f1696bd6\") " pod="openshift-image-registry/image-registry-5ccd8b79b5-rnb96" Jul 26 15:38:32.345469 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.345451 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/60d326a6-77da-4f72-965f-880f5c94f2b7-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-2q2tq\" (UID: \"60d326a6-77da-4f72-965f-880f5c94f2b7\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-2q2tq" Jul 26 15:38:32.356448 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.356425 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nfsl\" (UniqueName: \"kubernetes.io/projected/60d326a6-77da-4f72-965f-880f5c94f2b7-kube-api-access-6nfsl\") pod \"cluster-monitoring-operator-548b489cb8-2q2tq\" (UID: \"60d326a6-77da-4f72-965f-880f5c94f2b7\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-2q2tq" Jul 26 15:38:32.356548 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.356464 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gz7z5\" (UniqueName: \"kubernetes.io/projected/cb333fda-51e9-4488-acb3-85b027240581-kube-api-access-gz7z5\") pod \"volume-data-source-validator-549c6d54cf-6s9pz\" (UID: \"cb333fda-51e9-4488-acb3-85b027240581\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-6s9pz" Jul 26 15:38:32.356790 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.356771 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-tzgd6"] Jul 26 15:38:32.356917 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.356900 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-8wjqp" Jul 26 15:38:32.359268 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.359242 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 26 15:38:32.359359 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.359288 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 26 15:38:32.359359 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.359329 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-4wzxl\"" Jul 26 15:38:32.359453 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.359243 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 26 15:38:32.359453 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.359243 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 26 15:38:32.364505 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.364481 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 26 15:38:32.373189 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.373165 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5b79d74577-h8slk"] Jul 26 15:38:32.373278 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.373205 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-5xf7l"] Jul 26 15:38:32.373278 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.373227 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-5zpk2"] Jul 26 15:38:32.373278 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.373249 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-2xmhd"] Jul 26 15:38:32.373278 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.373268 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-8wjqp"] Jul 26 15:38:32.373453 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.373285 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-65h7n"] Jul 26 15:38:32.373453 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.373316 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-tzgd6"] Jul 26 15:38:32.373453 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.373364 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-tzgd6" Jul 26 15:38:32.376686 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.376664 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-jvz4h\"" Jul 26 15:38:32.376791 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.376701 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 26 15:38:32.376791 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.376724 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 26 15:38:32.376791 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.376761 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 26 15:38:32.377280 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.376992 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 26 15:38:32.397212 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.397174 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:38:32.446093 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.446052 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c2a7cc81-246f-4d1b-985d-22c611d6b339-config\") pod \"service-ca-operator-5d6c75f84f-2xmhd\" (UID: \"c2a7cc81-246f-4d1b-985d-22c611d6b339\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-2xmhd" Jul 26 15:38:32.446260 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.446107 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bf408cf7-787a-4839-ab2f-8220f1696bd6-image-registry-private-configuration\") pod \"image-registry-5ccd8b79b5-rnb96\" (UID: \"bf408cf7-787a-4839-ab2f-8220f1696bd6\") " pod="openshift-image-registry/image-registry-5ccd8b79b5-rnb96" Jul 26 15:38:32.446260 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.446230 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e6f72321-bfbd-415c-9e47-cb4ef4fa0d46-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-tzgd6\" (UID: \"e6f72321-bfbd-415c-9e47-cb4ef4fa0d46\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-tzgd6" Jul 26 15:38:32.446354 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.446282 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbxm4\" (UniqueName: \"kubernetes.io/projected/c2a7cc81-246f-4d1b-985d-22c611d6b339-kube-api-access-wbxm4\") pod \"service-ca-operator-5d6c75f84f-2xmhd\" (UID: \"c2a7cc81-246f-4d1b-985d-22c611d6b339\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-2xmhd" Jul 26 15:38:32.446354 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.446330 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/839598d1-d772-47cf-a460-5e88affbac0e-config-volume\") pod \"dns-default-65h7n\" (UID: \"839598d1-d772-47cf-a460-5e88affbac0e\") " pod="openshift-dns/dns-default-65h7n" Jul 26 15:38:32.446463 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.446429 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9af43da1-c25b-4687-820f-6a95afe88786-service-ca-bundle\") pod \"router-default-5b79d74577-h8slk\" (UID: \"9af43da1-c25b-4687-820f-6a95afe88786\") " pod="openshift-ingress/router-default-5b79d74577-h8slk" Jul 26 15:38:32.446502 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.446463 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8z4js\" (UniqueName: \"kubernetes.io/projected/d48d98c5-9029-4b14-ab01-2234e4ecc1f7-kube-api-access-8z4js\") pod \"insights-operator-5885fdb759-7qj5q\" (UID: \"d48d98c5-9029-4b14-ab01-2234e4ecc1f7\") " pod="openshift-insights/insights-operator-5885fdb759-7qj5q" Jul 26 15:38:32.446542 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.446502 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/9af43da1-c25b-4687-820f-6a95afe88786-default-certificate\") pod \"router-default-5b79d74577-h8slk\" (UID: \"9af43da1-c25b-4687-820f-6a95afe88786\") " pod="openshift-ingress/router-default-5b79d74577-h8slk" Jul 26 15:38:32.446542 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.446531 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9af43da1-c25b-4687-820f-6a95afe88786-metrics-certs\") pod \"router-default-5b79d74577-h8slk\" (UID: \"9af43da1-c25b-4687-820f-6a95afe88786\") " pod="openshift-ingress/router-default-5b79d74577-h8slk" Jul 26 15:38:32.446645 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.446555 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87h2r\" (UniqueName: \"kubernetes.io/projected/839598d1-d772-47cf-a460-5e88affbac0e-kube-api-access-87h2r\") pod \"dns-default-65h7n\" (UID: \"839598d1-d772-47cf-a460-5e88affbac0e\") " pod="openshift-dns/dns-default-65h7n" Jul 26 15:38:32.446645 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.446580 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vvnn\" (UniqueName: \"kubernetes.io/projected/b9353822-c6c4-47f1-8a41-ecdc92669f5f-kube-api-access-4vvnn\") pod \"cluster-samples-operator-6748489656-rrqbw\" (UID: \"b9353822-c6c4-47f1-8a41-ecdc92669f5f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-rrqbw" Jul 26 15:38:32.446645 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:32.446606 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9af43da1-c25b-4687-820f-6a95afe88786-service-ca-bundle podName:9af43da1-c25b-4687-820f-6a95afe88786 nodeName:}" failed. No retries permitted until 2026-07-26 15:38:32.946585667 +0000 UTC m=+35.685938969 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/9af43da1-c25b-4687-820f-6a95afe88786-service-ca-bundle") pod "router-default-5b79d74577-h8slk" (UID: "9af43da1-c25b-4687-820f-6a95afe88786") : configmap references non-existent config key: service-ca.crt Jul 26 15:38:32.446791 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.446652 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bf408cf7-787a-4839-ab2f-8220f1696bd6-registry-certificates\") pod \"image-registry-5ccd8b79b5-rnb96\" (UID: \"bf408cf7-787a-4839-ab2f-8220f1696bd6\") " pod="openshift-image-registry/image-registry-5ccd8b79b5-rnb96" Jul 26 15:38:32.446791 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:32.446690 2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 26 15:38:32.446791 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:32.446738 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9af43da1-c25b-4687-820f-6a95afe88786-metrics-certs podName:9af43da1-c25b-4687-820f-6a95afe88786 nodeName:}" failed. No retries permitted until 2026-07-26 15:38:32.946726656 +0000 UTC m=+35.686079947 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9af43da1-c25b-4687-820f-6a95afe88786-metrics-certs") pod "router-default-5b79d74577-h8slk" (UID: "9af43da1-c25b-4687-820f-6a95afe88786") : secret "router-metrics-certs-default" not found Jul 26 15:38:32.446791 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.446686 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/839598d1-d772-47cf-a460-5e88affbac0e-metrics-tls\") pod \"dns-default-65h7n\" (UID: \"839598d1-d772-47cf-a460-5e88affbac0e\") " pod="openshift-dns/dns-default-65h7n" Jul 26 15:38:32.446791 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.446777 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b6b8080d-0cee-4d0c-a05c-3e2f1b672a54-config\") pod \"console-operator-584dcb6b54-8wjqp\" (UID: \"b6b8080d-0cee-4d0c-a05c-3e2f1b672a54\") " pod="openshift-console-operator/console-operator-584dcb6b54-8wjqp" Jul 26 15:38:32.447026 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.446800 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c2a7cc81-246f-4d1b-985d-22c611d6b339-config\") pod \"service-ca-operator-5d6c75f84f-2xmhd\" (UID: \"c2a7cc81-246f-4d1b-985d-22c611d6b339\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-2xmhd" Jul 26 15:38:32.447026 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.446806 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d48d98c5-9029-4b14-ab01-2234e4ecc1f7-tmp\") pod \"insights-operator-5885fdb759-7qj5q\" (UID: \"d48d98c5-9029-4b14-ab01-2234e4ecc1f7\") " pod="openshift-insights/insights-operator-5885fdb759-7qj5q" Jul 26 15:38:32.447026 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.446846 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d48d98c5-9029-4b14-ab01-2234e4ecc1f7-serving-cert\") pod \"insights-operator-5885fdb759-7qj5q\" (UID: \"d48d98c5-9029-4b14-ab01-2234e4ecc1f7\") " pod="openshift-insights/insights-operator-5885fdb759-7qj5q" Jul 26 15:38:32.447026 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.446878 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/9af43da1-c25b-4687-820f-6a95afe88786-stats-auth\") pod \"router-default-5b79d74577-h8slk\" (UID: \"9af43da1-c25b-4687-820f-6a95afe88786\") " pod="openshift-ingress/router-default-5b79d74577-h8slk" Jul 26 15:38:32.447026 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.446908 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skmbt\" (UniqueName: \"kubernetes.io/projected/b6b8080d-0cee-4d0c-a05c-3e2f1b672a54-kube-api-access-skmbt\") pod \"console-operator-584dcb6b54-8wjqp\" (UID: \"b6b8080d-0cee-4d0c-a05c-3e2f1b672a54\") " pod="openshift-console-operator/console-operator-584dcb6b54-8wjqp" Jul 26 15:38:32.447026 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.446936 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9vjr\" (UniqueName: \"kubernetes.io/projected/e6f72321-bfbd-415c-9e47-cb4ef4fa0d46-kube-api-access-f9vjr\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-tzgd6\" (UID: \"e6f72321-bfbd-415c-9e47-cb4ef4fa0d46\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-tzgd6" Jul 26 15:38:32.447026 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.446973 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b9353822-c6c4-47f1-8a41-ecdc92669f5f-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-rrqbw\" (UID: \"b9353822-c6c4-47f1-8a41-ecdc92669f5f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-rrqbw" Jul 26 15:38:32.447026 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.447009 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgcrz\" (UniqueName: \"kubernetes.io/projected/9af43da1-c25b-4687-820f-6a95afe88786-kube-api-access-wgcrz\") pod \"router-default-5b79d74577-h8slk\" (UID: \"9af43da1-c25b-4687-820f-6a95afe88786\") " pod="openshift-ingress/router-default-5b79d74577-h8slk" Jul 26 15:38:32.447432 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.447040 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b6b8080d-0cee-4d0c-a05c-3e2f1b672a54-trusted-ca\") pod \"console-operator-584dcb6b54-8wjqp\" (UID: \"b6b8080d-0cee-4d0c-a05c-3e2f1b672a54\") " pod="openshift-console-operator/console-operator-584dcb6b54-8wjqp" Jul 26 15:38:32.447432 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.447090 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/d48d98c5-9029-4b14-ab01-2234e4ecc1f7-snapshots\") pod \"insights-operator-5885fdb759-7qj5q\" (UID: \"d48d98c5-9029-4b14-ab01-2234e4ecc1f7\") " pod="openshift-insights/insights-operator-5885fdb759-7qj5q" Jul 26 15:38:32.447432 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.447092 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d48d98c5-9029-4b14-ab01-2234e4ecc1f7-tmp\") pod \"insights-operator-5885fdb759-7qj5q\" (UID: \"d48d98c5-9029-4b14-ab01-2234e4ecc1f7\") " pod="openshift-insights/insights-operator-5885fdb759-7qj5q" Jul 26 15:38:32.447432 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.447115 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54z42\" (UniqueName: \"kubernetes.io/projected/bf408cf7-787a-4839-ab2f-8220f1696bd6-kube-api-access-54z42\") pod \"image-registry-5ccd8b79b5-rnb96\" (UID: \"bf408cf7-787a-4839-ab2f-8220f1696bd6\") " pod="openshift-image-registry/image-registry-5ccd8b79b5-rnb96" Jul 26 15:38:32.447432 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.447140 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b6b8080d-0cee-4d0c-a05c-3e2f1b672a54-serving-cert\") pod \"console-operator-584dcb6b54-8wjqp\" (UID: \"b6b8080d-0cee-4d0c-a05c-3e2f1b672a54\") " pod="openshift-console-operator/console-operator-584dcb6b54-8wjqp" Jul 26 15:38:32.447432 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.447181 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bf408cf7-787a-4839-ab2f-8220f1696bd6-installation-pull-secrets\") pod \"image-registry-5ccd8b79b5-rnb96\" (UID: \"bf408cf7-787a-4839-ab2f-8220f1696bd6\") " pod="openshift-image-registry/image-registry-5ccd8b79b5-rnb96" Jul 26 15:38:32.447432 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.447211 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d48d98c5-9029-4b14-ab01-2234e4ecc1f7-service-ca-bundle\") pod \"insights-operator-5885fdb759-7qj5q\" (UID: \"d48d98c5-9029-4b14-ab01-2234e4ecc1f7\") " pod="openshift-insights/insights-operator-5885fdb759-7qj5q" Jul 26 15:38:32.447432 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.447236 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bf408cf7-787a-4839-ab2f-8220f1696bd6-trusted-ca\") pod \"image-registry-5ccd8b79b5-rnb96\" (UID: \"bf408cf7-787a-4839-ab2f-8220f1696bd6\") " pod="openshift-image-registry/image-registry-5ccd8b79b5-rnb96" Jul 26 15:38:32.447432 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.447263 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5mxf\" (UniqueName: \"kubernetes.io/projected/904d1b9e-6aac-4226-aed5-f49fa3680cf5-kube-api-access-h5mxf\") pod \"network-check-source-7d88cb55d8-5zpk2\" (UID: \"904d1b9e-6aac-4226-aed5-f49fa3680cf5\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-5zpk2" Jul 26 15:38:32.447432 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.447302 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bf408cf7-787a-4839-ab2f-8220f1696bd6-ca-trust-extracted\") pod \"image-registry-5ccd8b79b5-rnb96\" (UID: \"bf408cf7-787a-4839-ab2f-8220f1696bd6\") " pod="openshift-image-registry/image-registry-5ccd8b79b5-rnb96" Jul 26 15:38:32.447432 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.447325 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bf408cf7-787a-4839-ab2f-8220f1696bd6-bound-sa-token\") pod \"image-registry-5ccd8b79b5-rnb96\" (UID: \"bf408cf7-787a-4839-ab2f-8220f1696bd6\") " pod="openshift-image-registry/image-registry-5ccd8b79b5-rnb96" Jul 26 15:38:32.447432 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.447364 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/839598d1-d772-47cf-a460-5e88affbac0e-tmp-dir\") pod \"dns-default-65h7n\" (UID: \"839598d1-d772-47cf-a460-5e88affbac0e\") " pod="openshift-dns/dns-default-65h7n" Jul 26 15:38:32.447432 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.447418 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d48d98c5-9029-4b14-ab01-2234e4ecc1f7-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-7qj5q\" (UID: \"d48d98c5-9029-4b14-ab01-2234e4ecc1f7\") " pod="openshift-insights/insights-operator-5885fdb759-7qj5q" Jul 26 15:38:32.448110 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.447445 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bf408cf7-787a-4839-ab2f-8220f1696bd6-registry-tls\") pod \"image-registry-5ccd8b79b5-rnb96\" (UID: \"bf408cf7-787a-4839-ab2f-8220f1696bd6\") " pod="openshift-image-registry/image-registry-5ccd8b79b5-rnb96" Jul 26 15:38:32.448110 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.447456 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bf408cf7-787a-4839-ab2f-8220f1696bd6-registry-certificates\") pod \"image-registry-5ccd8b79b5-rnb96\" (UID: \"bf408cf7-787a-4839-ab2f-8220f1696bd6\") " pod="openshift-image-registry/image-registry-5ccd8b79b5-rnb96" Jul 26 15:38:32.448110 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.447475 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fceea9e2-8f78-4c66-a5b4-3ed7666a93af-cert\") pod \"ingress-canary-5xf7l\" (UID: \"fceea9e2-8f78-4c66-a5b4-3ed7666a93af\") " pod="openshift-ingress-canary/ingress-canary-5xf7l" Jul 26 15:38:32.448110 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:32.447570 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 26 15:38:32.448110 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:32.447583 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5ccd8b79b5-rnb96: secret "image-registry-tls" not found Jul 26 15:38:32.448110 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:32.447623 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bf408cf7-787a-4839-ab2f-8220f1696bd6-registry-tls podName:bf408cf7-787a-4839-ab2f-8220f1696bd6 nodeName:}" failed. No retries permitted until 2026-07-26 15:38:32.947609319 +0000 UTC m=+35.686962611 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bf408cf7-787a-4839-ab2f-8220f1696bd6-registry-tls") pod "image-registry-5ccd8b79b5-rnb96" (UID: "bf408cf7-787a-4839-ab2f-8220f1696bd6") : secret "image-registry-tls" not found Jul 26 15:38:32.448110 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.447671 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kmzp\" (UniqueName: \"kubernetes.io/projected/fceea9e2-8f78-4c66-a5b4-3ed7666a93af-kube-api-access-7kmzp\") pod \"ingress-canary-5xf7l\" (UID: \"fceea9e2-8f78-4c66-a5b4-3ed7666a93af\") " pod="openshift-ingress-canary/ingress-canary-5xf7l" Jul 26 15:38:32.448110 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:32.447784 2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 26 15:38:32.448110 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:32.447831 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9353822-c6c4-47f1-8a41-ecdc92669f5f-samples-operator-tls podName:b9353822-c6c4-47f1-8a41-ecdc92669f5f nodeName:}" failed. No retries permitted until 2026-07-26 15:38:32.947816178 +0000 UTC m=+35.687169471 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b9353822-c6c4-47f1-8a41-ecdc92669f5f-samples-operator-tls") pod "cluster-samples-operator-6748489656-rrqbw" (UID: "b9353822-c6c4-47f1-8a41-ecdc92669f5f") : secret "samples-operator-tls" not found Jul 26 15:38:32.448689 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.448304 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d48d98c5-9029-4b14-ab01-2234e4ecc1f7-service-ca-bundle\") pod \"insights-operator-5885fdb759-7qj5q\" (UID: \"d48d98c5-9029-4b14-ab01-2234e4ecc1f7\") " pod="openshift-insights/insights-operator-5885fdb759-7qj5q" Jul 26 15:38:32.448689 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.448315 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d48d98c5-9029-4b14-ab01-2234e4ecc1f7-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-7qj5q\" (UID: \"d48d98c5-9029-4b14-ab01-2234e4ecc1f7\") " pod="openshift-insights/insights-operator-5885fdb759-7qj5q" Jul 26 15:38:32.448689 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.448371 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e6f72321-bfbd-415c-9e47-cb4ef4fa0d46-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-tzgd6\" (UID: \"e6f72321-bfbd-415c-9e47-cb4ef4fa0d46\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-tzgd6" Jul 26 15:38:32.448689 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.448586 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bf408cf7-787a-4839-ab2f-8220f1696bd6-ca-trust-extracted\") pod \"image-registry-5ccd8b79b5-rnb96\" (UID: \"bf408cf7-787a-4839-ab2f-8220f1696bd6\") " pod="openshift-image-registry/image-registry-5ccd8b79b5-rnb96" Jul 26 15:38:32.448689 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.448609 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bf408cf7-787a-4839-ab2f-8220f1696bd6-trusted-ca\") pod \"image-registry-5ccd8b79b5-rnb96\" (UID: \"bf408cf7-787a-4839-ab2f-8220f1696bd6\") " pod="openshift-image-registry/image-registry-5ccd8b79b5-rnb96" Jul 26 15:38:32.448914 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.448693 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c2a7cc81-246f-4d1b-985d-22c611d6b339-serving-cert\") pod \"service-ca-operator-5d6c75f84f-2xmhd\" (UID: \"c2a7cc81-246f-4d1b-985d-22c611d6b339\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-2xmhd" Jul 26 15:38:32.448914 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.448882 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/d48d98c5-9029-4b14-ab01-2234e4ecc1f7-snapshots\") pod \"insights-operator-5885fdb759-7qj5q\" (UID: \"d48d98c5-9029-4b14-ab01-2234e4ecc1f7\") " pod="openshift-insights/insights-operator-5885fdb759-7qj5q" Jul 26 15:38:32.449904 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.449867 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bf408cf7-787a-4839-ab2f-8220f1696bd6-image-registry-private-configuration\") pod \"image-registry-5ccd8b79b5-rnb96\" (UID: \"bf408cf7-787a-4839-ab2f-8220f1696bd6\") " pod="openshift-image-registry/image-registry-5ccd8b79b5-rnb96" Jul 26 15:38:32.450114 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.450089 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/9af43da1-c25b-4687-820f-6a95afe88786-default-certificate\") pod \"router-default-5b79d74577-h8slk\" (UID: \"9af43da1-c25b-4687-820f-6a95afe88786\") " pod="openshift-ingress/router-default-5b79d74577-h8slk" Jul 26 15:38:32.450516 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.450483 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/9af43da1-c25b-4687-820f-6a95afe88786-stats-auth\") pod \"router-default-5b79d74577-h8slk\" (UID: \"9af43da1-c25b-4687-820f-6a95afe88786\") " pod="openshift-ingress/router-default-5b79d74577-h8slk" Jul 26 15:38:32.450619 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.450604 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bf408cf7-787a-4839-ab2f-8220f1696bd6-installation-pull-secrets\") pod \"image-registry-5ccd8b79b5-rnb96\" (UID: \"bf408cf7-787a-4839-ab2f-8220f1696bd6\") " pod="openshift-image-registry/image-registry-5ccd8b79b5-rnb96" Jul 26 15:38:32.451046 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.451029 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d48d98c5-9029-4b14-ab01-2234e4ecc1f7-serving-cert\") pod \"insights-operator-5885fdb759-7qj5q\" (UID: \"d48d98c5-9029-4b14-ab01-2234e4ecc1f7\") " pod="openshift-insights/insights-operator-5885fdb759-7qj5q" Jul 26 15:38:32.451391 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.451372 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c2a7cc81-246f-4d1b-985d-22c611d6b339-serving-cert\") pod \"service-ca-operator-5d6c75f84f-2xmhd\" (UID: \"c2a7cc81-246f-4d1b-985d-22c611d6b339\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-2xmhd" Jul 26 15:38:32.455492 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.455448 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgcrz\" (UniqueName: \"kubernetes.io/projected/9af43da1-c25b-4687-820f-6a95afe88786-kube-api-access-wgcrz\") pod \"router-default-5b79d74577-h8slk\" (UID: \"9af43da1-c25b-4687-820f-6a95afe88786\") " pod="openshift-ingress/router-default-5b79d74577-h8slk" Jul 26 15:38:32.455871 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.455826 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbxm4\" (UniqueName: \"kubernetes.io/projected/c2a7cc81-246f-4d1b-985d-22c611d6b339-kube-api-access-wbxm4\") pod \"service-ca-operator-5d6c75f84f-2xmhd\" (UID: \"c2a7cc81-246f-4d1b-985d-22c611d6b339\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-2xmhd" Jul 26 15:38:32.456297 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.456271 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8z4js\" (UniqueName: \"kubernetes.io/projected/d48d98c5-9029-4b14-ab01-2234e4ecc1f7-kube-api-access-8z4js\") pod \"insights-operator-5885fdb759-7qj5q\" (UID: \"d48d98c5-9029-4b14-ab01-2234e4ecc1f7\") " pod="openshift-insights/insights-operator-5885fdb759-7qj5q" Jul 26 15:38:32.456857 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.456492 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bf408cf7-787a-4839-ab2f-8220f1696bd6-bound-sa-token\") pod \"image-registry-5ccd8b79b5-rnb96\" (UID: \"bf408cf7-787a-4839-ab2f-8220f1696bd6\") " pod="openshift-image-registry/image-registry-5ccd8b79b5-rnb96" Jul 26 15:38:32.456950 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.456872 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vvnn\" (UniqueName: \"kubernetes.io/projected/b9353822-c6c4-47f1-8a41-ecdc92669f5f-kube-api-access-4vvnn\") pod \"cluster-samples-operator-6748489656-rrqbw\" (UID: \"b9353822-c6c4-47f1-8a41-ecdc92669f5f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-rrqbw" Jul 26 15:38:32.457356 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.457338 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-54z42\" (UniqueName: \"kubernetes.io/projected/bf408cf7-787a-4839-ab2f-8220f1696bd6-kube-api-access-54z42\") pod \"image-registry-5ccd8b79b5-rnb96\" (UID: \"bf408cf7-787a-4839-ab2f-8220f1696bd6\") " pod="openshift-image-registry/image-registry-5ccd8b79b5-rnb96" Jul 26 15:38:32.463548 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.463524 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-6s9pz" Jul 26 15:38:32.497609 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.497572 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5885fdb759-7qj5q" Jul 26 15:38:32.549903 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.549863 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5mxf\" (UniqueName: \"kubernetes.io/projected/904d1b9e-6aac-4226-aed5-f49fa3680cf5-kube-api-access-h5mxf\") pod \"network-check-source-7d88cb55d8-5zpk2\" (UID: \"904d1b9e-6aac-4226-aed5-f49fa3680cf5\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-5zpk2" Jul 26 15:38:32.550075 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.549908 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/839598d1-d772-47cf-a460-5e88affbac0e-tmp-dir\") pod \"dns-default-65h7n\" (UID: \"839598d1-d772-47cf-a460-5e88affbac0e\") " pod="openshift-dns/dns-default-65h7n" Jul 26 15:38:32.550075 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.550058 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fceea9e2-8f78-4c66-a5b4-3ed7666a93af-cert\") pod \"ingress-canary-5xf7l\" (UID: \"fceea9e2-8f78-4c66-a5b4-3ed7666a93af\") " pod="openshift-ingress-canary/ingress-canary-5xf7l" Jul 26 15:38:32.550194 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.550099 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kmzp\" (UniqueName: \"kubernetes.io/projected/fceea9e2-8f78-4c66-a5b4-3ed7666a93af-kube-api-access-7kmzp\") pod \"ingress-canary-5xf7l\" (UID: \"fceea9e2-8f78-4c66-a5b4-3ed7666a93af\") " pod="openshift-ingress-canary/ingress-canary-5xf7l" Jul 26 15:38:32.550194 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.550124 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e6f72321-bfbd-415c-9e47-cb4ef4fa0d46-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-tzgd6\" (UID: \"e6f72321-bfbd-415c-9e47-cb4ef4fa0d46\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-tzgd6" Jul 26 15:38:32.550194 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.550162 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e6f72321-bfbd-415c-9e47-cb4ef4fa0d46-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-tzgd6\" (UID: \"e6f72321-bfbd-415c-9e47-cb4ef4fa0d46\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-tzgd6" Jul 26 15:38:32.550332 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.550200 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/839598d1-d772-47cf-a460-5e88affbac0e-config-volume\") pod \"dns-default-65h7n\" (UID: \"839598d1-d772-47cf-a460-5e88affbac0e\") " pod="openshift-dns/dns-default-65h7n" Jul 26 15:38:32.550332 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.550240 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/839598d1-d772-47cf-a460-5e88affbac0e-tmp-dir\") pod \"dns-default-65h7n\" (UID: \"839598d1-d772-47cf-a460-5e88affbac0e\") " pod="openshift-dns/dns-default-65h7n" Jul 26 15:38:32.550332 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.550266 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87h2r\" (UniqueName: \"kubernetes.io/projected/839598d1-d772-47cf-a460-5e88affbac0e-kube-api-access-87h2r\") pod \"dns-default-65h7n\" (UID: \"839598d1-d772-47cf-a460-5e88affbac0e\") " pod="openshift-dns/dns-default-65h7n" Jul 26 15:38:32.550471 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.550308 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/839598d1-d772-47cf-a460-5e88affbac0e-metrics-tls\") pod \"dns-default-65h7n\" (UID: \"839598d1-d772-47cf-a460-5e88affbac0e\") " pod="openshift-dns/dns-default-65h7n" Jul 26 15:38:32.550471 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.550356 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b6b8080d-0cee-4d0c-a05c-3e2f1b672a54-config\") pod \"console-operator-584dcb6b54-8wjqp\" (UID: \"b6b8080d-0cee-4d0c-a05c-3e2f1b672a54\") " pod="openshift-console-operator/console-operator-584dcb6b54-8wjqp" Jul 26 15:38:32.550471 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.550416 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skmbt\" (UniqueName: \"kubernetes.io/projected/b6b8080d-0cee-4d0c-a05c-3e2f1b672a54-kube-api-access-skmbt\") pod \"console-operator-584dcb6b54-8wjqp\" (UID: \"b6b8080d-0cee-4d0c-a05c-3e2f1b672a54\") " pod="openshift-console-operator/console-operator-584dcb6b54-8wjqp" Jul 26 15:38:32.550471 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.550445 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9vjr\" (UniqueName: \"kubernetes.io/projected/e6f72321-bfbd-415c-9e47-cb4ef4fa0d46-kube-api-access-f9vjr\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-tzgd6\" (UID: \"e6f72321-bfbd-415c-9e47-cb4ef4fa0d46\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-tzgd6" Jul 26 15:38:32.550660 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.550503 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b6b8080d-0cee-4d0c-a05c-3e2f1b672a54-trusted-ca\") pod \"console-operator-584dcb6b54-8wjqp\" (UID: \"b6b8080d-0cee-4d0c-a05c-3e2f1b672a54\") " pod="openshift-console-operator/console-operator-584dcb6b54-8wjqp" Jul 26 15:38:32.550750 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:32.550729 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 26 15:38:32.550813 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:32.550795 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/839598d1-d772-47cf-a460-5e88affbac0e-metrics-tls podName:839598d1-d772-47cf-a460-5e88affbac0e nodeName:}" failed. No retries permitted until 2026-07-26 15:38:33.05077616 +0000 UTC m=+35.790129453 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/839598d1-d772-47cf-a460-5e88affbac0e-metrics-tls") pod "dns-default-65h7n" (UID: "839598d1-d772-47cf-a460-5e88affbac0e") : secret "dns-default-metrics-tls" not found Jul 26 15:38:32.550948 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.550927 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b6b8080d-0cee-4d0c-a05c-3e2f1b672a54-serving-cert\") pod \"console-operator-584dcb6b54-8wjqp\" (UID: \"b6b8080d-0cee-4d0c-a05c-3e2f1b672a54\") " pod="openshift-console-operator/console-operator-584dcb6b54-8wjqp" Jul 26 15:38:32.551095 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.551070 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/839598d1-d772-47cf-a460-5e88affbac0e-config-volume\") pod \"dns-default-65h7n\" (UID: \"839598d1-d772-47cf-a460-5e88affbac0e\") " pod="openshift-dns/dns-default-65h7n" Jul 26 15:38:32.551154 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.551142 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e6f72321-bfbd-415c-9e47-cb4ef4fa0d46-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-tzgd6\" (UID: \"e6f72321-bfbd-415c-9e47-cb4ef4fa0d46\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-tzgd6" Jul 26 15:38:32.551208 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.551181 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b6b8080d-0cee-4d0c-a05c-3e2f1b672a54-config\") pod \"console-operator-584dcb6b54-8wjqp\" (UID: \"b6b8080d-0cee-4d0c-a05c-3e2f1b672a54\") " pod="openshift-console-operator/console-operator-584dcb6b54-8wjqp" Jul 26 15:38:32.551887 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.551864 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b6b8080d-0cee-4d0c-a05c-3e2f1b672a54-trusted-ca\") pod \"console-operator-584dcb6b54-8wjqp\" (UID: \"b6b8080d-0cee-4d0c-a05c-3e2f1b672a54\") " pod="openshift-console-operator/console-operator-584dcb6b54-8wjqp" Jul 26 15:38:32.552908 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.552884 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e6f72321-bfbd-415c-9e47-cb4ef4fa0d46-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-tzgd6\" (UID: \"e6f72321-bfbd-415c-9e47-cb4ef4fa0d46\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-tzgd6" Jul 26 15:38:32.553036 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.552980 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fceea9e2-8f78-4c66-a5b4-3ed7666a93af-cert\") pod \"ingress-canary-5xf7l\" (UID: \"fceea9e2-8f78-4c66-a5b4-3ed7666a93af\") " pod="openshift-ingress-canary/ingress-canary-5xf7l" Jul 26 15:38:32.553379 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.553355 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b6b8080d-0cee-4d0c-a05c-3e2f1b672a54-serving-cert\") pod \"console-operator-584dcb6b54-8wjqp\" (UID: \"b6b8080d-0cee-4d0c-a05c-3e2f1b672a54\") " pod="openshift-console-operator/console-operator-584dcb6b54-8wjqp" Jul 26 15:38:32.559066 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.558700 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9vjr\" (UniqueName: \"kubernetes.io/projected/e6f72321-bfbd-415c-9e47-cb4ef4fa0d46-kube-api-access-f9vjr\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-tzgd6\" (UID: \"e6f72321-bfbd-415c-9e47-cb4ef4fa0d46\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-tzgd6" Jul 26 15:38:32.559066 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.558994 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kmzp\" (UniqueName: \"kubernetes.io/projected/fceea9e2-8f78-4c66-a5b4-3ed7666a93af-kube-api-access-7kmzp\") pod \"ingress-canary-5xf7l\" (UID: \"fceea9e2-8f78-4c66-a5b4-3ed7666a93af\") " pod="openshift-ingress-canary/ingress-canary-5xf7l" Jul 26 15:38:32.559205 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.559073 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5mxf\" (UniqueName: \"kubernetes.io/projected/904d1b9e-6aac-4226-aed5-f49fa3680cf5-kube-api-access-h5mxf\") pod \"network-check-source-7d88cb55d8-5zpk2\" (UID: \"904d1b9e-6aac-4226-aed5-f49fa3680cf5\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-5zpk2" Jul 26 15:38:32.559915 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.559866 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-skmbt\" (UniqueName: \"kubernetes.io/projected/b6b8080d-0cee-4d0c-a05c-3e2f1b672a54-kube-api-access-skmbt\") pod \"console-operator-584dcb6b54-8wjqp\" (UID: \"b6b8080d-0cee-4d0c-a05c-3e2f1b672a54\") " pod="openshift-console-operator/console-operator-584dcb6b54-8wjqp" Jul 26 15:38:32.561108 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.561075 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-87h2r\" (UniqueName: \"kubernetes.io/projected/839598d1-d772-47cf-a460-5e88affbac0e-kube-api-access-87h2r\") pod \"dns-default-65h7n\" (UID: \"839598d1-d772-47cf-a460-5e88affbac0e\") " pod="openshift-dns/dns-default-65h7n" Jul 26 15:38:32.583004 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.582970 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-2xmhd" Jul 26 15:38:32.619099 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.619072 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-5xf7l" Jul 26 15:38:32.641548 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.641523 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-5zpk2" Jul 26 15:38:32.677142 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.677103 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-8wjqp" Jul 26 15:38:32.684271 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.684243 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-tzgd6" Jul 26 15:38:32.781219 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.781190 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kgk6w" Jul 26 15:38:32.781219 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.781210 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d2nnz" Jul 26 15:38:32.781483 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.781293 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l56b4" Jul 26 15:38:32.784338 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.783971 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 26 15:38:32.784338 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.784045 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8j62j\"" Jul 26 15:38:32.784338 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.784211 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-pkvmj\"" Jul 26 15:38:32.784338 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.784215 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 26 15:38:32.853429 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.853378 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/60d326a6-77da-4f72-965f-880f5c94f2b7-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-2q2tq\" (UID: \"60d326a6-77da-4f72-965f-880f5c94f2b7\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-2q2tq" Jul 26 15:38:32.853587 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:32.853532 2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 26 15:38:32.853658 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:32.853625 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60d326a6-77da-4f72-965f-880f5c94f2b7-cluster-monitoring-operator-tls podName:60d326a6-77da-4f72-965f-880f5c94f2b7 nodeName:}" failed. No retries permitted until 2026-07-26 15:38:33.85360515 +0000 UTC m=+36.592958438 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/60d326a6-77da-4f72-965f-880f5c94f2b7-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-2q2tq" (UID: "60d326a6-77da-4f72-965f-880f5c94f2b7") : secret "cluster-monitoring-operator-tls" not found Jul 26 15:38:32.954532 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.954440 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bf408cf7-787a-4839-ab2f-8220f1696bd6-registry-tls\") pod \"image-registry-5ccd8b79b5-rnb96\" (UID: \"bf408cf7-787a-4839-ab2f-8220f1696bd6\") " pod="openshift-image-registry/image-registry-5ccd8b79b5-rnb96" Jul 26 15:38:32.954710 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.954537 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9af43da1-c25b-4687-820f-6a95afe88786-service-ca-bundle\") pod \"router-default-5b79d74577-h8slk\" (UID: \"9af43da1-c25b-4687-820f-6a95afe88786\") " pod="openshift-ingress/router-default-5b79d74577-h8slk" Jul 26 15:38:32.954710 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:32.954575 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 26 15:38:32.954710 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:32.954599 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5ccd8b79b5-rnb96: secret "image-registry-tls" not found Jul 26 15:38:32.954710 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:32.954652 2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 26 15:38:32.954710 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:32.954671 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bf408cf7-787a-4839-ab2f-8220f1696bd6-registry-tls podName:bf408cf7-787a-4839-ab2f-8220f1696bd6 nodeName:}" failed. No retries permitted until 2026-07-26 15:38:33.954651996 +0000 UTC m=+36.694005289 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bf408cf7-787a-4839-ab2f-8220f1696bd6-registry-tls") pod "image-registry-5ccd8b79b5-rnb96" (UID: "bf408cf7-787a-4839-ab2f-8220f1696bd6") : secret "image-registry-tls" not found Jul 26 15:38:32.954710 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.954580 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9af43da1-c25b-4687-820f-6a95afe88786-metrics-certs\") pod \"router-default-5b79d74577-h8slk\" (UID: \"9af43da1-c25b-4687-820f-6a95afe88786\") " pod="openshift-ingress/router-default-5b79d74577-h8slk" Jul 26 15:38:32.954710 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:32.954705 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9af43da1-c25b-4687-820f-6a95afe88786-metrics-certs podName:9af43da1-c25b-4687-820f-6a95afe88786 nodeName:}" failed. No retries permitted until 2026-07-26 15:38:33.954689429 +0000 UTC m=+36.694042729 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9af43da1-c25b-4687-820f-6a95afe88786-metrics-certs") pod "router-default-5b79d74577-h8slk" (UID: "9af43da1-c25b-4687-820f-6a95afe88786") : secret "router-metrics-certs-default" not found Jul 26 15:38:32.955043 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:32.954728 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9af43da1-c25b-4687-820f-6a95afe88786-service-ca-bundle podName:9af43da1-c25b-4687-820f-6a95afe88786 nodeName:}" failed. No retries permitted until 2026-07-26 15:38:33.954721825 +0000 UTC m=+36.694075113 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/9af43da1-c25b-4687-820f-6a95afe88786-service-ca-bundle") pod "router-default-5b79d74577-h8slk" (UID: "9af43da1-c25b-4687-820f-6a95afe88786") : configmap references non-existent config key: service-ca.crt Jul 26 15:38:32.955043 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:32.954787 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b9353822-c6c4-47f1-8a41-ecdc92669f5f-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-rrqbw\" (UID: \"b9353822-c6c4-47f1-8a41-ecdc92669f5f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-rrqbw" Jul 26 15:38:32.955043 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:32.954948 2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 26 15:38:32.955043 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:32.955032 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9353822-c6c4-47f1-8a41-ecdc92669f5f-samples-operator-tls podName:b9353822-c6c4-47f1-8a41-ecdc92669f5f nodeName:}" failed. No retries permitted until 2026-07-26 15:38:33.955019328 +0000 UTC m=+36.694372616 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b9353822-c6c4-47f1-8a41-ecdc92669f5f-samples-operator-tls") pod "cluster-samples-operator-6748489656-rrqbw" (UID: "b9353822-c6c4-47f1-8a41-ecdc92669f5f") : secret "samples-operator-tls" not found Jul 26 15:38:33.055370 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:33.055331 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/839598d1-d772-47cf-a460-5e88affbac0e-metrics-tls\") pod \"dns-default-65h7n\" (UID: \"839598d1-d772-47cf-a460-5e88affbac0e\") " pod="openshift-dns/dns-default-65h7n" Jul 26 15:38:33.055538 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:33.055511 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 26 15:38:33.055587 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:33.055571 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/839598d1-d772-47cf-a460-5e88affbac0e-metrics-tls podName:839598d1-d772-47cf-a460-5e88affbac0e nodeName:}" failed. No retries permitted until 2026-07-26 15:38:34.055554973 +0000 UTC m=+36.794908260 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/839598d1-d772-47cf-a460-5e88affbac0e-metrics-tls") pod "dns-default-65h7n" (UID: "839598d1-d772-47cf-a460-5e88affbac0e") : secret "dns-default-metrics-tls" not found Jul 26 15:38:33.432937 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:33.432838 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:38:33.531824 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:33.531537 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:38:33.617571 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:33.617457 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-8wjqp"] Jul 26 15:38:33.625148 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:33.625111 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-tzgd6"] Jul 26 15:38:33.628028 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:33.628002 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-2xmhd"] Jul 26 15:38:33.631046 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:38:33.630979 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6b8080d_0cee_4d0c_a05c_3e2f1b672a54.slice/crio-5807caa27bfda0b4b47307080bae5f583d1f94184a6bc68b63e3e0284bc46a5f WatchSource:0}: Error finding container 5807caa27bfda0b4b47307080bae5f583d1f94184a6bc68b63e3e0284bc46a5f: Status 404 returned error can't find the container with id 5807caa27bfda0b4b47307080bae5f583d1f94184a6bc68b63e3e0284bc46a5f Jul 26 15:38:33.636610 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:33.636477 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-5zpk2"] Jul 26 15:38:33.640367 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:33.639656 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-6s9pz"] Jul 26 15:38:33.642072 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:38:33.642042 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod904d1b9e_6aac_4226_aed5_f49fa3680cf5.slice/crio-f33277da2b2204c84ddff604b4ba165e1da8da777e0c1149d291e038a6c5c781 WatchSource:0}: Error finding container f33277da2b2204c84ddff604b4ba165e1da8da777e0c1149d291e038a6c5c781: Status 404 returned error can't find the container with id f33277da2b2204c84ddff604b4ba165e1da8da777e0c1149d291e038a6c5c781 Jul 26 15:38:33.642667 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:38:33.642629 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb333fda_51e9_4488_acb3_85b027240581.slice/crio-434b271448adc5106866111483a8b9042bb72fd3f1736ec0e30fe2823fb72b6c WatchSource:0}: Error finding container 434b271448adc5106866111483a8b9042bb72fd3f1736ec0e30fe2823fb72b6c: Status 404 returned error can't find the container with id 434b271448adc5106866111483a8b9042bb72fd3f1736ec0e30fe2823fb72b6c Jul 26 15:38:33.650533 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:33.650507 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-5xf7l"] Jul 26 15:38:33.653917 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:33.653896 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5885fdb759-7qj5q"] Jul 26 15:38:33.656947 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:38:33.656796 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfceea9e2_8f78_4c66_a5b4_3ed7666a93af.slice/crio-5ef6f9ecff46635409d8019dbfbac9278a381cc617d71860682bb239e6affb16 WatchSource:0}: Error finding container 5ef6f9ecff46635409d8019dbfbac9278a381cc617d71860682bb239e6affb16: Status 404 returned error can't find the container with id 5ef6f9ecff46635409d8019dbfbac9278a381cc617d71860682bb239e6affb16 Jul 26 15:38:33.657822 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:38:33.657775 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd48d98c5_9029_4b14_ab01_2234e4ecc1f7.slice/crio-ed58039c6e344fcfa80203a3728bc403a5cbc5ada36bf85bb32cbcbc6aaaf448 WatchSource:0}: Error finding container ed58039c6e344fcfa80203a3728bc403a5cbc5ada36bf85bb32cbcbc6aaaf448: Status 404 returned error can't find the container with id ed58039c6e344fcfa80203a3728bc403a5cbc5ada36bf85bb32cbcbc6aaaf448 Jul 26 15:38:33.863708 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:33.863676 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/60d326a6-77da-4f72-965f-880f5c94f2b7-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-2q2tq\" (UID: \"60d326a6-77da-4f72-965f-880f5c94f2b7\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-2q2tq" Jul 26 15:38:33.863861 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:33.863825 2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 26 15:38:33.863910 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:33.863890 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60d326a6-77da-4f72-965f-880f5c94f2b7-cluster-monitoring-operator-tls podName:60d326a6-77da-4f72-965f-880f5c94f2b7 nodeName:}" failed. No retries permitted until 2026-07-26 15:38:35.863874019 +0000 UTC m=+38.603227316 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/60d326a6-77da-4f72-965f-880f5c94f2b7-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-2q2tq" (UID: "60d326a6-77da-4f72-965f-880f5c94f2b7") : secret "cluster-monitoring-operator-tls" not found Jul 26 15:38:33.964219 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:33.964133 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bf408cf7-787a-4839-ab2f-8220f1696bd6-registry-tls\") pod \"image-registry-5ccd8b79b5-rnb96\" (UID: \"bf408cf7-787a-4839-ab2f-8220f1696bd6\") " pod="openshift-image-registry/image-registry-5ccd8b79b5-rnb96" Jul 26 15:38:33.964219 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:33.964203 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9af43da1-c25b-4687-820f-6a95afe88786-service-ca-bundle\") pod \"router-default-5b79d74577-h8slk\" (UID: \"9af43da1-c25b-4687-820f-6a95afe88786\") " pod="openshift-ingress/router-default-5b79d74577-h8slk" Jul 26 15:38:33.964426 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:33.964232 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9af43da1-c25b-4687-820f-6a95afe88786-metrics-certs\") pod \"router-default-5b79d74577-h8slk\" (UID: \"9af43da1-c25b-4687-820f-6a95afe88786\") " pod="openshift-ingress/router-default-5b79d74577-h8slk" Jul 26 15:38:33.964426 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:33.964269 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b9353822-c6c4-47f1-8a41-ecdc92669f5f-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-rrqbw\" (UID: \"b9353822-c6c4-47f1-8a41-ecdc92669f5f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-rrqbw" Jul 26 15:38:33.964426 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:33.964339 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 26 15:38:33.964426 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:33.964355 2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 26 15:38:33.964426 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:33.964364 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5ccd8b79b5-rnb96: secret "image-registry-tls" not found Jul 26 15:38:33.964618 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:33.964438 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9af43da1-c25b-4687-820f-6a95afe88786-service-ca-bundle podName:9af43da1-c25b-4687-820f-6a95afe88786 nodeName:}" failed. No retries permitted until 2026-07-26 15:38:35.964425725 +0000 UTC m=+38.703779013 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/9af43da1-c25b-4687-820f-6a95afe88786-service-ca-bundle") pod "router-default-5b79d74577-h8slk" (UID: "9af43da1-c25b-4687-820f-6a95afe88786") : configmap references non-existent config key: service-ca.crt Jul 26 15:38:33.964618 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:33.964351 2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 26 15:38:33.964618 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:33.964517 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bf408cf7-787a-4839-ab2f-8220f1696bd6-registry-tls podName:bf408cf7-787a-4839-ab2f-8220f1696bd6 nodeName:}" failed. No retries permitted until 2026-07-26 15:38:35.964495696 +0000 UTC m=+38.703848986 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bf408cf7-787a-4839-ab2f-8220f1696bd6-registry-tls") pod "image-registry-5ccd8b79b5-rnb96" (UID: "bf408cf7-787a-4839-ab2f-8220f1696bd6") : secret "image-registry-tls" not found Jul 26 15:38:33.964618 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:33.964549 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9af43da1-c25b-4687-820f-6a95afe88786-metrics-certs podName:9af43da1-c25b-4687-820f-6a95afe88786 nodeName:}" failed. No retries permitted until 2026-07-26 15:38:35.964538627 +0000 UTC m=+38.703891920 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9af43da1-c25b-4687-820f-6a95afe88786-metrics-certs") pod "router-default-5b79d74577-h8slk" (UID: "9af43da1-c25b-4687-820f-6a95afe88786") : secret "router-metrics-certs-default" not found Jul 26 15:38:33.964618 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:33.964576 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9353822-c6c4-47f1-8a41-ecdc92669f5f-samples-operator-tls podName:b9353822-c6c4-47f1-8a41-ecdc92669f5f nodeName:}" failed. No retries permitted until 2026-07-26 15:38:35.964564847 +0000 UTC m=+38.703918138 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b9353822-c6c4-47f1-8a41-ecdc92669f5f-samples-operator-tls") pod "cluster-samples-operator-6748489656-rrqbw" (UID: "b9353822-c6c4-47f1-8a41-ecdc92669f5f") : secret "samples-operator-tls" not found Jul 26 15:38:34.051168 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:34.051130 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-6s9pz" event={"ID":"cb333fda-51e9-4488-acb3-85b027240581","Type":"ContainerStarted","Data":"434b271448adc5106866111483a8b9042bb72fd3f1736ec0e30fe2823fb72b6c"} Jul 26 15:38:34.052181 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:34.052158 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-7qj5q" event={"ID":"d48d98c5-9029-4b14-ab01-2234e4ecc1f7","Type":"ContainerStarted","Data":"ed58039c6e344fcfa80203a3728bc403a5cbc5ada36bf85bb32cbcbc6aaaf448"} Jul 26 15:38:34.053257 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:34.053236 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-2xmhd" event={"ID":"c2a7cc81-246f-4d1b-985d-22c611d6b339","Type":"ContainerStarted","Data":"64e930de5743cfc1a15a39c0befc3fead74d9b2982b7b6a2244031477d669dee"} Jul 26 15:38:34.054337 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:34.054308 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-8wjqp" event={"ID":"b6b8080d-0cee-4d0c-a05c-3e2f1b672a54","Type":"ContainerStarted","Data":"5807caa27bfda0b4b47307080bae5f583d1f94184a6bc68b63e3e0284bc46a5f"} Jul 26 15:38:34.056923 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:34.056898 2577 generic.go:356] "Generic (PLEG): container finished" podID="c2653055-9d7a-4aeb-9090-fb495a2b650b" containerID="3b10ed51bd529fdb2a66d711921a2ea3ac8e13534475f3382e8399006b03ecae" exitCode=0 Jul 26 15:38:34.057006 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:34.056974 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dzc7t" event={"ID":"c2653055-9d7a-4aeb-9090-fb495a2b650b","Type":"ContainerDied","Data":"3b10ed51bd529fdb2a66d711921a2ea3ac8e13534475f3382e8399006b03ecae"} Jul 26 15:38:34.058082 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:34.058064 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-tzgd6" event={"ID":"e6f72321-bfbd-415c-9e47-cb4ef4fa0d46","Type":"ContainerStarted","Data":"66112a800ac883ff5fa0145ff03d1808df45dab01a3ac32d59d15548ab2b85fe"} Jul 26 15:38:34.059075 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:34.059052 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-5xf7l" event={"ID":"fceea9e2-8f78-4c66-a5b4-3ed7666a93af","Type":"ContainerStarted","Data":"5ef6f9ecff46635409d8019dbfbac9278a381cc617d71860682bb239e6affb16"} Jul 26 15:38:34.060133 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:34.060112 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-5zpk2" event={"ID":"904d1b9e-6aac-4226-aed5-f49fa3680cf5","Type":"ContainerStarted","Data":"f33277da2b2204c84ddff604b4ba165e1da8da777e0c1149d291e038a6c5c781"} Jul 26 15:38:34.065006 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:34.064985 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/839598d1-d772-47cf-a460-5e88affbac0e-metrics-tls\") pod \"dns-default-65h7n\" (UID: \"839598d1-d772-47cf-a460-5e88affbac0e\") " pod="openshift-dns/dns-default-65h7n" Jul 26 15:38:34.065144 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:34.065129 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 26 15:38:34.065186 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:34.065179 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/839598d1-d772-47cf-a460-5e88affbac0e-metrics-tls podName:839598d1-d772-47cf-a460-5e88affbac0e nodeName:}" failed. No retries permitted until 2026-07-26 15:38:36.065164397 +0000 UTC m=+38.804517684 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/839598d1-d772-47cf-a460-5e88affbac0e-metrics-tls") pod "dns-default-65h7n" (UID: "839598d1-d772-47cf-a460-5e88affbac0e") : secret "dns-default-metrics-tls" not found Jul 26 15:38:34.425210 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:34.424052 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-5pdf8"] Jul 26 15:38:34.432973 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:34.432945 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-5pdf8" Jul 26 15:38:34.437289 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:34.436237 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-5ncdz\"" Jul 26 15:38:34.437289 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:34.436716 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Jul 26 15:38:34.437817 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:34.437792 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-5pdf8"] Jul 26 15:38:34.443238 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:34.442534 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Jul 26 15:38:34.563261 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:34.563228 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:38:34.570236 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:34.569939 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/3d7ef445-5c4a-4c4e-92f4-2da8ee29b868-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-5pdf8\" (UID: \"3d7ef445-5c4a-4c4e-92f4-2da8ee29b868\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-5pdf8" Jul 26 15:38:34.570236 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:34.570035 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3d7ef445-5c4a-4c4e-92f4-2da8ee29b868-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-5pdf8\" (UID: \"3d7ef445-5c4a-4c4e-92f4-2da8ee29b868\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-5pdf8" Jul 26 15:38:34.621657 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:34.621574 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:38:34.671191 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:34.671150 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/3d7ef445-5c4a-4c4e-92f4-2da8ee29b868-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-5pdf8\" (UID: \"3d7ef445-5c4a-4c4e-92f4-2da8ee29b868\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-5pdf8" Jul 26 15:38:34.671367 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:34.671235 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3d7ef445-5c4a-4c4e-92f4-2da8ee29b868-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-5pdf8\" (UID: \"3d7ef445-5c4a-4c4e-92f4-2da8ee29b868\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-5pdf8" Jul 26 15:38:34.671460 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:34.671435 2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 26 15:38:34.671512 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:34.671505 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d7ef445-5c4a-4c4e-92f4-2da8ee29b868-networking-console-plugin-cert podName:3d7ef445-5c4a-4c4e-92f4-2da8ee29b868 nodeName:}" failed. No retries permitted until 2026-07-26 15:38:35.171484067 +0000 UTC m=+37.910837357 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/3d7ef445-5c4a-4c4e-92f4-2da8ee29b868-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-5pdf8" (UID: "3d7ef445-5c4a-4c4e-92f4-2da8ee29b868") : secret "networking-console-plugin-cert" not found Jul 26 15:38:34.672291 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:34.672266 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/3d7ef445-5c4a-4c4e-92f4-2da8ee29b868-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-5pdf8\" (UID: \"3d7ef445-5c4a-4c4e-92f4-2da8ee29b868\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-5pdf8" Jul 26 15:38:35.074937 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:35.074901 2577 generic.go:356] "Generic (PLEG): container finished" podID="c2653055-9d7a-4aeb-9090-fb495a2b650b" containerID="00861e5a38160897da842591983409309a31378d81de54abd53720383d4a2f3b" exitCode=0 Jul 26 15:38:35.075140 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:35.074959 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dzc7t" event={"ID":"c2653055-9d7a-4aeb-9090-fb495a2b650b","Type":"ContainerDied","Data":"00861e5a38160897da842591983409309a31378d81de54abd53720383d4a2f3b"} Jul 26 15:38:35.175735 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:35.175246 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3d7ef445-5c4a-4c4e-92f4-2da8ee29b868-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-5pdf8\" (UID: \"3d7ef445-5c4a-4c4e-92f4-2da8ee29b868\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-5pdf8" Jul 26 15:38:35.177033 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:35.176142 2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 26 15:38:35.177033 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:35.176212 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d7ef445-5c4a-4c4e-92f4-2da8ee29b868-networking-console-plugin-cert podName:3d7ef445-5c4a-4c4e-92f4-2da8ee29b868 nodeName:}" failed. No retries permitted until 2026-07-26 15:38:36.176190587 +0000 UTC m=+38.915543875 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/3d7ef445-5c4a-4c4e-92f4-2da8ee29b868-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-5pdf8" (UID: "3d7ef445-5c4a-4c4e-92f4-2da8ee29b868") : secret "networking-console-plugin-cert" not found Jul 26 15:38:35.885449 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:35.884981 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/60d326a6-77da-4f72-965f-880f5c94f2b7-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-2q2tq\" (UID: \"60d326a6-77da-4f72-965f-880f5c94f2b7\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-2q2tq" Jul 26 15:38:35.885449 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:35.885384 2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 26 15:38:35.886069 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:35.885480 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60d326a6-77da-4f72-965f-880f5c94f2b7-cluster-monitoring-operator-tls podName:60d326a6-77da-4f72-965f-880f5c94f2b7 nodeName:}" failed. No retries permitted until 2026-07-26 15:38:39.885459048 +0000 UTC m=+42.624812339 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/60d326a6-77da-4f72-965f-880f5c94f2b7-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-2q2tq" (UID: "60d326a6-77da-4f72-965f-880f5c94f2b7") : secret "cluster-monitoring-operator-tls" not found Jul 26 15:38:35.986963 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:35.986230 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bf408cf7-787a-4839-ab2f-8220f1696bd6-registry-tls\") pod \"image-registry-5ccd8b79b5-rnb96\" (UID: \"bf408cf7-787a-4839-ab2f-8220f1696bd6\") " pod="openshift-image-registry/image-registry-5ccd8b79b5-rnb96" Jul 26 15:38:35.986963 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:35.986317 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9af43da1-c25b-4687-820f-6a95afe88786-service-ca-bundle\") pod \"router-default-5b79d74577-h8slk\" (UID: \"9af43da1-c25b-4687-820f-6a95afe88786\") " pod="openshift-ingress/router-default-5b79d74577-h8slk" Jul 26 15:38:35.986963 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:35.986361 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9af43da1-c25b-4687-820f-6a95afe88786-metrics-certs\") pod \"router-default-5b79d74577-h8slk\" (UID: \"9af43da1-c25b-4687-820f-6a95afe88786\") " pod="openshift-ingress/router-default-5b79d74577-h8slk" Jul 26 15:38:35.986963 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:35.986451 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b9353822-c6c4-47f1-8a41-ecdc92669f5f-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-rrqbw\" (UID: \"b9353822-c6c4-47f1-8a41-ecdc92669f5f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-rrqbw" Jul 26 15:38:35.986963 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:35.986570 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 26 15:38:35.986963 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:35.986594 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5ccd8b79b5-rnb96: secret "image-registry-tls" not found Jul 26 15:38:35.986963 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:35.986650 2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 26 15:38:35.986963 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:35.986787 2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 26 15:38:35.986963 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:35.986656 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bf408cf7-787a-4839-ab2f-8220f1696bd6-registry-tls podName:bf408cf7-787a-4839-ab2f-8220f1696bd6 nodeName:}" failed. No retries permitted until 2026-07-26 15:38:39.986636977 +0000 UTC m=+42.725990272 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bf408cf7-787a-4839-ab2f-8220f1696bd6-registry-tls") pod "image-registry-5ccd8b79b5-rnb96" (UID: "bf408cf7-787a-4839-ab2f-8220f1696bd6") : secret "image-registry-tls" not found Jul 26 15:38:35.986963 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:35.986859 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9353822-c6c4-47f1-8a41-ecdc92669f5f-samples-operator-tls podName:b9353822-c6c4-47f1-8a41-ecdc92669f5f nodeName:}" failed. No retries permitted until 2026-07-26 15:38:39.986843684 +0000 UTC m=+42.726196979 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b9353822-c6c4-47f1-8a41-ecdc92669f5f-samples-operator-tls") pod "cluster-samples-operator-6748489656-rrqbw" (UID: "b9353822-c6c4-47f1-8a41-ecdc92669f5f") : secret "samples-operator-tls" not found Jul 26 15:38:35.986963 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:35.986875 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9af43da1-c25b-4687-820f-6a95afe88786-service-ca-bundle podName:9af43da1-c25b-4687-820f-6a95afe88786 nodeName:}" failed. No retries permitted until 2026-07-26 15:38:39.98686571 +0000 UTC m=+42.726219004 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/9af43da1-c25b-4687-820f-6a95afe88786-service-ca-bundle") pod "router-default-5b79d74577-h8slk" (UID: "9af43da1-c25b-4687-820f-6a95afe88786") : configmap references non-existent config key: service-ca.crt Jul 26 15:38:35.986963 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:35.986890 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9af43da1-c25b-4687-820f-6a95afe88786-metrics-certs podName:9af43da1-c25b-4687-820f-6a95afe88786 nodeName:}" failed. No retries permitted until 2026-07-26 15:38:39.98688127 +0000 UTC m=+42.726234564 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9af43da1-c25b-4687-820f-6a95afe88786-metrics-certs") pod "router-default-5b79d74577-h8slk" (UID: "9af43da1-c25b-4687-820f-6a95afe88786") : secret "router-metrics-certs-default" not found Jul 26 15:38:36.085922 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:36.085061 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dzc7t" event={"ID":"c2653055-9d7a-4aeb-9090-fb495a2b650b","Type":"ContainerStarted","Data":"f4eec8332c895e134c09d3570dc5d48e3fe4def4107374ef79760261fbdc10c5"} Jul 26 15:38:36.087425 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:36.086835 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/839598d1-d772-47cf-a460-5e88affbac0e-metrics-tls\") pod \"dns-default-65h7n\" (UID: \"839598d1-d772-47cf-a460-5e88affbac0e\") " pod="openshift-dns/dns-default-65h7n" Jul 26 15:38:36.087425 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:36.087037 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 26 15:38:36.087425 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:36.087092 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/839598d1-d772-47cf-a460-5e88affbac0e-metrics-tls podName:839598d1-d772-47cf-a460-5e88affbac0e nodeName:}" failed. No retries permitted until 2026-07-26 15:38:40.087075194 +0000 UTC m=+42.826428486 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/839598d1-d772-47cf-a460-5e88affbac0e-metrics-tls") pod "dns-default-65h7n" (UID: "839598d1-d772-47cf-a460-5e88affbac0e") : secret "dns-default-metrics-tls" not found Jul 26 15:38:36.090958 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:36.090539 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-6s9pz" event={"ID":"cb333fda-51e9-4488-acb3-85b027240581","Type":"ContainerStarted","Data":"769ade3c06833156ca43faa3eba3b5b3850362f138ce9b21f226f7fbe9b4cb4c"} Jul 26 15:38:36.107719 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:36.107152 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-dzc7t" podStartSLOduration=7.31276205 podStartE2EDuration="38.107134574s" totalImagesPullingTime="30.794372524s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-26 15:37:58 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:38:36.106227392 +0000 UTC m=+38.845580704" watchObservedRunningTime="2026-07-26 15:38:36.107134574 +0000 UTC m=+38.846487884" Jul 26 15:38:36.120849 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:36.120595 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-6s9pz" podStartSLOduration=13.120223015 podStartE2EDuration="15.120576096s" totalImagesPullingTime="2.000353081s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:38:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:38:36.119991027 +0000 UTC m=+38.859344365" watchObservedRunningTime="2026-07-26 15:38:36.120576096 +0000 UTC m=+38.859929407" Jul 26 15:38:36.190505 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:36.187979 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3d7ef445-5c4a-4c4e-92f4-2da8ee29b868-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-5pdf8\" (UID: \"3d7ef445-5c4a-4c4e-92f4-2da8ee29b868\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-5pdf8" Jul 26 15:38:36.190505 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:36.189492 2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 26 15:38:36.190505 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:36.189557 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d7ef445-5c4a-4c4e-92f4-2da8ee29b868-networking-console-plugin-cert podName:3d7ef445-5c4a-4c4e-92f4-2da8ee29b868 nodeName:}" failed. No retries permitted until 2026-07-26 15:38:38.189537602 +0000 UTC m=+40.928890889 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/3d7ef445-5c4a-4c4e-92f4-2da8ee29b868-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-5pdf8" (UID: "3d7ef445-5c4a-4c4e-92f4-2da8ee29b868") : secret "networking-console-plugin-cert" not found Jul 26 15:38:36.233551 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:36.233373 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:38:36.305430 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:36.305380 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:38:36.526349 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:36.526316 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:38:36.579239 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:36.579157 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:38:38.207521 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:38.207483 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3d7ef445-5c4a-4c4e-92f4-2da8ee29b868-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-5pdf8\" (UID: \"3d7ef445-5c4a-4c4e-92f4-2da8ee29b868\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-5pdf8" Jul 26 15:38:38.208097 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:38.207678 2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 26 15:38:38.208097 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:38.207778 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d7ef445-5c4a-4c4e-92f4-2da8ee29b868-networking-console-plugin-cert podName:3d7ef445-5c4a-4c4e-92f4-2da8ee29b868 nodeName:}" failed. No retries permitted until 2026-07-26 15:38:42.20775724 +0000 UTC m=+44.947110535 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/3d7ef445-5c4a-4c4e-92f4-2da8ee29b868-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-5pdf8" (UID: "3d7ef445-5c4a-4c4e-92f4-2da8ee29b868") : secret "networking-console-plugin-cert" not found Jul 26 15:38:38.921036 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:38.921002 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:38:38.978346 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:38.978313 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:38:39.924530 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:39.924486 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/60d326a6-77da-4f72-965f-880f5c94f2b7-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-2q2tq\" (UID: \"60d326a6-77da-4f72-965f-880f5c94f2b7\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-2q2tq" Jul 26 15:38:39.924976 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:39.924659 2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 26 15:38:39.924976 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:39.924751 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60d326a6-77da-4f72-965f-880f5c94f2b7-cluster-monitoring-operator-tls podName:60d326a6-77da-4f72-965f-880f5c94f2b7 nodeName:}" failed. No retries permitted until 2026-07-26 15:38:47.924728033 +0000 UTC m=+50.664081327 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/60d326a6-77da-4f72-965f-880f5c94f2b7-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-2q2tq" (UID: "60d326a6-77da-4f72-965f-880f5c94f2b7") : secret "cluster-monitoring-operator-tls" not found Jul 26 15:38:40.025491 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:40.025457 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bf408cf7-787a-4839-ab2f-8220f1696bd6-registry-tls\") pod \"image-registry-5ccd8b79b5-rnb96\" (UID: \"bf408cf7-787a-4839-ab2f-8220f1696bd6\") " pod="openshift-image-registry/image-registry-5ccd8b79b5-rnb96" Jul 26 15:38:40.025686 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:40.025522 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9af43da1-c25b-4687-820f-6a95afe88786-service-ca-bundle\") pod \"router-default-5b79d74577-h8slk\" (UID: \"9af43da1-c25b-4687-820f-6a95afe88786\") " pod="openshift-ingress/router-default-5b79d74577-h8slk" Jul 26 15:38:40.025686 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:40.025565 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9af43da1-c25b-4687-820f-6a95afe88786-metrics-certs\") pod \"router-default-5b79d74577-h8slk\" (UID: \"9af43da1-c25b-4687-820f-6a95afe88786\") " pod="openshift-ingress/router-default-5b79d74577-h8slk" Jul 26 15:38:40.025686 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:40.025629 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b9353822-c6c4-47f1-8a41-ecdc92669f5f-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-rrqbw\" (UID: \"b9353822-c6c4-47f1-8a41-ecdc92669f5f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-rrqbw" Jul 26 15:38:40.025686 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:40.025640 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 26 15:38:40.025686 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:40.025663 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5ccd8b79b5-rnb96: secret "image-registry-tls" not found Jul 26 15:38:40.025880 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:40.025723 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bf408cf7-787a-4839-ab2f-8220f1696bd6-registry-tls podName:bf408cf7-787a-4839-ab2f-8220f1696bd6 nodeName:}" failed. No retries permitted until 2026-07-26 15:38:48.025704916 +0000 UTC m=+50.765058224 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bf408cf7-787a-4839-ab2f-8220f1696bd6-registry-tls") pod "image-registry-5ccd8b79b5-rnb96" (UID: "bf408cf7-787a-4839-ab2f-8220f1696bd6") : secret "image-registry-tls" not found Jul 26 15:38:40.025880 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:40.025730 2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 26 15:38:40.025880 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:40.025742 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9af43da1-c25b-4687-820f-6a95afe88786-service-ca-bundle podName:9af43da1-c25b-4687-820f-6a95afe88786 nodeName:}" failed. No retries permitted until 2026-07-26 15:38:48.025732695 +0000 UTC m=+50.765085997 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/9af43da1-c25b-4687-820f-6a95afe88786-service-ca-bundle") pod "router-default-5b79d74577-h8slk" (UID: "9af43da1-c25b-4687-820f-6a95afe88786") : configmap references non-existent config key: service-ca.crt Jul 26 15:38:40.025880 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:40.025749 2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 26 15:38:40.025880 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:40.025786 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9af43da1-c25b-4687-820f-6a95afe88786-metrics-certs podName:9af43da1-c25b-4687-820f-6a95afe88786 nodeName:}" failed. No retries permitted until 2026-07-26 15:38:48.025768272 +0000 UTC m=+50.765121562 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9af43da1-c25b-4687-820f-6a95afe88786-metrics-certs") pod "router-default-5b79d74577-h8slk" (UID: "9af43da1-c25b-4687-820f-6a95afe88786") : secret "router-metrics-certs-default" not found Jul 26 15:38:40.025880 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:40.025808 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9353822-c6c4-47f1-8a41-ecdc92669f5f-samples-operator-tls podName:b9353822-c6c4-47f1-8a41-ecdc92669f5f nodeName:}" failed. No retries permitted until 2026-07-26 15:38:48.025798017 +0000 UTC m=+50.765151314 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b9353822-c6c4-47f1-8a41-ecdc92669f5f-samples-operator-tls") pod "cluster-samples-operator-6748489656-rrqbw" (UID: "b9353822-c6c4-47f1-8a41-ecdc92669f5f") : secret "samples-operator-tls" not found Jul 26 15:38:40.126656 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:40.126616 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/839598d1-d772-47cf-a460-5e88affbac0e-metrics-tls\") pod \"dns-default-65h7n\" (UID: \"839598d1-d772-47cf-a460-5e88affbac0e\") " pod="openshift-dns/dns-default-65h7n" Jul 26 15:38:40.126819 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:40.126731 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 26 15:38:40.126819 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:40.126804 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/839598d1-d772-47cf-a460-5e88affbac0e-metrics-tls podName:839598d1-d772-47cf-a460-5e88affbac0e nodeName:}" failed. No retries permitted until 2026-07-26 15:38:48.126783357 +0000 UTC m=+50.866136645 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/839598d1-d772-47cf-a460-5e88affbac0e-metrics-tls") pod "dns-default-65h7n" (UID: "839598d1-d772-47cf-a460-5e88affbac0e") : secret "dns-default-metrics-tls" not found Jul 26 15:38:41.697565 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:41.697528 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:38:41.797054 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:41.797014 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:38:42.110505 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:42.110414 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-2xmhd" event={"ID":"c2a7cc81-246f-4d1b-985d-22c611d6b339","Type":"ContainerStarted","Data":"c76f07c03572ce3e491100818369003885b6c66fff54015a51bd0691770f62f4"} Jul 26 15:38:42.112261 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:42.112233 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-8wjqp_b6b8080d-0cee-4d0c-a05c-3e2f1b672a54/console-operator/0.log" Jul 26 15:38:42.112373 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:42.112279 2577 generic.go:356] "Generic (PLEG): container finished" podID="b6b8080d-0cee-4d0c-a05c-3e2f1b672a54" containerID="aaf2441534f721da7655ee9f601897d6ec60b81874a2cf33c684fb6ad4e17aff" exitCode=255 Jul 26 15:38:42.112456 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:42.112365 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-8wjqp" event={"ID":"b6b8080d-0cee-4d0c-a05c-3e2f1b672a54","Type":"ContainerDied","Data":"aaf2441534f721da7655ee9f601897d6ec60b81874a2cf33c684fb6ad4e17aff"} Jul 26 15:38:42.112593 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:42.112568 2577 scope.go:117] "RemoveContainer" containerID="aaf2441534f721da7655ee9f601897d6ec60b81874a2cf33c684fb6ad4e17aff" Jul 26 15:38:42.115529 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:42.114070 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-tzgd6" event={"ID":"e6f72321-bfbd-415c-9e47-cb4ef4fa0d46","Type":"ContainerStarted","Data":"3abae83a3773f72e1f903c6090637bbcf03e9a4ff04e853a9e50d6796128de41"} Jul 26 15:38:42.116461 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:42.116245 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-5xf7l" event={"ID":"fceea9e2-8f78-4c66-a5b4-3ed7666a93af","Type":"ContainerStarted","Data":"32176972f8ec8dd2a288a72d8f3a6ec20fda9dde141d51d83c96675acbc88f64"} Jul 26 15:38:42.117877 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:42.117854 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-5zpk2" event={"ID":"904d1b9e-6aac-4226-aed5-f49fa3680cf5","Type":"ContainerStarted","Data":"47872d6e2baa0fc259ff596d867a30b36515da70925651e37a014c1fb72f4ab1"} Jul 26 15:38:42.119454 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:42.119432 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-7qj5q" event={"ID":"d48d98c5-9029-4b14-ab01-2234e4ecc1f7","Type":"ContainerStarted","Data":"df899b7b15931ade877fa9540f33d1c7c4c702b0a4e37e663ad271d3a9ea7e75"} Jul 26 15:38:42.127850 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:42.126502 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-2xmhd" podStartSLOduration=13.484917148 podStartE2EDuration="21.126487327s" totalImagesPullingTime="7.641570179s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:38:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:38:42.126477652 +0000 UTC m=+44.865830984" watchObservedRunningTime="2026-07-26 15:38:42.126487327 +0000 UTC m=+44.865840644" Jul 26 15:38:42.163238 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:42.162346 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5885fdb759-7qj5q" podStartSLOduration=13.539554754 podStartE2EDuration="21.162325979s" totalImagesPullingTime="7.622771225s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:38:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:38:42.145501127 +0000 UTC m=+44.884854441" watchObservedRunningTime="2026-07-26 15:38:42.162325979 +0000 UTC m=+44.901679291" Jul 26 15:38:42.177898 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:42.177850 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-tzgd6" podStartSLOduration=13.53297702 podStartE2EDuration="21.177829357s" totalImagesPullingTime="7.644852337s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:38:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:38:42.177074774 +0000 UTC m=+44.916428086" watchObservedRunningTime="2026-07-26 15:38:42.177829357 +0000 UTC m=+44.917182670" Jul 26 15:38:42.218520 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:42.218049 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-5zpk2" podStartSLOduration=13.572101031 podStartE2EDuration="21.218036346s" totalImagesPullingTime="7.645935315s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:38:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:38:42.217388014 +0000 UTC m=+44.956741351" watchObservedRunningTime="2026-07-26 15:38:42.218036346 +0000 UTC m=+44.957389658" Jul 26 15:38:42.218758 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:42.218545 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-5xf7l" podStartSLOduration=2.595189514 podStartE2EDuration="10.218537452s" totalImagesPullingTime="7.623347938s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:38:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:38:42.196239611 +0000 UTC m=+44.935592922" watchObservedRunningTime="2026-07-26 15:38:42.218537452 +0000 UTC m=+44.957890765" Jul 26 15:38:42.252360 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:42.252324 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3d7ef445-5c4a-4c4e-92f4-2da8ee29b868-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-5pdf8\" (UID: \"3d7ef445-5c4a-4c4e-92f4-2da8ee29b868\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-5pdf8" Jul 26 15:38:42.252626 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:42.252609 2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 26 15:38:42.252715 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:42.252668 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d7ef445-5c4a-4c4e-92f4-2da8ee29b868-networking-console-plugin-cert podName:3d7ef445-5c4a-4c4e-92f4-2da8ee29b868 nodeName:}" failed. No retries permitted until 2026-07-26 15:38:50.252648509 +0000 UTC m=+52.992001824 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/3d7ef445-5c4a-4c4e-92f4-2da8ee29b868-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-5pdf8" (UID: "3d7ef445-5c4a-4c4e-92f4-2da8ee29b868") : secret "networking-console-plugin-cert" not found Jul 26 15:38:42.367638 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:42.367550 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:38:42.468521 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:42.468485 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:38:42.665444 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:42.665348 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:38:42.678102 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:42.678070 2577 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-584dcb6b54-8wjqp" Jul 26 15:38:42.678252 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:42.678120 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-8wjqp" Jul 26 15:38:42.765018 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:42.764980 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:38:42.979572 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:42.979450 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:38:43.062535 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:43.062493 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-5q8xh"] Jul 26 15:38:43.065904 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:43.065879 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:38:43.081843 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:43.081810 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-5q8xh"] Jul 26 15:38:43.082010 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:43.081947 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-5q8xh" Jul 26 15:38:43.084519 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:43.084492 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 26 15:38:43.085244 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:43.085219 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 26 15:38:43.090954 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:43.090929 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-fwcq5\"" Jul 26 15:38:43.124909 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:43.124844 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:38:43.125137 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:43.125115 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-8wjqp_b6b8080d-0cee-4d0c-a05c-3e2f1b672a54/console-operator/1.log" Jul 26 15:38:43.125618 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:43.125599 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-8wjqp_b6b8080d-0cee-4d0c-a05c-3e2f1b672a54/console-operator/0.log" Jul 26 15:38:43.125725 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:43.125636 2577 generic.go:356] "Generic (PLEG): container finished" podID="b6b8080d-0cee-4d0c-a05c-3e2f1b672a54" containerID="8d03b0ae032be9047d7c31246b62c2a28aba80a28b9fc6f8d388bc65d45d825e" exitCode=255 Jul 26 15:38:43.125781 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:43.125729 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-8wjqp" event={"ID":"b6b8080d-0cee-4d0c-a05c-3e2f1b672a54","Type":"ContainerDied","Data":"8d03b0ae032be9047d7c31246b62c2a28aba80a28b9fc6f8d388bc65d45d825e"} Jul 26 15:38:43.125829 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:43.125792 2577 scope.go:117] "RemoveContainer" containerID="aaf2441534f721da7655ee9f601897d6ec60b81874a2cf33c684fb6ad4e17aff" Jul 26 15:38:43.126042 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:43.126018 2577 scope.go:117] "RemoveContainer" containerID="8d03b0ae032be9047d7c31246b62c2a28aba80a28b9fc6f8d388bc65d45d825e" Jul 26 15:38:43.126220 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:43.126195 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-8wjqp_openshift-console-operator(b6b8080d-0cee-4d0c-a05c-3e2f1b672a54)\"" pod="openshift-console-operator/console-operator-584dcb6b54-8wjqp" podUID="b6b8080d-0cee-4d0c-a05c-3e2f1b672a54" Jul 26 15:38:43.164990 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:43.164937 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/76cd20bd-3a66-40ad-99c0-fefa173500d6-data-volume\") pod \"insights-runtime-extractor-5q8xh\" (UID: \"76cd20bd-3a66-40ad-99c0-fefa173500d6\") " pod="openshift-insights/insights-runtime-extractor-5q8xh" Jul 26 15:38:43.164990 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:43.164969 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/76cd20bd-3a66-40ad-99c0-fefa173500d6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5q8xh\" (UID: \"76cd20bd-3a66-40ad-99c0-fefa173500d6\") " pod="openshift-insights/insights-runtime-extractor-5q8xh" Jul 26 15:38:43.165205 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:43.165092 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/76cd20bd-3a66-40ad-99c0-fefa173500d6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5q8xh\" (UID: \"76cd20bd-3a66-40ad-99c0-fefa173500d6\") " pod="openshift-insights/insights-runtime-extractor-5q8xh" Jul 26 15:38:43.165205 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:43.165117 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/76cd20bd-3a66-40ad-99c0-fefa173500d6-crio-socket\") pod \"insights-runtime-extractor-5q8xh\" (UID: \"76cd20bd-3a66-40ad-99c0-fefa173500d6\") " pod="openshift-insights/insights-runtime-extractor-5q8xh" Jul 26 15:38:43.165671 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:43.165483 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvljz\" (UniqueName: \"kubernetes.io/projected/76cd20bd-3a66-40ad-99c0-fefa173500d6-kube-api-access-cvljz\") pod \"insights-runtime-extractor-5q8xh\" (UID: \"76cd20bd-3a66-40ad-99c0-fefa173500d6\") " pod="openshift-insights/insights-runtime-extractor-5q8xh" Jul 26 15:38:43.266340 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:43.266264 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/76cd20bd-3a66-40ad-99c0-fefa173500d6-data-volume\") pod \"insights-runtime-extractor-5q8xh\" (UID: \"76cd20bd-3a66-40ad-99c0-fefa173500d6\") " pod="openshift-insights/insights-runtime-extractor-5q8xh" Jul 26 15:38:43.266340 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:43.266301 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/76cd20bd-3a66-40ad-99c0-fefa173500d6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5q8xh\" (UID: \"76cd20bd-3a66-40ad-99c0-fefa173500d6\") " pod="openshift-insights/insights-runtime-extractor-5q8xh" Jul 26 15:38:43.266340 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:43.266326 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/76cd20bd-3a66-40ad-99c0-fefa173500d6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5q8xh\" (UID: \"76cd20bd-3a66-40ad-99c0-fefa173500d6\") " pod="openshift-insights/insights-runtime-extractor-5q8xh" Jul 26 15:38:43.266340 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:43.266342 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/76cd20bd-3a66-40ad-99c0-fefa173500d6-crio-socket\") pod \"insights-runtime-extractor-5q8xh\" (UID: \"76cd20bd-3a66-40ad-99c0-fefa173500d6\") " pod="openshift-insights/insights-runtime-extractor-5q8xh" Jul 26 15:38:43.266703 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:43.266427 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/76cd20bd-3a66-40ad-99c0-fefa173500d6-crio-socket\") pod \"insights-runtime-extractor-5q8xh\" (UID: \"76cd20bd-3a66-40ad-99c0-fefa173500d6\") " pod="openshift-insights/insights-runtime-extractor-5q8xh" Jul 26 15:38:43.266703 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:43.266426 2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 26 15:38:43.266703 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:43.266462 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 26 15:38:43.266703 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:43.266481 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvljz\" (UniqueName: \"kubernetes.io/projected/76cd20bd-3a66-40ad-99c0-fefa173500d6-kube-api-access-cvljz\") pod \"insights-runtime-extractor-5q8xh\" (UID: \"76cd20bd-3a66-40ad-99c0-fefa173500d6\") " pod="openshift-insights/insights-runtime-extractor-5q8xh" Jul 26 15:38:43.266703 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:43.266521 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/76cd20bd-3a66-40ad-99c0-fefa173500d6-kube-rbac-proxy-cm podName:76cd20bd-3a66-40ad-99c0-fefa173500d6 nodeName:}" failed. No retries permitted until 2026-07-26 15:38:43.766505285 +0000 UTC m=+46.505858572 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/76cd20bd-3a66-40ad-99c0-fefa173500d6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5q8xh" (UID: "76cd20bd-3a66-40ad-99c0-fefa173500d6") : configmap "kube-rbac-proxy" not found Jul 26 15:38:43.266703 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:43.266587 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/76cd20bd-3a66-40ad-99c0-fefa173500d6-data-volume\") pod \"insights-runtime-extractor-5q8xh\" (UID: \"76cd20bd-3a66-40ad-99c0-fefa173500d6\") " pod="openshift-insights/insights-runtime-extractor-5q8xh" Jul 26 15:38:43.266703 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:43.266617 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76cd20bd-3a66-40ad-99c0-fefa173500d6-insights-runtime-extractor-tls podName:76cd20bd-3a66-40ad-99c0-fefa173500d6 nodeName:}" failed. No retries permitted until 2026-07-26 15:38:43.766610245 +0000 UTC m=+46.505963534 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/76cd20bd-3a66-40ad-99c0-fefa173500d6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5q8xh" (UID: "76cd20bd-3a66-40ad-99c0-fefa173500d6") : secret "insights-runtime-extractor-tls" not found Jul 26 15:38:43.279045 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:43.279017 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvljz\" (UniqueName: \"kubernetes.io/projected/76cd20bd-3a66-40ad-99c0-fefa173500d6-kube-api-access-cvljz\") pod \"insights-runtime-extractor-5q8xh\" (UID: \"76cd20bd-3a66-40ad-99c0-fefa173500d6\") " pod="openshift-insights/insights-runtime-extractor-5q8xh" Jul 26 15:38:43.449821 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:43.449784 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:38:43.510701 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:43.510669 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:38:43.569188 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:43.569152 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:38:43.629846 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:43.629818 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:38:43.698529 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:43.698492 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:38:43.770074 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:43.770040 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/76cd20bd-3a66-40ad-99c0-fefa173500d6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5q8xh\" (UID: \"76cd20bd-3a66-40ad-99c0-fefa173500d6\") " pod="openshift-insights/insights-runtime-extractor-5q8xh" Jul 26 15:38:43.770580 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:43.770090 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/76cd20bd-3a66-40ad-99c0-fefa173500d6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5q8xh\" (UID: \"76cd20bd-3a66-40ad-99c0-fefa173500d6\") " pod="openshift-insights/insights-runtime-extractor-5q8xh" Jul 26 15:38:43.770580 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:43.770187 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 26 15:38:43.770580 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:43.770232 2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 26 15:38:43.770580 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:43.770250 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76cd20bd-3a66-40ad-99c0-fefa173500d6-insights-runtime-extractor-tls podName:76cd20bd-3a66-40ad-99c0-fefa173500d6 nodeName:}" failed. No retries permitted until 2026-07-26 15:38:44.770235677 +0000 UTC m=+47.509588966 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/76cd20bd-3a66-40ad-99c0-fefa173500d6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5q8xh" (UID: "76cd20bd-3a66-40ad-99c0-fefa173500d6") : secret "insights-runtime-extractor-tls" not found Jul 26 15:38:43.770580 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:43.770298 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/76cd20bd-3a66-40ad-99c0-fefa173500d6-kube-rbac-proxy-cm podName:76cd20bd-3a66-40ad-99c0-fefa173500d6 nodeName:}" failed. No retries permitted until 2026-07-26 15:38:44.770283491 +0000 UTC m=+47.509636780 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/76cd20bd-3a66-40ad-99c0-fefa173500d6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5q8xh" (UID: "76cd20bd-3a66-40ad-99c0-fefa173500d6") : configmap "kube-rbac-proxy" not found Jul 26 15:38:43.799444 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:43.799388 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:38:44.131024 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:44.130993 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-8wjqp_b6b8080d-0cee-4d0c-a05c-3e2f1b672a54/console-operator/1.log" Jul 26 15:38:44.131350 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:44.131332 2577 scope.go:117] "RemoveContainer" containerID="8d03b0ae032be9047d7c31246b62c2a28aba80a28b9fc6f8d388bc65d45d825e" Jul 26 15:38:44.131529 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:44.131509 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-8wjqp_openshift-console-operator(b6b8080d-0cee-4d0c-a05c-3e2f1b672a54)\"" pod="openshift-console-operator/console-operator-584dcb6b54-8wjqp" podUID="b6b8080d-0cee-4d0c-a05c-3e2f1b672a54" Jul 26 15:38:44.780140 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:44.780102 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/76cd20bd-3a66-40ad-99c0-fefa173500d6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5q8xh\" (UID: \"76cd20bd-3a66-40ad-99c0-fefa173500d6\") " pod="openshift-insights/insights-runtime-extractor-5q8xh" Jul 26 15:38:44.780140 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:44.780145 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/76cd20bd-3a66-40ad-99c0-fefa173500d6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5q8xh\" (UID: \"76cd20bd-3a66-40ad-99c0-fefa173500d6\") " pod="openshift-insights/insights-runtime-extractor-5q8xh" Jul 26 15:38:44.780595 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:44.780263 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 26 15:38:44.780595 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:44.780337 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76cd20bd-3a66-40ad-99c0-fefa173500d6-insights-runtime-extractor-tls podName:76cd20bd-3a66-40ad-99c0-fefa173500d6 nodeName:}" failed. No retries permitted until 2026-07-26 15:38:46.7803224 +0000 UTC m=+49.519675689 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/76cd20bd-3a66-40ad-99c0-fefa173500d6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5q8xh" (UID: "76cd20bd-3a66-40ad-99c0-fefa173500d6") : secret "insights-runtime-extractor-tls" not found Jul 26 15:38:44.780595 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:44.780274 2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 26 15:38:44.780595 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:44.780432 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/76cd20bd-3a66-40ad-99c0-fefa173500d6-kube-rbac-proxy-cm podName:76cd20bd-3a66-40ad-99c0-fefa173500d6 nodeName:}" failed. No retries permitted until 2026-07-26 15:38:46.780416876 +0000 UTC m=+49.519770187 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/76cd20bd-3a66-40ad-99c0-fefa173500d6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5q8xh" (UID: "76cd20bd-3a66-40ad-99c0-fefa173500d6") : configmap "kube-rbac-proxy" not found Jul 26 15:38:44.874533 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:44.874506 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-v4mgw_356a056a-d4d5-48ed-a10f-928b36becc81/dns-node-resolver/0.log" Jul 26 15:38:45.300674 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:45.300633 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:38:45.355869 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:45.355837 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:38:45.876410 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:45.876373 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-mfwqb_098dd2e5-9540-425f-ba4e-71ea0b064be7/node-ca/0.log" Jul 26 15:38:46.117620 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:46.117592 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:38:46.475302 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:46.475270 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-5xf7l_fceea9e2-8f78-4c66-a5b4-3ed7666a93af/serve-healthcheck-canary/0.log" Jul 26 15:38:46.797742 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:46.797707 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/76cd20bd-3a66-40ad-99c0-fefa173500d6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5q8xh\" (UID: \"76cd20bd-3a66-40ad-99c0-fefa173500d6\") " pod="openshift-insights/insights-runtime-extractor-5q8xh" Jul 26 15:38:46.797901 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:46.797750 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/76cd20bd-3a66-40ad-99c0-fefa173500d6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5q8xh\" (UID: \"76cd20bd-3a66-40ad-99c0-fefa173500d6\") " pod="openshift-insights/insights-runtime-extractor-5q8xh" Jul 26 15:38:46.797901 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:46.797826 2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 26 15:38:46.797901 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:46.797861 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 26 15:38:46.797901 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:46.797882 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/76cd20bd-3a66-40ad-99c0-fefa173500d6-kube-rbac-proxy-cm podName:76cd20bd-3a66-40ad-99c0-fefa173500d6 nodeName:}" failed. No retries permitted until 2026-07-26 15:38:50.797868251 +0000 UTC m=+53.537221539 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/76cd20bd-3a66-40ad-99c0-fefa173500d6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5q8xh" (UID: "76cd20bd-3a66-40ad-99c0-fefa173500d6") : configmap "kube-rbac-proxy" not found Jul 26 15:38:46.798084 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:46.797922 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76cd20bd-3a66-40ad-99c0-fefa173500d6-insights-runtime-extractor-tls podName:76cd20bd-3a66-40ad-99c0-fefa173500d6 nodeName:}" failed. No retries permitted until 2026-07-26 15:38:50.7979086 +0000 UTC m=+53.537261888 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/76cd20bd-3a66-40ad-99c0-fefa173500d6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5q8xh" (UID: "76cd20bd-3a66-40ad-99c0-fefa173500d6") : secret "insights-runtime-extractor-tls" not found Jul 26 15:38:47.669747 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:47.669713 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:38:48.010477 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:48.010361 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/60d326a6-77da-4f72-965f-880f5c94f2b7-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-2q2tq\" (UID: \"60d326a6-77da-4f72-965f-880f5c94f2b7\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-2q2tq" Jul 26 15:38:48.010624 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:48.010516 2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 26 15:38:48.010624 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:48.010583 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60d326a6-77da-4f72-965f-880f5c94f2b7-cluster-monitoring-operator-tls podName:60d326a6-77da-4f72-965f-880f5c94f2b7 nodeName:}" failed. No retries permitted until 2026-07-26 15:39:04.010568017 +0000 UTC m=+66.749921305 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/60d326a6-77da-4f72-965f-880f5c94f2b7-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-2q2tq" (UID: "60d326a6-77da-4f72-965f-880f5c94f2b7") : secret "cluster-monitoring-operator-tls" not found Jul 26 15:38:48.111741 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:48.111698 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bf408cf7-787a-4839-ab2f-8220f1696bd6-registry-tls\") pod \"image-registry-5ccd8b79b5-rnb96\" (UID: \"bf408cf7-787a-4839-ab2f-8220f1696bd6\") " pod="openshift-image-registry/image-registry-5ccd8b79b5-rnb96" Jul 26 15:38:48.111927 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:48.111856 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 26 15:38:48.111927 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:48.111877 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5ccd8b79b5-rnb96: secret "image-registry-tls" not found Jul 26 15:38:48.111927 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:48.111857 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9af43da1-c25b-4687-820f-6a95afe88786-service-ca-bundle\") pod \"router-default-5b79d74577-h8slk\" (UID: \"9af43da1-c25b-4687-820f-6a95afe88786\") " pod="openshift-ingress/router-default-5b79d74577-h8slk" Jul 26 15:38:48.112050 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:48.111929 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bf408cf7-787a-4839-ab2f-8220f1696bd6-registry-tls podName:bf408cf7-787a-4839-ab2f-8220f1696bd6 nodeName:}" failed. No retries permitted until 2026-07-26 15:39:04.111912426 +0000 UTC m=+66.851265718 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bf408cf7-787a-4839-ab2f-8220f1696bd6-registry-tls") pod "image-registry-5ccd8b79b5-rnb96" (UID: "bf408cf7-787a-4839-ab2f-8220f1696bd6") : secret "image-registry-tls" not found Jul 26 15:38:48.112050 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:48.111964 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9af43da1-c25b-4687-820f-6a95afe88786-service-ca-bundle podName:9af43da1-c25b-4687-820f-6a95afe88786 nodeName:}" failed. No retries permitted until 2026-07-26 15:39:04.111946688 +0000 UTC m=+66.851299979 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/9af43da1-c25b-4687-820f-6a95afe88786-service-ca-bundle") pod "router-default-5b79d74577-h8slk" (UID: "9af43da1-c25b-4687-820f-6a95afe88786") : configmap references non-existent config key: service-ca.crt Jul 26 15:38:48.112050 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:48.112019 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9af43da1-c25b-4687-820f-6a95afe88786-metrics-certs\") pod \"router-default-5b79d74577-h8slk\" (UID: \"9af43da1-c25b-4687-820f-6a95afe88786\") " pod="openshift-ingress/router-default-5b79d74577-h8slk" Jul 26 15:38:48.112208 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:48.112084 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b9353822-c6c4-47f1-8a41-ecdc92669f5f-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-rrqbw\" (UID: \"b9353822-c6c4-47f1-8a41-ecdc92669f5f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-rrqbw" Jul 26 15:38:48.112208 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:48.112159 2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 26 15:38:48.112288 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:48.112217 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9af43da1-c25b-4687-820f-6a95afe88786-metrics-certs podName:9af43da1-c25b-4687-820f-6a95afe88786 nodeName:}" failed. No retries permitted until 2026-07-26 15:39:04.112205593 +0000 UTC m=+66.851558891 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9af43da1-c25b-4687-820f-6a95afe88786-metrics-certs") pod "router-default-5b79d74577-h8slk" (UID: "9af43da1-c25b-4687-820f-6a95afe88786") : secret "router-metrics-certs-default" not found Jul 26 15:38:48.112288 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:48.112165 2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 26 15:38:48.112288 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:48.112247 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9353822-c6c4-47f1-8a41-ecdc92669f5f-samples-operator-tls podName:b9353822-c6c4-47f1-8a41-ecdc92669f5f nodeName:}" failed. No retries permitted until 2026-07-26 15:39:04.112242017 +0000 UTC m=+66.851595305 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b9353822-c6c4-47f1-8a41-ecdc92669f5f-samples-operator-tls") pod "cluster-samples-operator-6748489656-rrqbw" (UID: "b9353822-c6c4-47f1-8a41-ecdc92669f5f") : secret "samples-operator-tls" not found Jul 26 15:38:48.212830 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:48.212794 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/839598d1-d772-47cf-a460-5e88affbac0e-metrics-tls\") pod \"dns-default-65h7n\" (UID: \"839598d1-d772-47cf-a460-5e88affbac0e\") " pod="openshift-dns/dns-default-65h7n" Jul 26 15:38:48.213027 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:48.212931 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 26 15:38:48.213091 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:48.213031 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/839598d1-d772-47cf-a460-5e88affbac0e-metrics-tls podName:839598d1-d772-47cf-a460-5e88affbac0e nodeName:}" failed. No retries permitted until 2026-07-26 15:39:04.213010073 +0000 UTC m=+66.952363364 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/839598d1-d772-47cf-a460-5e88affbac0e-metrics-tls") pod "dns-default-65h7n" (UID: "839598d1-d772-47cf-a460-5e88affbac0e") : secret "dns-default-metrics-tls" not found Jul 26 15:38:49.215864 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:49.215829 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:38:49.735086 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:49.735018 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:38:49.909540 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:49.909486 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:38:50.002005 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:50.001915 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:38:50.065692 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:50.065642 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:38:50.333530 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:50.333495 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3d7ef445-5c4a-4c4e-92f4-2da8ee29b868-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-5pdf8\" (UID: \"3d7ef445-5c4a-4c4e-92f4-2da8ee29b868\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-5pdf8" Jul 26 15:38:50.333918 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:50.333655 2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 26 15:38:50.333918 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:50.333722 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d7ef445-5c4a-4c4e-92f4-2da8ee29b868-networking-console-plugin-cert podName:3d7ef445-5c4a-4c4e-92f4-2da8ee29b868 nodeName:}" failed. No retries permitted until 2026-07-26 15:39:06.333707015 +0000 UTC m=+69.073060303 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/3d7ef445-5c4a-4c4e-92f4-2da8ee29b868-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-5pdf8" (UID: "3d7ef445-5c4a-4c4e-92f4-2da8ee29b868") : secret "networking-console-plugin-cert" not found Jul 26 15:38:50.435689 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:50.435652 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:38:50.838098 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:50.838055 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/76cd20bd-3a66-40ad-99c0-fefa173500d6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5q8xh\" (UID: \"76cd20bd-3a66-40ad-99c0-fefa173500d6\") " pod="openshift-insights/insights-runtime-extractor-5q8xh" Jul 26 15:38:50.838296 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:50.838104 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/76cd20bd-3a66-40ad-99c0-fefa173500d6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5q8xh\" (UID: \"76cd20bd-3a66-40ad-99c0-fefa173500d6\") " pod="openshift-insights/insights-runtime-extractor-5q8xh" Jul 26 15:38:50.838296 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:50.838236 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 26 15:38:50.838464 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:38:50.838313 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76cd20bd-3a66-40ad-99c0-fefa173500d6-insights-runtime-extractor-tls podName:76cd20bd-3a66-40ad-99c0-fefa173500d6 nodeName:}" failed. No retries permitted until 2026-07-26 15:38:58.838293863 +0000 UTC m=+61.577647151 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/76cd20bd-3a66-40ad-99c0-fefa173500d6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5q8xh" (UID: "76cd20bd-3a66-40ad-99c0-fefa173500d6") : secret "insights-runtime-extractor-tls" not found Jul 26 15:38:50.838644 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:50.838626 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/76cd20bd-3a66-40ad-99c0-fefa173500d6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5q8xh\" (UID: \"76cd20bd-3a66-40ad-99c0-fefa173500d6\") " pod="openshift-insights/insights-runtime-extractor-5q8xh" Jul 26 15:38:51.430873 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:51.430829 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:38:51.633664 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:51.633627 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:38:52.032350 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:52.032304 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:38:52.236974 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:52.236938 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:38:52.580139 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:52.580103 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:38:52.677831 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:52.677802 2577 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-584dcb6b54-8wjqp" Jul 26 15:38:52.677960 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:52.677852 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-8wjqp" Jul 26 15:38:52.678222 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:52.678208 2577 scope.go:117] "RemoveContainer" containerID="8d03b0ae032be9047d7c31246b62c2a28aba80a28b9fc6f8d388bc65d45d825e" Jul 26 15:38:52.830314 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:52.830208 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:38:53.034711 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:53.034676 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:38:53.164588 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:53.164517 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-8wjqp_b6b8080d-0cee-4d0c-a05c-3e2f1b672a54/console-operator/1.log" Jul 26 15:38:53.164588 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:53.164570 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-8wjqp" event={"ID":"b6b8080d-0cee-4d0c-a05c-3e2f1b672a54","Type":"ContainerStarted","Data":"8a3be216becd953bd4d974be8b7ad358b8d01948c04b4ade7dfd6dddc72fbd3f"} Jul 26 15:38:53.164867 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:53.164844 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-8wjqp" Jul 26 15:38:53.181333 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:53.181269 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-584dcb6b54-8wjqp" podStartSLOduration=24.532417611 podStartE2EDuration="32.181252229s" totalImagesPullingTime="7.648834618s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:38:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:38:53.180730254 +0000 UTC m=+55.920083587" watchObservedRunningTime="2026-07-26 15:38:53.181252229 +0000 UTC m=+55.920605538" Jul 26 15:38:53.233620 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:53.233580 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:38:53.366359 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:53.366330 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-584dcb6b54-8wjqp" Jul 26 15:38:53.433211 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:53.433119 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:38:53.538441 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:53.538392 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:38:53.640503 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:53.640458 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:38:53.693757 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:53.693667 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:38:53.745765 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:53.745731 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:38:53.848975 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:53.848939 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:38:54.023946 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:54.023917 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:38:54.126247 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:54.126211 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:38:54.233390 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:54.233344 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:38:54.430844 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:54.430751 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:38:54.488001 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:54.487967 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:38:54.642378 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:54.642344 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:38:54.743278 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:54.743186 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:38:54.843865 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:54.843817 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:38:55.228589 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:55.228559 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:38:55.283894 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:55.283863 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:38:55.429678 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:55.429643 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:38:55.632983 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:55.632944 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:38:55.833337 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:55.833301 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:38:56.023973 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:56.023945 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:38:56.234970 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:56.234933 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:38:56.332839 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:56.332753 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:38:56.832260 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:56.832222 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:38:57.232103 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:57.232015 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:38:57.633720 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:57.633680 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:38:57.829363 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:57.829332 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:38:58.046365 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:58.046334 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ztlp6" Jul 26 15:38:58.232437 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:58.232385 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:38:58.832412 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:58.832361 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:38:58.913606 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:58.913569 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/76cd20bd-3a66-40ad-99c0-fefa173500d6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5q8xh\" (UID: \"76cd20bd-3a66-40ad-99c0-fefa173500d6\") " pod="openshift-insights/insights-runtime-extractor-5q8xh" Jul 26 15:38:58.915865 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:58.915835 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/76cd20bd-3a66-40ad-99c0-fefa173500d6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5q8xh\" (UID: \"76cd20bd-3a66-40ad-99c0-fefa173500d6\") " pod="openshift-insights/insights-runtime-extractor-5q8xh" Jul 26 15:38:59.001855 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:59.001826 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-fwcq5\"" Jul 26 15:38:59.009705 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:59.009685 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-5q8xh" Jul 26 15:38:59.133953 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:59.133922 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-5q8xh"] Jul 26 15:38:59.137133 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:38:59.137104 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76cd20bd_3a66_40ad_99c0_fefa173500d6.slice/crio-71a5de0076c34d6c90e3094aac4c6c4696159cfea9deb3479101045d434f1e75 WatchSource:0}: Error finding container 71a5de0076c34d6c90e3094aac4c6c4696159cfea9deb3479101045d434f1e75: Status 404 returned error can't find the container with id 71a5de0076c34d6c90e3094aac4c6c4696159cfea9deb3479101045d434f1e75 Jul 26 15:38:59.182775 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:59.182747 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5q8xh" event={"ID":"76cd20bd-3a66-40ad-99c0-fefa173500d6","Type":"ContainerStarted","Data":"71a5de0076c34d6c90e3094aac4c6c4696159cfea9deb3479101045d434f1e75"} Jul 26 15:38:59.236801 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:59.236766 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:38:59.631974 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:38:59.631940 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:39:00.034347 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:00.034309 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:39:00.226456 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:00.226420 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:39:00.621430 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:00.621370 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:39:00.722754 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:00.722713 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:39:00.835386 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:00.835350 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:39:02.449979 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:02.449945 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fztmw\" (UniqueName: \"kubernetes.io/projected/f3e728c9-4710-4f6c-977f-caf52a2bfcaf-kube-api-access-fztmw\") pod \"network-check-target-d2nnz\" (UID: \"f3e728c9-4710-4f6c-977f-caf52a2bfcaf\") " pod="openshift-network-diagnostics/network-check-target-d2nnz" Jul 26 15:39:02.450429 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:02.450004 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f6bd9013-635a-4af8-b0ee-621a280a459a-original-pull-secret\") pod \"global-pull-secret-syncer-l56b4\" (UID: \"f6bd9013-635a-4af8-b0ee-621a280a459a\") " pod="kube-system/global-pull-secret-syncer-l56b4" Jul 26 15:39:02.452618 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:02.452590 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 26 15:39:02.452818 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:02.452795 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fztmw\" (UniqueName: \"kubernetes.io/projected/f3e728c9-4710-4f6c-977f-caf52a2bfcaf-kube-api-access-fztmw\") pod \"network-check-target-d2nnz\" (UID: \"f3e728c9-4710-4f6c-977f-caf52a2bfcaf\") " pod="openshift-network-diagnostics/network-check-target-d2nnz" Jul 26 15:39:02.463065 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:02.463038 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f6bd9013-635a-4af8-b0ee-621a280a459a-original-pull-secret\") pod \"global-pull-secret-syncer-l56b4\" (UID: \"f6bd9013-635a-4af8-b0ee-621a280a459a\") " pod="kube-system/global-pull-secret-syncer-l56b4" Jul 26 15:39:02.501391 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:02.501358 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l56b4" Jul 26 15:39:02.510673 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:02.510644 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-pkvmj\"" Jul 26 15:39:02.519496 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:02.519469 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d2nnz" Jul 26 15:39:02.551380 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:02.551341 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5f442b1-a918-43a5-9870-f2183fe5085a-metrics-certs\") pod \"network-metrics-daemon-kgk6w\" (UID: \"b5f442b1-a918-43a5-9870-f2183fe5085a\") " pod="openshift-multus/network-metrics-daemon-kgk6w" Jul 26 15:39:02.553786 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:02.553761 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 26 15:39:02.564263 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:02.564237 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5f442b1-a918-43a5-9870-f2183fe5085a-metrics-certs\") pod \"network-metrics-daemon-kgk6w\" (UID: \"b5f442b1-a918-43a5-9870-f2183fe5085a\") " pod="openshift-multus/network-metrics-daemon-kgk6w" Jul 26 15:39:02.797269 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:02.797235 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8j62j\"" Jul 26 15:39:02.805809 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:02.805783 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kgk6w" Jul 26 15:39:03.984726 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:03.984653 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-l56b4"] Jul 26 15:39:03.990904 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:39:03.990687 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6bd9013_635a_4af8_b0ee_621a280a459a.slice/crio-ce431daff1f7597e9b7cf5f2d21aa92591c4c221cf913efb351ab421a39d3380 WatchSource:0}: Error finding container ce431daff1f7597e9b7cf5f2d21aa92591c4c221cf913efb351ab421a39d3380: Status 404 returned error can't find the container with id ce431daff1f7597e9b7cf5f2d21aa92591c4c221cf913efb351ab421a39d3380 Jul 26 15:39:04.000790 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:04.000755 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-kgk6w"] Jul 26 15:39:04.004496 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:39:04.004376 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5f442b1_a918_43a5_9870_f2183fe5085a.slice/crio-2e23f16880fea4acafe2f191e9c38644344159005c80f4b685098a1e26fe161e WatchSource:0}: Error finding container 2e23f16880fea4acafe2f191e9c38644344159005c80f4b685098a1e26fe161e: Status 404 returned error can't find the container with id 2e23f16880fea4acafe2f191e9c38644344159005c80f4b685098a1e26fe161e Jul 26 15:39:04.022205 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:04.022174 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-d2nnz"] Jul 26 15:39:04.026726 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:39:04.026692 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3e728c9_4710_4f6c_977f_caf52a2bfcaf.slice/crio-f42ad7b51599ff4d2e3027a0c06dc4125d2b35119100fb3732bd7be4710a816f WatchSource:0}: Error finding container f42ad7b51599ff4d2e3027a0c06dc4125d2b35119100fb3732bd7be4710a816f: Status 404 returned error can't find the container with id f42ad7b51599ff4d2e3027a0c06dc4125d2b35119100fb3732bd7be4710a816f Jul 26 15:39:04.067020 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:04.066996 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/60d326a6-77da-4f72-965f-880f5c94f2b7-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-2q2tq\" (UID: \"60d326a6-77da-4f72-965f-880f5c94f2b7\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-2q2tq" Jul 26 15:39:04.069178 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:04.069158 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/60d326a6-77da-4f72-965f-880f5c94f2b7-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-2q2tq\" (UID: \"60d326a6-77da-4f72-965f-880f5c94f2b7\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-2q2tq" Jul 26 15:39:04.167776 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:04.167687 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bf408cf7-787a-4839-ab2f-8220f1696bd6-registry-tls\") pod \"image-registry-5ccd8b79b5-rnb96\" (UID: \"bf408cf7-787a-4839-ab2f-8220f1696bd6\") " pod="openshift-image-registry/image-registry-5ccd8b79b5-rnb96" Jul 26 15:39:04.167776 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:04.167739 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9af43da1-c25b-4687-820f-6a95afe88786-service-ca-bundle\") pod \"router-default-5b79d74577-h8slk\" (UID: \"9af43da1-c25b-4687-820f-6a95afe88786\") " pod="openshift-ingress/router-default-5b79d74577-h8slk" Jul 26 15:39:04.167776 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:04.167758 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9af43da1-c25b-4687-820f-6a95afe88786-metrics-certs\") pod \"router-default-5b79d74577-h8slk\" (UID: \"9af43da1-c25b-4687-820f-6a95afe88786\") " pod="openshift-ingress/router-default-5b79d74577-h8slk" Jul 26 15:39:04.168047 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:04.167799 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b9353822-c6c4-47f1-8a41-ecdc92669f5f-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-rrqbw\" (UID: \"b9353822-c6c4-47f1-8a41-ecdc92669f5f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-rrqbw" Jul 26 15:39:04.169954 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:04.169569 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9af43da1-c25b-4687-820f-6a95afe88786-service-ca-bundle\") pod \"router-default-5b79d74577-h8slk\" (UID: \"9af43da1-c25b-4687-820f-6a95afe88786\") " pod="openshift-ingress/router-default-5b79d74577-h8slk" Jul 26 15:39:04.173252 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:04.173229 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bf408cf7-787a-4839-ab2f-8220f1696bd6-registry-tls\") pod \"image-registry-5ccd8b79b5-rnb96\" (UID: \"bf408cf7-787a-4839-ab2f-8220f1696bd6\") " pod="openshift-image-registry/image-registry-5ccd8b79b5-rnb96" Jul 26 15:39:04.173378 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:04.173292 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b9353822-c6c4-47f1-8a41-ecdc92669f5f-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-rrqbw\" (UID: \"b9353822-c6c4-47f1-8a41-ecdc92669f5f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-rrqbw" Jul 26 15:39:04.174158 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:04.174142 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9af43da1-c25b-4687-820f-6a95afe88786-metrics-certs\") pod \"router-default-5b79d74577-h8slk\" (UID: \"9af43da1-c25b-4687-820f-6a95afe88786\") " pod="openshift-ingress/router-default-5b79d74577-h8slk" Jul 26 15:39:04.203160 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:04.203128 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5q8xh" event={"ID":"76cd20bd-3a66-40ad-99c0-fefa173500d6","Type":"ContainerStarted","Data":"1f377e3205c5eb163946cd13046982878cbd4d88a411293fd04fe58407d83b1f"} Jul 26 15:39:04.204621 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:04.204595 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-d2nnz" event={"ID":"f3e728c9-4710-4f6c-977f-caf52a2bfcaf","Type":"ContainerStarted","Data":"abaafdbeab5206c7289283ce516af9e4fb88939aa5d1d3546d3497e18239f85b"} Jul 26 15:39:04.204773 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:04.204636 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-d2nnz" event={"ID":"f3e728c9-4710-4f6c-977f-caf52a2bfcaf","Type":"ContainerStarted","Data":"f42ad7b51599ff4d2e3027a0c06dc4125d2b35119100fb3732bd7be4710a816f"} Jul 26 15:39:04.204773 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:04.204711 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-d2nnz" Jul 26 15:39:04.205890 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:04.205869 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-kgk6w" event={"ID":"b5f442b1-a918-43a5-9870-f2183fe5085a","Type":"ContainerStarted","Data":"2e23f16880fea4acafe2f191e9c38644344159005c80f4b685098a1e26fe161e"} Jul 26 15:39:04.206844 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:04.206826 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-l56b4" event={"ID":"f6bd9013-635a-4af8-b0ee-621a280a459a","Type":"ContainerStarted","Data":"ce431daff1f7597e9b7cf5f2d21aa92591c4c221cf913efb351ab421a39d3380"} Jul 26 15:39:04.219320 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:04.219282 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-d2nnz" podStartSLOduration=67.219270953 podStartE2EDuration="1m7.219270953s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:37:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:39:04.217646067 +0000 UTC m=+66.956999396" watchObservedRunningTime="2026-07-26 15:39:04.219270953 +0000 UTC m=+66.958624241" Jul 26 15:39:04.268988 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:04.268956 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/839598d1-d772-47cf-a460-5e88affbac0e-metrics-tls\") pod \"dns-default-65h7n\" (UID: \"839598d1-d772-47cf-a460-5e88affbac0e\") " pod="openshift-dns/dns-default-65h7n" Jul 26 15:39:04.271200 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:04.271174 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/839598d1-d772-47cf-a460-5e88affbac0e-metrics-tls\") pod \"dns-default-65h7n\" (UID: \"839598d1-d772-47cf-a460-5e88affbac0e\") " pod="openshift-dns/dns-default-65h7n" Jul 26 15:39:04.283340 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:04.283315 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-x7hq5\"" Jul 26 15:39:04.291184 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:04.291167 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-2q2tq" Jul 26 15:39:04.322680 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:04.322648 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-wk5h4\"" Jul 26 15:39:04.330437 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:04.330386 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5ccd8b79b5-rnb96" Jul 26 15:39:04.342195 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:04.342174 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-fmt2r\"" Jul 26 15:39:04.349993 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:04.349919 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-rrqbw" Jul 26 15:39:04.367966 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:04.367941 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-lbb85\"" Jul 26 15:39:04.376150 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:04.376051 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5b79d74577-h8slk" Jul 26 15:39:04.400934 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:04.400714 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-t6bwt\"" Jul 26 15:39:04.408980 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:04.408722 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-65h7n" Jul 26 15:39:04.442086 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:04.442056 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-2q2tq"] Jul 26 15:39:04.448113 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:39:04.448028 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60d326a6_77da_4f72_965f_880f5c94f2b7.slice/crio-df2bbd29bbd6f3fa2eaf7b7525289b0f708927879e051deddf8a919ffd8b2f3a WatchSource:0}: Error finding container df2bbd29bbd6f3fa2eaf7b7525289b0f708927879e051deddf8a919ffd8b2f3a: Status 404 returned error can't find the container with id df2bbd29bbd6f3fa2eaf7b7525289b0f708927879e051deddf8a919ffd8b2f3a Jul 26 15:39:04.515067 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:04.514792 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5ccd8b79b5-rnb96"] Jul 26 15:39:04.524997 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:39:04.524960 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf408cf7_787a_4839_ab2f_8220f1696bd6.slice/crio-a1e44cca231a9629a11051332e5876857c7d842262b8babc814a4b0f80fbcaa4 WatchSource:0}: Error finding container a1e44cca231a9629a11051332e5876857c7d842262b8babc814a4b0f80fbcaa4: Status 404 returned error can't find the container with id a1e44cca231a9629a11051332e5876857c7d842262b8babc814a4b0f80fbcaa4 Jul 26 15:39:04.550442 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:04.550414 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-rrqbw"] Jul 26 15:39:04.601609 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:04.601546 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5b79d74577-h8slk"] Jul 26 15:39:04.606788 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:39:04.606758 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9af43da1_c25b_4687_820f_6a95afe88786.slice/crio-27600a15f3732d20b1d22198a82e5a5558d000ad1ba44e9e9cd92f7a8496a71a WatchSource:0}: Error finding container 27600a15f3732d20b1d22198a82e5a5558d000ad1ba44e9e9cd92f7a8496a71a: Status 404 returned error can't find the container with id 27600a15f3732d20b1d22198a82e5a5558d000ad1ba44e9e9cd92f7a8496a71a Jul 26 15:39:04.629952 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:04.629924 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-65h7n"] Jul 26 15:39:04.643274 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:39:04.643242 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod839598d1_d772_47cf_a460_5e88affbac0e.slice/crio-42ce36da57f1b47cf5a4d402e45512b2093314e027a8565d0deba3260f6d81aa WatchSource:0}: Error finding container 42ce36da57f1b47cf5a4d402e45512b2093314e027a8565d0deba3260f6d81aa: Status 404 returned error can't find the container with id 42ce36da57f1b47cf5a4d402e45512b2093314e027a8565d0deba3260f6d81aa Jul 26 15:39:05.214454 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:05.213658 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5b79d74577-h8slk" event={"ID":"9af43da1-c25b-4687-820f-6a95afe88786","Type":"ContainerStarted","Data":"9903f3d5d1a65bc9847071312ddad1eae2ec78a3ccbfce7d4596acff650b7085"} Jul 26 15:39:05.214454 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:05.213719 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5b79d74577-h8slk" event={"ID":"9af43da1-c25b-4687-820f-6a95afe88786","Type":"ContainerStarted","Data":"27600a15f3732d20b1d22198a82e5a5558d000ad1ba44e9e9cd92f7a8496a71a"} Jul 26 15:39:05.220203 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:05.219597 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5ccd8b79b5-rnb96" event={"ID":"bf408cf7-787a-4839-ab2f-8220f1696bd6","Type":"ContainerStarted","Data":"9356871b4716b2377f6b6b8a69668e9e40e2abb18da39859f2ec0e1b7a2e0c91"} Jul 26 15:39:05.220203 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:05.219683 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5ccd8b79b5-rnb96" event={"ID":"bf408cf7-787a-4839-ab2f-8220f1696bd6","Type":"ContainerStarted","Data":"a1e44cca231a9629a11051332e5876857c7d842262b8babc814a4b0f80fbcaa4"} Jul 26 15:39:05.220203 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:05.219912 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5ccd8b79b5-rnb96" Jul 26 15:39:05.223308 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:05.223233 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-2q2tq" event={"ID":"60d326a6-77da-4f72-965f-880f5c94f2b7","Type":"ContainerStarted","Data":"df2bbd29bbd6f3fa2eaf7b7525289b0f708927879e051deddf8a919ffd8b2f3a"} Jul 26 15:39:05.225681 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:05.225654 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-65h7n" event={"ID":"839598d1-d772-47cf-a460-5e88affbac0e","Type":"ContainerStarted","Data":"42ce36da57f1b47cf5a4d402e45512b2093314e027a8565d0deba3260f6d81aa"} Jul 26 15:39:05.227934 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:05.227835 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-rrqbw" event={"ID":"b9353822-c6c4-47f1-8a41-ecdc92669f5f","Type":"ContainerStarted","Data":"e58d8cb6736ffd6f14a319661869029395a869e53880188aa41eb656e5d1e454"} Jul 26 15:39:05.233365 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:05.233317 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-5b79d74577-h8slk" podStartSLOduration=44.233298728 podStartE2EDuration="44.233298728s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:38:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:39:05.232170138 +0000 UTC m=+67.971523472" watchObservedRunningTime="2026-07-26 15:39:05.233298728 +0000 UTC m=+67.972652038" Jul 26 15:39:05.251837 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:05.251761 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5ccd8b79b5-rnb96" podStartSLOduration=67.251742034 podStartE2EDuration="1m7.251742034s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:37:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:39:05.250734702 +0000 UTC m=+67.990088016" watchObservedRunningTime="2026-07-26 15:39:05.251742034 +0000 UTC m=+67.991095346" Jul 26 15:39:05.376579 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:05.376541 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-5b79d74577-h8slk" Jul 26 15:39:05.385042 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:05.385014 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-5b79d74577-h8slk" Jul 26 15:39:05.649902 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:05.649830 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:39:05.706739 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:05.706699 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:39:06.239715 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:06.239081 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-kgk6w" event={"ID":"b5f442b1-a918-43a5-9870-f2183fe5085a","Type":"ContainerStarted","Data":"d9ba707c9a46ef15c54d4e6570a124221b06931c6b7c6ffd0274ccdfaf55f0f6"} Jul 26 15:39:06.239715 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:06.239155 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-kgk6w" event={"ID":"b5f442b1-a918-43a5-9870-f2183fe5085a","Type":"ContainerStarted","Data":"7827ea46e54e47ae02f07d2bf30a0f215f5fe1fa5f4b464fb816bdced5bec54a"} Jul 26 15:39:06.239715 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:06.239189 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-5b79d74577-h8slk" Jul 26 15:39:06.241307 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:06.241239 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-5b79d74577-h8slk" Jul 26 15:39:06.279419 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:06.279235 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-kgk6w" podStartSLOduration=66.567112865 podStartE2EDuration="1m8.279218187s" totalImagesPullingTime="1.712105322s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:37:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:39:06.2787197 +0000 UTC m=+69.018073028" watchObservedRunningTime="2026-07-26 15:39:06.279218187 +0000 UTC m=+69.018571496" Jul 26 15:39:06.391743 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:06.390119 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3d7ef445-5c4a-4c4e-92f4-2da8ee29b868-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-5pdf8\" (UID: \"3d7ef445-5c4a-4c4e-92f4-2da8ee29b868\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-5pdf8" Jul 26 15:39:06.394441 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:06.394383 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3d7ef445-5c4a-4c4e-92f4-2da8ee29b868-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-5pdf8\" (UID: \"3d7ef445-5c4a-4c4e-92f4-2da8ee29b868\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-5pdf8" Jul 26 15:39:06.555895 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:06.555863 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-5ncdz\"" Jul 26 15:39:06.564005 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:06.563976 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-5pdf8" Jul 26 15:39:07.458151 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:07.458084 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:39:07.559918 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:07.559861 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:39:09.330690 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:09.330651 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7bb9f8c5fb-r487g"] Jul 26 15:39:09.334334 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:09.334305 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:39:09.344722 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:09.344056 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7bb9f8c5fb-r487g" Jul 26 15:39:09.344722 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:09.344384 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7bb9f8c5fb-r487g"] Jul 26 15:39:09.346692 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:09.346431 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 26 15:39:09.348868 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:09.347794 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 26 15:39:09.348868 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:09.348010 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 26 15:39:09.348868 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:09.348215 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 26 15:39:09.428707 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:09.428673 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-7c97dc4555-rt6ww"] Jul 26 15:39:09.445773 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:09.444673 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5796bccdbd-vl5xj"] Jul 26 15:39:09.445773 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:09.444904 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-rt6ww" Jul 26 15:39:09.445773 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:09.444934 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:39:09.447222 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:09.447186 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-mw62p\"" Jul 26 15:39:09.450112 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:09.449652 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 26 15:39:09.450112 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:09.449908 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 26 15:39:09.454950 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:09.454925 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-rt6ww"] Jul 26 15:39:09.455061 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:09.454969 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5796bccdbd-vl5xj"] Jul 26 15:39:09.455124 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:09.455068 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5796bccdbd-vl5xj" Jul 26 15:39:09.458832 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:09.458811 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 26 15:39:09.458976 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:09.458813 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 26 15:39:09.459201 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:09.459183 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-rr5w4\"" Jul 26 15:39:09.459299 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:09.459236 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 26 15:39:09.459599 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:09.459553 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 26 15:39:09.464703 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:09.464623 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 26 15:39:09.517976 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:09.517934 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2123f11d-8547-4520-8d6c-d73bc66afa3c-tmp\") pod \"klusterlet-addon-workmgr-7bb9f8c5fb-r487g\" (UID: \"2123f11d-8547-4520-8d6c-d73bc66afa3c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7bb9f8c5fb-r487g" Jul 26 15:39:09.518155 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:09.517992 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/2123f11d-8547-4520-8d6c-d73bc66afa3c-klusterlet-config\") pod \"klusterlet-addon-workmgr-7bb9f8c5fb-r487g\" (UID: \"2123f11d-8547-4520-8d6c-d73bc66afa3c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7bb9f8c5fb-r487g" Jul 26 15:39:09.518155 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:09.518119 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5swrm\" (UniqueName: \"kubernetes.io/projected/2123f11d-8547-4520-8d6c-d73bc66afa3c-kube-api-access-5swrm\") pod \"klusterlet-addon-workmgr-7bb9f8c5fb-r487g\" (UID: \"2123f11d-8547-4520-8d6c-d73bc66afa3c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7bb9f8c5fb-r487g" Jul 26 15:39:09.529781 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:09.529746 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:39:09.618749 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:09.618679 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szgm7\" (UniqueName: \"kubernetes.io/projected/3a06ab69-b69e-44df-b0c3-939c0d6c423e-kube-api-access-szgm7\") pod \"console-5796bccdbd-vl5xj\" (UID: \"3a06ab69-b69e-44df-b0c3-939c0d6c423e\") " pod="openshift-console/console-5796bccdbd-vl5xj" Jul 26 15:39:09.618749 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:09.618712 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3a06ab69-b69e-44df-b0c3-939c0d6c423e-service-ca\") pod \"console-5796bccdbd-vl5xj\" (UID: \"3a06ab69-b69e-44df-b0c3-939c0d6c423e\") " pod="openshift-console/console-5796bccdbd-vl5xj" Jul 26 15:39:09.618749 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:09.618739 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5swrm\" (UniqueName: \"kubernetes.io/projected/2123f11d-8547-4520-8d6c-d73bc66afa3c-kube-api-access-5swrm\") pod \"klusterlet-addon-workmgr-7bb9f8c5fb-r487g\" (UID: \"2123f11d-8547-4520-8d6c-d73bc66afa3c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7bb9f8c5fb-r487g" Jul 26 15:39:09.618955 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:09.618792 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjm7t\" (UniqueName: \"kubernetes.io/projected/3a938ae2-0e14-4c59-8347-52d26294c17f-kube-api-access-qjm7t\") pod \"downloads-7c97dc4555-rt6ww\" (UID: \"3a938ae2-0e14-4c59-8347-52d26294c17f\") " pod="openshift-console/downloads-7c97dc4555-rt6ww" Jul 26 15:39:09.618955 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:09.618833 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3a06ab69-b69e-44df-b0c3-939c0d6c423e-oauth-serving-cert\") pod \"console-5796bccdbd-vl5xj\" (UID: \"3a06ab69-b69e-44df-b0c3-939c0d6c423e\") " pod="openshift-console/console-5796bccdbd-vl5xj" Jul 26 15:39:09.618955 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:09.618863 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3a06ab69-b69e-44df-b0c3-939c0d6c423e-console-serving-cert\") pod \"console-5796bccdbd-vl5xj\" (UID: \"3a06ab69-b69e-44df-b0c3-939c0d6c423e\") " pod="openshift-console/console-5796bccdbd-vl5xj" Jul 26 15:39:09.618955 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:09.618893 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2123f11d-8547-4520-8d6c-d73bc66afa3c-tmp\") pod \"klusterlet-addon-workmgr-7bb9f8c5fb-r487g\" (UID: \"2123f11d-8547-4520-8d6c-d73bc66afa3c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7bb9f8c5fb-r487g" Jul 26 15:39:09.618955 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:09.618935 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3a06ab69-b69e-44df-b0c3-939c0d6c423e-console-config\") pod \"console-5796bccdbd-vl5xj\" (UID: \"3a06ab69-b69e-44df-b0c3-939c0d6c423e\") " pod="openshift-console/console-5796bccdbd-vl5xj" Jul 26 15:39:09.619198 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:09.618984 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3a06ab69-b69e-44df-b0c3-939c0d6c423e-console-oauth-config\") pod \"console-5796bccdbd-vl5xj\" (UID: \"3a06ab69-b69e-44df-b0c3-939c0d6c423e\") " pod="openshift-console/console-5796bccdbd-vl5xj" Jul 26 15:39:09.619198 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:09.619016 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/2123f11d-8547-4520-8d6c-d73bc66afa3c-klusterlet-config\") pod \"klusterlet-addon-workmgr-7bb9f8c5fb-r487g\" (UID: \"2123f11d-8547-4520-8d6c-d73bc66afa3c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7bb9f8c5fb-r487g" Jul 26 15:39:09.619198 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:09.619165 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2123f11d-8547-4520-8d6c-d73bc66afa3c-tmp\") pod \"klusterlet-addon-workmgr-7bb9f8c5fb-r487g\" (UID: \"2123f11d-8547-4520-8d6c-d73bc66afa3c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7bb9f8c5fb-r487g" Jul 26 15:39:09.621007 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:09.620970 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:39:09.621681 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:09.621661 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/2123f11d-8547-4520-8d6c-d73bc66afa3c-klusterlet-config\") pod \"klusterlet-addon-workmgr-7bb9f8c5fb-r487g\" (UID: \"2123f11d-8547-4520-8d6c-d73bc66afa3c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7bb9f8c5fb-r487g" Jul 26 15:39:09.627082 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:09.627060 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5swrm\" (UniqueName: \"kubernetes.io/projected/2123f11d-8547-4520-8d6c-d73bc66afa3c-kube-api-access-5swrm\") pod \"klusterlet-addon-workmgr-7bb9f8c5fb-r487g\" (UID: \"2123f11d-8547-4520-8d6c-d73bc66afa3c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7bb9f8c5fb-r487g" Jul 26 15:39:09.661029 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:09.660989 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7bb9f8c5fb-r487g" Jul 26 15:39:09.719751 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:09.719722 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3a06ab69-b69e-44df-b0c3-939c0d6c423e-oauth-serving-cert\") pod \"console-5796bccdbd-vl5xj\" (UID: \"3a06ab69-b69e-44df-b0c3-939c0d6c423e\") " pod="openshift-console/console-5796bccdbd-vl5xj" Jul 26 15:39:09.719926 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:09.719763 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3a06ab69-b69e-44df-b0c3-939c0d6c423e-console-serving-cert\") pod \"console-5796bccdbd-vl5xj\" (UID: \"3a06ab69-b69e-44df-b0c3-939c0d6c423e\") " pod="openshift-console/console-5796bccdbd-vl5xj" Jul 26 15:39:09.719926 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:09.719896 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3a06ab69-b69e-44df-b0c3-939c0d6c423e-console-config\") pod \"console-5796bccdbd-vl5xj\" (UID: \"3a06ab69-b69e-44df-b0c3-939c0d6c423e\") " pod="openshift-console/console-5796bccdbd-vl5xj" Jul 26 15:39:09.720051 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:09.719944 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3a06ab69-b69e-44df-b0c3-939c0d6c423e-console-oauth-config\") pod \"console-5796bccdbd-vl5xj\" (UID: \"3a06ab69-b69e-44df-b0c3-939c0d6c423e\") " pod="openshift-console/console-5796bccdbd-vl5xj" Jul 26 15:39:09.720051 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:09.720008 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szgm7\" (UniqueName: \"kubernetes.io/projected/3a06ab69-b69e-44df-b0c3-939c0d6c423e-kube-api-access-szgm7\") pod \"console-5796bccdbd-vl5xj\" (UID: \"3a06ab69-b69e-44df-b0c3-939c0d6c423e\") " pod="openshift-console/console-5796bccdbd-vl5xj" Jul 26 15:39:09.720051 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:09.720041 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3a06ab69-b69e-44df-b0c3-939c0d6c423e-service-ca\") pod \"console-5796bccdbd-vl5xj\" (UID: \"3a06ab69-b69e-44df-b0c3-939c0d6c423e\") " pod="openshift-console/console-5796bccdbd-vl5xj" Jul 26 15:39:09.720195 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:09.720079 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjm7t\" (UniqueName: \"kubernetes.io/projected/3a938ae2-0e14-4c59-8347-52d26294c17f-kube-api-access-qjm7t\") pod \"downloads-7c97dc4555-rt6ww\" (UID: \"3a938ae2-0e14-4c59-8347-52d26294c17f\") " pod="openshift-console/downloads-7c97dc4555-rt6ww" Jul 26 15:39:09.720476 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:09.720453 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3a06ab69-b69e-44df-b0c3-939c0d6c423e-oauth-serving-cert\") pod \"console-5796bccdbd-vl5xj\" (UID: \"3a06ab69-b69e-44df-b0c3-939c0d6c423e\") " pod="openshift-console/console-5796bccdbd-vl5xj" Jul 26 15:39:09.720606 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:09.720588 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3a06ab69-b69e-44df-b0c3-939c0d6c423e-console-config\") pod \"console-5796bccdbd-vl5xj\" (UID: \"3a06ab69-b69e-44df-b0c3-939c0d6c423e\") " pod="openshift-console/console-5796bccdbd-vl5xj" Jul 26 15:39:09.720766 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:09.720749 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3a06ab69-b69e-44df-b0c3-939c0d6c423e-service-ca\") pod \"console-5796bccdbd-vl5xj\" (UID: \"3a06ab69-b69e-44df-b0c3-939c0d6c423e\") " pod="openshift-console/console-5796bccdbd-vl5xj" Jul 26 15:39:09.722308 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:09.722287 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3a06ab69-b69e-44df-b0c3-939c0d6c423e-console-serving-cert\") pod \"console-5796bccdbd-vl5xj\" (UID: \"3a06ab69-b69e-44df-b0c3-939c0d6c423e\") " pod="openshift-console/console-5796bccdbd-vl5xj" Jul 26 15:39:09.722807 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:09.722788 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3a06ab69-b69e-44df-b0c3-939c0d6c423e-console-oauth-config\") pod \"console-5796bccdbd-vl5xj\" (UID: \"3a06ab69-b69e-44df-b0c3-939c0d6c423e\") " pod="openshift-console/console-5796bccdbd-vl5xj" Jul 26 15:39:09.725685 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:09.725663 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:39:09.731513 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:09.731489 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-szgm7\" (UniqueName: \"kubernetes.io/projected/3a06ab69-b69e-44df-b0c3-939c0d6c423e-kube-api-access-szgm7\") pod \"console-5796bccdbd-vl5xj\" (UID: \"3a06ab69-b69e-44df-b0c3-939c0d6c423e\") " pod="openshift-console/console-5796bccdbd-vl5xj" Jul 26 15:39:09.731600 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:09.731573 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjm7t\" (UniqueName: \"kubernetes.io/projected/3a938ae2-0e14-4c59-8347-52d26294c17f-kube-api-access-qjm7t\") pod \"downloads-7c97dc4555-rt6ww\" (UID: \"3a938ae2-0e14-4c59-8347-52d26294c17f\") " pod="openshift-console/downloads-7c97dc4555-rt6ww" Jul 26 15:39:09.759463 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:09.759435 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-rt6ww" Jul 26 15:39:09.768270 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:09.768242 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5796bccdbd-vl5xj" Jul 26 15:39:12.842744 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:12.838707 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-rt6ww"] Jul 26 15:39:12.898589 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:12.897957 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5796bccdbd-vl5xj"] Jul 26 15:39:12.907998 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:12.907969 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-5pdf8"] Jul 26 15:39:12.910463 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:12.910430 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7bb9f8c5fb-r487g"] Jul 26 15:39:12.991845 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:39:12.991796 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a938ae2_0e14_4c59_8347_52d26294c17f.slice/crio-aadffaa8cd9eebbc2990105c28dc06d009b412831320e9f7322e4a78fcf3c68a WatchSource:0}: Error finding container aadffaa8cd9eebbc2990105c28dc06d009b412831320e9f7322e4a78fcf3c68a: Status 404 returned error can't find the container with id aadffaa8cd9eebbc2990105c28dc06d009b412831320e9f7322e4a78fcf3c68a Jul 26 15:39:12.992868 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:39:12.992831 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a06ab69_b69e_44df_b0c3_939c0d6c423e.slice/crio-ab76a7b97fb43f272782218ebc65bee04e2f40e6877415cdb373077dc53d6291 WatchSource:0}: Error finding container ab76a7b97fb43f272782218ebc65bee04e2f40e6877415cdb373077dc53d6291: Status 404 returned error can't find the container with id ab76a7b97fb43f272782218ebc65bee04e2f40e6877415cdb373077dc53d6291 Jul 26 15:39:12.994208 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:39:12.993796 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d7ef445_5c4a_4c4e_92f4_2da8ee29b868.slice/crio-9547ddaf95926f8a3478a0039985af4c962a544d086554e7f0569e9ffc2fbb6a WatchSource:0}: Error finding container 9547ddaf95926f8a3478a0039985af4c962a544d086554e7f0569e9ffc2fbb6a: Status 404 returned error can't find the container with id 9547ddaf95926f8a3478a0039985af4c962a544d086554e7f0569e9ffc2fbb6a Jul 26 15:39:12.994708 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:39:12.994663 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2123f11d_8547_4520_8d6c_d73bc66afa3c.slice/crio-896a1b0e71b017508eb8cffe07471346b522b9a4282398fda052f20bb6daf529 WatchSource:0}: Error finding container 896a1b0e71b017508eb8cffe07471346b522b9a4282398fda052f20bb6daf529: Status 404 returned error can't find the container with id 896a1b0e71b017508eb8cffe07471346b522b9a4282398fda052f20bb6daf529 Jul 26 15:39:13.129075 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:13.128931 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:39:13.232326 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:13.232281 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:39:13.271476 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:13.271436 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-2q2tq" event={"ID":"60d326a6-77da-4f72-965f-880f5c94f2b7","Type":"ContainerStarted","Data":"cb074287c356c12b3702df82dda87e1b96a3d8e5c32924e2f7e44fb06187bfed"} Jul 26 15:39:13.273002 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:13.272970 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-rt6ww" event={"ID":"3a938ae2-0e14-4c59-8347-52d26294c17f","Type":"ContainerStarted","Data":"aadffaa8cd9eebbc2990105c28dc06d009b412831320e9f7322e4a78fcf3c68a"} Jul 26 15:39:13.275000 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:13.274972 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-rrqbw" event={"ID":"b9353822-c6c4-47f1-8a41-ecdc92669f5f","Type":"ContainerStarted","Data":"56a2c066bce5b7a5159ae02f2da31d04c69759865d0ddbf916b70926988fd4c2"} Jul 26 15:39:13.275088 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:13.275029 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-rrqbw" event={"ID":"b9353822-c6c4-47f1-8a41-ecdc92669f5f","Type":"ContainerStarted","Data":"651b7e8be68ec0d43ee9eb4402863648a01ceeb48b5f1265486af6ffaf4b6cdd"} Jul 26 15:39:13.276932 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:13.276907 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5q8xh" event={"ID":"76cd20bd-3a66-40ad-99c0-fefa173500d6","Type":"ContainerStarted","Data":"362903c0dc7678fc6edaf2203d6287843206c664ac596081b4b590a5cd382452"} Jul 26 15:39:13.278272 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:13.278248 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5796bccdbd-vl5xj" event={"ID":"3a06ab69-b69e-44df-b0c3-939c0d6c423e","Type":"ContainerStarted","Data":"ab76a7b97fb43f272782218ebc65bee04e2f40e6877415cdb373077dc53d6291"} Jul 26 15:39:13.279443 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:13.279420 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7bb9f8c5fb-r487g" event={"ID":"2123f11d-8547-4520-8d6c-d73bc66afa3c","Type":"ContainerStarted","Data":"896a1b0e71b017508eb8cffe07471346b522b9a4282398fda052f20bb6daf529"} Jul 26 15:39:13.281105 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:13.281080 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-l56b4" event={"ID":"f6bd9013-635a-4af8-b0ee-621a280a459a","Type":"ContainerStarted","Data":"123ae9f99c24b8e8ee68f11a04d633b5344a170130f527cbf67dfc7cfcf20622"} Jul 26 15:39:13.283131 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:13.283103 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-65h7n" event={"ID":"839598d1-d772-47cf-a460-5e88affbac0e","Type":"ContainerStarted","Data":"05f650e958a6031f3675365a437cf934e9dd8955b4a777d23adba622a6cdd579"} Jul 26 15:39:13.283236 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:13.283157 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-65h7n" Jul 26 15:39:13.283236 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:13.283184 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-65h7n" event={"ID":"839598d1-d772-47cf-a460-5e88affbac0e","Type":"ContainerStarted","Data":"16f7f946c2809cd48e1aa608d295cad5121fb9aa677c8a8a2e350022721bfd8b"} Jul 26 15:39:13.284094 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:13.284071 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-5pdf8" event={"ID":"3d7ef445-5c4a-4c4e-92f4-2da8ee29b868","Type":"ContainerStarted","Data":"9547ddaf95926f8a3478a0039985af4c962a544d086554e7f0569e9ffc2fbb6a"} Jul 26 15:39:13.291366 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:13.291317 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-2q2tq" podStartSLOduration=44.142197366 podStartE2EDuration="52.291279097s" totalImagesPullingTime="8.149081731s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:38:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:39:13.290579007 +0000 UTC m=+76.029932331" watchObservedRunningTime="2026-07-26 15:39:13.291279097 +0000 UTC m=+76.030632408" Jul 26 15:39:13.304663 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:13.304623 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:39:13.307310 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:13.307286 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-9rkgx"] Jul 26 15:39:13.310436 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:13.310379 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-l56b4" podStartSLOduration=66.658859002 podStartE2EDuration="1m15.310364579s" totalImagesPullingTime="8.651505577s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-26 15:37:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:39:13.309328793 +0000 UTC m=+76.048682103" watchObservedRunningTime="2026-07-26 15:39:13.310364579 +0000 UTC m=+76.049717866" Jul 26 15:39:13.312690 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:13.312672 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-9rkgx" Jul 26 15:39:13.314649 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:13.314632 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 26 15:39:13.314764 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:13.314725 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-79db8\"" Jul 26 15:39:13.317833 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:13.317808 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-9rkgx"] Jul 26 15:39:13.325145 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:13.325090 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-65h7n" podStartSLOduration=33.350793343 podStartE2EDuration="41.325072461s" totalImagesPullingTime="7.974279118s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:38:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:39:13.324155203 +0000 UTC m=+76.063508539" watchObservedRunningTime="2026-07-26 15:39:13.325072461 +0000 UTC m=+76.064425772" Jul 26 15:39:13.340742 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:13.340693 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-rrqbw" podStartSLOduration=44.356023177 podStartE2EDuration="52.34067908s" totalImagesPullingTime="7.984655903s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:38:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:39:13.339460984 +0000 UTC m=+76.078814320" watchObservedRunningTime="2026-07-26 15:39:13.34067908 +0000 UTC m=+76.080032390" Jul 26 15:39:13.359008 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:13.358976 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/ef70e16b-47fb-49f0-ab9d-d54f5cce38fb-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-9rkgx\" (UID: \"ef70e16b-47fb-49f0-ab9d-d54f5cce38fb\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-9rkgx" Jul 26 15:39:13.450259 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:13.450174 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:39:13.460420 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:13.460330 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/ef70e16b-47fb-49f0-ab9d-d54f5cce38fb-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-9rkgx\" (UID: \"ef70e16b-47fb-49f0-ab9d-d54f5cce38fb\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-9rkgx" Jul 26 15:39:13.460572 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:39:13.460476 2577 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found Jul 26 15:39:13.460572 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:39:13.460539 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef70e16b-47fb-49f0-ab9d-d54f5cce38fb-tls-certificates podName:ef70e16b-47fb-49f0-ab9d-d54f5cce38fb nodeName:}" failed. No retries permitted until 2026-07-26 15:39:13.960521101 +0000 UTC m=+76.699874396 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/ef70e16b-47fb-49f0-ab9d-d54f5cce38fb-tls-certificates") pod "prometheus-operator-admission-webhook-7f8d78f54b-9rkgx" (UID: "ef70e16b-47fb-49f0-ab9d-d54f5cce38fb") : secret "prometheus-operator-admission-webhook-tls" not found Jul 26 15:39:13.551269 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:13.551230 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:39:13.668242 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:13.668206 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:39:13.762625 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:13.761910 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:39:13.964840 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:13.964794 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/ef70e16b-47fb-49f0-ab9d-d54f5cce38fb-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-9rkgx\" (UID: \"ef70e16b-47fb-49f0-ab9d-d54f5cce38fb\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-9rkgx" Jul 26 15:39:13.965278 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:39:13.964991 2577 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found Jul 26 15:39:13.965278 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:39:13.965062 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef70e16b-47fb-49f0-ab9d-d54f5cce38fb-tls-certificates podName:ef70e16b-47fb-49f0-ab9d-d54f5cce38fb nodeName:}" failed. No retries permitted until 2026-07-26 15:39:14.965042574 +0000 UTC m=+77.704395869 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/ef70e16b-47fb-49f0-ab9d-d54f5cce38fb-tls-certificates") pod "prometheus-operator-admission-webhook-7f8d78f54b-9rkgx" (UID: "ef70e16b-47fb-49f0-ab9d-d54f5cce38fb") : secret "prometheus-operator-admission-webhook-tls" not found Jul 26 15:39:14.045581 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:14.045550 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:39:14.447115 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:14.445635 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6c9b4dbd6-4vw85"] Jul 26 15:39:14.474684 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:14.474613 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6c9b4dbd6-4vw85"] Jul 26 15:39:14.474858 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:14.474734 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c9b4dbd6-4vw85" Jul 26 15:39:14.486215 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:14.486188 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 26 15:39:14.572073 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:14.572016 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7133ee3d-c624-49a5-9055-7321df9f2a50-console-oauth-config\") pod \"console-6c9b4dbd6-4vw85\" (UID: \"7133ee3d-c624-49a5-9055-7321df9f2a50\") " pod="openshift-console/console-6c9b4dbd6-4vw85" Jul 26 15:39:14.572247 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:14.572120 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4skf4\" (UniqueName: \"kubernetes.io/projected/7133ee3d-c624-49a5-9055-7321df9f2a50-kube-api-access-4skf4\") pod \"console-6c9b4dbd6-4vw85\" (UID: \"7133ee3d-c624-49a5-9055-7321df9f2a50\") " pod="openshift-console/console-6c9b4dbd6-4vw85" Jul 26 15:39:14.572247 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:14.572151 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7133ee3d-c624-49a5-9055-7321df9f2a50-console-serving-cert\") pod \"console-6c9b4dbd6-4vw85\" (UID: \"7133ee3d-c624-49a5-9055-7321df9f2a50\") " pod="openshift-console/console-6c9b4dbd6-4vw85" Jul 26 15:39:14.572247 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:14.572185 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7133ee3d-c624-49a5-9055-7321df9f2a50-oauth-serving-cert\") pod \"console-6c9b4dbd6-4vw85\" (UID: \"7133ee3d-c624-49a5-9055-7321df9f2a50\") " pod="openshift-console/console-6c9b4dbd6-4vw85" Jul 26 15:39:14.572247 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:14.572239 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7133ee3d-c624-49a5-9055-7321df9f2a50-console-config\") pod \"console-6c9b4dbd6-4vw85\" (UID: \"7133ee3d-c624-49a5-9055-7321df9f2a50\") " pod="openshift-console/console-6c9b4dbd6-4vw85" Jul 26 15:39:14.572486 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:14.572274 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7133ee3d-c624-49a5-9055-7321df9f2a50-trusted-ca-bundle\") pod \"console-6c9b4dbd6-4vw85\" (UID: \"7133ee3d-c624-49a5-9055-7321df9f2a50\") " pod="openshift-console/console-6c9b4dbd6-4vw85" Jul 26 15:39:14.572486 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:14.572305 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7133ee3d-c624-49a5-9055-7321df9f2a50-service-ca\") pod \"console-6c9b4dbd6-4vw85\" (UID: \"7133ee3d-c624-49a5-9055-7321df9f2a50\") " pod="openshift-console/console-6c9b4dbd6-4vw85" Jul 26 15:39:14.673064 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:14.672992 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7133ee3d-c624-49a5-9055-7321df9f2a50-oauth-serving-cert\") pod \"console-6c9b4dbd6-4vw85\" (UID: \"7133ee3d-c624-49a5-9055-7321df9f2a50\") " pod="openshift-console/console-6c9b4dbd6-4vw85" Jul 26 15:39:14.673247 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:14.673115 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7133ee3d-c624-49a5-9055-7321df9f2a50-console-config\") pod \"console-6c9b4dbd6-4vw85\" (UID: \"7133ee3d-c624-49a5-9055-7321df9f2a50\") " pod="openshift-console/console-6c9b4dbd6-4vw85" Jul 26 15:39:14.673247 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:14.673150 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7133ee3d-c624-49a5-9055-7321df9f2a50-trusted-ca-bundle\") pod \"console-6c9b4dbd6-4vw85\" (UID: \"7133ee3d-c624-49a5-9055-7321df9f2a50\") " pod="openshift-console/console-6c9b4dbd6-4vw85" Jul 26 15:39:14.673247 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:14.673179 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7133ee3d-c624-49a5-9055-7321df9f2a50-service-ca\") pod \"console-6c9b4dbd6-4vw85\" (UID: \"7133ee3d-c624-49a5-9055-7321df9f2a50\") " pod="openshift-console/console-6c9b4dbd6-4vw85" Jul 26 15:39:14.673247 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:14.673215 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7133ee3d-c624-49a5-9055-7321df9f2a50-console-oauth-config\") pod \"console-6c9b4dbd6-4vw85\" (UID: \"7133ee3d-c624-49a5-9055-7321df9f2a50\") " pod="openshift-console/console-6c9b4dbd6-4vw85" Jul 26 15:39:14.673487 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:14.673296 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4skf4\" (UniqueName: \"kubernetes.io/projected/7133ee3d-c624-49a5-9055-7321df9f2a50-kube-api-access-4skf4\") pod \"console-6c9b4dbd6-4vw85\" (UID: \"7133ee3d-c624-49a5-9055-7321df9f2a50\") " pod="openshift-console/console-6c9b4dbd6-4vw85" Jul 26 15:39:14.673487 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:14.673325 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7133ee3d-c624-49a5-9055-7321df9f2a50-console-serving-cert\") pod \"console-6c9b4dbd6-4vw85\" (UID: \"7133ee3d-c624-49a5-9055-7321df9f2a50\") " pod="openshift-console/console-6c9b4dbd6-4vw85" Jul 26 15:39:14.676376 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:14.673901 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7133ee3d-c624-49a5-9055-7321df9f2a50-oauth-serving-cert\") pod \"console-6c9b4dbd6-4vw85\" (UID: \"7133ee3d-c624-49a5-9055-7321df9f2a50\") " pod="openshift-console/console-6c9b4dbd6-4vw85" Jul 26 15:39:14.676376 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:14.674486 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7133ee3d-c624-49a5-9055-7321df9f2a50-service-ca\") pod \"console-6c9b4dbd6-4vw85\" (UID: \"7133ee3d-c624-49a5-9055-7321df9f2a50\") " pod="openshift-console/console-6c9b4dbd6-4vw85" Jul 26 15:39:14.676376 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:14.675031 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7133ee3d-c624-49a5-9055-7321df9f2a50-console-config\") pod \"console-6c9b4dbd6-4vw85\" (UID: \"7133ee3d-c624-49a5-9055-7321df9f2a50\") " pod="openshift-console/console-6c9b4dbd6-4vw85" Jul 26 15:39:14.676376 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:14.675761 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7133ee3d-c624-49a5-9055-7321df9f2a50-trusted-ca-bundle\") pod \"console-6c9b4dbd6-4vw85\" (UID: \"7133ee3d-c624-49a5-9055-7321df9f2a50\") " pod="openshift-console/console-6c9b4dbd6-4vw85" Jul 26 15:39:14.689987 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:14.689930 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7133ee3d-c624-49a5-9055-7321df9f2a50-console-serving-cert\") pod \"console-6c9b4dbd6-4vw85\" (UID: \"7133ee3d-c624-49a5-9055-7321df9f2a50\") " pod="openshift-console/console-6c9b4dbd6-4vw85" Jul 26 15:39:14.692445 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:14.692304 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4skf4\" (UniqueName: \"kubernetes.io/projected/7133ee3d-c624-49a5-9055-7321df9f2a50-kube-api-access-4skf4\") pod \"console-6c9b4dbd6-4vw85\" (UID: \"7133ee3d-c624-49a5-9055-7321df9f2a50\") " pod="openshift-console/console-6c9b4dbd6-4vw85" Jul 26 15:39:14.710791 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:14.710691 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7133ee3d-c624-49a5-9055-7321df9f2a50-console-oauth-config\") pod \"console-6c9b4dbd6-4vw85\" (UID: \"7133ee3d-c624-49a5-9055-7321df9f2a50\") " pod="openshift-console/console-6c9b4dbd6-4vw85" Jul 26 15:39:14.792024 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:14.791795 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c9b4dbd6-4vw85" Jul 26 15:39:14.976510 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:14.976429 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/ef70e16b-47fb-49f0-ab9d-d54f5cce38fb-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-9rkgx\" (UID: \"ef70e16b-47fb-49f0-ab9d-d54f5cce38fb\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-9rkgx" Jul 26 15:39:14.988435 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:14.988360 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/ef70e16b-47fb-49f0-ab9d-d54f5cce38fb-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-9rkgx\" (UID: \"ef70e16b-47fb-49f0-ab9d-d54f5cce38fb\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-9rkgx" Jul 26 15:39:15.123200 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:15.123169 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-9rkgx" Jul 26 15:39:15.286895 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:15.286787 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:39:15.289689 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:15.289653 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6c9b4dbd6-4vw85"] Jul 26 15:39:15.351024 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:15.350967 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-9rkgx"] Jul 26 15:39:15.353770 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:15.353695 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:39:15.537326 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:39:15.537287 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7133ee3d_c624_49a5_9055_7321df9f2a50.slice/crio-bc115fca6ddfb67dd4f3934b45e9fcf10771e7c914f40dce8390497c60286f70 WatchSource:0}: Error finding container bc115fca6ddfb67dd4f3934b45e9fcf10771e7c914f40dce8390497c60286f70: Status 404 returned error can't find the container with id bc115fca6ddfb67dd4f3934b45e9fcf10771e7c914f40dce8390497c60286f70 Jul 26 15:39:15.538949 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:39:15.538912 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef70e16b_47fb_49f0_ab9d_d54f5cce38fb.slice/crio-d0602d4cb3fb133d932aa5f335905a2c34a0634196a35b92112faa55b9179857 WatchSource:0}: Error finding container d0602d4cb3fb133d932aa5f335905a2c34a0634196a35b92112faa55b9179857: Status 404 returned error can't find the container with id d0602d4cb3fb133d932aa5f335905a2c34a0634196a35b92112faa55b9179857 Jul 26 15:39:15.623223 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:15.623026 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:39:15.678946 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:15.678916 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:39:15.849297 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:15.849196 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:39:15.949351 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:15.949318 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:39:16.236026 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:16.235858 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:39:16.306220 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:16.306180 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-9rkgx" event={"ID":"ef70e16b-47fb-49f0-ab9d-d54f5cce38fb","Type":"ContainerStarted","Data":"d0602d4cb3fb133d932aa5f335905a2c34a0634196a35b92112faa55b9179857"} Jul 26 15:39:16.309336 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:16.309303 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-5pdf8" event={"ID":"3d7ef445-5c4a-4c4e-92f4-2da8ee29b868","Type":"ContainerStarted","Data":"b2f2c3186c3ef244de3f243c55f9ff92fdfe104898a244bd40facec253b905f1"} Jul 26 15:39:16.312380 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:16.312308 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c9b4dbd6-4vw85" event={"ID":"7133ee3d-c624-49a5-9055-7321df9f2a50","Type":"ContainerStarted","Data":"bc115fca6ddfb67dd4f3934b45e9fcf10771e7c914f40dce8390497c60286f70"} Jul 26 15:39:16.325554 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:16.325279 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-74c9f884b7-5pdf8" podStartSLOduration=39.764977035 podStartE2EDuration="42.325257409s" totalImagesPullingTime="2.560280374s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:38:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:39:16.32346423 +0000 UTC m=+79.062817541" watchObservedRunningTime="2026-07-26 15:39:16.325257409 +0000 UTC m=+79.064610720" Jul 26 15:39:16.329774 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:16.329598 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:39:16.457080 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:16.457043 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:39:16.517373 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:16.517262 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:39:18.405456 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:18.405414 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:39:18.455315 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:18.455283 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:39:19.757614 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:19.757574 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:39:19.857482 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:19.857063 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:39:20.270465 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:20.270424 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:39:20.372371 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:20.372190 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:39:20.670567 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:20.670500 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:39:22.347946 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:22.347905 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5q8xh" event={"ID":"76cd20bd-3a66-40ad-99c0-fefa173500d6","Type":"ContainerStarted","Data":"62a999eb14cfc249ce17d9772589dacbfc292701d82bda143f91a6830b9c2e1e"} Jul 26 15:39:22.369206 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:22.369146 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-5q8xh" podStartSLOduration=16.683880199 podStartE2EDuration="39.369123112s" totalImagesPullingTime="22.685242913s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-26 15:38:43 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:39:22.367615824 +0000 UTC m=+85.106969137" watchObservedRunningTime="2026-07-26 15:39:22.369123112 +0000 UTC m=+85.108476425" Jul 26 15:39:22.886506 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:22.886452 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:39:22.990265 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:22.990229 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:39:23.085003 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:23.084967 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:39:23.294550 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:23.294514 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-65h7n" Jul 26 15:39:23.355996 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:23.355959 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5796bccdbd-vl5xj" event={"ID":"3a06ab69-b69e-44df-b0c3-939c0d6c423e","Type":"ContainerStarted","Data":"863b87afb855aa0ec0f1bc293dfe91f0a0301cc4433ee99ef22fda51d9926074"} Jul 26 15:39:23.358182 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:23.358129 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7bb9f8c5fb-r487g" event={"ID":"2123f11d-8547-4520-8d6c-d73bc66afa3c","Type":"ContainerStarted","Data":"47916b485b7258ef921207d754a729880494cf4c9f55a99eb15da88dfbdc1a83"} Jul 26 15:39:23.358541 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:23.358515 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7bb9f8c5fb-r487g" Jul 26 15:39:23.360485 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:23.360454 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-9rkgx" event={"ID":"ef70e16b-47fb-49f0-ab9d-d54f5cce38fb","Type":"ContainerStarted","Data":"8b19570223b2b3d3bb73f6b1e886e718a6fad0806eddcf0e13d8c0f3aa2ded84"} Jul 26 15:39:23.360614 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:23.360527 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7bb9f8c5fb-r487g" Jul 26 15:39:23.360614 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:23.360553 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-9rkgx" Jul 26 15:39:23.363474 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:23.363442 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c9b4dbd6-4vw85" event={"ID":"7133ee3d-c624-49a5-9055-7321df9f2a50","Type":"ContainerStarted","Data":"bfb45a9c251eabd0016364c1ccfa3def603d4c57fea52360c5127df609528c01"} Jul 26 15:39:23.366120 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:23.366098 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-9rkgx" Jul 26 15:39:23.373409 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:23.373348 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5796bccdbd-vl5xj" podStartSLOduration=5.320585073 podStartE2EDuration="14.373329204s" totalImagesPullingTime="9.052744131s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:39:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:39:23.371356064 +0000 UTC m=+86.110709400" watchObservedRunningTime="2026-07-26 15:39:23.373329204 +0000 UTC m=+86.112682516" Jul 26 15:39:23.388512 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:23.388462 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6c9b4dbd6-4vw85" podStartSLOduration=2.584835638 podStartE2EDuration="9.388445947s" totalImagesPullingTime="6.803610309s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:39:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:39:23.387547741 +0000 UTC m=+86.126901054" watchObservedRunningTime="2026-07-26 15:39:23.388445947 +0000 UTC m=+86.127799257" Jul 26 15:39:23.406422 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:23.406342 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7bb9f8c5fb-r487g" podStartSLOduration=5.047904494 podStartE2EDuration="14.406323242s" totalImagesPullingTime="9.358418748s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:39:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:39:23.405155885 +0000 UTC m=+86.144509218" watchObservedRunningTime="2026-07-26 15:39:23.406323242 +0000 UTC m=+86.145676555" Jul 26 15:39:23.420136 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:23.420083 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-9rkgx" podStartSLOduration=3.560393477 podStartE2EDuration="10.42006304s" totalImagesPullingTime="6.859669563s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:39:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:39:23.418812276 +0000 UTC m=+86.158165584" watchObservedRunningTime="2026-07-26 15:39:23.42006304 +0000 UTC m=+86.159416349" Jul 26 15:39:24.336050 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:24.336012 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:39:24.417493 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:24.417453 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:39:24.609272 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:24.609172 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:39:24.793244 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:24.793208 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6c9b4dbd6-4vw85" Jul 26 15:39:24.793438 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:24.793273 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6c9b4dbd6-4vw85" Jul 26 15:39:24.799626 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:24.799602 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6c9b4dbd6-4vw85" Jul 26 15:39:25.328830 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:25.328747 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:39:25.378063 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:25.378033 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6c9b4dbd6-4vw85" Jul 26 15:39:25.384511 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:25.384478 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:39:25.424078 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:25.424045 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5796bccdbd-vl5xj"] Jul 26 15:39:25.559639 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:25.559602 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:39:25.658600 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:25.658500 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:39:25.716089 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:25.716052 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:39:26.244494 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:26.244466 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5ccd8b79b5-rnb96" Jul 26 15:39:26.404453 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:26.404391 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:39:26.460290 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:26.460234 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:39:26.925386 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:26.925347 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:39:28.253702 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:28.253653 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:39:28.304371 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:28.304333 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:39:29.482684 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:29.482619 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:39:29.701487 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:29.701446 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:39:29.752233 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:29.752146 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-s5rxf"] Jul 26 15:39:29.777810 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:29.777771 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:39:29.851436 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:29.851371 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:39:29.946936 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:29.946897 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:39:29.980132 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:29.980046 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5796bccdbd-vl5xj" Jul 26 15:39:29.980300 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:29.980227 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-s5rxf" Jul 26 15:39:29.983445 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:29.983279 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 26 15:39:29.983577 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:29.983477 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 26 15:39:29.984180 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:29.984141 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-5fmmc\"" Jul 26 15:39:29.984180 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:29.984176 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 26 15:39:29.984422 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:29.984379 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 26 15:39:30.005032 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:30.004946 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:39:30.013059 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:30.013033 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e8a7bf77-ea86-49c6-9111-73932e0359bb-sys\") pod \"node-exporter-s5rxf\" (UID: \"e8a7bf77-ea86-49c6-9111-73932e0359bb\") " pod="openshift-monitoring/node-exporter-s5rxf" Jul 26 15:39:30.013211 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:30.013065 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e8a7bf77-ea86-49c6-9111-73932e0359bb-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-s5rxf\" (UID: \"e8a7bf77-ea86-49c6-9111-73932e0359bb\") " pod="openshift-monitoring/node-exporter-s5rxf" Jul 26 15:39:30.013211 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:30.013099 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e8a7bf77-ea86-49c6-9111-73932e0359bb-root\") pod \"node-exporter-s5rxf\" (UID: \"e8a7bf77-ea86-49c6-9111-73932e0359bb\") " pod="openshift-monitoring/node-exporter-s5rxf" Jul 26 15:39:30.013211 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:30.013128 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e8a7bf77-ea86-49c6-9111-73932e0359bb-node-exporter-textfile\") pod \"node-exporter-s5rxf\" (UID: \"e8a7bf77-ea86-49c6-9111-73932e0359bb\") " pod="openshift-monitoring/node-exporter-s5rxf" Jul 26 15:39:30.013211 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:30.013159 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e8a7bf77-ea86-49c6-9111-73932e0359bb-node-exporter-wtmp\") pod \"node-exporter-s5rxf\" (UID: \"e8a7bf77-ea86-49c6-9111-73932e0359bb\") " pod="openshift-monitoring/node-exporter-s5rxf" Jul 26 15:39:30.013211 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:30.013175 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wt9dp\" (UniqueName: \"kubernetes.io/projected/e8a7bf77-ea86-49c6-9111-73932e0359bb-kube-api-access-wt9dp\") pod \"node-exporter-s5rxf\" (UID: \"e8a7bf77-ea86-49c6-9111-73932e0359bb\") " pod="openshift-monitoring/node-exporter-s5rxf" Jul 26 15:39:30.013449 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:30.013327 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e8a7bf77-ea86-49c6-9111-73932e0359bb-metrics-client-ca\") pod \"node-exporter-s5rxf\" (UID: \"e8a7bf77-ea86-49c6-9111-73932e0359bb\") " pod="openshift-monitoring/node-exporter-s5rxf" Jul 26 15:39:30.013449 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:30.013385 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e8a7bf77-ea86-49c6-9111-73932e0359bb-node-exporter-tls\") pod \"node-exporter-s5rxf\" (UID: \"e8a7bf77-ea86-49c6-9111-73932e0359bb\") " pod="openshift-monitoring/node-exporter-s5rxf" Jul 26 15:39:30.013449 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:30.013439 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e8a7bf77-ea86-49c6-9111-73932e0359bb-node-exporter-accelerators-collector-config\") pod \"node-exporter-s5rxf\" (UID: \"e8a7bf77-ea86-49c6-9111-73932e0359bb\") " pod="openshift-monitoring/node-exporter-s5rxf" Jul 26 15:39:30.114722 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:30.114689 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e8a7bf77-ea86-49c6-9111-73932e0359bb-node-exporter-tls\") pod \"node-exporter-s5rxf\" (UID: \"e8a7bf77-ea86-49c6-9111-73932e0359bb\") " pod="openshift-monitoring/node-exporter-s5rxf" Jul 26 15:39:30.114912 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:30.114738 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e8a7bf77-ea86-49c6-9111-73932e0359bb-node-exporter-accelerators-collector-config\") pod \"node-exporter-s5rxf\" (UID: \"e8a7bf77-ea86-49c6-9111-73932e0359bb\") " pod="openshift-monitoring/node-exporter-s5rxf" Jul 26 15:39:30.114912 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:30.114777 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e8a7bf77-ea86-49c6-9111-73932e0359bb-sys\") pod \"node-exporter-s5rxf\" (UID: \"e8a7bf77-ea86-49c6-9111-73932e0359bb\") " pod="openshift-monitoring/node-exporter-s5rxf" Jul 26 15:39:30.114912 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:30.114798 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e8a7bf77-ea86-49c6-9111-73932e0359bb-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-s5rxf\" (UID: \"e8a7bf77-ea86-49c6-9111-73932e0359bb\") " pod="openshift-monitoring/node-exporter-s5rxf" Jul 26 15:39:30.114912 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:30.114825 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e8a7bf77-ea86-49c6-9111-73932e0359bb-root\") pod \"node-exporter-s5rxf\" (UID: \"e8a7bf77-ea86-49c6-9111-73932e0359bb\") " pod="openshift-monitoring/node-exporter-s5rxf" Jul 26 15:39:30.114912 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:30.114852 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e8a7bf77-ea86-49c6-9111-73932e0359bb-node-exporter-textfile\") pod \"node-exporter-s5rxf\" (UID: \"e8a7bf77-ea86-49c6-9111-73932e0359bb\") " pod="openshift-monitoring/node-exporter-s5rxf" Jul 26 15:39:30.114912 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:30.114890 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e8a7bf77-ea86-49c6-9111-73932e0359bb-node-exporter-wtmp\") pod \"node-exporter-s5rxf\" (UID: \"e8a7bf77-ea86-49c6-9111-73932e0359bb\") " pod="openshift-monitoring/node-exporter-s5rxf" Jul 26 15:39:30.114912 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:30.114888 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e8a7bf77-ea86-49c6-9111-73932e0359bb-sys\") pod \"node-exporter-s5rxf\" (UID: \"e8a7bf77-ea86-49c6-9111-73932e0359bb\") " pod="openshift-monitoring/node-exporter-s5rxf" Jul 26 15:39:30.114912 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:30.114912 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wt9dp\" (UniqueName: \"kubernetes.io/projected/e8a7bf77-ea86-49c6-9111-73932e0359bb-kube-api-access-wt9dp\") pod \"node-exporter-s5rxf\" (UID: \"e8a7bf77-ea86-49c6-9111-73932e0359bb\") " pod="openshift-monitoring/node-exporter-s5rxf" Jul 26 15:39:30.115303 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:30.114973 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e8a7bf77-ea86-49c6-9111-73932e0359bb-metrics-client-ca\") pod \"node-exporter-s5rxf\" (UID: \"e8a7bf77-ea86-49c6-9111-73932e0359bb\") " pod="openshift-monitoring/node-exporter-s5rxf" Jul 26 15:39:30.115303 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:30.114997 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e8a7bf77-ea86-49c6-9111-73932e0359bb-root\") pod \"node-exporter-s5rxf\" (UID: \"e8a7bf77-ea86-49c6-9111-73932e0359bb\") " pod="openshift-monitoring/node-exporter-s5rxf" Jul 26 15:39:30.115303 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:30.115110 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e8a7bf77-ea86-49c6-9111-73932e0359bb-node-exporter-wtmp\") pod \"node-exporter-s5rxf\" (UID: \"e8a7bf77-ea86-49c6-9111-73932e0359bb\") " pod="openshift-monitoring/node-exporter-s5rxf" Jul 26 15:39:30.115495 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:30.115426 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e8a7bf77-ea86-49c6-9111-73932e0359bb-node-exporter-textfile\") pod \"node-exporter-s5rxf\" (UID: \"e8a7bf77-ea86-49c6-9111-73932e0359bb\") " pod="openshift-monitoring/node-exporter-s5rxf" Jul 26 15:39:30.115703 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:30.115677 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e8a7bf77-ea86-49c6-9111-73932e0359bb-node-exporter-accelerators-collector-config\") pod \"node-exporter-s5rxf\" (UID: \"e8a7bf77-ea86-49c6-9111-73932e0359bb\") " pod="openshift-monitoring/node-exporter-s5rxf" Jul 26 15:39:30.115784 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:30.115701 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e8a7bf77-ea86-49c6-9111-73932e0359bb-metrics-client-ca\") pod \"node-exporter-s5rxf\" (UID: \"e8a7bf77-ea86-49c6-9111-73932e0359bb\") " pod="openshift-monitoring/node-exporter-s5rxf" Jul 26 15:39:30.117715 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:30.117682 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e8a7bf77-ea86-49c6-9111-73932e0359bb-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-s5rxf\" (UID: \"e8a7bf77-ea86-49c6-9111-73932e0359bb\") " pod="openshift-monitoring/node-exporter-s5rxf" Jul 26 15:39:30.117821 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:30.117797 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e8a7bf77-ea86-49c6-9111-73932e0359bb-node-exporter-tls\") pod \"node-exporter-s5rxf\" (UID: \"e8a7bf77-ea86-49c6-9111-73932e0359bb\") " pod="openshift-monitoring/node-exporter-s5rxf" Jul 26 15:39:30.128658 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:30.128636 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wt9dp\" (UniqueName: \"kubernetes.io/projected/e8a7bf77-ea86-49c6-9111-73932e0359bb-kube-api-access-wt9dp\") pod \"node-exporter-s5rxf\" (UID: \"e8a7bf77-ea86-49c6-9111-73932e0359bb\") " pod="openshift-monitoring/node-exporter-s5rxf" Jul 26 15:39:30.153652 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:30.153614 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:39:30.259973 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:30.259888 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:39:30.294472 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:30.294440 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-s5rxf" Jul 26 15:39:30.351773 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:30.351736 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:39:30.620518 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:30.620463 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:39:30.720199 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:30.720162 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:39:30.781944 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:30.781905 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:39:30.928739 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:30.928654 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:39:32.358048 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:39:32.358012 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8a7bf77_ea86_49c6_9111_73932e0359bb.slice/crio-abd9b4ce79fcede237e485679f67ef97d6695386dd7b2e1f443c8d096311e2bb WatchSource:0}: Error finding container abd9b4ce79fcede237e485679f67ef97d6695386dd7b2e1f443c8d096311e2bb: Status 404 returned error can't find the container with id abd9b4ce79fcede237e485679f67ef97d6695386dd7b2e1f443c8d096311e2bb Jul 26 15:39:32.405743 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:32.405695 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-s5rxf" event={"ID":"e8a7bf77-ea86-49c6-9111-73932e0359bb","Type":"ContainerStarted","Data":"abd9b4ce79fcede237e485679f67ef97d6695386dd7b2e1f443c8d096311e2bb"} Jul 26 15:39:32.413142 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:32.413112 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:39:32.470316 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:32.470280 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:39:32.493963 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:32.493932 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5ccd8b79b5-rnb96"] Jul 26 15:39:32.579694 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:32.579660 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:39:32.683002 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:32.682923 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:39:32.782547 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:32.782514 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:39:32.882819 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:32.882784 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:39:32.983567 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:32.983518 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:39:33.084046 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:33.084008 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:39:33.411350 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:33.411296 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-rt6ww" event={"ID":"3a938ae2-0e14-4c59-8347-52d26294c17f","Type":"ContainerStarted","Data":"52efb7fedf5d2d549047b41ab040b48f855c18b165f69d7ba1e535ae488cf7a0"} Jul 26 15:39:33.411808 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:33.411499 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-7c97dc4555-rt6ww" Jul 26 15:39:33.419162 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:33.419122 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-7c97dc4555-rt6ww" Jul 26 15:39:33.428869 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:33.428824 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-7c97dc4555-rt6ww" podStartSLOduration=4.5500173109999995 podStartE2EDuration="24.428803896s" totalImagesPullingTime="19.878786585s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:39:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:39:33.426632962 +0000 UTC m=+96.165986272" watchObservedRunningTime="2026-07-26 15:39:33.428803896 +0000 UTC m=+96.168157205" Jul 26 15:39:34.418285 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:34.418242 2577 generic.go:356] "Generic (PLEG): container finished" podID="e8a7bf77-ea86-49c6-9111-73932e0359bb" containerID="0f43b11d2020f2a70c8d46c475d78367016f5a974ab65298fa8540ce7848d4f3" exitCode=0 Jul 26 15:39:34.418791 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:34.418326 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-s5rxf" event={"ID":"e8a7bf77-ea86-49c6-9111-73932e0359bb","Type":"ContainerDied","Data":"0f43b11d2020f2a70c8d46c475d78367016f5a974ab65298fa8540ce7848d4f3"} Jul 26 15:39:34.529653 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:34.529560 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:39:34.607208 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:34.607122 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:39:34.714104 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:34.714069 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:39:34.821847 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:34.821808 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:39:34.916044 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:34.916006 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:39:35.231082 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:35.231045 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-d2nnz" Jul 26 15:39:35.425987 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:35.425948 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-s5rxf" event={"ID":"e8a7bf77-ea86-49c6-9111-73932e0359bb","Type":"ContainerStarted","Data":"9a23b94343016a44680c8d752b820f2267506a77e3e5295d1ec9152c7927c755"} Jul 26 15:39:35.426494 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:35.426010 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-s5rxf" event={"ID":"e8a7bf77-ea86-49c6-9111-73932e0359bb","Type":"ContainerStarted","Data":"865e23cec4faba6acc95b705f9046399fc7d5438d3a46640c3a19112e3225b24"} Jul 26 15:39:35.443722 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:35.443663 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-s5rxf" podStartSLOduration=5.267125776 podStartE2EDuration="6.44364457s" totalImagesPullingTime="1.088168569s" totalInitContainerRuntime="88.350225ms" isStatefulPod=true podCreationTimestamp="2026-07-26 15:39:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:39:35.442594894 +0000 UTC m=+98.181948225" watchObservedRunningTime="2026-07-26 15:39:35.44364457 +0000 UTC m=+98.182997880" Jul 26 15:39:36.051641 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:36.051601 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:39:36.172182 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:36.172143 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:39:36.274219 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:36.274181 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:39:40.874328 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:40.874292 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-57d79d964d-2g5z4"] Jul 26 15:39:40.980999 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:40.980957 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-57d79d964d-2g5z4"] Jul 26 15:39:40.981175 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:40.981090 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57d79d964d-2g5z4" Jul 26 15:39:41.110974 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:41.110928 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtn97\" (UniqueName: \"kubernetes.io/projected/86d9d032-2be9-4f10-9b69-09fe21a8a430-kube-api-access-mtn97\") pod \"console-57d79d964d-2g5z4\" (UID: \"86d9d032-2be9-4f10-9b69-09fe21a8a430\") " pod="openshift-console/console-57d79d964d-2g5z4" Jul 26 15:39:41.111143 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:41.111000 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86d9d032-2be9-4f10-9b69-09fe21a8a430-trusted-ca-bundle\") pod \"console-57d79d964d-2g5z4\" (UID: \"86d9d032-2be9-4f10-9b69-09fe21a8a430\") " pod="openshift-console/console-57d79d964d-2g5z4" Jul 26 15:39:41.111143 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:41.111035 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/86d9d032-2be9-4f10-9b69-09fe21a8a430-console-oauth-config\") pod \"console-57d79d964d-2g5z4\" (UID: \"86d9d032-2be9-4f10-9b69-09fe21a8a430\") " pod="openshift-console/console-57d79d964d-2g5z4" Jul 26 15:39:41.111143 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:41.111073 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/86d9d032-2be9-4f10-9b69-09fe21a8a430-console-config\") pod \"console-57d79d964d-2g5z4\" (UID: \"86d9d032-2be9-4f10-9b69-09fe21a8a430\") " pod="openshift-console/console-57d79d964d-2g5z4" Jul 26 15:39:41.111143 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:41.111126 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/86d9d032-2be9-4f10-9b69-09fe21a8a430-oauth-serving-cert\") pod \"console-57d79d964d-2g5z4\" (UID: \"86d9d032-2be9-4f10-9b69-09fe21a8a430\") " pod="openshift-console/console-57d79d964d-2g5z4" Jul 26 15:39:41.111335 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:41.111181 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/86d9d032-2be9-4f10-9b69-09fe21a8a430-console-serving-cert\") pod \"console-57d79d964d-2g5z4\" (UID: \"86d9d032-2be9-4f10-9b69-09fe21a8a430\") " pod="openshift-console/console-57d79d964d-2g5z4" Jul 26 15:39:41.111335 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:41.111213 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/86d9d032-2be9-4f10-9b69-09fe21a8a430-service-ca\") pod \"console-57d79d964d-2g5z4\" (UID: \"86d9d032-2be9-4f10-9b69-09fe21a8a430\") " pod="openshift-console/console-57d79d964d-2g5z4" Jul 26 15:39:41.212268 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:41.211832 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtn97\" (UniqueName: \"kubernetes.io/projected/86d9d032-2be9-4f10-9b69-09fe21a8a430-kube-api-access-mtn97\") pod \"console-57d79d964d-2g5z4\" (UID: \"86d9d032-2be9-4f10-9b69-09fe21a8a430\") " pod="openshift-console/console-57d79d964d-2g5z4" Jul 26 15:39:41.212578 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:41.212553 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86d9d032-2be9-4f10-9b69-09fe21a8a430-trusted-ca-bundle\") pod \"console-57d79d964d-2g5z4\" (UID: \"86d9d032-2be9-4f10-9b69-09fe21a8a430\") " pod="openshift-console/console-57d79d964d-2g5z4" Jul 26 15:39:41.212746 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:41.212728 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/86d9d032-2be9-4f10-9b69-09fe21a8a430-console-oauth-config\") pod \"console-57d79d964d-2g5z4\" (UID: \"86d9d032-2be9-4f10-9b69-09fe21a8a430\") " pod="openshift-console/console-57d79d964d-2g5z4" Jul 26 15:39:41.212871 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:41.212851 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/86d9d032-2be9-4f10-9b69-09fe21a8a430-console-config\") pod \"console-57d79d964d-2g5z4\" (UID: \"86d9d032-2be9-4f10-9b69-09fe21a8a430\") " pod="openshift-console/console-57d79d964d-2g5z4" Jul 26 15:39:41.212995 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:41.212877 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/86d9d032-2be9-4f10-9b69-09fe21a8a430-oauth-serving-cert\") pod \"console-57d79d964d-2g5z4\" (UID: \"86d9d032-2be9-4f10-9b69-09fe21a8a430\") " pod="openshift-console/console-57d79d964d-2g5z4" Jul 26 15:39:41.212995 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:41.212915 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/86d9d032-2be9-4f10-9b69-09fe21a8a430-console-serving-cert\") pod \"console-57d79d964d-2g5z4\" (UID: \"86d9d032-2be9-4f10-9b69-09fe21a8a430\") " pod="openshift-console/console-57d79d964d-2g5z4" Jul 26 15:39:41.212995 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:41.212939 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/86d9d032-2be9-4f10-9b69-09fe21a8a430-service-ca\") pod \"console-57d79d964d-2g5z4\" (UID: \"86d9d032-2be9-4f10-9b69-09fe21a8a430\") " pod="openshift-console/console-57d79d964d-2g5z4" Jul 26 15:39:41.213816 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:41.213793 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/86d9d032-2be9-4f10-9b69-09fe21a8a430-service-ca\") pod \"console-57d79d964d-2g5z4\" (UID: \"86d9d032-2be9-4f10-9b69-09fe21a8a430\") " pod="openshift-console/console-57d79d964d-2g5z4" Jul 26 15:39:41.214853 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:41.214830 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86d9d032-2be9-4f10-9b69-09fe21a8a430-trusted-ca-bundle\") pod \"console-57d79d964d-2g5z4\" (UID: \"86d9d032-2be9-4f10-9b69-09fe21a8a430\") " pod="openshift-console/console-57d79d964d-2g5z4" Jul 26 15:39:41.215838 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:41.215815 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/86d9d032-2be9-4f10-9b69-09fe21a8a430-oauth-serving-cert\") pod \"console-57d79d964d-2g5z4\" (UID: \"86d9d032-2be9-4f10-9b69-09fe21a8a430\") " pod="openshift-console/console-57d79d964d-2g5z4" Jul 26 15:39:41.216206 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:41.216183 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/86d9d032-2be9-4f10-9b69-09fe21a8a430-console-config\") pod \"console-57d79d964d-2g5z4\" (UID: \"86d9d032-2be9-4f10-9b69-09fe21a8a430\") " pod="openshift-console/console-57d79d964d-2g5z4" Jul 26 15:39:41.217768 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:41.217746 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/86d9d032-2be9-4f10-9b69-09fe21a8a430-console-oauth-config\") pod \"console-57d79d964d-2g5z4\" (UID: \"86d9d032-2be9-4f10-9b69-09fe21a8a430\") " pod="openshift-console/console-57d79d964d-2g5z4" Jul 26 15:39:41.218634 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:41.218588 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/86d9d032-2be9-4f10-9b69-09fe21a8a430-console-serving-cert\") pod \"console-57d79d964d-2g5z4\" (UID: \"86d9d032-2be9-4f10-9b69-09fe21a8a430\") " pod="openshift-console/console-57d79d964d-2g5z4" Jul 26 15:39:41.223050 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:41.223028 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtn97\" (UniqueName: \"kubernetes.io/projected/86d9d032-2be9-4f10-9b69-09fe21a8a430-kube-api-access-mtn97\") pod \"console-57d79d964d-2g5z4\" (UID: \"86d9d032-2be9-4f10-9b69-09fe21a8a430\") " pod="openshift-console/console-57d79d964d-2g5z4" Jul 26 15:39:41.292354 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:41.292322 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57d79d964d-2g5z4" Jul 26 15:39:41.297582 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:41.297550 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:39:41.404096 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:41.404012 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:39:41.484431 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:41.484356 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-57d79d964d-2g5z4"] Jul 26 15:39:41.486922 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:39:41.486892 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86d9d032_2be9_4f10_9b69_09fe21a8a430.slice/crio-656bfe96fa4501234f7149c08c5563688f74e8247c80bfd4f6324ce013c8e0c2 WatchSource:0}: Error finding container 656bfe96fa4501234f7149c08c5563688f74e8247c80bfd4f6324ce013c8e0c2: Status 404 returned error can't find the container with id 656bfe96fa4501234f7149c08c5563688f74e8247c80bfd4f6324ce013c8e0c2 Jul 26 15:39:41.848440 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:41.848384 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:39:42.462604 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:42.462563 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57d79d964d-2g5z4" event={"ID":"86d9d032-2be9-4f10-9b69-09fe21a8a430","Type":"ContainerStarted","Data":"7bfd1b27d58825734a99b31259f69ff0a167417eb753d8641b6a15fa460d5497"} Jul 26 15:39:42.463057 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:42.462623 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57d79d964d-2g5z4" event={"ID":"86d9d032-2be9-4f10-9b69-09fe21a8a430","Type":"ContainerStarted","Data":"656bfe96fa4501234f7149c08c5563688f74e8247c80bfd4f6324ce013c8e0c2"} Jul 26 15:39:42.494755 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:42.494701 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-57d79d964d-2g5z4" podStartSLOduration=2.4946841490000002 podStartE2EDuration="2.494684149s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:39:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:39:42.493936784 +0000 UTC m=+105.233290096" watchObservedRunningTime="2026-07-26 15:39:42.494684149 +0000 UTC m=+105.234037459" Jul 26 15:39:42.827096 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:42.827062 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:39:42.872907 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:42.872873 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:39:45.621033 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:45.621003 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:39:45.670787 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:45.670747 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:39:50.448764 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:50.448701 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5796bccdbd-vl5xj" podUID="3a06ab69-b69e-44df-b0c3-939c0d6c423e" containerName="console" containerID="cri-o://863b87afb855aa0ec0f1bc293dfe91f0a0301cc4433ee99ef22fda51d9926074" gracePeriod=15 Jul 26 15:39:50.715744 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:50.715669 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5796bccdbd-vl5xj" Jul 26 15:39:50.798924 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:50.798890 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3a06ab69-b69e-44df-b0c3-939c0d6c423e-console-oauth-config\") pod \"3a06ab69-b69e-44df-b0c3-939c0d6c423e\" (UID: \"3a06ab69-b69e-44df-b0c3-939c0d6c423e\") " Jul 26 15:39:50.798924 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:50.798928 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3a06ab69-b69e-44df-b0c3-939c0d6c423e-service-ca\") pod \"3a06ab69-b69e-44df-b0c3-939c0d6c423e\" (UID: \"3a06ab69-b69e-44df-b0c3-939c0d6c423e\") " Jul 26 15:39:50.799133 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:50.798947 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3a06ab69-b69e-44df-b0c3-939c0d6c423e-console-config\") pod \"3a06ab69-b69e-44df-b0c3-939c0d6c423e\" (UID: \"3a06ab69-b69e-44df-b0c3-939c0d6c423e\") " Jul 26 15:39:50.799133 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:50.798997 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3a06ab69-b69e-44df-b0c3-939c0d6c423e-oauth-serving-cert\") pod \"3a06ab69-b69e-44df-b0c3-939c0d6c423e\" (UID: \"3a06ab69-b69e-44df-b0c3-939c0d6c423e\") " Jul 26 15:39:50.799133 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:50.799023 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-szgm7\" (UniqueName: \"kubernetes.io/projected/3a06ab69-b69e-44df-b0c3-939c0d6c423e-kube-api-access-szgm7\") pod \"3a06ab69-b69e-44df-b0c3-939c0d6c423e\" (UID: \"3a06ab69-b69e-44df-b0c3-939c0d6c423e\") " Jul 26 15:39:50.799133 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:50.799093 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3a06ab69-b69e-44df-b0c3-939c0d6c423e-console-serving-cert\") pod \"3a06ab69-b69e-44df-b0c3-939c0d6c423e\" (UID: \"3a06ab69-b69e-44df-b0c3-939c0d6c423e\") " Jul 26 15:39:50.799363 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:50.799339 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a06ab69-b69e-44df-b0c3-939c0d6c423e-console-config" (OuterVolumeSpecName: "console-config") pod "3a06ab69-b69e-44df-b0c3-939c0d6c423e" (UID: "3a06ab69-b69e-44df-b0c3-939c0d6c423e"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 15:39:50.799460 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:50.799422 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a06ab69-b69e-44df-b0c3-939c0d6c423e-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "3a06ab69-b69e-44df-b0c3-939c0d6c423e" (UID: "3a06ab69-b69e-44df-b0c3-939c0d6c423e"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 15:39:50.799522 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:50.799490 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a06ab69-b69e-44df-b0c3-939c0d6c423e-service-ca" (OuterVolumeSpecName: "service-ca") pod "3a06ab69-b69e-44df-b0c3-939c0d6c423e" (UID: "3a06ab69-b69e-44df-b0c3-939c0d6c423e"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 15:39:50.801351 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:50.801326 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a06ab69-b69e-44df-b0c3-939c0d6c423e-kube-api-access-szgm7" (OuterVolumeSpecName: "kube-api-access-szgm7") pod "3a06ab69-b69e-44df-b0c3-939c0d6c423e" (UID: "3a06ab69-b69e-44df-b0c3-939c0d6c423e"). InnerVolumeSpecName "kube-api-access-szgm7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 15:39:50.801351 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:50.801338 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a06ab69-b69e-44df-b0c3-939c0d6c423e-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "3a06ab69-b69e-44df-b0c3-939c0d6c423e" (UID: "3a06ab69-b69e-44df-b0c3-939c0d6c423e"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 26 15:39:50.801499 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:50.801415 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a06ab69-b69e-44df-b0c3-939c0d6c423e-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "3a06ab69-b69e-44df-b0c3-939c0d6c423e" (UID: "3a06ab69-b69e-44df-b0c3-939c0d6c423e"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 26 15:39:50.900655 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:50.900622 2577 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3a06ab69-b69e-44df-b0c3-939c0d6c423e-console-config\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 15:39:50.900655 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:50.900660 2577 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3a06ab69-b69e-44df-b0c3-939c0d6c423e-oauth-serving-cert\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 15:39:50.900853 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:50.900671 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szgm7\" (UniqueName: \"kubernetes.io/projected/3a06ab69-b69e-44df-b0c3-939c0d6c423e-kube-api-access-szgm7\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 15:39:50.900853 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:50.900680 2577 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3a06ab69-b69e-44df-b0c3-939c0d6c423e-console-serving-cert\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 15:39:50.900853 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:50.900688 2577 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3a06ab69-b69e-44df-b0c3-939c0d6c423e-console-oauth-config\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 15:39:50.900853 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:50.900697 2577 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3a06ab69-b69e-44df-b0c3-939c0d6c423e-service-ca\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 15:39:51.292930 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:51.292888 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-57d79d964d-2g5z4" Jul 26 15:39:51.293199 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:51.292957 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-57d79d964d-2g5z4" Jul 26 15:39:51.297655 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:51.297631 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-57d79d964d-2g5z4" Jul 26 15:39:51.497911 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:51.497877 2577 generic.go:356] "Generic (PLEG): container finished" podID="3a06ab69-b69e-44df-b0c3-939c0d6c423e" containerID="863b87afb855aa0ec0f1bc293dfe91f0a0301cc4433ee99ef22fda51d9926074" exitCode=0 Jul 26 15:39:51.497911 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:51.497915 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5796bccdbd-vl5xj" Jul 26 15:39:51.498579 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:51.497969 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5796bccdbd-vl5xj" event={"ID":"3a06ab69-b69e-44df-b0c3-939c0d6c423e","Type":"ContainerDied","Data":"863b87afb855aa0ec0f1bc293dfe91f0a0301cc4433ee99ef22fda51d9926074"} Jul 26 15:39:51.498579 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:51.498021 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5796bccdbd-vl5xj" event={"ID":"3a06ab69-b69e-44df-b0c3-939c0d6c423e","Type":"ContainerDied","Data":"ab76a7b97fb43f272782218ebc65bee04e2f40e6877415cdb373077dc53d6291"} Jul 26 15:39:51.498579 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:51.498047 2577 scope.go:117] "RemoveContainer" containerID="863b87afb855aa0ec0f1bc293dfe91f0a0301cc4433ee99ef22fda51d9926074" Jul 26 15:39:51.504916 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:51.504880 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-57d79d964d-2g5z4" Jul 26 15:39:51.511665 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:51.511647 2577 scope.go:117] "RemoveContainer" containerID="863b87afb855aa0ec0f1bc293dfe91f0a0301cc4433ee99ef22fda51d9926074" Jul 26 15:39:51.511939 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:39:51.511917 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"863b87afb855aa0ec0f1bc293dfe91f0a0301cc4433ee99ef22fda51d9926074\": container with ID starting with 863b87afb855aa0ec0f1bc293dfe91f0a0301cc4433ee99ef22fda51d9926074 not found: ID does not exist" containerID="863b87afb855aa0ec0f1bc293dfe91f0a0301cc4433ee99ef22fda51d9926074" Jul 26 15:39:51.512007 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:51.511947 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"863b87afb855aa0ec0f1bc293dfe91f0a0301cc4433ee99ef22fda51d9926074"} err="failed to get container status \"863b87afb855aa0ec0f1bc293dfe91f0a0301cc4433ee99ef22fda51d9926074\": rpc error: code = NotFound desc = could not find container \"863b87afb855aa0ec0f1bc293dfe91f0a0301cc4433ee99ef22fda51d9926074\": container with ID starting with 863b87afb855aa0ec0f1bc293dfe91f0a0301cc4433ee99ef22fda51d9926074 not found: ID does not exist" Jul 26 15:39:51.534697 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:51.534669 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5796bccdbd-vl5xj"] Jul 26 15:39:51.539219 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:51.539187 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5796bccdbd-vl5xj"] Jul 26 15:39:51.548127 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:51.548069 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6c9b4dbd6-4vw85"] Jul 26 15:39:51.645521 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:51.645483 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:39:51.698852 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:51.698821 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:39:51.753993 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:51.753941 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:39:51.786379 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:51.786350 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a06ab69-b69e-44df-b0c3-939c0d6c423e" path="/var/lib/kubelet/pods/3a06ab69-b69e-44df-b0c3-939c0d6c423e/volumes" Jul 26 15:39:57.513655 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:57.513556 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-5ccd8b79b5-rnb96" podUID="bf408cf7-787a-4839-ab2f-8220f1696bd6" containerName="registry" containerID="cri-o://9356871b4716b2377f6b6b8a69668e9e40e2abb18da39859f2ec0e1b7a2e0c91" gracePeriod=30 Jul 26 15:39:57.525701 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:57.525672 2577 generic.go:356] "Generic (PLEG): container finished" podID="c2a7cc81-246f-4d1b-985d-22c611d6b339" containerID="c76f07c03572ce3e491100818369003885b6c66fff54015a51bd0691770f62f4" exitCode=0 Jul 26 15:39:57.525836 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:57.525752 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-2xmhd" event={"ID":"c2a7cc81-246f-4d1b-985d-22c611d6b339","Type":"ContainerDied","Data":"c76f07c03572ce3e491100818369003885b6c66fff54015a51bd0691770f62f4"} Jul 26 15:39:57.526152 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:57.526134 2577 scope.go:117] "RemoveContainer" containerID="c76f07c03572ce3e491100818369003885b6c66fff54015a51bd0691770f62f4" Jul 26 15:39:57.812936 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:57.812905 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5ccd8b79b5-rnb96" Jul 26 15:39:57.967519 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:57.967481 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bf408cf7-787a-4839-ab2f-8220f1696bd6-bound-sa-token\") pod \"bf408cf7-787a-4839-ab2f-8220f1696bd6\" (UID: \"bf408cf7-787a-4839-ab2f-8220f1696bd6\") " Jul 26 15:39:57.967717 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:57.967545 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bf408cf7-787a-4839-ab2f-8220f1696bd6-installation-pull-secrets\") pod \"bf408cf7-787a-4839-ab2f-8220f1696bd6\" (UID: \"bf408cf7-787a-4839-ab2f-8220f1696bd6\") " Jul 26 15:39:57.967717 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:57.967581 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bf408cf7-787a-4839-ab2f-8220f1696bd6-registry-tls\") pod \"bf408cf7-787a-4839-ab2f-8220f1696bd6\" (UID: \"bf408cf7-787a-4839-ab2f-8220f1696bd6\") " Jul 26 15:39:57.967717 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:57.967616 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-54z42\" (UniqueName: \"kubernetes.io/projected/bf408cf7-787a-4839-ab2f-8220f1696bd6-kube-api-access-54z42\") pod \"bf408cf7-787a-4839-ab2f-8220f1696bd6\" (UID: \"bf408cf7-787a-4839-ab2f-8220f1696bd6\") " Jul 26 15:39:57.967717 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:57.967661 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bf408cf7-787a-4839-ab2f-8220f1696bd6-image-registry-private-configuration\") pod \"bf408cf7-787a-4839-ab2f-8220f1696bd6\" (UID: \"bf408cf7-787a-4839-ab2f-8220f1696bd6\") " Jul 26 15:39:57.967717 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:57.967689 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bf408cf7-787a-4839-ab2f-8220f1696bd6-trusted-ca\") pod \"bf408cf7-787a-4839-ab2f-8220f1696bd6\" (UID: \"bf408cf7-787a-4839-ab2f-8220f1696bd6\") " Jul 26 15:39:57.967961 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:57.967721 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bf408cf7-787a-4839-ab2f-8220f1696bd6-ca-trust-extracted\") pod \"bf408cf7-787a-4839-ab2f-8220f1696bd6\" (UID: \"bf408cf7-787a-4839-ab2f-8220f1696bd6\") " Jul 26 15:39:57.967961 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:57.967753 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bf408cf7-787a-4839-ab2f-8220f1696bd6-registry-certificates\") pod \"bf408cf7-787a-4839-ab2f-8220f1696bd6\" (UID: \"bf408cf7-787a-4839-ab2f-8220f1696bd6\") " Jul 26 15:39:57.968468 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:57.968439 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf408cf7-787a-4839-ab2f-8220f1696bd6-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "bf408cf7-787a-4839-ab2f-8220f1696bd6" (UID: "bf408cf7-787a-4839-ab2f-8220f1696bd6"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 15:39:57.968738 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:57.968518 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf408cf7-787a-4839-ab2f-8220f1696bd6-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "bf408cf7-787a-4839-ab2f-8220f1696bd6" (UID: "bf408cf7-787a-4839-ab2f-8220f1696bd6"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 15:39:57.970464 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:57.970432 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf408cf7-787a-4839-ab2f-8220f1696bd6-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "bf408cf7-787a-4839-ab2f-8220f1696bd6" (UID: "bf408cf7-787a-4839-ab2f-8220f1696bd6"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 26 15:39:57.970565 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:57.970497 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf408cf7-787a-4839-ab2f-8220f1696bd6-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "bf408cf7-787a-4839-ab2f-8220f1696bd6" (UID: "bf408cf7-787a-4839-ab2f-8220f1696bd6"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 15:39:57.970565 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:57.970515 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf408cf7-787a-4839-ab2f-8220f1696bd6-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "bf408cf7-787a-4839-ab2f-8220f1696bd6" (UID: "bf408cf7-787a-4839-ab2f-8220f1696bd6"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 26 15:39:57.970662 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:57.970569 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf408cf7-787a-4839-ab2f-8220f1696bd6-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "bf408cf7-787a-4839-ab2f-8220f1696bd6" (UID: "bf408cf7-787a-4839-ab2f-8220f1696bd6"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 15:39:57.970702 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:57.970678 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf408cf7-787a-4839-ab2f-8220f1696bd6-kube-api-access-54z42" (OuterVolumeSpecName: "kube-api-access-54z42") pod "bf408cf7-787a-4839-ab2f-8220f1696bd6" (UID: "bf408cf7-787a-4839-ab2f-8220f1696bd6"). InnerVolumeSpecName "kube-api-access-54z42". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 15:39:57.975989 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:57.975963 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf408cf7-787a-4839-ab2f-8220f1696bd6-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "bf408cf7-787a-4839-ab2f-8220f1696bd6" (UID: "bf408cf7-787a-4839-ab2f-8220f1696bd6"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:39:58.069102 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:58.069033 2577 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bf408cf7-787a-4839-ab2f-8220f1696bd6-registry-tls\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 15:39:58.069102 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:58.069055 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54z42\" (UniqueName: \"kubernetes.io/projected/bf408cf7-787a-4839-ab2f-8220f1696bd6-kube-api-access-54z42\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 15:39:58.069102 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:58.069066 2577 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bf408cf7-787a-4839-ab2f-8220f1696bd6-image-registry-private-configuration\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 15:39:58.069102 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:58.069076 2577 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bf408cf7-787a-4839-ab2f-8220f1696bd6-trusted-ca\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 15:39:58.069102 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:58.069085 2577 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bf408cf7-787a-4839-ab2f-8220f1696bd6-ca-trust-extracted\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 15:39:58.069102 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:58.069094 2577 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bf408cf7-787a-4839-ab2f-8220f1696bd6-registry-certificates\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 15:39:58.069102 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:58.069102 2577 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bf408cf7-787a-4839-ab2f-8220f1696bd6-bound-sa-token\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 15:39:58.069102 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:58.069110 2577 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bf408cf7-787a-4839-ab2f-8220f1696bd6-installation-pull-secrets\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 15:39:58.530463 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:58.530425 2577 generic.go:356] "Generic (PLEG): container finished" podID="bf408cf7-787a-4839-ab2f-8220f1696bd6" containerID="9356871b4716b2377f6b6b8a69668e9e40e2abb18da39859f2ec0e1b7a2e0c91" exitCode=0 Jul 26 15:39:58.530948 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:58.530469 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5ccd8b79b5-rnb96" Jul 26 15:39:58.530948 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:58.530509 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5ccd8b79b5-rnb96" event={"ID":"bf408cf7-787a-4839-ab2f-8220f1696bd6","Type":"ContainerDied","Data":"9356871b4716b2377f6b6b8a69668e9e40e2abb18da39859f2ec0e1b7a2e0c91"} Jul 26 15:39:58.530948 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:58.530555 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5ccd8b79b5-rnb96" event={"ID":"bf408cf7-787a-4839-ab2f-8220f1696bd6","Type":"ContainerDied","Data":"a1e44cca231a9629a11051332e5876857c7d842262b8babc814a4b0f80fbcaa4"} Jul 26 15:39:58.530948 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:58.530580 2577 scope.go:117] "RemoveContainer" containerID="9356871b4716b2377f6b6b8a69668e9e40e2abb18da39859f2ec0e1b7a2e0c91" Jul 26 15:39:58.532707 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:58.532680 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-2xmhd" event={"ID":"c2a7cc81-246f-4d1b-985d-22c611d6b339","Type":"ContainerStarted","Data":"012190a2f0e923c35f356a3bd134edfd97ceddbd5d7239c37b6cb006765d1bec"} Jul 26 15:39:58.539954 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:58.539937 2577 scope.go:117] "RemoveContainer" containerID="9356871b4716b2377f6b6b8a69668e9e40e2abb18da39859f2ec0e1b7a2e0c91" Jul 26 15:39:58.540208 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:39:58.540192 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9356871b4716b2377f6b6b8a69668e9e40e2abb18da39859f2ec0e1b7a2e0c91\": container with ID starting with 9356871b4716b2377f6b6b8a69668e9e40e2abb18da39859f2ec0e1b7a2e0c91 not found: ID does not exist" containerID="9356871b4716b2377f6b6b8a69668e9e40e2abb18da39859f2ec0e1b7a2e0c91" Jul 26 15:39:58.540254 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:58.540214 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9356871b4716b2377f6b6b8a69668e9e40e2abb18da39859f2ec0e1b7a2e0c91"} err="failed to get container status \"9356871b4716b2377f6b6b8a69668e9e40e2abb18da39859f2ec0e1b7a2e0c91\": rpc error: code = NotFound desc = could not find container \"9356871b4716b2377f6b6b8a69668e9e40e2abb18da39859f2ec0e1b7a2e0c91\": container with ID starting with 9356871b4716b2377f6b6b8a69668e9e40e2abb18da39859f2ec0e1b7a2e0c91 not found: ID does not exist" Jul 26 15:39:58.563922 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:58.563891 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5ccd8b79b5-rnb96"] Jul 26 15:39:58.569449 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:58.569420 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5ccd8b79b5-rnb96"] Jul 26 15:39:59.786776 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:39:59.786736 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bf408cf7-787a-4839-ab2f-8220f1696bd6" path="/var/lib/kubelet/pods/bf408cf7-787a-4839-ab2f-8220f1696bd6/volumes" Jul 26 15:40:00.568228 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:40:00.568188 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:40:00.669683 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:40:00.669646 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:40:00.728374 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:40:00.728340 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:40:07.568631 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:40:07.568598 2577 generic.go:356] "Generic (PLEG): container finished" podID="d48d98c5-9029-4b14-ab01-2234e4ecc1f7" containerID="df899b7b15931ade877fa9540f33d1c7c4c702b0a4e37e663ad271d3a9ea7e75" exitCode=0 Jul 26 15:40:07.569306 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:40:07.568678 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-7qj5q" event={"ID":"d48d98c5-9029-4b14-ab01-2234e4ecc1f7","Type":"ContainerDied","Data":"df899b7b15931ade877fa9540f33d1c7c4c702b0a4e37e663ad271d3a9ea7e75"} Jul 26 15:40:07.569306 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:40:07.569106 2577 scope.go:117] "RemoveContainer" containerID="df899b7b15931ade877fa9540f33d1c7c4c702b0a4e37e663ad271d3a9ea7e75" Jul 26 15:40:08.574088 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:40:08.574053 2577 generic.go:356] "Generic (PLEG): container finished" podID="e6f72321-bfbd-415c-9e47-cb4ef4fa0d46" containerID="3abae83a3773f72e1f903c6090637bbcf03e9a4ff04e853a9e50d6796128de41" exitCode=0 Jul 26 15:40:08.574621 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:40:08.574135 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-tzgd6" event={"ID":"e6f72321-bfbd-415c-9e47-cb4ef4fa0d46","Type":"ContainerDied","Data":"3abae83a3773f72e1f903c6090637bbcf03e9a4ff04e853a9e50d6796128de41"} Jul 26 15:40:08.574621 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:40:08.574583 2577 scope.go:117] "RemoveContainer" containerID="3abae83a3773f72e1f903c6090637bbcf03e9a4ff04e853a9e50d6796128de41" Jul 26 15:40:08.575961 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:40:08.575943 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-7qj5q" event={"ID":"d48d98c5-9029-4b14-ab01-2234e4ecc1f7","Type":"ContainerStarted","Data":"bcebbd53cc459a8f2506a66452ff895b987886e8787a35c5aeb94a636b77a155"} Jul 26 15:40:09.581720 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:40:09.581689 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-tzgd6" event={"ID":"e6f72321-bfbd-415c-9e47-cb4ef4fa0d46","Type":"ContainerStarted","Data":"f2748ff6f120da6f32238592a10ea0bd38fcc07e39fbbf56790ba019e477bcb3"} Jul 26 15:40:16.568231 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:40:16.568166 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6c9b4dbd6-4vw85" podUID="7133ee3d-c624-49a5-9055-7321df9f2a50" containerName="console" containerID="cri-o://bfb45a9c251eabd0016364c1ccfa3def603d4c57fea52360c5127df609528c01" gracePeriod=15 Jul 26 15:40:16.839326 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:40:16.839253 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c9b4dbd6-4vw85" Jul 26 15:40:16.924363 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:40:16.924327 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7133ee3d-c624-49a5-9055-7321df9f2a50-oauth-serving-cert\") pod \"7133ee3d-c624-49a5-9055-7321df9f2a50\" (UID: \"7133ee3d-c624-49a5-9055-7321df9f2a50\") " Jul 26 15:40:16.924575 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:40:16.924385 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7133ee3d-c624-49a5-9055-7321df9f2a50-console-serving-cert\") pod \"7133ee3d-c624-49a5-9055-7321df9f2a50\" (UID: \"7133ee3d-c624-49a5-9055-7321df9f2a50\") " Jul 26 15:40:16.924575 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:40:16.924436 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7133ee3d-c624-49a5-9055-7321df9f2a50-console-oauth-config\") pod \"7133ee3d-c624-49a5-9055-7321df9f2a50\" (UID: \"7133ee3d-c624-49a5-9055-7321df9f2a50\") " Jul 26 15:40:16.924575 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:40:16.924467 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4skf4\" (UniqueName: \"kubernetes.io/projected/7133ee3d-c624-49a5-9055-7321df9f2a50-kube-api-access-4skf4\") pod \"7133ee3d-c624-49a5-9055-7321df9f2a50\" (UID: \"7133ee3d-c624-49a5-9055-7321df9f2a50\") " Jul 26 15:40:16.924575 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:40:16.924493 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7133ee3d-c624-49a5-9055-7321df9f2a50-console-config\") pod \"7133ee3d-c624-49a5-9055-7321df9f2a50\" (UID: \"7133ee3d-c624-49a5-9055-7321df9f2a50\") " Jul 26 15:40:16.924786 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:40:16.924650 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7133ee3d-c624-49a5-9055-7321df9f2a50-service-ca\") pod \"7133ee3d-c624-49a5-9055-7321df9f2a50\" (UID: \"7133ee3d-c624-49a5-9055-7321df9f2a50\") " Jul 26 15:40:16.924786 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:40:16.924716 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7133ee3d-c624-49a5-9055-7321df9f2a50-trusted-ca-bundle\") pod \"7133ee3d-c624-49a5-9055-7321df9f2a50\" (UID: \"7133ee3d-c624-49a5-9055-7321df9f2a50\") " Jul 26 15:40:16.924912 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:40:16.924747 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7133ee3d-c624-49a5-9055-7321df9f2a50-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "7133ee3d-c624-49a5-9055-7321df9f2a50" (UID: "7133ee3d-c624-49a5-9055-7321df9f2a50"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 15:40:16.924977 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:40:16.924918 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7133ee3d-c624-49a5-9055-7321df9f2a50-console-config" (OuterVolumeSpecName: "console-config") pod "7133ee3d-c624-49a5-9055-7321df9f2a50" (UID: "7133ee3d-c624-49a5-9055-7321df9f2a50"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 15:40:16.925029 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:40:16.924972 2577 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7133ee3d-c624-49a5-9055-7321df9f2a50-oauth-serving-cert\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 15:40:16.925078 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:40:16.925016 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7133ee3d-c624-49a5-9055-7321df9f2a50-service-ca" (OuterVolumeSpecName: "service-ca") pod "7133ee3d-c624-49a5-9055-7321df9f2a50" (UID: "7133ee3d-c624-49a5-9055-7321df9f2a50"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 15:40:16.925134 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:40:16.925106 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7133ee3d-c624-49a5-9055-7321df9f2a50-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "7133ee3d-c624-49a5-9055-7321df9f2a50" (UID: "7133ee3d-c624-49a5-9055-7321df9f2a50"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 15:40:16.926667 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:40:16.926641 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7133ee3d-c624-49a5-9055-7321df9f2a50-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "7133ee3d-c624-49a5-9055-7321df9f2a50" (UID: "7133ee3d-c624-49a5-9055-7321df9f2a50"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 26 15:40:16.926667 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:40:16.926652 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7133ee3d-c624-49a5-9055-7321df9f2a50-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "7133ee3d-c624-49a5-9055-7321df9f2a50" (UID: "7133ee3d-c624-49a5-9055-7321df9f2a50"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 26 15:40:16.926796 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:40:16.926742 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7133ee3d-c624-49a5-9055-7321df9f2a50-kube-api-access-4skf4" (OuterVolumeSpecName: "kube-api-access-4skf4") pod "7133ee3d-c624-49a5-9055-7321df9f2a50" (UID: "7133ee3d-c624-49a5-9055-7321df9f2a50"). InnerVolumeSpecName "kube-api-access-4skf4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 15:40:17.026299 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:40:17.026259 2577 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7133ee3d-c624-49a5-9055-7321df9f2a50-service-ca\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 15:40:17.026299 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:40:17.026291 2577 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7133ee3d-c624-49a5-9055-7321df9f2a50-trusted-ca-bundle\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 15:40:17.026299 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:40:17.026302 2577 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7133ee3d-c624-49a5-9055-7321df9f2a50-console-serving-cert\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 15:40:17.026299 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:40:17.026310 2577 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7133ee3d-c624-49a5-9055-7321df9f2a50-console-oauth-config\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 15:40:17.026581 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:40:17.026319 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4skf4\" (UniqueName: \"kubernetes.io/projected/7133ee3d-c624-49a5-9055-7321df9f2a50-kube-api-access-4skf4\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 15:40:17.026581 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:40:17.026328 2577 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7133ee3d-c624-49a5-9055-7321df9f2a50-console-config\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 15:40:17.612996 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:40:17.612955 2577 generic.go:356] "Generic (PLEG): container finished" podID="7133ee3d-c624-49a5-9055-7321df9f2a50" containerID="bfb45a9c251eabd0016364c1ccfa3def603d4c57fea52360c5127df609528c01" exitCode=0 Jul 26 15:40:17.612996 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:40:17.613003 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c9b4dbd6-4vw85" Jul 26 15:40:17.613528 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:40:17.613033 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c9b4dbd6-4vw85" event={"ID":"7133ee3d-c624-49a5-9055-7321df9f2a50","Type":"ContainerDied","Data":"bfb45a9c251eabd0016364c1ccfa3def603d4c57fea52360c5127df609528c01"} Jul 26 15:40:17.613528 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:40:17.613088 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c9b4dbd6-4vw85" event={"ID":"7133ee3d-c624-49a5-9055-7321df9f2a50","Type":"ContainerDied","Data":"bc115fca6ddfb67dd4f3934b45e9fcf10771e7c914f40dce8390497c60286f70"} Jul 26 15:40:17.613528 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:40:17.613108 2577 scope.go:117] "RemoveContainer" containerID="bfb45a9c251eabd0016364c1ccfa3def603d4c57fea52360c5127df609528c01" Jul 26 15:40:17.621770 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:40:17.621741 2577 scope.go:117] "RemoveContainer" containerID="bfb45a9c251eabd0016364c1ccfa3def603d4c57fea52360c5127df609528c01" Jul 26 15:40:17.622039 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:40:17.622020 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfb45a9c251eabd0016364c1ccfa3def603d4c57fea52360c5127df609528c01\": container with ID starting with bfb45a9c251eabd0016364c1ccfa3def603d4c57fea52360c5127df609528c01 not found: ID does not exist" containerID="bfb45a9c251eabd0016364c1ccfa3def603d4c57fea52360c5127df609528c01" Jul 26 15:40:17.622099 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:40:17.622047 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfb45a9c251eabd0016364c1ccfa3def603d4c57fea52360c5127df609528c01"} err="failed to get container status \"bfb45a9c251eabd0016364c1ccfa3def603d4c57fea52360c5127df609528c01\": rpc error: code = NotFound desc = could not find container \"bfb45a9c251eabd0016364c1ccfa3def603d4c57fea52360c5127df609528c01\": container with ID starting with bfb45a9c251eabd0016364c1ccfa3def603d4c57fea52360c5127df609528c01 not found: ID does not exist" Jul 26 15:40:17.632565 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:40:17.632538 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6c9b4dbd6-4vw85"] Jul 26 15:40:17.634139 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:40:17.634120 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6c9b4dbd6-4vw85"] Jul 26 15:40:17.786593 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:40:17.786563 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7133ee3d-c624-49a5-9055-7321df9f2a50" path="/var/lib/kubelet/pods/7133ee3d-c624-49a5-9055-7321df9f2a50/volumes" Jul 26 15:40:25.349551 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:40:25.349499 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:40:45.838905 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:40:45.838869 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:40:45.944273 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:40:45.944239 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:40:50.711312 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:40:50.711222 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:40:52.239168 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:40:52.239138 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-54449ffcf6-56rkx"] Jul 26 15:40:52.239631 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:40:52.239429 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7133ee3d-c624-49a5-9055-7321df9f2a50" containerName="console" Jul 26 15:40:52.239631 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:40:52.239442 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7133ee3d-c624-49a5-9055-7321df9f2a50" containerName="console" Jul 26 15:40:52.239631 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:40:52.239458 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a06ab69-b69e-44df-b0c3-939c0d6c423e" containerName="console" Jul 26 15:40:52.239631 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:40:52.239464 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a06ab69-b69e-44df-b0c3-939c0d6c423e" containerName="console" Jul 26 15:40:52.239631 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:40:52.239475 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf408cf7-787a-4839-ab2f-8220f1696bd6" containerName="registry" Jul 26 15:40:52.239631 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:40:52.239481 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf408cf7-787a-4839-ab2f-8220f1696bd6" containerName="registry" Jul 26 15:40:52.239631 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:40:52.239532 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a06ab69-b69e-44df-b0c3-939c0d6c423e" containerName="console" Jul 26 15:40:52.239631 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:40:52.239539 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="7133ee3d-c624-49a5-9055-7321df9f2a50" containerName="console" Jul 26 15:40:52.239631 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:40:52.239547 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf408cf7-787a-4839-ab2f-8220f1696bd6" containerName="registry" Jul 26 15:40:52.241724 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:40:52.241702 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54449ffcf6-56rkx" Jul 26 15:40:52.257097 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:40:52.257052 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-54449ffcf6-56rkx"] Jul 26 15:40:52.294755 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:40:52.294719 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:40:52.314816 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:40:52.314780 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/331ec89f-a6dd-459d-8582-be410b9176fe-service-ca\") pod \"console-54449ffcf6-56rkx\" (UID: \"331ec89f-a6dd-459d-8582-be410b9176fe\") " pod="openshift-console/console-54449ffcf6-56rkx" Jul 26 15:40:52.314968 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:40:52.314834 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/331ec89f-a6dd-459d-8582-be410b9176fe-trusted-ca-bundle\") pod \"console-54449ffcf6-56rkx\" (UID: \"331ec89f-a6dd-459d-8582-be410b9176fe\") " pod="openshift-console/console-54449ffcf6-56rkx" Jul 26 15:40:52.314968 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:40:52.314851 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/331ec89f-a6dd-459d-8582-be410b9176fe-oauth-serving-cert\") pod \"console-54449ffcf6-56rkx\" (UID: \"331ec89f-a6dd-459d-8582-be410b9176fe\") " pod="openshift-console/console-54449ffcf6-56rkx" Jul 26 15:40:52.314968 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:40:52.314881 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/331ec89f-a6dd-459d-8582-be410b9176fe-console-oauth-config\") pod \"console-54449ffcf6-56rkx\" (UID: \"331ec89f-a6dd-459d-8582-be410b9176fe\") " pod="openshift-console/console-54449ffcf6-56rkx" Jul 26 15:40:52.314968 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:40:52.314905 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/331ec89f-a6dd-459d-8582-be410b9176fe-console-serving-cert\") pod \"console-54449ffcf6-56rkx\" (UID: \"331ec89f-a6dd-459d-8582-be410b9176fe\") " pod="openshift-console/console-54449ffcf6-56rkx" Jul 26 15:40:52.314968 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:40:52.314931 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nprcd\" (UniqueName: \"kubernetes.io/projected/331ec89f-a6dd-459d-8582-be410b9176fe-kube-api-access-nprcd\") pod \"console-54449ffcf6-56rkx\" (UID: \"331ec89f-a6dd-459d-8582-be410b9176fe\") " pod="openshift-console/console-54449ffcf6-56rkx" Jul 26 15:40:52.315149 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:40:52.314972 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/331ec89f-a6dd-459d-8582-be410b9176fe-console-config\") pod \"console-54449ffcf6-56rkx\" (UID: \"331ec89f-a6dd-459d-8582-be410b9176fe\") " pod="openshift-console/console-54449ffcf6-56rkx" Jul 26 15:40:52.345842 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:40:52.345812 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:40:52.416161 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:40:52.416128 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/331ec89f-a6dd-459d-8582-be410b9176fe-console-serving-cert\") pod \"console-54449ffcf6-56rkx\" (UID: \"331ec89f-a6dd-459d-8582-be410b9176fe\") " pod="openshift-console/console-54449ffcf6-56rkx" Jul 26 15:40:52.416351 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:40:52.416167 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nprcd\" (UniqueName: \"kubernetes.io/projected/331ec89f-a6dd-459d-8582-be410b9176fe-kube-api-access-nprcd\") pod \"console-54449ffcf6-56rkx\" (UID: \"331ec89f-a6dd-459d-8582-be410b9176fe\") " pod="openshift-console/console-54449ffcf6-56rkx" Jul 26 15:40:52.416351 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:40:52.416190 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/331ec89f-a6dd-459d-8582-be410b9176fe-console-config\") pod \"console-54449ffcf6-56rkx\" (UID: \"331ec89f-a6dd-459d-8582-be410b9176fe\") " pod="openshift-console/console-54449ffcf6-56rkx" Jul 26 15:40:52.416351 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:40:52.416213 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/331ec89f-a6dd-459d-8582-be410b9176fe-service-ca\") pod \"console-54449ffcf6-56rkx\" (UID: \"331ec89f-a6dd-459d-8582-be410b9176fe\") " pod="openshift-console/console-54449ffcf6-56rkx" Jul 26 15:40:52.416351 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:40:52.416253 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/331ec89f-a6dd-459d-8582-be410b9176fe-trusted-ca-bundle\") pod \"console-54449ffcf6-56rkx\" (UID: \"331ec89f-a6dd-459d-8582-be410b9176fe\") " pod="openshift-console/console-54449ffcf6-56rkx" Jul 26 15:40:52.416351 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:40:52.416267 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/331ec89f-a6dd-459d-8582-be410b9176fe-oauth-serving-cert\") pod \"console-54449ffcf6-56rkx\" (UID: \"331ec89f-a6dd-459d-8582-be410b9176fe\") " pod="openshift-console/console-54449ffcf6-56rkx" Jul 26 15:40:52.416351 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:40:52.416287 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/331ec89f-a6dd-459d-8582-be410b9176fe-console-oauth-config\") pod \"console-54449ffcf6-56rkx\" (UID: \"331ec89f-a6dd-459d-8582-be410b9176fe\") " pod="openshift-console/console-54449ffcf6-56rkx" Jul 26 15:40:52.417020 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:40:52.416996 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/331ec89f-a6dd-459d-8582-be410b9176fe-oauth-serving-cert\") pod \"console-54449ffcf6-56rkx\" (UID: \"331ec89f-a6dd-459d-8582-be410b9176fe\") " pod="openshift-console/console-54449ffcf6-56rkx" Jul 26 15:40:52.417192 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:40:52.417170 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/331ec89f-a6dd-459d-8582-be410b9176fe-console-config\") pod \"console-54449ffcf6-56rkx\" (UID: \"331ec89f-a6dd-459d-8582-be410b9176fe\") " pod="openshift-console/console-54449ffcf6-56rkx" Jul 26 15:40:52.417257 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:40:52.417170 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/331ec89f-a6dd-459d-8582-be410b9176fe-service-ca\") pod \"console-54449ffcf6-56rkx\" (UID: \"331ec89f-a6dd-459d-8582-be410b9176fe\") " pod="openshift-console/console-54449ffcf6-56rkx" Jul 26 15:40:52.417257 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:40:52.417230 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/331ec89f-a6dd-459d-8582-be410b9176fe-trusted-ca-bundle\") pod \"console-54449ffcf6-56rkx\" (UID: \"331ec89f-a6dd-459d-8582-be410b9176fe\") " pod="openshift-console/console-54449ffcf6-56rkx" Jul 26 15:40:52.418689 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:40:52.418667 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/331ec89f-a6dd-459d-8582-be410b9176fe-console-oauth-config\") pod \"console-54449ffcf6-56rkx\" (UID: \"331ec89f-a6dd-459d-8582-be410b9176fe\") " pod="openshift-console/console-54449ffcf6-56rkx" Jul 26 15:40:52.418797 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:40:52.418780 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/331ec89f-a6dd-459d-8582-be410b9176fe-console-serving-cert\") pod \"console-54449ffcf6-56rkx\" (UID: \"331ec89f-a6dd-459d-8582-be410b9176fe\") " pod="openshift-console/console-54449ffcf6-56rkx" Jul 26 15:40:52.426238 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:40:52.426218 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nprcd\" (UniqueName: \"kubernetes.io/projected/331ec89f-a6dd-459d-8582-be410b9176fe-kube-api-access-nprcd\") pod \"console-54449ffcf6-56rkx\" (UID: \"331ec89f-a6dd-459d-8582-be410b9176fe\") " pod="openshift-console/console-54449ffcf6-56rkx" Jul 26 15:40:52.438834 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:40:52.438803 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:40:52.485998 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:40:52.485964 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:40:52.551906 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:40:52.551877 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54449ffcf6-56rkx" Jul 26 15:40:52.684966 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:40:52.684915 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-54449ffcf6-56rkx"] Jul 26 15:40:52.687532 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:40:52.687504 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod331ec89f_a6dd_459d_8582_be410b9176fe.slice/crio-ffff0cb0d7a5de8aacc7624fe75184390e7b84b81c6fc3535bfc192b750775c8 WatchSource:0}: Error finding container ffff0cb0d7a5de8aacc7624fe75184390e7b84b81c6fc3535bfc192b750775c8: Status 404 returned error can't find the container with id ffff0cb0d7a5de8aacc7624fe75184390e7b84b81c6fc3535bfc192b750775c8 Jul 26 15:40:52.749274 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:40:52.749247 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54449ffcf6-56rkx" event={"ID":"331ec89f-a6dd-459d-8582-be410b9176fe","Type":"ContainerStarted","Data":"a489cf23ee712e1e617a52e2af3e9cd61484ae26811998dbf5ab2262ae6cf5f1"} Jul 26 15:40:52.749411 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:40:52.749295 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54449ffcf6-56rkx" event={"ID":"331ec89f-a6dd-459d-8582-be410b9176fe","Type":"ContainerStarted","Data":"ffff0cb0d7a5de8aacc7624fe75184390e7b84b81c6fc3535bfc192b750775c8"} Jul 26 15:40:52.764867 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:40:52.764818 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-54449ffcf6-56rkx" podStartSLOduration=0.764797376 podStartE2EDuration="764.797376ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:40:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:40:52.764385309 +0000 UTC m=+175.503738640" watchObservedRunningTime="2026-07-26 15:40:52.764797376 +0000 UTC m=+175.504150689" Jul 26 15:40:53.035124 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:40:53.035084 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:40:57.658674 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:40:57.658633 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:40:57.710600 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:40:57.710571 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:41:00.634369 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:41:00.634332 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:41:00.734453 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:41:00.734414 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:41:02.552641 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:41:02.552603 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-54449ffcf6-56rkx" Jul 26 15:41:02.553106 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:41:02.552667 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-54449ffcf6-56rkx" Jul 26 15:41:02.557511 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:41:02.557485 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-54449ffcf6-56rkx" Jul 26 15:41:02.793003 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:41:02.792974 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-54449ffcf6-56rkx" Jul 26 15:41:02.842216 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:41:02.837902 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-57d79d964d-2g5z4"] Jul 26 15:41:02.942835 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:41:02.942805 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:41:02.990533 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:41:02.990500 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:41:27.865095 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:41:27.865026 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-57d79d964d-2g5z4" podUID="86d9d032-2be9-4f10-9b69-09fe21a8a430" containerName="console" containerID="cri-o://7bfd1b27d58825734a99b31259f69ff0a167417eb753d8641b6a15fa460d5497" gracePeriod=15 Jul 26 15:41:28.104577 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:41:28.104554 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57d79d964d-2g5z4" Jul 26 15:41:28.203441 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:41:28.203326 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/86d9d032-2be9-4f10-9b69-09fe21a8a430-console-serving-cert\") pod \"86d9d032-2be9-4f10-9b69-09fe21a8a430\" (UID: \"86d9d032-2be9-4f10-9b69-09fe21a8a430\") " Jul 26 15:41:28.203441 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:41:28.203388 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/86d9d032-2be9-4f10-9b69-09fe21a8a430-console-oauth-config\") pod \"86d9d032-2be9-4f10-9b69-09fe21a8a430\" (UID: \"86d9d032-2be9-4f10-9b69-09fe21a8a430\") " Jul 26 15:41:28.203441 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:41:28.203439 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86d9d032-2be9-4f10-9b69-09fe21a8a430-trusted-ca-bundle\") pod \"86d9d032-2be9-4f10-9b69-09fe21a8a430\" (UID: \"86d9d032-2be9-4f10-9b69-09fe21a8a430\") " Jul 26 15:41:28.203706 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:41:28.203460 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/86d9d032-2be9-4f10-9b69-09fe21a8a430-service-ca\") pod \"86d9d032-2be9-4f10-9b69-09fe21a8a430\" (UID: \"86d9d032-2be9-4f10-9b69-09fe21a8a430\") " Jul 26 15:41:28.203706 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:41:28.203482 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mtn97\" (UniqueName: \"kubernetes.io/projected/86d9d032-2be9-4f10-9b69-09fe21a8a430-kube-api-access-mtn97\") pod \"86d9d032-2be9-4f10-9b69-09fe21a8a430\" (UID: \"86d9d032-2be9-4f10-9b69-09fe21a8a430\") " Jul 26 15:41:28.203706 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:41:28.203514 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/86d9d032-2be9-4f10-9b69-09fe21a8a430-oauth-serving-cert\") pod \"86d9d032-2be9-4f10-9b69-09fe21a8a430\" (UID: \"86d9d032-2be9-4f10-9b69-09fe21a8a430\") " Jul 26 15:41:28.203706 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:41:28.203544 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/86d9d032-2be9-4f10-9b69-09fe21a8a430-console-config\") pod \"86d9d032-2be9-4f10-9b69-09fe21a8a430\" (UID: \"86d9d032-2be9-4f10-9b69-09fe21a8a430\") " Jul 26 15:41:28.204012 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:41:28.203984 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86d9d032-2be9-4f10-9b69-09fe21a8a430-service-ca" (OuterVolumeSpecName: "service-ca") pod "86d9d032-2be9-4f10-9b69-09fe21a8a430" (UID: "86d9d032-2be9-4f10-9b69-09fe21a8a430"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 15:41:28.204012 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:41:28.203990 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86d9d032-2be9-4f10-9b69-09fe21a8a430-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "86d9d032-2be9-4f10-9b69-09fe21a8a430" (UID: "86d9d032-2be9-4f10-9b69-09fe21a8a430"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 15:41:28.204137 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:41:28.203996 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86d9d032-2be9-4f10-9b69-09fe21a8a430-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "86d9d032-2be9-4f10-9b69-09fe21a8a430" (UID: "86d9d032-2be9-4f10-9b69-09fe21a8a430"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 15:41:28.204137 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:41:28.204043 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86d9d032-2be9-4f10-9b69-09fe21a8a430-console-config" (OuterVolumeSpecName: "console-config") pod "86d9d032-2be9-4f10-9b69-09fe21a8a430" (UID: "86d9d032-2be9-4f10-9b69-09fe21a8a430"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 15:41:28.205655 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:41:28.205631 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86d9d032-2be9-4f10-9b69-09fe21a8a430-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "86d9d032-2be9-4f10-9b69-09fe21a8a430" (UID: "86d9d032-2be9-4f10-9b69-09fe21a8a430"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 26 15:41:28.205835 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:41:28.205651 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86d9d032-2be9-4f10-9b69-09fe21a8a430-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "86d9d032-2be9-4f10-9b69-09fe21a8a430" (UID: "86d9d032-2be9-4f10-9b69-09fe21a8a430"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 26 15:41:28.205835 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:41:28.205650 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86d9d032-2be9-4f10-9b69-09fe21a8a430-kube-api-access-mtn97" (OuterVolumeSpecName: "kube-api-access-mtn97") pod "86d9d032-2be9-4f10-9b69-09fe21a8a430" (UID: "86d9d032-2be9-4f10-9b69-09fe21a8a430"). InnerVolumeSpecName "kube-api-access-mtn97". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 15:41:28.304219 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:41:28.304181 2577 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/86d9d032-2be9-4f10-9b69-09fe21a8a430-console-oauth-config\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 15:41:28.304219 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:41:28.304213 2577 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86d9d032-2be9-4f10-9b69-09fe21a8a430-trusted-ca-bundle\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 15:41:28.304219 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:41:28.304224 2577 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/86d9d032-2be9-4f10-9b69-09fe21a8a430-service-ca\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 15:41:28.304481 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:41:28.304233 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtn97\" (UniqueName: \"kubernetes.io/projected/86d9d032-2be9-4f10-9b69-09fe21a8a430-kube-api-access-mtn97\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 15:41:28.304481 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:41:28.304243 2577 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/86d9d032-2be9-4f10-9b69-09fe21a8a430-oauth-serving-cert\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 15:41:28.304481 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:41:28.304252 2577 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/86d9d032-2be9-4f10-9b69-09fe21a8a430-console-config\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 15:41:28.304481 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:41:28.304260 2577 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/86d9d032-2be9-4f10-9b69-09fe21a8a430-console-serving-cert\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 15:41:28.884947 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:41:28.884905 2577 generic.go:356] "Generic (PLEG): container finished" podID="86d9d032-2be9-4f10-9b69-09fe21a8a430" containerID="7bfd1b27d58825734a99b31259f69ff0a167417eb753d8641b6a15fa460d5497" exitCode=0 Jul 26 15:41:28.884947 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:41:28.884948 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57d79d964d-2g5z4" Jul 26 15:41:28.885504 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:41:28.884984 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57d79d964d-2g5z4" event={"ID":"86d9d032-2be9-4f10-9b69-09fe21a8a430","Type":"ContainerDied","Data":"7bfd1b27d58825734a99b31259f69ff0a167417eb753d8641b6a15fa460d5497"} Jul 26 15:41:28.885504 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:41:28.885047 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57d79d964d-2g5z4" event={"ID":"86d9d032-2be9-4f10-9b69-09fe21a8a430","Type":"ContainerDied","Data":"656bfe96fa4501234f7149c08c5563688f74e8247c80bfd4f6324ce013c8e0c2"} Jul 26 15:41:28.885504 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:41:28.885122 2577 scope.go:117] "RemoveContainer" containerID="7bfd1b27d58825734a99b31259f69ff0a167417eb753d8641b6a15fa460d5497" Jul 26 15:41:28.896324 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:41:28.896301 2577 scope.go:117] "RemoveContainer" containerID="7bfd1b27d58825734a99b31259f69ff0a167417eb753d8641b6a15fa460d5497" Jul 26 15:41:28.896722 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:41:28.896699 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7bfd1b27d58825734a99b31259f69ff0a167417eb753d8641b6a15fa460d5497\": container with ID starting with 7bfd1b27d58825734a99b31259f69ff0a167417eb753d8641b6a15fa460d5497 not found: ID does not exist" containerID="7bfd1b27d58825734a99b31259f69ff0a167417eb753d8641b6a15fa460d5497" Jul 26 15:41:28.896839 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:41:28.896729 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bfd1b27d58825734a99b31259f69ff0a167417eb753d8641b6a15fa460d5497"} err="failed to get container status \"7bfd1b27d58825734a99b31259f69ff0a167417eb753d8641b6a15fa460d5497\": rpc error: code = NotFound desc = could not find container \"7bfd1b27d58825734a99b31259f69ff0a167417eb753d8641b6a15fa460d5497\": container with ID starting with 7bfd1b27d58825734a99b31259f69ff0a167417eb753d8641b6a15fa460d5497 not found: ID does not exist" Jul 26 15:41:28.906628 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:41:28.906605 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-57d79d964d-2g5z4"] Jul 26 15:41:28.910021 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:41:28.910001 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-57d79d964d-2g5z4"] Jul 26 15:41:29.786168 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:41:29.786131 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="86d9d032-2be9-4f10-9b69-09fe21a8a430" path="/var/lib/kubelet/pods/86d9d032-2be9-4f10-9b69-09fe21a8a430/volumes" Jul 26 15:41:38.911996 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:41:38.911961 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:41:38.966836 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:41:38.966801 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:41:45.390909 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:41:45.390870 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:41:45.486921 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:41:45.486890 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:41:46.031632 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:41:46.031600 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gr7zx"] Jul 26 15:41:46.031938 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:41:46.031924 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86d9d032-2be9-4f10-9b69-09fe21a8a430" containerName="console" Jul 26 15:41:46.031980 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:41:46.031941 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="86d9d032-2be9-4f10-9b69-09fe21a8a430" containerName="console" Jul 26 15:41:46.032014 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:41:46.032007 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="86d9d032-2be9-4f10-9b69-09fe21a8a430" containerName="console" Jul 26 15:41:46.034978 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:41:46.034950 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gr7zx" Jul 26 15:41:46.037344 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:41:46.037322 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 26 15:41:46.038213 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:41:46.038196 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 26 15:41:46.038268 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:41:46.038197 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-xn298\"" Jul 26 15:41:46.042549 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:41:46.042529 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gr7zx"] Jul 26 15:41:46.148903 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:41:46.148861 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/05327d3b-8c35-440c-9057-c4c9339956c9-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gr7zx\" (UID: \"05327d3b-8c35-440c-9057-c4c9339956c9\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gr7zx" Jul 26 15:41:46.149078 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:41:46.148915 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fk8q\" (UniqueName: \"kubernetes.io/projected/05327d3b-8c35-440c-9057-c4c9339956c9-kube-api-access-4fk8q\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gr7zx\" (UID: \"05327d3b-8c35-440c-9057-c4c9339956c9\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gr7zx" Jul 26 15:41:46.149078 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:41:46.148994 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/05327d3b-8c35-440c-9057-c4c9339956c9-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gr7zx\" (UID: \"05327d3b-8c35-440c-9057-c4c9339956c9\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gr7zx" Jul 26 15:41:46.249886 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:41:46.249848 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/05327d3b-8c35-440c-9057-c4c9339956c9-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gr7zx\" (UID: \"05327d3b-8c35-440c-9057-c4c9339956c9\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gr7zx" Jul 26 15:41:46.250045 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:41:46.249904 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fk8q\" (UniqueName: \"kubernetes.io/projected/05327d3b-8c35-440c-9057-c4c9339956c9-kube-api-access-4fk8q\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gr7zx\" (UID: \"05327d3b-8c35-440c-9057-c4c9339956c9\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gr7zx" Jul 26 15:41:46.250045 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:41:46.249966 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/05327d3b-8c35-440c-9057-c4c9339956c9-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gr7zx\" (UID: \"05327d3b-8c35-440c-9057-c4c9339956c9\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gr7zx" Jul 26 15:41:46.250327 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:41:46.250306 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/05327d3b-8c35-440c-9057-c4c9339956c9-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gr7zx\" (UID: \"05327d3b-8c35-440c-9057-c4c9339956c9\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gr7zx" Jul 26 15:41:46.250360 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:41:46.250315 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/05327d3b-8c35-440c-9057-c4c9339956c9-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gr7zx\" (UID: \"05327d3b-8c35-440c-9057-c4c9339956c9\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gr7zx" Jul 26 15:41:46.257595 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:41:46.257567 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fk8q\" (UniqueName: \"kubernetes.io/projected/05327d3b-8c35-440c-9057-c4c9339956c9-kube-api-access-4fk8q\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gr7zx\" (UID: \"05327d3b-8c35-440c-9057-c4c9339956c9\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gr7zx" Jul 26 15:41:46.345594 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:41:46.345493 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gr7zx" Jul 26 15:41:46.476000 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:41:46.475963 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gr7zx"] Jul 26 15:41:46.479417 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:41:46.479364 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05327d3b_8c35_440c_9057_c4c9339956c9.slice/crio-811293488b33d52e31067ee44adc3b0940508baae454dc5591d39316d277a54e WatchSource:0}: Error finding container 811293488b33d52e31067ee44adc3b0940508baae454dc5591d39316d277a54e: Status 404 returned error can't find the container with id 811293488b33d52e31067ee44adc3b0940508baae454dc5591d39316d277a54e Jul 26 15:41:46.962322 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:41:46.962276 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gr7zx" event={"ID":"05327d3b-8c35-440c-9057-c4c9339956c9","Type":"ContainerStarted","Data":"811293488b33d52e31067ee44adc3b0940508baae454dc5591d39316d277a54e"} Jul 26 15:41:52.843835 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:41:52.843787 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:41:52.988104 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:41:52.988067 2577 generic.go:356] "Generic (PLEG): container finished" podID="05327d3b-8c35-440c-9057-c4c9339956c9" containerID="18162e1111d3d413b0549acdcd4e540c80eeb969c95c6984a20c5ce8a09ecb7e" exitCode=0 Jul 26 15:41:52.988288 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:41:52.988114 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gr7zx" event={"ID":"05327d3b-8c35-440c-9057-c4c9339956c9","Type":"ContainerDied","Data":"18162e1111d3d413b0549acdcd4e540c80eeb969c95c6984a20c5ce8a09ecb7e"} Jul 26 15:41:56.002640 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:41:56.002601 2577 generic.go:356] "Generic (PLEG): container finished" podID="05327d3b-8c35-440c-9057-c4c9339956c9" containerID="5de075b00c862f8009f3fc990e8857cc0f26d1a3603c0ceb12645765a45d1947" exitCode=0 Jul 26 15:41:56.003067 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:41:56.002688 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gr7zx" event={"ID":"05327d3b-8c35-440c-9057-c4c9339956c9","Type":"ContainerDied","Data":"5de075b00c862f8009f3fc990e8857cc0f26d1a3603c0ceb12645765a45d1947"} Jul 26 15:42:03.034681 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:03.034642 2577 generic.go:356] "Generic (PLEG): container finished" podID="05327d3b-8c35-440c-9057-c4c9339956c9" containerID="693d9780953947f1a1ecc2ae7af8dddb5f74dcde661ac420e02d64997693c721" exitCode=0 Jul 26 15:42:03.035085 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:03.034729 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gr7zx" event={"ID":"05327d3b-8c35-440c-9057-c4c9339956c9","Type":"ContainerDied","Data":"693d9780953947f1a1ecc2ae7af8dddb5f74dcde661ac420e02d64997693c721"} Jul 26 15:42:04.160054 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:04.160027 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gr7zx" Jul 26 15:42:04.305307 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:04.305217 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4fk8q\" (UniqueName: \"kubernetes.io/projected/05327d3b-8c35-440c-9057-c4c9339956c9-kube-api-access-4fk8q\") pod \"05327d3b-8c35-440c-9057-c4c9339956c9\" (UID: \"05327d3b-8c35-440c-9057-c4c9339956c9\") " Jul 26 15:42:04.305307 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:04.305262 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/05327d3b-8c35-440c-9057-c4c9339956c9-bundle\") pod \"05327d3b-8c35-440c-9057-c4c9339956c9\" (UID: \"05327d3b-8c35-440c-9057-c4c9339956c9\") " Jul 26 15:42:04.305534 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:04.305330 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/05327d3b-8c35-440c-9057-c4c9339956c9-util\") pod \"05327d3b-8c35-440c-9057-c4c9339956c9\" (UID: \"05327d3b-8c35-440c-9057-c4c9339956c9\") " Jul 26 15:42:04.305986 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:04.305954 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05327d3b-8c35-440c-9057-c4c9339956c9-bundle" (OuterVolumeSpecName: "bundle") pod "05327d3b-8c35-440c-9057-c4c9339956c9" (UID: "05327d3b-8c35-440c-9057-c4c9339956c9"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:42:04.307517 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:04.307495 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05327d3b-8c35-440c-9057-c4c9339956c9-kube-api-access-4fk8q" (OuterVolumeSpecName: "kube-api-access-4fk8q") pod "05327d3b-8c35-440c-9057-c4c9339956c9" (UID: "05327d3b-8c35-440c-9057-c4c9339956c9"). InnerVolumeSpecName "kube-api-access-4fk8q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 15:42:04.310611 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:04.310580 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05327d3b-8c35-440c-9057-c4c9339956c9-util" (OuterVolumeSpecName: "util") pod "05327d3b-8c35-440c-9057-c4c9339956c9" (UID: "05327d3b-8c35-440c-9057-c4c9339956c9"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:42:04.406013 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:04.405977 2577 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/05327d3b-8c35-440c-9057-c4c9339956c9-bundle\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 15:42:04.406013 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:04.406006 2577 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/05327d3b-8c35-440c-9057-c4c9339956c9-util\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 15:42:04.406013 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:04.406016 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fk8q\" (UniqueName: \"kubernetes.io/projected/05327d3b-8c35-440c-9057-c4c9339956c9-kube-api-access-4fk8q\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 15:42:05.043819 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:05.043792 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gr7zx" Jul 26 15:42:05.043983 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:05.043792 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gr7zx" event={"ID":"05327d3b-8c35-440c-9057-c4c9339956c9","Type":"ContainerDied","Data":"811293488b33d52e31067ee44adc3b0940508baae454dc5591d39316d277a54e"} Jul 26 15:42:05.043983 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:05.043919 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="811293488b33d52e31067ee44adc3b0940508baae454dc5591d39316d277a54e" Jul 26 15:42:06.814658 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:06.814612 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:42:08.196371 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:08.196327 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:42:08.812390 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:08.812348 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-z7vrb"] Jul 26 15:42:08.812996 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:08.812971 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05327d3b-8c35-440c-9057-c4c9339956c9" containerName="util" Jul 26 15:42:08.813133 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:08.813000 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="05327d3b-8c35-440c-9057-c4c9339956c9" containerName="util" Jul 26 15:42:08.813133 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:08.813044 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05327d3b-8c35-440c-9057-c4c9339956c9" containerName="pull" Jul 26 15:42:08.813133 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:08.813054 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="05327d3b-8c35-440c-9057-c4c9339956c9" containerName="pull" Jul 26 15:42:08.813133 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:08.813099 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05327d3b-8c35-440c-9057-c4c9339956c9" containerName="extract" Jul 26 15:42:08.813133 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:08.813109 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="05327d3b-8c35-440c-9057-c4c9339956c9" containerName="extract" Jul 26 15:42:08.813439 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:08.813244 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="05327d3b-8c35-440c-9057-c4c9339956c9" containerName="extract" Jul 26 15:42:08.817747 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:08.817726 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-z7vrb" Jul 26 15:42:08.820159 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:08.820138 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Jul 26 15:42:08.820267 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:08.820139 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Jul 26 15:42:08.820267 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:08.820184 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-rgk5m\"" Jul 26 15:42:08.828439 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:08.828414 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-z7vrb"] Jul 26 15:42:08.942620 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:08.942584 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2fa1220d-007b-4a62-9b52-268a0fcc848f-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-z7vrb\" (UID: \"2fa1220d-007b-4a62-9b52-268a0fcc848f\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-z7vrb" Jul 26 15:42:08.942840 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:08.942793 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdrts\" (UniqueName: \"kubernetes.io/projected/2fa1220d-007b-4a62-9b52-268a0fcc848f-kube-api-access-zdrts\") pod \"cert-manager-operator-controller-manager-55c444bf86-z7vrb\" (UID: \"2fa1220d-007b-4a62-9b52-268a0fcc848f\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-z7vrb" Jul 26 15:42:09.044147 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:09.044112 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2fa1220d-007b-4a62-9b52-268a0fcc848f-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-z7vrb\" (UID: \"2fa1220d-007b-4a62-9b52-268a0fcc848f\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-z7vrb" Jul 26 15:42:09.044301 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:09.044213 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdrts\" (UniqueName: \"kubernetes.io/projected/2fa1220d-007b-4a62-9b52-268a0fcc848f-kube-api-access-zdrts\") pod \"cert-manager-operator-controller-manager-55c444bf86-z7vrb\" (UID: \"2fa1220d-007b-4a62-9b52-268a0fcc848f\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-z7vrb" Jul 26 15:42:09.044648 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:09.044627 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2fa1220d-007b-4a62-9b52-268a0fcc848f-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-z7vrb\" (UID: \"2fa1220d-007b-4a62-9b52-268a0fcc848f\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-z7vrb" Jul 26 15:42:09.053029 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:09.053004 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdrts\" (UniqueName: \"kubernetes.io/projected/2fa1220d-007b-4a62-9b52-268a0fcc848f-kube-api-access-zdrts\") pod \"cert-manager-operator-controller-manager-55c444bf86-z7vrb\" (UID: \"2fa1220d-007b-4a62-9b52-268a0fcc848f\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-z7vrb" Jul 26 15:42:09.126378 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:09.126292 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-z7vrb" Jul 26 15:42:09.264859 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:09.264827 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-z7vrb"] Jul 26 15:42:09.269346 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:42:09.269313 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fa1220d_007b_4a62_9b52_268a0fcc848f.slice/crio-78bcbf2b7b4b2164d0a77ff39a679a75febfdd2fa2f5f600da959fb1a573c46b WatchSource:0}: Error finding container 78bcbf2b7b4b2164d0a77ff39a679a75febfdd2fa2f5f600da959fb1a573c46b: Status 404 returned error can't find the container with id 78bcbf2b7b4b2164d0a77ff39a679a75febfdd2fa2f5f600da959fb1a573c46b Jul 26 15:42:10.065220 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:10.065177 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-z7vrb" event={"ID":"2fa1220d-007b-4a62-9b52-268a0fcc848f","Type":"ContainerStarted","Data":"78bcbf2b7b4b2164d0a77ff39a679a75febfdd2fa2f5f600da959fb1a573c46b"} Jul 26 15:42:10.623457 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:10.623416 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:42:10.723673 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:10.723628 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:42:11.071852 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:11.071807 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-z7vrb" event={"ID":"2fa1220d-007b-4a62-9b52-268a0fcc848f","Type":"ContainerStarted","Data":"88e7d49cd5b2a8f873882682d8cde7e2817e020da5aa037c79559d101c310d79"} Jul 26 15:42:11.101328 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:11.101267 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-z7vrb" podStartSLOduration=1.522200969 podStartE2EDuration="3.101251462s" totalImagesPullingTime="1.579050493s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:42:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:42:11.101007423 +0000 UTC m=+253.840360767" watchObservedRunningTime="2026-07-26 15:42:11.101251462 +0000 UTC m=+253.840604772" Jul 26 15:42:11.467332 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:11.467290 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:42:11.587182 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:11.587141 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:42:11.724455 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:11.724344 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:42:12.244171 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:12.244130 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwk28r"] Jul 26 15:42:12.246482 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:12.246465 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwk28r" Jul 26 15:42:12.248745 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:12.248721 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 26 15:42:12.249658 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:12.249638 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 26 15:42:12.249751 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:12.249689 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-xn298\"" Jul 26 15:42:12.255838 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:12.255810 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwk28r"] Jul 26 15:42:12.373312 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:12.373275 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rj2zr\" (UniqueName: \"kubernetes.io/projected/27f4c49d-269e-4818-a19d-22b7a7c15a1e-kube-api-access-rj2zr\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwk28r\" (UID: \"27f4c49d-269e-4818-a19d-22b7a7c15a1e\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwk28r" Jul 26 15:42:12.373312 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:12.373318 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/27f4c49d-269e-4818-a19d-22b7a7c15a1e-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwk28r\" (UID: \"27f4c49d-269e-4818-a19d-22b7a7c15a1e\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwk28r" Jul 26 15:42:12.373554 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:12.373384 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/27f4c49d-269e-4818-a19d-22b7a7c15a1e-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwk28r\" (UID: \"27f4c49d-269e-4818-a19d-22b7a7c15a1e\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwk28r" Jul 26 15:42:12.474757 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:12.474718 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rj2zr\" (UniqueName: \"kubernetes.io/projected/27f4c49d-269e-4818-a19d-22b7a7c15a1e-kube-api-access-rj2zr\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwk28r\" (UID: \"27f4c49d-269e-4818-a19d-22b7a7c15a1e\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwk28r" Jul 26 15:42:12.474757 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:12.474764 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/27f4c49d-269e-4818-a19d-22b7a7c15a1e-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwk28r\" (UID: \"27f4c49d-269e-4818-a19d-22b7a7c15a1e\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwk28r" Jul 26 15:42:12.474932 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:12.474798 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/27f4c49d-269e-4818-a19d-22b7a7c15a1e-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwk28r\" (UID: \"27f4c49d-269e-4818-a19d-22b7a7c15a1e\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwk28r" Jul 26 15:42:12.475255 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:12.475232 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/27f4c49d-269e-4818-a19d-22b7a7c15a1e-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwk28r\" (UID: \"27f4c49d-269e-4818-a19d-22b7a7c15a1e\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwk28r" Jul 26 15:42:12.475293 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:12.475240 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/27f4c49d-269e-4818-a19d-22b7a7c15a1e-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwk28r\" (UID: \"27f4c49d-269e-4818-a19d-22b7a7c15a1e\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwk28r" Jul 26 15:42:12.490661 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:12.490634 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rj2zr\" (UniqueName: \"kubernetes.io/projected/27f4c49d-269e-4818-a19d-22b7a7c15a1e-kube-api-access-rj2zr\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwk28r\" (UID: \"27f4c49d-269e-4818-a19d-22b7a7c15a1e\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwk28r" Jul 26 15:42:12.555720 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:12.555687 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwk28r" Jul 26 15:42:12.687878 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:12.687732 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwk28r"] Jul 26 15:42:12.690470 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:42:12.690439 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27f4c49d_269e_4818_a19d_22b7a7c15a1e.slice/crio-6985fb658e04d7a25036f3aa66511be5912402055266a6a65ac717a48474badc WatchSource:0}: Error finding container 6985fb658e04d7a25036f3aa66511be5912402055266a6a65ac717a48474badc: Status 404 returned error can't find the container with id 6985fb658e04d7a25036f3aa66511be5912402055266a6a65ac717a48474badc Jul 26 15:42:13.082011 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:13.081973 2577 generic.go:356] "Generic (PLEG): container finished" podID="27f4c49d-269e-4818-a19d-22b7a7c15a1e" containerID="29d7b2cb2b123967606bbab8ce212ea447aa300e806a6454d9bbb410b4180330" exitCode=0 Jul 26 15:42:13.082383 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:13.082049 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwk28r" event={"ID":"27f4c49d-269e-4818-a19d-22b7a7c15a1e","Type":"ContainerDied","Data":"29d7b2cb2b123967606bbab8ce212ea447aa300e806a6454d9bbb410b4180330"} Jul 26 15:42:13.082383 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:13.082100 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwk28r" event={"ID":"27f4c49d-269e-4818-a19d-22b7a7c15a1e","Type":"ContainerStarted","Data":"6985fb658e04d7a25036f3aa66511be5912402055266a6a65ac717a48474badc"} Jul 26 15:42:14.019523 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:14.019491 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:42:14.244561 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:14.244506 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:42:16.096868 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:16.096837 2577 generic.go:356] "Generic (PLEG): container finished" podID="27f4c49d-269e-4818-a19d-22b7a7c15a1e" containerID="d397a205f3dab4558638f3c43be3f1f1dd014e8c5aa6fd576cb67c8922110b63" exitCode=0 Jul 26 15:42:16.097224 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:16.096918 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwk28r" event={"ID":"27f4c49d-269e-4818-a19d-22b7a7c15a1e","Type":"ContainerDied","Data":"d397a205f3dab4558638f3c43be3f1f1dd014e8c5aa6fd576cb67c8922110b63"} Jul 26 15:42:17.102951 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:17.102917 2577 generic.go:356] "Generic (PLEG): container finished" podID="27f4c49d-269e-4818-a19d-22b7a7c15a1e" containerID="8ba1e82cf14382e7b490121ff148f5a5bcf677fc0c12ddd5101b14950c53befb" exitCode=0 Jul 26 15:42:17.103329 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:17.103002 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwk28r" event={"ID":"27f4c49d-269e-4818-a19d-22b7a7c15a1e","Type":"ContainerDied","Data":"8ba1e82cf14382e7b490121ff148f5a5bcf677fc0c12ddd5101b14950c53befb"} Jul 26 15:42:17.608917 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:17.608876 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:42:17.710854 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:17.710820 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:42:18.232198 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:18.232171 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwk28r" Jul 26 15:42:18.324459 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:18.324422 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/27f4c49d-269e-4818-a19d-22b7a7c15a1e-bundle\") pod \"27f4c49d-269e-4818-a19d-22b7a7c15a1e\" (UID: \"27f4c49d-269e-4818-a19d-22b7a7c15a1e\") " Jul 26 15:42:18.324659 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:18.324482 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/27f4c49d-269e-4818-a19d-22b7a7c15a1e-util\") pod \"27f4c49d-269e-4818-a19d-22b7a7c15a1e\" (UID: \"27f4c49d-269e-4818-a19d-22b7a7c15a1e\") " Jul 26 15:42:18.324659 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:18.324544 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rj2zr\" (UniqueName: \"kubernetes.io/projected/27f4c49d-269e-4818-a19d-22b7a7c15a1e-kube-api-access-rj2zr\") pod \"27f4c49d-269e-4818-a19d-22b7a7c15a1e\" (UID: \"27f4c49d-269e-4818-a19d-22b7a7c15a1e\") " Jul 26 15:42:18.324897 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:18.324871 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27f4c49d-269e-4818-a19d-22b7a7c15a1e-bundle" (OuterVolumeSpecName: "bundle") pod "27f4c49d-269e-4818-a19d-22b7a7c15a1e" (UID: "27f4c49d-269e-4818-a19d-22b7a7c15a1e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:42:18.326650 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:18.326628 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27f4c49d-269e-4818-a19d-22b7a7c15a1e-kube-api-access-rj2zr" (OuterVolumeSpecName: "kube-api-access-rj2zr") pod "27f4c49d-269e-4818-a19d-22b7a7c15a1e" (UID: "27f4c49d-269e-4818-a19d-22b7a7c15a1e"). InnerVolumeSpecName "kube-api-access-rj2zr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 15:42:18.329596 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:18.329560 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27f4c49d-269e-4818-a19d-22b7a7c15a1e-util" (OuterVolumeSpecName: "util") pod "27f4c49d-269e-4818-a19d-22b7a7c15a1e" (UID: "27f4c49d-269e-4818-a19d-22b7a7c15a1e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:42:18.425855 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:18.425772 2577 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/27f4c49d-269e-4818-a19d-22b7a7c15a1e-util\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 15:42:18.425855 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:18.425805 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rj2zr\" (UniqueName: \"kubernetes.io/projected/27f4c49d-269e-4818-a19d-22b7a7c15a1e-kube-api-access-rj2zr\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 15:42:18.425855 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:18.425816 2577 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/27f4c49d-269e-4818-a19d-22b7a7c15a1e-bundle\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 15:42:19.113685 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:19.113654 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwk28r" Jul 26 15:42:19.113841 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:19.113683 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwk28r" event={"ID":"27f4c49d-269e-4818-a19d-22b7a7c15a1e","Type":"ContainerDied","Data":"6985fb658e04d7a25036f3aa66511be5912402055266a6a65ac717a48474badc"} Jul 26 15:42:19.113841 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:19.113729 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6985fb658e04d7a25036f3aa66511be5912402055266a6a65ac717a48474badc" Jul 26 15:42:21.630815 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:21.630727 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:42:22.623191 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:22.623155 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:42:22.720566 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:22.720528 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:42:23.573083 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:23.573048 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-dq29z"] Jul 26 15:42:23.573347 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:23.573335 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27f4c49d-269e-4818-a19d-22b7a7c15a1e" containerName="util" Jul 26 15:42:23.573420 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:23.573349 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="27f4c49d-269e-4818-a19d-22b7a7c15a1e" containerName="util" Jul 26 15:42:23.573420 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:23.573370 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27f4c49d-269e-4818-a19d-22b7a7c15a1e" containerName="pull" Jul 26 15:42:23.573420 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:23.573376 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="27f4c49d-269e-4818-a19d-22b7a7c15a1e" containerName="pull" Jul 26 15:42:23.573420 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:23.573383 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27f4c49d-269e-4818-a19d-22b7a7c15a1e" containerName="extract" Jul 26 15:42:23.573420 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:23.573389 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="27f4c49d-269e-4818-a19d-22b7a7c15a1e" containerName="extract" Jul 26 15:42:23.573569 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:23.573451 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="27f4c49d-269e-4818-a19d-22b7a7c15a1e" containerName="extract" Jul 26 15:42:23.576246 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:23.576229 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-dq29z" Jul 26 15:42:23.579201 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:23.579170 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 26 15:42:23.579326 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:23.579169 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 26 15:42:23.580032 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:23.580016 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-j544b\"" Jul 26 15:42:23.586227 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:23.586200 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-dq29z"] Jul 26 15:42:23.667369 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:23.667330 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rsf7\" (UniqueName: \"kubernetes.io/projected/11cd225a-0889-4329-a58f-2d81d38d2203-kube-api-access-4rsf7\") pod \"cert-manager-55b4d954d5-dq29z\" (UID: \"11cd225a-0889-4329-a58f-2d81d38d2203\") " pod="cert-manager/cert-manager-55b4d954d5-dq29z" Jul 26 15:42:23.667369 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:23.667371 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/11cd225a-0889-4329-a58f-2d81d38d2203-bound-sa-token\") pod \"cert-manager-55b4d954d5-dq29z\" (UID: \"11cd225a-0889-4329-a58f-2d81d38d2203\") " pod="cert-manager/cert-manager-55b4d954d5-dq29z" Jul 26 15:42:23.768685 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:23.768647 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rsf7\" (UniqueName: \"kubernetes.io/projected/11cd225a-0889-4329-a58f-2d81d38d2203-kube-api-access-4rsf7\") pod \"cert-manager-55b4d954d5-dq29z\" (UID: \"11cd225a-0889-4329-a58f-2d81d38d2203\") " pod="cert-manager/cert-manager-55b4d954d5-dq29z" Jul 26 15:42:23.769053 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:23.768690 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/11cd225a-0889-4329-a58f-2d81d38d2203-bound-sa-token\") pod \"cert-manager-55b4d954d5-dq29z\" (UID: \"11cd225a-0889-4329-a58f-2d81d38d2203\") " pod="cert-manager/cert-manager-55b4d954d5-dq29z" Jul 26 15:42:23.779055 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:23.779026 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rsf7\" (UniqueName: \"kubernetes.io/projected/11cd225a-0889-4329-a58f-2d81d38d2203-kube-api-access-4rsf7\") pod \"cert-manager-55b4d954d5-dq29z\" (UID: \"11cd225a-0889-4329-a58f-2d81d38d2203\") " pod="cert-manager/cert-manager-55b4d954d5-dq29z" Jul 26 15:42:23.779622 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:23.779604 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/11cd225a-0889-4329-a58f-2d81d38d2203-bound-sa-token\") pod \"cert-manager-55b4d954d5-dq29z\" (UID: \"11cd225a-0889-4329-a58f-2d81d38d2203\") " pod="cert-manager/cert-manager-55b4d954d5-dq29z" Jul 26 15:42:23.895748 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:23.895657 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-dq29z" Jul 26 15:42:24.023904 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:24.023872 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-dq29z"] Jul 26 15:42:24.026841 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:42:24.026817 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod11cd225a_0889_4329_a58f_2d81d38d2203.slice/crio-f77d09c2fc387cc7f552ec11f7bc5e54fa7e158de438090433732b42f76cbbe0 WatchSource:0}: Error finding container f77d09c2fc387cc7f552ec11f7bc5e54fa7e158de438090433732b42f76cbbe0: Status 404 returned error can't find the container with id f77d09c2fc387cc7f552ec11f7bc5e54fa7e158de438090433732b42f76cbbe0 Jul 26 15:42:24.134020 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:24.133979 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-dq29z" event={"ID":"11cd225a-0889-4329-a58f-2d81d38d2203","Type":"ContainerStarted","Data":"f77d09c2fc387cc7f552ec11f7bc5e54fa7e158de438090433732b42f76cbbe0"} Jul 26 15:42:27.148115 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:27.148019 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-dq29z" event={"ID":"11cd225a-0889-4329-a58f-2d81d38d2203","Type":"ContainerStarted","Data":"9749aec9b25a8b8c9ea701e1a6d36c5e4d92fee6b4e56bd93d993b30d9ead488"} Jul 26 15:42:27.164196 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:27.164140 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-dq29z" podStartSLOduration=1.3738816919999999 podStartE2EDuration="4.164120381s" totalImagesPullingTime="2.790238689s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:42:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:42:27.161767548 +0000 UTC m=+269.901120858" watchObservedRunningTime="2026-07-26 15:42:27.164120381 +0000 UTC m=+269.903473691" Jul 26 15:42:29.753292 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:29.753248 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:42:29.856405 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:29.856371 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:42:31.615993 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:31.615963 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:42:31.702620 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:31.699778 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbmlnt"] Jul 26 15:42:31.704972 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:31.704937 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbmlnt" Jul 26 15:42:31.707362 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:31.707339 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 26 15:42:31.707486 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:31.707383 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-xn298\"" Jul 26 15:42:31.707486 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:31.707383 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 26 15:42:31.710315 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:31.710292 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbmlnt"] Jul 26 15:42:31.732709 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:31.732679 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dee8be2b-3235-4e79-96e0-2fc33bcac868-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbmlnt\" (UID: \"dee8be2b-3235-4e79-96e0-2fc33bcac868\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbmlnt" Jul 26 15:42:31.732855 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:31.732795 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsdxq\" (UniqueName: \"kubernetes.io/projected/dee8be2b-3235-4e79-96e0-2fc33bcac868-kube-api-access-wsdxq\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbmlnt\" (UID: \"dee8be2b-3235-4e79-96e0-2fc33bcac868\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbmlnt" Jul 26 15:42:31.732925 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:31.732865 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dee8be2b-3235-4e79-96e0-2fc33bcac868-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbmlnt\" (UID: \"dee8be2b-3235-4e79-96e0-2fc33bcac868\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbmlnt" Jul 26 15:42:31.833464 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:31.833425 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dee8be2b-3235-4e79-96e0-2fc33bcac868-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbmlnt\" (UID: \"dee8be2b-3235-4e79-96e0-2fc33bcac868\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbmlnt" Jul 26 15:42:31.833648 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:31.833506 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsdxq\" (UniqueName: \"kubernetes.io/projected/dee8be2b-3235-4e79-96e0-2fc33bcac868-kube-api-access-wsdxq\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbmlnt\" (UID: \"dee8be2b-3235-4e79-96e0-2fc33bcac868\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbmlnt" Jul 26 15:42:31.833648 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:31.833543 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dee8be2b-3235-4e79-96e0-2fc33bcac868-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbmlnt\" (UID: \"dee8be2b-3235-4e79-96e0-2fc33bcac868\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbmlnt" Jul 26 15:42:31.833915 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:31.833891 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dee8be2b-3235-4e79-96e0-2fc33bcac868-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbmlnt\" (UID: \"dee8be2b-3235-4e79-96e0-2fc33bcac868\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbmlnt" Jul 26 15:42:31.833951 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:31.833901 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dee8be2b-3235-4e79-96e0-2fc33bcac868-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbmlnt\" (UID: \"dee8be2b-3235-4e79-96e0-2fc33bcac868\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbmlnt" Jul 26 15:42:31.841219 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:31.841190 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsdxq\" (UniqueName: \"kubernetes.io/projected/dee8be2b-3235-4e79-96e0-2fc33bcac868-kube-api-access-wsdxq\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbmlnt\" (UID: \"dee8be2b-3235-4e79-96e0-2fc33bcac868\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbmlnt" Jul 26 15:42:32.015865 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:32.015771 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbmlnt" Jul 26 15:42:32.140964 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:32.140825 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbmlnt"] Jul 26 15:42:32.143656 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:42:32.143626 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddee8be2b_3235_4e79_96e0_2fc33bcac868.slice/crio-dc3e9b1e02ca84899b99eebba28d928550bf9c9684147127b7531455f22506d0 WatchSource:0}: Error finding container dc3e9b1e02ca84899b99eebba28d928550bf9c9684147127b7531455f22506d0: Status 404 returned error can't find the container with id dc3e9b1e02ca84899b99eebba28d928550bf9c9684147127b7531455f22506d0 Jul 26 15:42:32.170836 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:32.170805 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbmlnt" event={"ID":"dee8be2b-3235-4e79-96e0-2fc33bcac868","Type":"ContainerStarted","Data":"dc3e9b1e02ca84899b99eebba28d928550bf9c9684147127b7531455f22506d0"} Jul 26 15:42:33.176751 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:33.176665 2577 generic.go:356] "Generic (PLEG): container finished" podID="dee8be2b-3235-4e79-96e0-2fc33bcac868" containerID="e7d6234527ebbd5ae294b533df823048c925f46f1ddcdd15ec6cc94e1de15f79" exitCode=0 Jul 26 15:42:33.177242 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:33.176752 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbmlnt" event={"ID":"dee8be2b-3235-4e79-96e0-2fc33bcac868","Type":"ContainerDied","Data":"e7d6234527ebbd5ae294b533df823048c925f46f1ddcdd15ec6cc94e1de15f79"} Jul 26 15:42:34.183040 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:34.183007 2577 generic.go:356] "Generic (PLEG): container finished" podID="dee8be2b-3235-4e79-96e0-2fc33bcac868" containerID="f9b5ea06828b622464293b5614506ce17f69bb4634f814d761da3bde5623f20c" exitCode=0 Jul 26 15:42:34.183457 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:34.183102 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbmlnt" event={"ID":"dee8be2b-3235-4e79-96e0-2fc33bcac868","Type":"ContainerDied","Data":"f9b5ea06828b622464293b5614506ce17f69bb4634f814d761da3bde5623f20c"} Jul 26 15:42:34.585480 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:34.585388 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:42:34.638615 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:34.638572 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:42:35.189837 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:35.189802 2577 generic.go:356] "Generic (PLEG): container finished" podID="dee8be2b-3235-4e79-96e0-2fc33bcac868" containerID="c26b8b4ff7e8414efa81c06d7da0ff4d027880ef8e8902824a4c35385f9f2d3a" exitCode=0 Jul 26 15:42:35.189837 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:35.189841 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbmlnt" event={"ID":"dee8be2b-3235-4e79-96e0-2fc33bcac868","Type":"ContainerDied","Data":"c26b8b4ff7e8414efa81c06d7da0ff4d027880ef8e8902824a4c35385f9f2d3a"} Jul 26 15:42:36.346444 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:36.346407 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbmlnt" Jul 26 15:42:36.470905 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:36.470808 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dee8be2b-3235-4e79-96e0-2fc33bcac868-util\") pod \"dee8be2b-3235-4e79-96e0-2fc33bcac868\" (UID: \"dee8be2b-3235-4e79-96e0-2fc33bcac868\") " Jul 26 15:42:36.470905 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:36.470872 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dee8be2b-3235-4e79-96e0-2fc33bcac868-bundle\") pod \"dee8be2b-3235-4e79-96e0-2fc33bcac868\" (UID: \"dee8be2b-3235-4e79-96e0-2fc33bcac868\") " Jul 26 15:42:36.471141 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:36.470927 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wsdxq\" (UniqueName: \"kubernetes.io/projected/dee8be2b-3235-4e79-96e0-2fc33bcac868-kube-api-access-wsdxq\") pod \"dee8be2b-3235-4e79-96e0-2fc33bcac868\" (UID: \"dee8be2b-3235-4e79-96e0-2fc33bcac868\") " Jul 26 15:42:36.471605 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:36.471577 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dee8be2b-3235-4e79-96e0-2fc33bcac868-bundle" (OuterVolumeSpecName: "bundle") pod "dee8be2b-3235-4e79-96e0-2fc33bcac868" (UID: "dee8be2b-3235-4e79-96e0-2fc33bcac868"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:42:36.473022 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:36.473001 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dee8be2b-3235-4e79-96e0-2fc33bcac868-kube-api-access-wsdxq" (OuterVolumeSpecName: "kube-api-access-wsdxq") pod "dee8be2b-3235-4e79-96e0-2fc33bcac868" (UID: "dee8be2b-3235-4e79-96e0-2fc33bcac868"). InnerVolumeSpecName "kube-api-access-wsdxq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 15:42:36.476477 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:36.476440 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dee8be2b-3235-4e79-96e0-2fc33bcac868-util" (OuterVolumeSpecName: "util") pod "dee8be2b-3235-4e79-96e0-2fc33bcac868" (UID: "dee8be2b-3235-4e79-96e0-2fc33bcac868"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:42:36.571757 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:36.571718 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wsdxq\" (UniqueName: \"kubernetes.io/projected/dee8be2b-3235-4e79-96e0-2fc33bcac868-kube-api-access-wsdxq\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 15:42:36.571757 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:36.571750 2577 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dee8be2b-3235-4e79-96e0-2fc33bcac868-util\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 15:42:36.571757 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:36.571760 2577 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dee8be2b-3235-4e79-96e0-2fc33bcac868-bundle\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 15:42:37.201553 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:37.201521 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbmlnt" Jul 26 15:42:37.201720 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:37.201520 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbmlnt" event={"ID":"dee8be2b-3235-4e79-96e0-2fc33bcac868","Type":"ContainerDied","Data":"dc3e9b1e02ca84899b99eebba28d928550bf9c9684147127b7531455f22506d0"} Jul 26 15:42:37.201720 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:37.201647 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dc3e9b1e02ca84899b99eebba28d928550bf9c9684147127b7531455f22506d0" Jul 26 15:42:43.489772 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:43.489727 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:42:43.590302 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:43.590260 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:42:44.031202 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:44.031157 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:42:44.246829 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:44.246794 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:42:44.342630 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:44.342545 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:42:44.365197 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:44.365167 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-66d4685587-5rddj"] Jul 26 15:42:44.365496 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:44.365473 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dee8be2b-3235-4e79-96e0-2fc33bcac868" containerName="pull" Jul 26 15:42:44.365496 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:44.365498 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="dee8be2b-3235-4e79-96e0-2fc33bcac868" containerName="pull" Jul 26 15:42:44.365583 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:44.365514 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dee8be2b-3235-4e79-96e0-2fc33bcac868" containerName="util" Jul 26 15:42:44.365583 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:44.365520 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="dee8be2b-3235-4e79-96e0-2fc33bcac868" containerName="util" Jul 26 15:42:44.365583 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:44.365530 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dee8be2b-3235-4e79-96e0-2fc33bcac868" containerName="extract" Jul 26 15:42:44.365583 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:44.365537 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="dee8be2b-3235-4e79-96e0-2fc33bcac868" containerName="extract" Jul 26 15:42:44.365701 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:44.365592 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="dee8be2b-3235-4e79-96e0-2fc33bcac868" containerName="extract" Jul 26 15:42:44.371661 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:44.371638 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-66d4685587-5rddj" Jul 26 15:42:44.374022 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:44.373998 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 26 15:42:44.374274 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:44.374250 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 26 15:42:44.374412 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:44.374321 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-tgn28\"" Jul 26 15:42:44.374412 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:44.374382 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 26 15:42:44.374544 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:44.374413 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 26 15:42:44.386204 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:44.386173 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-66d4685587-5rddj"] Jul 26 15:42:44.433936 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:44.433900 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/881b4d20-4178-492f-8183-7d1011be3602-apiservice-cert\") pod \"opendatahub-operator-controller-manager-66d4685587-5rddj\" (UID: \"881b4d20-4178-492f-8183-7d1011be3602\") " pod="opendatahub/opendatahub-operator-controller-manager-66d4685587-5rddj" Jul 26 15:42:44.433936 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:44.433941 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/881b4d20-4178-492f-8183-7d1011be3602-manifests\") pod \"opendatahub-operator-controller-manager-66d4685587-5rddj\" (UID: \"881b4d20-4178-492f-8183-7d1011be3602\") " pod="opendatahub/opendatahub-operator-controller-manager-66d4685587-5rddj" Jul 26 15:42:44.434140 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:44.433985 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfs98\" (UniqueName: \"kubernetes.io/projected/881b4d20-4178-492f-8183-7d1011be3602-kube-api-access-dfs98\") pod \"opendatahub-operator-controller-manager-66d4685587-5rddj\" (UID: \"881b4d20-4178-492f-8183-7d1011be3602\") " pod="opendatahub/opendatahub-operator-controller-manager-66d4685587-5rddj" Jul 26 15:42:44.434140 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:44.434093 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/881b4d20-4178-492f-8183-7d1011be3602-tmp\") pod \"opendatahub-operator-controller-manager-66d4685587-5rddj\" (UID: \"881b4d20-4178-492f-8183-7d1011be3602\") " pod="opendatahub/opendatahub-operator-controller-manager-66d4685587-5rddj" Jul 26 15:42:44.434208 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:44.434154 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/881b4d20-4178-492f-8183-7d1011be3602-webhook-cert\") pod \"opendatahub-operator-controller-manager-66d4685587-5rddj\" (UID: \"881b4d20-4178-492f-8183-7d1011be3602\") " pod="opendatahub/opendatahub-operator-controller-manager-66d4685587-5rddj" Jul 26 15:42:44.494058 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:44.494023 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:42:44.535239 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:44.535203 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/881b4d20-4178-492f-8183-7d1011be3602-apiservice-cert\") pod \"opendatahub-operator-controller-manager-66d4685587-5rddj\" (UID: \"881b4d20-4178-492f-8183-7d1011be3602\") " pod="opendatahub/opendatahub-operator-controller-manager-66d4685587-5rddj" Jul 26 15:42:44.535239 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:44.535244 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/881b4d20-4178-492f-8183-7d1011be3602-manifests\") pod \"opendatahub-operator-controller-manager-66d4685587-5rddj\" (UID: \"881b4d20-4178-492f-8183-7d1011be3602\") " pod="opendatahub/opendatahub-operator-controller-manager-66d4685587-5rddj" Jul 26 15:42:44.535479 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:44.535270 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfs98\" (UniqueName: \"kubernetes.io/projected/881b4d20-4178-492f-8183-7d1011be3602-kube-api-access-dfs98\") pod \"opendatahub-operator-controller-manager-66d4685587-5rddj\" (UID: \"881b4d20-4178-492f-8183-7d1011be3602\") " pod="opendatahub/opendatahub-operator-controller-manager-66d4685587-5rddj" Jul 26 15:42:44.535479 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:44.535339 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/881b4d20-4178-492f-8183-7d1011be3602-tmp\") pod \"opendatahub-operator-controller-manager-66d4685587-5rddj\" (UID: \"881b4d20-4178-492f-8183-7d1011be3602\") " pod="opendatahub/opendatahub-operator-controller-manager-66d4685587-5rddj" Jul 26 15:42:44.535479 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:44.535392 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/881b4d20-4178-492f-8183-7d1011be3602-webhook-cert\") pod \"opendatahub-operator-controller-manager-66d4685587-5rddj\" (UID: \"881b4d20-4178-492f-8183-7d1011be3602\") " pod="opendatahub/opendatahub-operator-controller-manager-66d4685587-5rddj" Jul 26 15:42:44.535692 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:44.535666 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/881b4d20-4178-492f-8183-7d1011be3602-manifests\") pod \"opendatahub-operator-controller-manager-66d4685587-5rddj\" (UID: \"881b4d20-4178-492f-8183-7d1011be3602\") " pod="opendatahub/opendatahub-operator-controller-manager-66d4685587-5rddj" Jul 26 15:42:44.537658 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:44.537626 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/881b4d20-4178-492f-8183-7d1011be3602-tmp\") pod \"opendatahub-operator-controller-manager-66d4685587-5rddj\" (UID: \"881b4d20-4178-492f-8183-7d1011be3602\") " pod="opendatahub/opendatahub-operator-controller-manager-66d4685587-5rddj" Jul 26 15:42:44.537907 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:44.537889 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/881b4d20-4178-492f-8183-7d1011be3602-apiservice-cert\") pod \"opendatahub-operator-controller-manager-66d4685587-5rddj\" (UID: \"881b4d20-4178-492f-8183-7d1011be3602\") " pod="opendatahub/opendatahub-operator-controller-manager-66d4685587-5rddj" Jul 26 15:42:44.537907 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:44.537897 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/881b4d20-4178-492f-8183-7d1011be3602-webhook-cert\") pod \"opendatahub-operator-controller-manager-66d4685587-5rddj\" (UID: \"881b4d20-4178-492f-8183-7d1011be3602\") " pod="opendatahub/opendatahub-operator-controller-manager-66d4685587-5rddj" Jul 26 15:42:44.542740 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:44.542721 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfs98\" (UniqueName: \"kubernetes.io/projected/881b4d20-4178-492f-8183-7d1011be3602-kube-api-access-dfs98\") pod \"opendatahub-operator-controller-manager-66d4685587-5rddj\" (UID: \"881b4d20-4178-492f-8183-7d1011be3602\") " pod="opendatahub/opendatahub-operator-controller-manager-66d4685587-5rddj" Jul 26 15:42:44.590248 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:44.590214 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:42:44.681466 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:44.681339 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-66d4685587-5rddj" Jul 26 15:42:44.690175 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:44.690138 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:42:44.793651 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:44.793574 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:42:44.820252 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:44.820227 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-66d4685587-5rddj"] Jul 26 15:42:44.822586 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:42:44.822555 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod881b4d20_4178_492f_8183_7d1011be3602.slice/crio-05c8630108aaa8b5dd61ccada05fe88816fd4fcccaa7d6adb57ddd361e63740d WatchSource:0}: Error finding container 05c8630108aaa8b5dd61ccada05fe88816fd4fcccaa7d6adb57ddd361e63740d: Status 404 returned error can't find the container with id 05c8630108aaa8b5dd61ccada05fe88816fd4fcccaa7d6adb57ddd361e63740d Jul 26 15:42:44.893537 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:44.893507 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:42:44.994996 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:44.994906 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:42:45.239134 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:45.239100 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-66d4685587-5rddj" event={"ID":"881b4d20-4178-492f-8183-7d1011be3602","Type":"ContainerStarted","Data":"05c8630108aaa8b5dd61ccada05fe88816fd4fcccaa7d6adb57ddd361e63740d"} Jul 26 15:42:48.255491 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:48.255387 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-66d4685587-5rddj" event={"ID":"881b4d20-4178-492f-8183-7d1011be3602","Type":"ContainerStarted","Data":"5fc63ca9d6609c80e8e16e43aa6927e374f1c55e9e7f357487261470a9edb76c"} Jul 26 15:42:49.261836 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:49.261749 2577 generic.go:356] "Generic (PLEG): container finished" podID="881b4d20-4178-492f-8183-7d1011be3602" containerID="5fc63ca9d6609c80e8e16e43aa6927e374f1c55e9e7f357487261470a9edb76c" exitCode=0 Jul 26 15:42:49.262264 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:49.261828 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-66d4685587-5rddj" event={"ID":"881b4d20-4178-492f-8183-7d1011be3602","Type":"ContainerDied","Data":"5fc63ca9d6609c80e8e16e43aa6927e374f1c55e9e7f357487261470a9edb76c"} Jul 26 15:42:50.267505 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:50.267462 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-66d4685587-5rddj" event={"ID":"881b4d20-4178-492f-8183-7d1011be3602","Type":"ContainerStarted","Data":"9f7c3559030a64ffee0550b32207c2c2ef42771f47bd91b57fb36f5fa152e773"} Jul 26 15:42:50.267988 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:50.267623 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-66d4685587-5rddj" Jul 26 15:42:50.290007 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:50.289954 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-66d4685587-5rddj" podStartSLOduration=1.875494365 podStartE2EDuration="6.289937268s" totalImagesPullingTime="3.330594666s" totalInitContainerRuntime="1.083848237s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:42:44 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:42:50.287392128 +0000 UTC m=+293.026745438" watchObservedRunningTime="2026-07-26 15:42:50.289937268 +0000 UTC m=+293.029290578" Jul 26 15:42:51.815224 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:51.815188 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:42:51.927599 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:51.927561 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:42:57.685547 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:57.685517 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-8wjqp_b6b8080d-0cee-4d0c-a05c-3e2f1b672a54/console-operator/1.log" Jul 26 15:42:57.690221 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:57.690199 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ztlp6_cbe2cd41-6ebc-4722-a813-57d83fa1d242/ovn-acl-logging/0.log" Jul 26 15:42:57.697249 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:57.697213 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-8wjqp_b6b8080d-0cee-4d0c-a05c-3e2f1b672a54/console-operator/1.log" Jul 26 15:42:57.702260 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:57.702239 2577 kubelet.go:1648] "Image garbage collection succeeded" Jul 26 15:42:57.703155 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:57.703132 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ztlp6_cbe2cd41-6ebc-4722-a813-57d83fa1d242/ovn-acl-logging/0.log" Jul 26 15:42:59.481025 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:42:59.480987 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:43:00.831564 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:43:00.831532 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:43:01.275015 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:43:01.274988 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-66d4685587-5rddj" Jul 26 15:43:03.203196 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:43:03.203160 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:43:03.288469 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:43:03.288431 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:43:03.786897 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:43:03.786858 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-rg7qh"] Jul 26 15:43:03.789185 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:43:03.789161 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-rg7qh" Jul 26 15:43:03.791856 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:43:03.791830 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-wlgmb\"" Jul 26 15:43:03.791994 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:43:03.791977 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 26 15:43:03.798272 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:43:03.798248 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-rg7qh"] Jul 26 15:43:03.819146 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:43:03.819118 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:43:03.908142 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:43:03.908111 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cvpt\" (UniqueName: \"kubernetes.io/projected/f24e879c-5719-4461-a065-c15b3bfc8513-kube-api-access-4cvpt\") pod \"odh-model-controller-7998c67dc4-rg7qh\" (UID: \"f24e879c-5719-4461-a065-c15b3bfc8513\") " pod="opendatahub/odh-model-controller-7998c67dc4-rg7qh" Jul 26 15:43:03.908318 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:43:03.908160 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f24e879c-5719-4461-a065-c15b3bfc8513-cert\") pod \"odh-model-controller-7998c67dc4-rg7qh\" (UID: \"f24e879c-5719-4461-a065-c15b3bfc8513\") " pod="opendatahub/odh-model-controller-7998c67dc4-rg7qh" Jul 26 15:43:03.918024 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:43:03.917988 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:43:04.009250 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:43:04.009216 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cvpt\" (UniqueName: \"kubernetes.io/projected/f24e879c-5719-4461-a065-c15b3bfc8513-kube-api-access-4cvpt\") pod \"odh-model-controller-7998c67dc4-rg7qh\" (UID: \"f24e879c-5719-4461-a065-c15b3bfc8513\") " pod="opendatahub/odh-model-controller-7998c67dc4-rg7qh" Jul 26 15:43:04.009434 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:43:04.009265 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f24e879c-5719-4461-a065-c15b3bfc8513-cert\") pod \"odh-model-controller-7998c67dc4-rg7qh\" (UID: \"f24e879c-5719-4461-a065-c15b3bfc8513\") " pod="opendatahub/odh-model-controller-7998c67dc4-rg7qh" Jul 26 15:43:04.009434 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:43:04.009386 2577 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Jul 26 15:43:04.009510 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:43:04.009473 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f24e879c-5719-4461-a065-c15b3bfc8513-cert podName:f24e879c-5719-4461-a065-c15b3bfc8513 nodeName:}" failed. No retries permitted until 2026-07-26 15:43:04.509453931 +0000 UTC m=+307.248807234 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/f24e879c-5719-4461-a065-c15b3bfc8513-cert") pod "odh-model-controller-7998c67dc4-rg7qh" (UID: "f24e879c-5719-4461-a065-c15b3bfc8513") : secret "odh-model-controller-webhook-cert" not found Jul 26 15:43:04.017522 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:43:04.017478 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:43:04.018118 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:43:04.018091 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cvpt\" (UniqueName: \"kubernetes.io/projected/f24e879c-5719-4461-a065-c15b3bfc8513-kube-api-access-4cvpt\") pod \"odh-model-controller-7998c67dc4-rg7qh\" (UID: \"f24e879c-5719-4461-a065-c15b3bfc8513\") " pod="opendatahub/odh-model-controller-7998c67dc4-rg7qh" Jul 26 15:43:04.276877 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:43:04.276845 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:43:04.342967 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:43:04.342933 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-mblzh"] Jul 26 15:43:04.345373 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:43:04.345354 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-mblzh" Jul 26 15:43:04.347770 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:43:04.347745 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Jul 26 15:43:04.362358 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:43:04.362332 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-mblzh"] Jul 26 15:43:04.366273 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:43:04.366246 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-54c98bf85c-znvqm"] Jul 26 15:43:04.368523 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:43:04.368501 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-54c98bf85c-znvqm" Jul 26 15:43:04.368641 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:43:04.368611 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:43:04.371455 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:43:04.371435 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 26 15:43:04.371551 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:43:04.371438 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 26 15:43:04.371551 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:43:04.371471 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 26 15:43:04.371785 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:43:04.371768 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-hqcjz\"" Jul 26 15:43:04.371881 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:43:04.371862 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 26 15:43:04.371956 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:43:04.371892 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 26 15:43:04.381858 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:43:04.381833 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-54c98bf85c-znvqm"] Jul 26 15:43:04.514771 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:43:04.514739 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f24e879c-5719-4461-a065-c15b3bfc8513-cert\") pod \"odh-model-controller-7998c67dc4-rg7qh\" (UID: \"f24e879c-5719-4461-a065-c15b3bfc8513\") " pod="opendatahub/odh-model-controller-7998c67dc4-rg7qh" Jul 26 15:43:04.514771 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:43:04.514774 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/534eb529-23ca-407a-9c54-bae5531d8360-cert\") pod \"lws-controller-manager-54c98bf85c-znvqm\" (UID: \"534eb529-23ca-407a-9c54-bae5531d8360\") " pod="openshift-lws-operator/lws-controller-manager-54c98bf85c-znvqm" Jul 26 15:43:04.514974 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:43:04.514793 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f230bc42-6931-4613-ac67-8e30d1b81094-tls-certs\") pod \"model-serving-api-54d89bd5fc-mblzh\" (UID: \"f230bc42-6931-4613-ac67-8e30d1b81094\") " pod="opendatahub/model-serving-api-54d89bd5fc-mblzh" Jul 26 15:43:04.514974 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:43:04.514900 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/534eb529-23ca-407a-9c54-bae5531d8360-manager-config\") pod \"lws-controller-manager-54c98bf85c-znvqm\" (UID: \"534eb529-23ca-407a-9c54-bae5531d8360\") " pod="openshift-lws-operator/lws-controller-manager-54c98bf85c-znvqm" Jul 26 15:43:04.514974 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:43:04.514921 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqbw5\" (UniqueName: \"kubernetes.io/projected/534eb529-23ca-407a-9c54-bae5531d8360-kube-api-access-sqbw5\") pod \"lws-controller-manager-54c98bf85c-znvqm\" (UID: \"534eb529-23ca-407a-9c54-bae5531d8360\") " pod="openshift-lws-operator/lws-controller-manager-54c98bf85c-znvqm" Jul 26 15:43:04.515114 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:43:04.514975 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bc8vr\" (UniqueName: \"kubernetes.io/projected/f230bc42-6931-4613-ac67-8e30d1b81094-kube-api-access-bc8vr\") pod \"model-serving-api-54d89bd5fc-mblzh\" (UID: \"f230bc42-6931-4613-ac67-8e30d1b81094\") " pod="opendatahub/model-serving-api-54d89bd5fc-mblzh" Jul 26 15:43:04.515114 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:43:04.514998 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/534eb529-23ca-407a-9c54-bae5531d8360-metrics-cert\") pod \"lws-controller-manager-54c98bf85c-znvqm\" (UID: \"534eb529-23ca-407a-9c54-bae5531d8360\") " pod="openshift-lws-operator/lws-controller-manager-54c98bf85c-znvqm" Jul 26 15:43:04.517155 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:43:04.517137 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f24e879c-5719-4461-a065-c15b3bfc8513-cert\") pod \"odh-model-controller-7998c67dc4-rg7qh\" (UID: \"f24e879c-5719-4461-a065-c15b3bfc8513\") " pod="opendatahub/odh-model-controller-7998c67dc4-rg7qh" Jul 26 15:43:04.615804 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:43:04.615714 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/534eb529-23ca-407a-9c54-bae5531d8360-cert\") pod \"lws-controller-manager-54c98bf85c-znvqm\" (UID: \"534eb529-23ca-407a-9c54-bae5531d8360\") " pod="openshift-lws-operator/lws-controller-manager-54c98bf85c-znvqm" Jul 26 15:43:04.615804 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:43:04.615750 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f230bc42-6931-4613-ac67-8e30d1b81094-tls-certs\") pod \"model-serving-api-54d89bd5fc-mblzh\" (UID: \"f230bc42-6931-4613-ac67-8e30d1b81094\") " pod="opendatahub/model-serving-api-54d89bd5fc-mblzh" Jul 26 15:43:04.615804 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:43:04.615771 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/534eb529-23ca-407a-9c54-bae5531d8360-manager-config\") pod \"lws-controller-manager-54c98bf85c-znvqm\" (UID: \"534eb529-23ca-407a-9c54-bae5531d8360\") " pod="openshift-lws-operator/lws-controller-manager-54c98bf85c-znvqm" Jul 26 15:43:04.615804 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:43:04.615788 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqbw5\" (UniqueName: \"kubernetes.io/projected/534eb529-23ca-407a-9c54-bae5531d8360-kube-api-access-sqbw5\") pod \"lws-controller-manager-54c98bf85c-znvqm\" (UID: \"534eb529-23ca-407a-9c54-bae5531d8360\") " pod="openshift-lws-operator/lws-controller-manager-54c98bf85c-znvqm" Jul 26 15:43:04.616125 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:43:04.615835 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bc8vr\" (UniqueName: \"kubernetes.io/projected/f230bc42-6931-4613-ac67-8e30d1b81094-kube-api-access-bc8vr\") pod \"model-serving-api-54d89bd5fc-mblzh\" (UID: \"f230bc42-6931-4613-ac67-8e30d1b81094\") " pod="opendatahub/model-serving-api-54d89bd5fc-mblzh" Jul 26 15:43:04.616125 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:43:04.615855 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/534eb529-23ca-407a-9c54-bae5531d8360-metrics-cert\") pod \"lws-controller-manager-54c98bf85c-znvqm\" (UID: \"534eb529-23ca-407a-9c54-bae5531d8360\") " pod="openshift-lws-operator/lws-controller-manager-54c98bf85c-znvqm" Jul 26 15:43:04.616125 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:43:04.615888 2577 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found Jul 26 15:43:04.616125 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:43:04.615969 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f230bc42-6931-4613-ac67-8e30d1b81094-tls-certs podName:f230bc42-6931-4613-ac67-8e30d1b81094 nodeName:}" failed. No retries permitted until 2026-07-26 15:43:05.11594559 +0000 UTC m=+307.855298882 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/f230bc42-6931-4613-ac67-8e30d1b81094-tls-certs") pod "model-serving-api-54d89bd5fc-mblzh" (UID: "f230bc42-6931-4613-ac67-8e30d1b81094") : secret "model-serving-api-tls" not found Jul 26 15:43:04.616494 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:43:04.616466 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/534eb529-23ca-407a-9c54-bae5531d8360-manager-config\") pod \"lws-controller-manager-54c98bf85c-znvqm\" (UID: \"534eb529-23ca-407a-9c54-bae5531d8360\") " pod="openshift-lws-operator/lws-controller-manager-54c98bf85c-znvqm" Jul 26 15:43:04.618552 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:43:04.618530 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/534eb529-23ca-407a-9c54-bae5531d8360-cert\") pod \"lws-controller-manager-54c98bf85c-znvqm\" (UID: \"534eb529-23ca-407a-9c54-bae5531d8360\") " pod="openshift-lws-operator/lws-controller-manager-54c98bf85c-znvqm" Jul 26 15:43:04.618642 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:43:04.618533 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/534eb529-23ca-407a-9c54-bae5531d8360-metrics-cert\") pod \"lws-controller-manager-54c98bf85c-znvqm\" (UID: \"534eb529-23ca-407a-9c54-bae5531d8360\") " pod="openshift-lws-operator/lws-controller-manager-54c98bf85c-znvqm" Jul 26 15:43:04.624007 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:43:04.623983 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bc8vr\" (UniqueName: \"kubernetes.io/projected/f230bc42-6931-4613-ac67-8e30d1b81094-kube-api-access-bc8vr\") pod \"model-serving-api-54d89bd5fc-mblzh\" (UID: \"f230bc42-6931-4613-ac67-8e30d1b81094\") " pod="opendatahub/model-serving-api-54d89bd5fc-mblzh" Jul 26 15:43:04.624095 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:43:04.624018 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqbw5\" (UniqueName: \"kubernetes.io/projected/534eb529-23ca-407a-9c54-bae5531d8360-kube-api-access-sqbw5\") pod \"lws-controller-manager-54c98bf85c-znvqm\" (UID: \"534eb529-23ca-407a-9c54-bae5531d8360\") " pod="openshift-lws-operator/lws-controller-manager-54c98bf85c-znvqm" Jul 26 15:43:04.678332 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:43:04.678281 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-54c98bf85c-znvqm" Jul 26 15:43:04.699541 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:43:04.699489 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-rg7qh" Jul 26 15:43:04.825162 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:43:04.825133 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-54c98bf85c-znvqm"] Jul 26 15:43:04.826565 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:43:04.826537 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod534eb529_23ca_407a_9c54_bae5531d8360.slice/crio-bd87f86a3df456b5405b11cc50f915825b9fe288a9bd25fa7a686cfa60d3d238 WatchSource:0}: Error finding container bd87f86a3df456b5405b11cc50f915825b9fe288a9bd25fa7a686cfa60d3d238: Status 404 returned error can't find the container with id bd87f86a3df456b5405b11cc50f915825b9fe288a9bd25fa7a686cfa60d3d238 Jul 26 15:43:04.828748 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:43:04.828730 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 26 15:43:04.838556 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:43:04.838482 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-rg7qh"] Jul 26 15:43:04.841131 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:43:04.841108 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf24e879c_5719_4461_a065_c15b3bfc8513.slice/crio-7bd97d007ebc6c40786598d6e076ffaf6219075af8489ac78daf163a42badc6e WatchSource:0}: Error finding container 7bd97d007ebc6c40786598d6e076ffaf6219075af8489ac78daf163a42badc6e: Status 404 returned error can't find the container with id 7bd97d007ebc6c40786598d6e076ffaf6219075af8489ac78daf163a42badc6e Jul 26 15:43:04.903736 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:43:04.903646 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:43:05.121171 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:43:05.121134 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f230bc42-6931-4613-ac67-8e30d1b81094-tls-certs\") pod \"model-serving-api-54d89bd5fc-mblzh\" (UID: \"f230bc42-6931-4613-ac67-8e30d1b81094\") " pod="opendatahub/model-serving-api-54d89bd5fc-mblzh" Jul 26 15:43:05.123604 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:43:05.123575 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f230bc42-6931-4613-ac67-8e30d1b81094-tls-certs\") pod \"model-serving-api-54d89bd5fc-mblzh\" (UID: \"f230bc42-6931-4613-ac67-8e30d1b81094\") " pod="opendatahub/model-serving-api-54d89bd5fc-mblzh" Jul 26 15:43:05.255630 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:43:05.255539 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-mblzh" Jul 26 15:43:05.345347 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:43:05.345314 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-54c98bf85c-znvqm" event={"ID":"534eb529-23ca-407a-9c54-bae5531d8360","Type":"ContainerStarted","Data":"bd87f86a3df456b5405b11cc50f915825b9fe288a9bd25fa7a686cfa60d3d238"} Jul 26 15:43:05.346545 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:43:05.346521 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-rg7qh" event={"ID":"f24e879c-5719-4461-a065-c15b3bfc8513","Type":"ContainerStarted","Data":"7bd97d007ebc6c40786598d6e076ffaf6219075af8489ac78daf163a42badc6e"} Jul 26 15:43:05.398797 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:43:05.398769 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-mblzh"] Jul 26 15:43:05.401342 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:43:05.401304 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf230bc42_6931_4613_ac67_8e30d1b81094.slice/crio-eee21ac9380d7c8f92046277c6baacfe864fd0d52542810bf7c435a5ec02e1fb WatchSource:0}: Error finding container eee21ac9380d7c8f92046277c6baacfe864fd0d52542810bf7c435a5ec02e1fb: Status 404 returned error can't find the container with id eee21ac9380d7c8f92046277c6baacfe864fd0d52542810bf7c435a5ec02e1fb Jul 26 15:43:06.355201 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:43:06.355110 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-mblzh" event={"ID":"f230bc42-6931-4613-ac67-8e30d1b81094","Type":"ContainerStarted","Data":"eee21ac9380d7c8f92046277c6baacfe864fd0d52542810bf7c435a5ec02e1fb"} Jul 26 15:43:06.822543 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:43:06.822413 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:43:06.876186 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:43:06.876141 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:43:09.372821 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:43:09.372786 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-mblzh" event={"ID":"f230bc42-6931-4613-ac67-8e30d1b81094","Type":"ContainerStarted","Data":"33fd874b7ffa075e9138cb6a2c1449747bc255999b0e60f897c25feba3f43c08"} Jul 26 15:43:09.373348 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:43:09.372873 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-mblzh" Jul 26 15:43:09.374303 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:43:09.374274 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-54c98bf85c-znvqm" event={"ID":"534eb529-23ca-407a-9c54-bae5531d8360","Type":"ContainerStarted","Data":"af2dbc337df6474acbab07e2dd987a0efe34ede8387d13e3d742511582fb6a48"} Jul 26 15:43:09.374472 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:43:09.374325 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-54c98bf85c-znvqm" Jul 26 15:43:09.375716 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:43:09.375691 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-rg7qh" event={"ID":"f24e879c-5719-4461-a065-c15b3bfc8513","Type":"ContainerStarted","Data":"dca46efefc0c9a0c64e6a8723011b60b7aebf5248559f6000fb2c6c8ea4bcc7d"} Jul 26 15:43:09.375821 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:43:09.375746 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-rg7qh" Jul 26 15:43:09.395283 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:43:09.395232 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-mblzh" podStartSLOduration=2.291291949 podStartE2EDuration="5.395218779s" totalImagesPullingTime="3.10392683s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:43:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:43:09.394035351 +0000 UTC m=+312.133388675" watchObservedRunningTime="2026-07-26 15:43:09.395218779 +0000 UTC m=+312.134572089" Jul 26 15:43:09.433330 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:43:09.433269 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-54c98bf85c-znvqm" podStartSLOduration=1.7591770119999999 podStartE2EDuration="5.433252707s" totalImagesPullingTime="3.674075695s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:43:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:43:09.432711716 +0000 UTC m=+312.172065027" watchObservedRunningTime="2026-07-26 15:43:09.433252707 +0000 UTC m=+312.172606016" Jul 26 15:43:09.462440 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:43:09.462353 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-rg7qh" podStartSLOduration=2.786092919 podStartE2EDuration="6.462330299s" totalImagesPullingTime="3.67623738s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:43:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:43:09.459622303 +0000 UTC m=+312.198975634" watchObservedRunningTime="2026-07-26 15:43:09.462330299 +0000 UTC m=+312.201683618" Jul 26 15:43:13.076892 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:43:13.076845 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:43:13.189265 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:43:13.189231 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:43:13.286499 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:43:13.286464 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:43:13.418668 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:43:13.418566 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:43:13.570592 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:43:13.570557 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:43:13.664496 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:43:13.664457 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:43:13.993129 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:43:13.993082 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:43:14.289764 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:43:14.289721 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:43:16.162217 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:43:16.162175 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:43:16.248791 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:43:16.248752 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:43:17.793035 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:43:17.792998 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:43:20.383301 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:43:20.383270 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-rg7qh" Jul 26 15:43:20.383729 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:43:20.383438 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-54c98bf85c-znvqm" Jul 26 15:43:20.385516 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:43:20.385490 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-mblzh" Jul 26 15:43:21.582164 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:43:21.582125 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:43:21.682635 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:43:21.682602 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:43:22.145506 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:43:22.145470 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rx9js"] Jul 26 15:43:22.150796 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:43:22.150774 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rx9js" Jul 26 15:43:22.154414 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:43:22.154371 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-kqnw2\"" Jul 26 15:43:22.155022 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:43:22.155002 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 26 15:43:22.172312 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:43:22.172277 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rx9js"] Jul 26 15:43:22.245944 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:43:22.245904 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:43:22.271082 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:43:22.271046 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b03798f1-6dbf-4cee-bf15-062b468c5206-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-rx9js\" (UID: \"b03798f1-6dbf-4cee-bf15-062b468c5206\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rx9js" Jul 26 15:43:22.271246 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:43:22.271111 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b03798f1-6dbf-4cee-bf15-062b468c5206-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-rx9js\" (UID: \"b03798f1-6dbf-4cee-bf15-062b468c5206\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rx9js" Jul 26 15:43:22.271246 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:43:22.271138 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b03798f1-6dbf-4cee-bf15-062b468c5206-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-rx9js\" (UID: \"b03798f1-6dbf-4cee-bf15-062b468c5206\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rx9js" Jul 26 15:43:22.271246 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:43:22.271211 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b03798f1-6dbf-4cee-bf15-062b468c5206-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-rx9js\" (UID: \"b03798f1-6dbf-4cee-bf15-062b468c5206\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rx9js" Jul 26 15:43:22.271375 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:43:22.271252 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b03798f1-6dbf-4cee-bf15-062b468c5206-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-rx9js\" (UID: \"b03798f1-6dbf-4cee-bf15-062b468c5206\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rx9js" Jul 26 15:43:22.271375 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:43:22.271273 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b03798f1-6dbf-4cee-bf15-062b468c5206-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-rx9js\" (UID: \"b03798f1-6dbf-4cee-bf15-062b468c5206\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rx9js" Jul 26 15:43:22.271375 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:43:22.271349 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gs5pp\" (UniqueName: \"kubernetes.io/projected/b03798f1-6dbf-4cee-bf15-062b468c5206-kube-api-access-gs5pp\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-rx9js\" (UID: \"b03798f1-6dbf-4cee-bf15-062b468c5206\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rx9js" Jul 26 15:43:22.271375 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:43:22.271368 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b03798f1-6dbf-4cee-bf15-062b468c5206-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-rx9js\" (UID: \"b03798f1-6dbf-4cee-bf15-062b468c5206\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rx9js" Jul 26 15:43:22.271536 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:43:22.271412 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b03798f1-6dbf-4cee-bf15-062b468c5206-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-rx9js\" (UID: \"b03798f1-6dbf-4cee-bf15-062b468c5206\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rx9js" Jul 26 15:43:22.372756 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:43:22.372717 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gs5pp\" (UniqueName: \"kubernetes.io/projected/b03798f1-6dbf-4cee-bf15-062b468c5206-kube-api-access-gs5pp\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-rx9js\" (UID: \"b03798f1-6dbf-4cee-bf15-062b468c5206\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rx9js" Jul 26 15:43:22.372756 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:43:22.372757 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b03798f1-6dbf-4cee-bf15-062b468c5206-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-rx9js\" (UID: \"b03798f1-6dbf-4cee-bf15-062b468c5206\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rx9js" Jul 26 15:43:22.372994 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:43:22.372787 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b03798f1-6dbf-4cee-bf15-062b468c5206-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-rx9js\" (UID: \"b03798f1-6dbf-4cee-bf15-062b468c5206\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rx9js" Jul 26 15:43:22.372994 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:43:22.372823 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b03798f1-6dbf-4cee-bf15-062b468c5206-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-rx9js\" (UID: \"b03798f1-6dbf-4cee-bf15-062b468c5206\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rx9js" Jul 26 15:43:22.372994 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:43:22.372854 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b03798f1-6dbf-4cee-bf15-062b468c5206-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-rx9js\" (UID: \"b03798f1-6dbf-4cee-bf15-062b468c5206\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rx9js" Jul 26 15:43:22.372994 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:43:22.372883 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b03798f1-6dbf-4cee-bf15-062b468c5206-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-rx9js\" (UID: \"b03798f1-6dbf-4cee-bf15-062b468c5206\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rx9js" Jul 26 15:43:22.372994 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:43:22.372932 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b03798f1-6dbf-4cee-bf15-062b468c5206-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-rx9js\" (UID: \"b03798f1-6dbf-4cee-bf15-062b468c5206\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rx9js" Jul 26 15:43:22.372994 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:43:22.372963 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b03798f1-6dbf-4cee-bf15-062b468c5206-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-rx9js\" (UID: \"b03798f1-6dbf-4cee-bf15-062b468c5206\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rx9js" Jul 26 15:43:22.372994 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:43:22.372991 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b03798f1-6dbf-4cee-bf15-062b468c5206-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-rx9js\" (UID: \"b03798f1-6dbf-4cee-bf15-062b468c5206\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rx9js" Jul 26 15:43:22.373385 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:43:22.373284 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b03798f1-6dbf-4cee-bf15-062b468c5206-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-rx9js\" (UID: \"b03798f1-6dbf-4cee-bf15-062b468c5206\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rx9js" Jul 26 15:43:22.373385 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:43:22.373296 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b03798f1-6dbf-4cee-bf15-062b468c5206-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-rx9js\" (UID: \"b03798f1-6dbf-4cee-bf15-062b468c5206\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rx9js" Jul 26 15:43:22.373385 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:43:22.373369 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b03798f1-6dbf-4cee-bf15-062b468c5206-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-rx9js\" (UID: \"b03798f1-6dbf-4cee-bf15-062b468c5206\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rx9js" Jul 26 15:43:22.373677 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:43:22.373648 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b03798f1-6dbf-4cee-bf15-062b468c5206-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-rx9js\" (UID: \"b03798f1-6dbf-4cee-bf15-062b468c5206\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rx9js" Jul 26 15:43:22.374064 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:43:22.374040 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b03798f1-6dbf-4cee-bf15-062b468c5206-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-rx9js\" (UID: \"b03798f1-6dbf-4cee-bf15-062b468c5206\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rx9js" Jul 26 15:43:22.375287 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:43:22.375267 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b03798f1-6dbf-4cee-bf15-062b468c5206-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-rx9js\" (UID: \"b03798f1-6dbf-4cee-bf15-062b468c5206\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rx9js" Jul 26 15:43:22.375528 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:43:22.375511 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b03798f1-6dbf-4cee-bf15-062b468c5206-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-rx9js\" (UID: \"b03798f1-6dbf-4cee-bf15-062b468c5206\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rx9js" Jul 26 15:43:22.382377 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:43:22.382354 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b03798f1-6dbf-4cee-bf15-062b468c5206-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-rx9js\" (UID: \"b03798f1-6dbf-4cee-bf15-062b468c5206\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rx9js" Jul 26 15:43:22.382502 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:43:22.382478 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gs5pp\" (UniqueName: \"kubernetes.io/projected/b03798f1-6dbf-4cee-bf15-062b468c5206-kube-api-access-gs5pp\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-rx9js\" (UID: \"b03798f1-6dbf-4cee-bf15-062b468c5206\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rx9js" Jul 26 15:43:22.449891 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:43:22.449796 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:43:22.462616 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:43:22.462588 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rx9js" Jul 26 15:43:22.553002 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:43:22.552975 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:43:22.600088 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:43:22.600063 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rx9js"] Jul 26 15:43:22.602291 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:43:22.602265 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb03798f1_6dbf_4cee_bf15_062b468c5206.slice/crio-804292a405db0e96673fa4da37f05bb9c4e74be3c6628dd69f6fd1e5917dbbf1 WatchSource:0}: Error finding container 804292a405db0e96673fa4da37f05bb9c4e74be3c6628dd69f6fd1e5917dbbf1: Status 404 returned error can't find the container with id 804292a405db0e96673fa4da37f05bb9c4e74be3c6628dd69f6fd1e5917dbbf1 Jul 26 15:43:23.446705 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:43:23.446666 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rx9js" event={"ID":"b03798f1-6dbf-4cee-bf15-062b468c5206","Type":"ContainerStarted","Data":"804292a405db0e96673fa4da37f05bb9c4e74be3c6628dd69f6fd1e5917dbbf1"} Jul 26 15:43:25.092149 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:43:25.092111 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:43:25.191337 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:43:25.191293 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:43:26.187483 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:43:26.187442 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 15:43:26.187855 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:43:26.187535 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 15:43:26.187855 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:43:26.187583 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 15:43:26.465613 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:43:26.465517 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rx9js" event={"ID":"b03798f1-6dbf-4cee-bf15-062b468c5206","Type":"ContainerStarted","Data":"57d9093c8333227fde5f7af4405f9b06c07523dd117a0ffea10d55b04081257b"} Jul 26 15:43:26.493033 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:43:26.492971 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rx9js" podStartSLOduration=0.910002161 podStartE2EDuration="4.492947828s" totalImagesPullingTime="3.582945667s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:43:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:43:26.491142796 +0000 UTC m=+329.230496114" watchObservedRunningTime="2026-07-26 15:43:26.492947828 +0000 UTC m=+329.232301140" Jul 26 15:43:27.463481 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:43:27.463433 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rx9js" Jul 26 15:43:27.468457 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:43:27.468432 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rx9js" Jul 26 15:43:27.470263 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:43:27.470242 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rx9js" Jul 26 15:43:27.470949 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:43:27.470930 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rx9js" Jul 26 15:44:04.678779 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:04.678654 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:44:04.762206 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:04.762168 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:44:04.788458 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:04.788418 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-tvl5t"] Jul 26 15:44:04.798695 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:04.798663 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-tvl5t" Jul 26 15:44:04.801565 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:04.801538 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-hcf6p\"" Jul 26 15:44:04.802934 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:04.802908 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 26 15:44:04.803155 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:04.803138 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 26 15:44:04.803991 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:04.803966 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-tvl5t"] Jul 26 15:44:04.837970 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:04.837938 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98ggj\" (UniqueName: \"kubernetes.io/projected/f344b88f-a527-463a-9475-aee8a54d2b59-kube-api-access-98ggj\") pod \"kuadrant-operator-catalog-tvl5t\" (UID: \"f344b88f-a527-463a-9475-aee8a54d2b59\") " pod="kuadrant-system/kuadrant-operator-catalog-tvl5t" Jul 26 15:44:04.838768 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:04.838739 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:44:04.939319 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:04.939225 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98ggj\" (UniqueName: \"kubernetes.io/projected/f344b88f-a527-463a-9475-aee8a54d2b59-kube-api-access-98ggj\") pod \"kuadrant-operator-catalog-tvl5t\" (UID: \"f344b88f-a527-463a-9475-aee8a54d2b59\") " pod="kuadrant-system/kuadrant-operator-catalog-tvl5t" Jul 26 15:44:04.947240 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:04.947219 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-98ggj\" (UniqueName: \"kubernetes.io/projected/f344b88f-a527-463a-9475-aee8a54d2b59-kube-api-access-98ggj\") pod \"kuadrant-operator-catalog-tvl5t\" (UID: \"f344b88f-a527-463a-9475-aee8a54d2b59\") " pod="kuadrant-system/kuadrant-operator-catalog-tvl5t" Jul 26 15:44:05.110985 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:05.110949 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-tvl5t" Jul 26 15:44:05.161751 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:05.161720 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-tvl5t"] Jul 26 15:44:05.252844 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:05.252819 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-tvl5t"] Jul 26 15:44:05.255348 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:44:05.255316 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf344b88f_a527_463a_9475_aee8a54d2b59.slice/crio-29f415e6c0d21bcac168f49475cdeaf1891a6f63dadcf7ffbf8355c83cecee03 WatchSource:0}: Error finding container 29f415e6c0d21bcac168f49475cdeaf1891a6f63dadcf7ffbf8355c83cecee03: Status 404 returned error can't find the container with id 29f415e6c0d21bcac168f49475cdeaf1891a6f63dadcf7ffbf8355c83cecee03 Jul 26 15:44:05.360659 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:05.360622 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-s5mxx"] Jul 26 15:44:05.365632 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:05.365612 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-s5mxx" Jul 26 15:44:05.371527 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:05.371499 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-s5mxx"] Jul 26 15:44:05.443420 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:05.443352 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcc95\" (UniqueName: \"kubernetes.io/projected/e3fb5dbe-c712-4c56-82fc-dbcb11fad428-kube-api-access-vcc95\") pod \"kuadrant-operator-catalog-s5mxx\" (UID: \"e3fb5dbe-c712-4c56-82fc-dbcb11fad428\") " pod="kuadrant-system/kuadrant-operator-catalog-s5mxx" Jul 26 15:44:05.544511 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:05.544478 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcc95\" (UniqueName: \"kubernetes.io/projected/e3fb5dbe-c712-4c56-82fc-dbcb11fad428-kube-api-access-vcc95\") pod \"kuadrant-operator-catalog-s5mxx\" (UID: \"e3fb5dbe-c712-4c56-82fc-dbcb11fad428\") " pod="kuadrant-system/kuadrant-operator-catalog-s5mxx" Jul 26 15:44:05.552720 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:05.552690 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcc95\" (UniqueName: \"kubernetes.io/projected/e3fb5dbe-c712-4c56-82fc-dbcb11fad428-kube-api-access-vcc95\") pod \"kuadrant-operator-catalog-s5mxx\" (UID: \"e3fb5dbe-c712-4c56-82fc-dbcb11fad428\") " pod="kuadrant-system/kuadrant-operator-catalog-s5mxx" Jul 26 15:44:05.656170 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:05.656137 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-tvl5t" event={"ID":"f344b88f-a527-463a-9475-aee8a54d2b59","Type":"ContainerStarted","Data":"29f415e6c0d21bcac168f49475cdeaf1891a6f63dadcf7ffbf8355c83cecee03"} Jul 26 15:44:05.675873 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:05.675844 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-s5mxx" Jul 26 15:44:05.813879 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:05.813854 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-s5mxx"] Jul 26 15:44:05.816172 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:44:05.816143 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3fb5dbe_c712_4c56_82fc_dbcb11fad428.slice/crio-9692d94bbea2734b7eb3ab28342d1e79d1306cea84fce8845959da8acc81ae78 WatchSource:0}: Error finding container 9692d94bbea2734b7eb3ab28342d1e79d1306cea84fce8845959da8acc81ae78: Status 404 returned error can't find the container with id 9692d94bbea2734b7eb3ab28342d1e79d1306cea84fce8845959da8acc81ae78 Jul 26 15:44:06.663078 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:06.663035 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-s5mxx" event={"ID":"e3fb5dbe-c712-4c56-82fc-dbcb11fad428","Type":"ContainerStarted","Data":"9692d94bbea2734b7eb3ab28342d1e79d1306cea84fce8845959da8acc81ae78"} Jul 26 15:44:07.669307 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:07.669270 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-tvl5t" event={"ID":"f344b88f-a527-463a-9475-aee8a54d2b59","Type":"ContainerStarted","Data":"c7cd1089d54a1eda3e02f2cb7ebf8841d7261ed09c9f47732086c0c0d588eddf"} Jul 26 15:44:07.669307 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:07.669296 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-tvl5t" podUID="f344b88f-a527-463a-9475-aee8a54d2b59" containerName="registry-server" containerID="cri-o://c7cd1089d54a1eda3e02f2cb7ebf8841d7261ed09c9f47732086c0c0d588eddf" gracePeriod=2 Jul 26 15:44:07.670833 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:07.670808 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-s5mxx" event={"ID":"e3fb5dbe-c712-4c56-82fc-dbcb11fad428","Type":"ContainerStarted","Data":"8c8dd46a75509af6981ebbbe4150dd62e9166cabe5eac6a29d9ee74c32f37dc7"} Jul 26 15:44:07.686246 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:07.686199 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-tvl5t" podStartSLOduration=1.5005167400000001 podStartE2EDuration="3.686183172s" totalImagesPullingTime="2.185666432s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:44:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:44:07.684872266 +0000 UTC m=+370.424225577" watchObservedRunningTime="2026-07-26 15:44:07.686183172 +0000 UTC m=+370.425536460" Jul 26 15:44:07.699708 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:07.699659 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-s5mxx" podStartSLOduration=1.07220803 podStartE2EDuration="2.699639405s" totalImagesPullingTime="1.627431375s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:44:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:44:07.69904368 +0000 UTC m=+370.438396984" watchObservedRunningTime="2026-07-26 15:44:07.699639405 +0000 UTC m=+370.438992716" Jul 26 15:44:07.919778 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:07.919747 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-tvl5t" Jul 26 15:44:07.968045 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:07.967965 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-98ggj\" (UniqueName: \"kubernetes.io/projected/f344b88f-a527-463a-9475-aee8a54d2b59-kube-api-access-98ggj\") pod \"f344b88f-a527-463a-9475-aee8a54d2b59\" (UID: \"f344b88f-a527-463a-9475-aee8a54d2b59\") " Jul 26 15:44:07.970095 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:07.970071 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f344b88f-a527-463a-9475-aee8a54d2b59-kube-api-access-98ggj" (OuterVolumeSpecName: "kube-api-access-98ggj") pod "f344b88f-a527-463a-9475-aee8a54d2b59" (UID: "f344b88f-a527-463a-9475-aee8a54d2b59"). InnerVolumeSpecName "kube-api-access-98ggj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 15:44:08.069273 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:08.069230 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98ggj\" (UniqueName: \"kubernetes.io/projected/f344b88f-a527-463a-9475-aee8a54d2b59-kube-api-access-98ggj\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 15:44:08.676816 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:08.676774 2577 generic.go:356] "Generic (PLEG): container finished" podID="f344b88f-a527-463a-9475-aee8a54d2b59" containerID="c7cd1089d54a1eda3e02f2cb7ebf8841d7261ed09c9f47732086c0c0d588eddf" exitCode=0 Jul 26 15:44:08.677233 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:08.676828 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-tvl5t" Jul 26 15:44:08.677233 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:08.676854 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-tvl5t" event={"ID":"f344b88f-a527-463a-9475-aee8a54d2b59","Type":"ContainerDied","Data":"c7cd1089d54a1eda3e02f2cb7ebf8841d7261ed09c9f47732086c0c0d588eddf"} Jul 26 15:44:08.677233 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:08.676933 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-tvl5t" event={"ID":"f344b88f-a527-463a-9475-aee8a54d2b59","Type":"ContainerDied","Data":"29f415e6c0d21bcac168f49475cdeaf1891a6f63dadcf7ffbf8355c83cecee03"} Jul 26 15:44:08.677233 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:08.676957 2577 scope.go:117] "RemoveContainer" containerID="c7cd1089d54a1eda3e02f2cb7ebf8841d7261ed09c9f47732086c0c0d588eddf" Jul 26 15:44:08.686460 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:08.686438 2577 scope.go:117] "RemoveContainer" containerID="c7cd1089d54a1eda3e02f2cb7ebf8841d7261ed09c9f47732086c0c0d588eddf" Jul 26 15:44:08.686716 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:44:08.686696 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7cd1089d54a1eda3e02f2cb7ebf8841d7261ed09c9f47732086c0c0d588eddf\": container with ID starting with c7cd1089d54a1eda3e02f2cb7ebf8841d7261ed09c9f47732086c0c0d588eddf not found: ID does not exist" containerID="c7cd1089d54a1eda3e02f2cb7ebf8841d7261ed09c9f47732086c0c0d588eddf" Jul 26 15:44:08.686766 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:08.686723 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7cd1089d54a1eda3e02f2cb7ebf8841d7261ed09c9f47732086c0c0d588eddf"} err="failed to get container status \"c7cd1089d54a1eda3e02f2cb7ebf8841d7261ed09c9f47732086c0c0d588eddf\": rpc error: code = NotFound desc = could not find container \"c7cd1089d54a1eda3e02f2cb7ebf8841d7261ed09c9f47732086c0c0d588eddf\": container with ID starting with c7cd1089d54a1eda3e02f2cb7ebf8841d7261ed09c9f47732086c0c0d588eddf not found: ID does not exist" Jul 26 15:44:08.699047 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:08.699021 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-tvl5t"] Jul 26 15:44:08.702942 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:08.702919 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-tvl5t"] Jul 26 15:44:09.786203 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:09.786169 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f344b88f-a527-463a-9475-aee8a54d2b59" path="/var/lib/kubelet/pods/f344b88f-a527-463a-9475-aee8a54d2b59/volumes" Jul 26 15:44:15.676595 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:15.676558 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-s5mxx" Jul 26 15:44:15.677107 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:15.676623 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-s5mxx" Jul 26 15:44:15.698260 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:15.698234 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-s5mxx" Jul 26 15:44:15.732536 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:15.732507 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-s5mxx" Jul 26 15:44:20.110416 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:20.110359 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hdscz"] Jul 26 15:44:20.110800 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:20.110704 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f344b88f-a527-463a-9475-aee8a54d2b59" containerName="registry-server" Jul 26 15:44:20.110800 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:20.110716 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f344b88f-a527-463a-9475-aee8a54d2b59" containerName="registry-server" Jul 26 15:44:20.110875 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:20.110804 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="f344b88f-a527-463a-9475-aee8a54d2b59" containerName="registry-server" Jul 26 15:44:20.115222 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:20.115203 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hdscz" Jul 26 15:44:20.117750 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:20.117732 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-sjb8w\"" Jul 26 15:44:20.121300 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:20.121276 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hdscz"] Jul 26 15:44:20.170917 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:20.170848 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkqbz\" (UniqueName: \"kubernetes.io/projected/ac43ac70-9ab9-4bdd-ba8a-ea16fe1c6936-kube-api-access-dkqbz\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hdscz\" (UID: \"ac43ac70-9ab9-4bdd-ba8a-ea16fe1c6936\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hdscz" Jul 26 15:44:20.170917 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:20.170923 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ac43ac70-9ab9-4bdd-ba8a-ea16fe1c6936-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hdscz\" (UID: \"ac43ac70-9ab9-4bdd-ba8a-ea16fe1c6936\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hdscz" Jul 26 15:44:20.171147 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:20.170977 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ac43ac70-9ab9-4bdd-ba8a-ea16fe1c6936-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hdscz\" (UID: \"ac43ac70-9ab9-4bdd-ba8a-ea16fe1c6936\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hdscz" Jul 26 15:44:20.271723 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:20.271679 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ac43ac70-9ab9-4bdd-ba8a-ea16fe1c6936-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hdscz\" (UID: \"ac43ac70-9ab9-4bdd-ba8a-ea16fe1c6936\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hdscz" Jul 26 15:44:20.271892 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:20.271776 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkqbz\" (UniqueName: \"kubernetes.io/projected/ac43ac70-9ab9-4bdd-ba8a-ea16fe1c6936-kube-api-access-dkqbz\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hdscz\" (UID: \"ac43ac70-9ab9-4bdd-ba8a-ea16fe1c6936\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hdscz" Jul 26 15:44:20.271892 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:20.271806 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ac43ac70-9ab9-4bdd-ba8a-ea16fe1c6936-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hdscz\" (UID: \"ac43ac70-9ab9-4bdd-ba8a-ea16fe1c6936\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hdscz" Jul 26 15:44:20.272204 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:20.272182 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ac43ac70-9ab9-4bdd-ba8a-ea16fe1c6936-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hdscz\" (UID: \"ac43ac70-9ab9-4bdd-ba8a-ea16fe1c6936\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hdscz" Jul 26 15:44:20.272252 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:20.272238 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ac43ac70-9ab9-4bdd-ba8a-ea16fe1c6936-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hdscz\" (UID: \"ac43ac70-9ab9-4bdd-ba8a-ea16fe1c6936\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hdscz" Jul 26 15:44:20.280168 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:20.280145 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkqbz\" (UniqueName: \"kubernetes.io/projected/ac43ac70-9ab9-4bdd-ba8a-ea16fe1c6936-kube-api-access-dkqbz\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hdscz\" (UID: \"ac43ac70-9ab9-4bdd-ba8a-ea16fe1c6936\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hdscz" Jul 26 15:44:20.425111 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:20.425029 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hdscz" Jul 26 15:44:20.559131 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:20.559096 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hdscz"] Jul 26 15:44:20.561980 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:44:20.561949 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac43ac70_9ab9_4bdd_ba8a_ea16fe1c6936.slice/crio-73cc27de0ec1971a41770e2e7915d433d36006ecd4cbb8b6850a0ed879e3b55c WatchSource:0}: Error finding container 73cc27de0ec1971a41770e2e7915d433d36006ecd4cbb8b6850a0ed879e3b55c: Status 404 returned error can't find the container with id 73cc27de0ec1971a41770e2e7915d433d36006ecd4cbb8b6850a0ed879e3b55c Jul 26 15:44:20.709149 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:20.705710 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xndcq"] Jul 26 15:44:20.711474 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:20.711211 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xndcq" Jul 26 15:44:20.716307 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:20.716281 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xndcq"] Jul 26 15:44:20.734693 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:20.734664 2577 generic.go:356] "Generic (PLEG): container finished" podID="ac43ac70-9ab9-4bdd-ba8a-ea16fe1c6936" containerID="b32e6aa20a62239a6df851a329d28a3c9840361fce4c66ac5db996b0d5623afc" exitCode=0 Jul 26 15:44:20.734827 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:20.734726 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hdscz" event={"ID":"ac43ac70-9ab9-4bdd-ba8a-ea16fe1c6936","Type":"ContainerDied","Data":"b32e6aa20a62239a6df851a329d28a3c9840361fce4c66ac5db996b0d5623afc"} Jul 26 15:44:20.734827 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:20.734763 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hdscz" event={"ID":"ac43ac70-9ab9-4bdd-ba8a-ea16fe1c6936","Type":"ContainerStarted","Data":"73cc27de0ec1971a41770e2e7915d433d36006ecd4cbb8b6850a0ed879e3b55c"} Jul 26 15:44:20.777382 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:20.777346 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1435a9c3-05a1-4b00-9365-4db5b74bb22c-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xndcq\" (UID: \"1435a9c3-05a1-4b00-9365-4db5b74bb22c\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xndcq" Jul 26 15:44:20.777602 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:20.777479 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1435a9c3-05a1-4b00-9365-4db5b74bb22c-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xndcq\" (UID: \"1435a9c3-05a1-4b00-9365-4db5b74bb22c\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xndcq" Jul 26 15:44:20.777602 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:20.777549 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sft8n\" (UniqueName: \"kubernetes.io/projected/1435a9c3-05a1-4b00-9365-4db5b74bb22c-kube-api-access-sft8n\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xndcq\" (UID: \"1435a9c3-05a1-4b00-9365-4db5b74bb22c\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xndcq" Jul 26 15:44:20.878754 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:20.878706 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1435a9c3-05a1-4b00-9365-4db5b74bb22c-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xndcq\" (UID: \"1435a9c3-05a1-4b00-9365-4db5b74bb22c\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xndcq" Jul 26 15:44:20.878754 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:20.878763 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sft8n\" (UniqueName: \"kubernetes.io/projected/1435a9c3-05a1-4b00-9365-4db5b74bb22c-kube-api-access-sft8n\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xndcq\" (UID: \"1435a9c3-05a1-4b00-9365-4db5b74bb22c\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xndcq" Jul 26 15:44:20.879034 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:20.878810 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1435a9c3-05a1-4b00-9365-4db5b74bb22c-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xndcq\" (UID: \"1435a9c3-05a1-4b00-9365-4db5b74bb22c\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xndcq" Jul 26 15:44:20.879303 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:20.879280 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1435a9c3-05a1-4b00-9365-4db5b74bb22c-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xndcq\" (UID: \"1435a9c3-05a1-4b00-9365-4db5b74bb22c\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xndcq" Jul 26 15:44:20.879382 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:20.879302 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1435a9c3-05a1-4b00-9365-4db5b74bb22c-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xndcq\" (UID: \"1435a9c3-05a1-4b00-9365-4db5b74bb22c\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xndcq" Jul 26 15:44:20.887212 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:20.887183 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sft8n\" (UniqueName: \"kubernetes.io/projected/1435a9c3-05a1-4b00-9365-4db5b74bb22c-kube-api-access-sft8n\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xndcq\" (UID: \"1435a9c3-05a1-4b00-9365-4db5b74bb22c\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xndcq" Jul 26 15:44:21.028478 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:21.028447 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xndcq" Jul 26 15:44:21.157787 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:21.157744 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xndcq"] Jul 26 15:44:21.161339 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:44:21.161311 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1435a9c3_05a1_4b00_9365_4db5b74bb22c.slice/crio-56acc269313bdd76e3ed378641aacb7fd84b368478c3d68b1e1ce8103c85975f WatchSource:0}: Error finding container 56acc269313bdd76e3ed378641aacb7fd84b368478c3d68b1e1ce8103c85975f: Status 404 returned error can't find the container with id 56acc269313bdd76e3ed378641aacb7fd84b368478c3d68b1e1ce8103c85975f Jul 26 15:44:21.304924 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:21.304845 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73g75nl"] Jul 26 15:44:21.308349 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:21.308328 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73g75nl" Jul 26 15:44:21.315964 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:21.315932 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73g75nl"] Jul 26 15:44:21.383737 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:21.383704 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cb6ec50d-8695-49c0-8eab-bad100286052-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73g75nl\" (UID: \"cb6ec50d-8695-49c0-8eab-bad100286052\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73g75nl" Jul 26 15:44:21.383916 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:21.383785 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jq585\" (UniqueName: \"kubernetes.io/projected/cb6ec50d-8695-49c0-8eab-bad100286052-kube-api-access-jq585\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73g75nl\" (UID: \"cb6ec50d-8695-49c0-8eab-bad100286052\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73g75nl" Jul 26 15:44:21.383916 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:21.383804 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cb6ec50d-8695-49c0-8eab-bad100286052-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73g75nl\" (UID: \"cb6ec50d-8695-49c0-8eab-bad100286052\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73g75nl" Jul 26 15:44:21.484852 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:21.484813 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jq585\" (UniqueName: \"kubernetes.io/projected/cb6ec50d-8695-49c0-8eab-bad100286052-kube-api-access-jq585\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73g75nl\" (UID: \"cb6ec50d-8695-49c0-8eab-bad100286052\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73g75nl" Jul 26 15:44:21.484852 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:21.484854 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cb6ec50d-8695-49c0-8eab-bad100286052-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73g75nl\" (UID: \"cb6ec50d-8695-49c0-8eab-bad100286052\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73g75nl" Jul 26 15:44:21.485061 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:21.484980 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cb6ec50d-8695-49c0-8eab-bad100286052-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73g75nl\" (UID: \"cb6ec50d-8695-49c0-8eab-bad100286052\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73g75nl" Jul 26 15:44:21.485320 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:21.485306 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cb6ec50d-8695-49c0-8eab-bad100286052-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73g75nl\" (UID: \"cb6ec50d-8695-49c0-8eab-bad100286052\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73g75nl" Jul 26 15:44:21.485362 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:21.485342 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cb6ec50d-8695-49c0-8eab-bad100286052-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73g75nl\" (UID: \"cb6ec50d-8695-49c0-8eab-bad100286052\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73g75nl" Jul 26 15:44:21.492361 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:21.492337 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jq585\" (UniqueName: \"kubernetes.io/projected/cb6ec50d-8695-49c0-8eab-bad100286052-kube-api-access-jq585\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73g75nl\" (UID: \"cb6ec50d-8695-49c0-8eab-bad100286052\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73g75nl" Jul 26 15:44:21.628821 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:21.628736 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73g75nl" Jul 26 15:44:21.712190 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:21.712157 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1674ps"] Jul 26 15:44:21.743899 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:21.743868 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1674ps"] Jul 26 15:44:21.744131 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:21.744032 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1674ps" Jul 26 15:44:21.752572 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:21.752539 2577 generic.go:356] "Generic (PLEG): container finished" podID="1435a9c3-05a1-4b00-9365-4db5b74bb22c" containerID="9efb5d40372477537b08755d44c2d466e278020ff1181620f281c4bee7bab02f" exitCode=0 Jul 26 15:44:21.752742 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:21.752640 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xndcq" event={"ID":"1435a9c3-05a1-4b00-9365-4db5b74bb22c","Type":"ContainerDied","Data":"9efb5d40372477537b08755d44c2d466e278020ff1181620f281c4bee7bab02f"} Jul 26 15:44:21.752742 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:21.752684 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xndcq" event={"ID":"1435a9c3-05a1-4b00-9365-4db5b74bb22c","Type":"ContainerStarted","Data":"56acc269313bdd76e3ed378641aacb7fd84b368478c3d68b1e1ce8103c85975f"} Jul 26 15:44:21.787301 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:21.787261 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5n5tm\" (UniqueName: \"kubernetes.io/projected/b8633ddc-0c3b-4ae5-b5a1-36716c97108f-kube-api-access-5n5tm\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1674ps\" (UID: \"b8633ddc-0c3b-4ae5-b5a1-36716c97108f\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1674ps" Jul 26 15:44:21.787301 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:21.787308 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b8633ddc-0c3b-4ae5-b5a1-36716c97108f-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1674ps\" (UID: \"b8633ddc-0c3b-4ae5-b5a1-36716c97108f\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1674ps" Jul 26 15:44:21.787574 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:21.787439 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b8633ddc-0c3b-4ae5-b5a1-36716c97108f-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1674ps\" (UID: \"b8633ddc-0c3b-4ae5-b5a1-36716c97108f\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1674ps" Jul 26 15:44:21.799166 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:21.799143 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73g75nl"] Jul 26 15:44:21.855340 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:44:21.855307 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb6ec50d_8695_49c0_8eab_bad100286052.slice/crio-eac0ddfaa12254c6a5de70ef6a7e17dc01fa3d955c41d4953aefe6504fedc304 WatchSource:0}: Error finding container eac0ddfaa12254c6a5de70ef6a7e17dc01fa3d955c41d4953aefe6504fedc304: Status 404 returned error can't find the container with id eac0ddfaa12254c6a5de70ef6a7e17dc01fa3d955c41d4953aefe6504fedc304 Jul 26 15:44:21.888579 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:21.888555 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5n5tm\" (UniqueName: \"kubernetes.io/projected/b8633ddc-0c3b-4ae5-b5a1-36716c97108f-kube-api-access-5n5tm\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1674ps\" (UID: \"b8633ddc-0c3b-4ae5-b5a1-36716c97108f\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1674ps" Jul 26 15:44:21.888704 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:21.888654 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b8633ddc-0c3b-4ae5-b5a1-36716c97108f-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1674ps\" (UID: \"b8633ddc-0c3b-4ae5-b5a1-36716c97108f\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1674ps" Jul 26 15:44:21.888778 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:21.888758 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b8633ddc-0c3b-4ae5-b5a1-36716c97108f-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1674ps\" (UID: \"b8633ddc-0c3b-4ae5-b5a1-36716c97108f\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1674ps" Jul 26 15:44:21.889310 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:21.889078 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b8633ddc-0c3b-4ae5-b5a1-36716c97108f-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1674ps\" (UID: \"b8633ddc-0c3b-4ae5-b5a1-36716c97108f\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1674ps" Jul 26 15:44:21.889310 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:21.889105 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b8633ddc-0c3b-4ae5-b5a1-36716c97108f-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1674ps\" (UID: \"b8633ddc-0c3b-4ae5-b5a1-36716c97108f\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1674ps" Jul 26 15:44:21.899461 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:21.899438 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5n5tm\" (UniqueName: \"kubernetes.io/projected/b8633ddc-0c3b-4ae5-b5a1-36716c97108f-kube-api-access-5n5tm\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1674ps\" (UID: \"b8633ddc-0c3b-4ae5-b5a1-36716c97108f\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1674ps" Jul 26 15:44:22.058281 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:22.058252 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1674ps" Jul 26 15:44:22.188947 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:22.188923 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1674ps"] Jul 26 15:44:22.190656 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:44:22.190624 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8633ddc_0c3b_4ae5_b5a1_36716c97108f.slice/crio-4d688660feaa9e01c07e9d4b6c22d0e1f578dd27369f5b6c80a4e3e567ca3e51 WatchSource:0}: Error finding container 4d688660feaa9e01c07e9d4b6c22d0e1f578dd27369f5b6c80a4e3e567ca3e51: Status 404 returned error can't find the container with id 4d688660feaa9e01c07e9d4b6c22d0e1f578dd27369f5b6c80a4e3e567ca3e51 Jul 26 15:44:22.759242 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:22.759151 2577 generic.go:356] "Generic (PLEG): container finished" podID="cb6ec50d-8695-49c0-8eab-bad100286052" containerID="39979f83591723a6d205cd56f83de1077c1e71f4008788b46479500460848592" exitCode=0 Jul 26 15:44:22.759426 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:22.759241 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73g75nl" event={"ID":"cb6ec50d-8695-49c0-8eab-bad100286052","Type":"ContainerDied","Data":"39979f83591723a6d205cd56f83de1077c1e71f4008788b46479500460848592"} Jul 26 15:44:22.759426 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:22.759307 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73g75nl" event={"ID":"cb6ec50d-8695-49c0-8eab-bad100286052","Type":"ContainerStarted","Data":"eac0ddfaa12254c6a5de70ef6a7e17dc01fa3d955c41d4953aefe6504fedc304"} Jul 26 15:44:22.761136 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:22.761113 2577 generic.go:356] "Generic (PLEG): container finished" podID="1435a9c3-05a1-4b00-9365-4db5b74bb22c" containerID="74d76a7f1ed1c5a8fd58c8b420a9922ce5eb0e4831a9480b0448c0c4f79bdd4b" exitCode=0 Jul 26 15:44:22.761228 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:22.761205 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xndcq" event={"ID":"1435a9c3-05a1-4b00-9365-4db5b74bb22c","Type":"ContainerDied","Data":"74d76a7f1ed1c5a8fd58c8b420a9922ce5eb0e4831a9480b0448c0c4f79bdd4b"} Jul 26 15:44:22.763266 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:22.763240 2577 generic.go:356] "Generic (PLEG): container finished" podID="ac43ac70-9ab9-4bdd-ba8a-ea16fe1c6936" containerID="05edcddd51d9c86b49d9c0f2fb348c26729cff6fe7da20c30f742e9f2aa170eb" exitCode=0 Jul 26 15:44:22.763360 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:22.763342 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hdscz" event={"ID":"ac43ac70-9ab9-4bdd-ba8a-ea16fe1c6936","Type":"ContainerDied","Data":"05edcddd51d9c86b49d9c0f2fb348c26729cff6fe7da20c30f742e9f2aa170eb"} Jul 26 15:44:22.764976 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:22.764958 2577 generic.go:356] "Generic (PLEG): container finished" podID="b8633ddc-0c3b-4ae5-b5a1-36716c97108f" containerID="050604675e7a99b080652e532b50ff71e6a334c1bf4c8177739ffe5f48a9a14c" exitCode=0 Jul 26 15:44:22.765086 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:22.765038 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1674ps" event={"ID":"b8633ddc-0c3b-4ae5-b5a1-36716c97108f","Type":"ContainerDied","Data":"050604675e7a99b080652e532b50ff71e6a334c1bf4c8177739ffe5f48a9a14c"} Jul 26 15:44:22.765086 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:22.765069 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1674ps" event={"ID":"b8633ddc-0c3b-4ae5-b5a1-36716c97108f","Type":"ContainerStarted","Data":"4d688660feaa9e01c07e9d4b6c22d0e1f578dd27369f5b6c80a4e3e567ca3e51"} Jul 26 15:44:23.771598 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:23.771571 2577 generic.go:356] "Generic (PLEG): container finished" podID="b8633ddc-0c3b-4ae5-b5a1-36716c97108f" containerID="5c4828448027adf50311ad464615242df5853dc9738f03f427ccf2c85e3c170b" exitCode=0 Jul 26 15:44:23.771934 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:23.771650 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1674ps" event={"ID":"b8633ddc-0c3b-4ae5-b5a1-36716c97108f","Type":"ContainerDied","Data":"5c4828448027adf50311ad464615242df5853dc9738f03f427ccf2c85e3c170b"} Jul 26 15:44:23.773487 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:23.773464 2577 generic.go:356] "Generic (PLEG): container finished" podID="cb6ec50d-8695-49c0-8eab-bad100286052" containerID="8d9eeb8296a99be812ee0a600ea94e6a9b9cd8b0cdf0ce92d709e3696fdbb162" exitCode=0 Jul 26 15:44:23.773594 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:23.773545 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73g75nl" event={"ID":"cb6ec50d-8695-49c0-8eab-bad100286052","Type":"ContainerDied","Data":"8d9eeb8296a99be812ee0a600ea94e6a9b9cd8b0cdf0ce92d709e3696fdbb162"} Jul 26 15:44:23.775702 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:23.775680 2577 generic.go:356] "Generic (PLEG): container finished" podID="1435a9c3-05a1-4b00-9365-4db5b74bb22c" containerID="a38cb5be6316543c1e314730c207c94e3548ae0e08e0cbdfe490222088e98041" exitCode=0 Jul 26 15:44:23.775802 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:23.775730 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xndcq" event={"ID":"1435a9c3-05a1-4b00-9365-4db5b74bb22c","Type":"ContainerDied","Data":"a38cb5be6316543c1e314730c207c94e3548ae0e08e0cbdfe490222088e98041"} Jul 26 15:44:23.778036 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:23.778014 2577 generic.go:356] "Generic (PLEG): container finished" podID="ac43ac70-9ab9-4bdd-ba8a-ea16fe1c6936" containerID="7111f3cf7fb565f54035084be070193f5abc2e770d3704d4c4c312b7a55aa7bf" exitCode=0 Jul 26 15:44:23.778124 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:23.778085 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hdscz" event={"ID":"ac43ac70-9ab9-4bdd-ba8a-ea16fe1c6936","Type":"ContainerDied","Data":"7111f3cf7fb565f54035084be070193f5abc2e770d3704d4c4c312b7a55aa7bf"} Jul 26 15:44:24.785790 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:24.785747 2577 generic.go:356] "Generic (PLEG): container finished" podID="b8633ddc-0c3b-4ae5-b5a1-36716c97108f" containerID="3b37ec5d08bfcf63124f61e178c315ff65ce76bad735d5978bf58018930a867e" exitCode=0 Jul 26 15:44:24.786209 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:24.785824 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1674ps" event={"ID":"b8633ddc-0c3b-4ae5-b5a1-36716c97108f","Type":"ContainerDied","Data":"3b37ec5d08bfcf63124f61e178c315ff65ce76bad735d5978bf58018930a867e"} Jul 26 15:44:24.787689 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:24.787665 2577 generic.go:356] "Generic (PLEG): container finished" podID="cb6ec50d-8695-49c0-8eab-bad100286052" containerID="93e3decdcfd77580263e2090cc68d9a366ada5a72f438ebfd420a3519b63772b" exitCode=0 Jul 26 15:44:24.787795 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:24.787733 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73g75nl" event={"ID":"cb6ec50d-8695-49c0-8eab-bad100286052","Type":"ContainerDied","Data":"93e3decdcfd77580263e2090cc68d9a366ada5a72f438ebfd420a3519b63772b"} Jul 26 15:44:24.930022 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:24.929997 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hdscz" Jul 26 15:44:24.966649 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:24.966624 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xndcq" Jul 26 15:44:25.020298 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:25.020265 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1435a9c3-05a1-4b00-9365-4db5b74bb22c-util\") pod \"1435a9c3-05a1-4b00-9365-4db5b74bb22c\" (UID: \"1435a9c3-05a1-4b00-9365-4db5b74bb22c\") " Jul 26 15:44:25.020298 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:25.020302 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ac43ac70-9ab9-4bdd-ba8a-ea16fe1c6936-bundle\") pod \"ac43ac70-9ab9-4bdd-ba8a-ea16fe1c6936\" (UID: \"ac43ac70-9ab9-4bdd-ba8a-ea16fe1c6936\") " Jul 26 15:44:25.020562 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:25.020328 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ac43ac70-9ab9-4bdd-ba8a-ea16fe1c6936-util\") pod \"ac43ac70-9ab9-4bdd-ba8a-ea16fe1c6936\" (UID: \"ac43ac70-9ab9-4bdd-ba8a-ea16fe1c6936\") " Jul 26 15:44:25.020562 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:25.020349 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sft8n\" (UniqueName: \"kubernetes.io/projected/1435a9c3-05a1-4b00-9365-4db5b74bb22c-kube-api-access-sft8n\") pod \"1435a9c3-05a1-4b00-9365-4db5b74bb22c\" (UID: \"1435a9c3-05a1-4b00-9365-4db5b74bb22c\") " Jul 26 15:44:25.020562 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:25.020452 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1435a9c3-05a1-4b00-9365-4db5b74bb22c-bundle\") pod \"1435a9c3-05a1-4b00-9365-4db5b74bb22c\" (UID: \"1435a9c3-05a1-4b00-9365-4db5b74bb22c\") " Jul 26 15:44:25.020720 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:25.020561 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dkqbz\" (UniqueName: \"kubernetes.io/projected/ac43ac70-9ab9-4bdd-ba8a-ea16fe1c6936-kube-api-access-dkqbz\") pod \"ac43ac70-9ab9-4bdd-ba8a-ea16fe1c6936\" (UID: \"ac43ac70-9ab9-4bdd-ba8a-ea16fe1c6936\") " Jul 26 15:44:25.020939 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:25.020910 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1435a9c3-05a1-4b00-9365-4db5b74bb22c-bundle" (OuterVolumeSpecName: "bundle") pod "1435a9c3-05a1-4b00-9365-4db5b74bb22c" (UID: "1435a9c3-05a1-4b00-9365-4db5b74bb22c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:44:25.021083 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:25.020995 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac43ac70-9ab9-4bdd-ba8a-ea16fe1c6936-bundle" (OuterVolumeSpecName: "bundle") pod "ac43ac70-9ab9-4bdd-ba8a-ea16fe1c6936" (UID: "ac43ac70-9ab9-4bdd-ba8a-ea16fe1c6936"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:44:25.022778 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:25.022750 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1435a9c3-05a1-4b00-9365-4db5b74bb22c-kube-api-access-sft8n" (OuterVolumeSpecName: "kube-api-access-sft8n") pod "1435a9c3-05a1-4b00-9365-4db5b74bb22c" (UID: "1435a9c3-05a1-4b00-9365-4db5b74bb22c"). InnerVolumeSpecName "kube-api-access-sft8n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 15:44:25.023095 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:25.023063 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac43ac70-9ab9-4bdd-ba8a-ea16fe1c6936-kube-api-access-dkqbz" (OuterVolumeSpecName: "kube-api-access-dkqbz") pod "ac43ac70-9ab9-4bdd-ba8a-ea16fe1c6936" (UID: "ac43ac70-9ab9-4bdd-ba8a-ea16fe1c6936"). InnerVolumeSpecName "kube-api-access-dkqbz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 15:44:25.026539 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:25.026514 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac43ac70-9ab9-4bdd-ba8a-ea16fe1c6936-util" (OuterVolumeSpecName: "util") pod "ac43ac70-9ab9-4bdd-ba8a-ea16fe1c6936" (UID: "ac43ac70-9ab9-4bdd-ba8a-ea16fe1c6936"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:44:25.026669 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:25.026649 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1435a9c3-05a1-4b00-9365-4db5b74bb22c-util" (OuterVolumeSpecName: "util") pod "1435a9c3-05a1-4b00-9365-4db5b74bb22c" (UID: "1435a9c3-05a1-4b00-9365-4db5b74bb22c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:44:25.121626 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:25.121546 2577 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1435a9c3-05a1-4b00-9365-4db5b74bb22c-bundle\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 15:44:25.121626 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:25.121579 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkqbz\" (UniqueName: \"kubernetes.io/projected/ac43ac70-9ab9-4bdd-ba8a-ea16fe1c6936-kube-api-access-dkqbz\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 15:44:25.121626 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:25.121591 2577 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1435a9c3-05a1-4b00-9365-4db5b74bb22c-util\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 15:44:25.121626 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:25.121602 2577 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ac43ac70-9ab9-4bdd-ba8a-ea16fe1c6936-bundle\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 15:44:25.121626 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:25.121609 2577 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ac43ac70-9ab9-4bdd-ba8a-ea16fe1c6936-util\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 15:44:25.121626 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:25.121617 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sft8n\" (UniqueName: \"kubernetes.io/projected/1435a9c3-05a1-4b00-9365-4db5b74bb22c-kube-api-access-sft8n\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 15:44:25.794910 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:25.794879 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xndcq" Jul 26 15:44:25.795439 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:25.794881 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xndcq" event={"ID":"1435a9c3-05a1-4b00-9365-4db5b74bb22c","Type":"ContainerDied","Data":"56acc269313bdd76e3ed378641aacb7fd84b368478c3d68b1e1ce8103c85975f"} Jul 26 15:44:25.795439 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:25.795010 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="56acc269313bdd76e3ed378641aacb7fd84b368478c3d68b1e1ce8103c85975f" Jul 26 15:44:25.797178 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:25.797157 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hdscz" Jul 26 15:44:25.797289 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:25.797191 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hdscz" event={"ID":"ac43ac70-9ab9-4bdd-ba8a-ea16fe1c6936","Type":"ContainerDied","Data":"73cc27de0ec1971a41770e2e7915d433d36006ecd4cbb8b6850a0ed879e3b55c"} Jul 26 15:44:25.797289 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:25.797229 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="73cc27de0ec1971a41770e2e7915d433d36006ecd4cbb8b6850a0ed879e3b55c" Jul 26 15:44:25.943147 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:25.943122 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1674ps" Jul 26 15:44:25.972702 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:25.972662 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73g75nl" Jul 26 15:44:26.030623 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:26.030590 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cb6ec50d-8695-49c0-8eab-bad100286052-util\") pod \"cb6ec50d-8695-49c0-8eab-bad100286052\" (UID: \"cb6ec50d-8695-49c0-8eab-bad100286052\") " Jul 26 15:44:26.030802 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:26.030669 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jq585\" (UniqueName: \"kubernetes.io/projected/cb6ec50d-8695-49c0-8eab-bad100286052-kube-api-access-jq585\") pod \"cb6ec50d-8695-49c0-8eab-bad100286052\" (UID: \"cb6ec50d-8695-49c0-8eab-bad100286052\") " Jul 26 15:44:26.030802 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:26.030712 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cb6ec50d-8695-49c0-8eab-bad100286052-bundle\") pod \"cb6ec50d-8695-49c0-8eab-bad100286052\" (UID: \"cb6ec50d-8695-49c0-8eab-bad100286052\") " Jul 26 15:44:26.030802 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:26.030734 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b8633ddc-0c3b-4ae5-b5a1-36716c97108f-bundle\") pod \"b8633ddc-0c3b-4ae5-b5a1-36716c97108f\" (UID: \"b8633ddc-0c3b-4ae5-b5a1-36716c97108f\") " Jul 26 15:44:26.030802 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:26.030758 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b8633ddc-0c3b-4ae5-b5a1-36716c97108f-util\") pod \"b8633ddc-0c3b-4ae5-b5a1-36716c97108f\" (UID: \"b8633ddc-0c3b-4ae5-b5a1-36716c97108f\") " Jul 26 15:44:26.031032 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:26.030824 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5n5tm\" (UniqueName: \"kubernetes.io/projected/b8633ddc-0c3b-4ae5-b5a1-36716c97108f-kube-api-access-5n5tm\") pod \"b8633ddc-0c3b-4ae5-b5a1-36716c97108f\" (UID: \"b8633ddc-0c3b-4ae5-b5a1-36716c97108f\") " Jul 26 15:44:26.031637 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:26.031444 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8633ddc-0c3b-4ae5-b5a1-36716c97108f-bundle" (OuterVolumeSpecName: "bundle") pod "b8633ddc-0c3b-4ae5-b5a1-36716c97108f" (UID: "b8633ddc-0c3b-4ae5-b5a1-36716c97108f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:44:26.031637 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:26.031499 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb6ec50d-8695-49c0-8eab-bad100286052-bundle" (OuterVolumeSpecName: "bundle") pod "cb6ec50d-8695-49c0-8eab-bad100286052" (UID: "cb6ec50d-8695-49c0-8eab-bad100286052"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:44:26.034755 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:26.034726 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb6ec50d-8695-49c0-8eab-bad100286052-kube-api-access-jq585" (OuterVolumeSpecName: "kube-api-access-jq585") pod "cb6ec50d-8695-49c0-8eab-bad100286052" (UID: "cb6ec50d-8695-49c0-8eab-bad100286052"). InnerVolumeSpecName "kube-api-access-jq585". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 15:44:26.034847 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:26.034772 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8633ddc-0c3b-4ae5-b5a1-36716c97108f-kube-api-access-5n5tm" (OuterVolumeSpecName: "kube-api-access-5n5tm") pod "b8633ddc-0c3b-4ae5-b5a1-36716c97108f" (UID: "b8633ddc-0c3b-4ae5-b5a1-36716c97108f"). InnerVolumeSpecName "kube-api-access-5n5tm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 15:44:26.039278 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:26.039243 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8633ddc-0c3b-4ae5-b5a1-36716c97108f-util" (OuterVolumeSpecName: "util") pod "b8633ddc-0c3b-4ae5-b5a1-36716c97108f" (UID: "b8633ddc-0c3b-4ae5-b5a1-36716c97108f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:44:26.040040 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:26.040017 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb6ec50d-8695-49c0-8eab-bad100286052-util" (OuterVolumeSpecName: "util") pod "cb6ec50d-8695-49c0-8eab-bad100286052" (UID: "cb6ec50d-8695-49c0-8eab-bad100286052"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:44:26.131623 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:26.131539 2577 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cb6ec50d-8695-49c0-8eab-bad100286052-bundle\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 15:44:26.131623 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:26.131567 2577 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b8633ddc-0c3b-4ae5-b5a1-36716c97108f-bundle\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 15:44:26.131623 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:26.131575 2577 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b8633ddc-0c3b-4ae5-b5a1-36716c97108f-util\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 15:44:26.131623 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:26.131583 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5n5tm\" (UniqueName: \"kubernetes.io/projected/b8633ddc-0c3b-4ae5-b5a1-36716c97108f-kube-api-access-5n5tm\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 15:44:26.131623 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:26.131594 2577 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cb6ec50d-8695-49c0-8eab-bad100286052-util\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 15:44:26.131623 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:26.131603 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jq585\" (UniqueName: \"kubernetes.io/projected/cb6ec50d-8695-49c0-8eab-bad100286052-kube-api-access-jq585\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 15:44:26.804357 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:26.804323 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1674ps" Jul 26 15:44:26.804820 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:26.804325 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1674ps" event={"ID":"b8633ddc-0c3b-4ae5-b5a1-36716c97108f","Type":"ContainerDied","Data":"4d688660feaa9e01c07e9d4b6c22d0e1f578dd27369f5b6c80a4e3e567ca3e51"} Jul 26 15:44:26.804820 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:26.804447 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4d688660feaa9e01c07e9d4b6c22d0e1f578dd27369f5b6c80a4e3e567ca3e51" Jul 26 15:44:26.806165 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:26.806136 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73g75nl" event={"ID":"cb6ec50d-8695-49c0-8eab-bad100286052","Type":"ContainerDied","Data":"eac0ddfaa12254c6a5de70ef6a7e17dc01fa3d955c41d4953aefe6504fedc304"} Jul 26 15:44:26.806282 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:26.806183 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eac0ddfaa12254c6a5de70ef6a7e17dc01fa3d955c41d4953aefe6504fedc304" Jul 26 15:44:26.806282 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:26.806183 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73g75nl" Jul 26 15:44:33.112379 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:33.112343 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:44:33.711657 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:33.711620 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:44:34.741220 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:34.741183 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:44:34.849617 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:34.849572 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:44:37.192355 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:37.192323 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-5qlgd"] Jul 26 15:44:37.192845 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:37.192814 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac43ac70-9ab9-4bdd-ba8a-ea16fe1c6936" containerName="pull" Jul 26 15:44:37.192845 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:37.192835 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac43ac70-9ab9-4bdd-ba8a-ea16fe1c6936" containerName="pull" Jul 26 15:44:37.192979 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:37.192851 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8633ddc-0c3b-4ae5-b5a1-36716c97108f" containerName="pull" Jul 26 15:44:37.192979 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:37.192859 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8633ddc-0c3b-4ae5-b5a1-36716c97108f" containerName="pull" Jul 26 15:44:37.192979 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:37.192869 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8633ddc-0c3b-4ae5-b5a1-36716c97108f" containerName="extract" Jul 26 15:44:37.192979 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:37.192877 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8633ddc-0c3b-4ae5-b5a1-36716c97108f" containerName="extract" Jul 26 15:44:37.192979 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:37.192889 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1435a9c3-05a1-4b00-9365-4db5b74bb22c" containerName="pull" Jul 26 15:44:37.192979 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:37.192898 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="1435a9c3-05a1-4b00-9365-4db5b74bb22c" containerName="pull" Jul 26 15:44:37.192979 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:37.192910 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb6ec50d-8695-49c0-8eab-bad100286052" containerName="util" Jul 26 15:44:37.192979 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:37.192918 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb6ec50d-8695-49c0-8eab-bad100286052" containerName="util" Jul 26 15:44:37.192979 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:37.192927 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb6ec50d-8695-49c0-8eab-bad100286052" containerName="extract" Jul 26 15:44:37.192979 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:37.192937 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb6ec50d-8695-49c0-8eab-bad100286052" containerName="extract" Jul 26 15:44:37.192979 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:37.192951 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1435a9c3-05a1-4b00-9365-4db5b74bb22c" containerName="util" Jul 26 15:44:37.192979 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:37.192959 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="1435a9c3-05a1-4b00-9365-4db5b74bb22c" containerName="util" Jul 26 15:44:37.192979 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:37.192972 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac43ac70-9ab9-4bdd-ba8a-ea16fe1c6936" containerName="util" Jul 26 15:44:37.192979 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:37.192980 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac43ac70-9ab9-4bdd-ba8a-ea16fe1c6936" containerName="util" Jul 26 15:44:37.193653 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:37.192989 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8633ddc-0c3b-4ae5-b5a1-36716c97108f" containerName="util" Jul 26 15:44:37.193653 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:37.192997 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8633ddc-0c3b-4ae5-b5a1-36716c97108f" containerName="util" Jul 26 15:44:37.193653 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:37.193007 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1435a9c3-05a1-4b00-9365-4db5b74bb22c" containerName="extract" Jul 26 15:44:37.193653 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:37.193016 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="1435a9c3-05a1-4b00-9365-4db5b74bb22c" containerName="extract" Jul 26 15:44:37.193653 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:37.193031 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac43ac70-9ab9-4bdd-ba8a-ea16fe1c6936" containerName="extract" Jul 26 15:44:37.193653 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:37.193040 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac43ac70-9ab9-4bdd-ba8a-ea16fe1c6936" containerName="extract" Jul 26 15:44:37.193653 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:37.193049 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb6ec50d-8695-49c0-8eab-bad100286052" containerName="pull" Jul 26 15:44:37.193653 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:37.193058 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb6ec50d-8695-49c0-8eab-bad100286052" containerName="pull" Jul 26 15:44:37.193653 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:37.193150 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac43ac70-9ab9-4bdd-ba8a-ea16fe1c6936" containerName="extract" Jul 26 15:44:37.193653 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:37.193163 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8633ddc-0c3b-4ae5-b5a1-36716c97108f" containerName="extract" Jul 26 15:44:37.193653 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:37.193173 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb6ec50d-8695-49c0-8eab-bad100286052" containerName="extract" Jul 26 15:44:37.193653 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:37.193184 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="1435a9c3-05a1-4b00-9365-4db5b74bb22c" containerName="extract" Jul 26 15:44:37.197716 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:37.197696 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-5qlgd" Jul 26 15:44:37.202445 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:37.202419 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-9wpmq\"" Jul 26 15:44:37.216686 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:37.216660 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-5qlgd"] Jul 26 15:44:37.325391 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:37.325352 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrwwt\" (UniqueName: \"kubernetes.io/projected/8fa87b24-e0fc-4e83-adc1-d9fcf1395fc6-kube-api-access-vrwwt\") pod \"authorino-operator-75cc8d5cb7-5qlgd\" (UID: \"8fa87b24-e0fc-4e83-adc1-d9fcf1395fc6\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-5qlgd" Jul 26 15:44:37.426616 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:37.426577 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrwwt\" (UniqueName: \"kubernetes.io/projected/8fa87b24-e0fc-4e83-adc1-d9fcf1395fc6-kube-api-access-vrwwt\") pod \"authorino-operator-75cc8d5cb7-5qlgd\" (UID: \"8fa87b24-e0fc-4e83-adc1-d9fcf1395fc6\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-5qlgd" Jul 26 15:44:37.434748 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:37.434716 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrwwt\" (UniqueName: \"kubernetes.io/projected/8fa87b24-e0fc-4e83-adc1-d9fcf1395fc6-kube-api-access-vrwwt\") pod \"authorino-operator-75cc8d5cb7-5qlgd\" (UID: \"8fa87b24-e0fc-4e83-adc1-d9fcf1395fc6\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-5qlgd" Jul 26 15:44:37.507865 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:37.507783 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-5qlgd" Jul 26 15:44:37.638042 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:37.637991 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-5qlgd"] Jul 26 15:44:37.640371 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:44:37.640348 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fa87b24_e0fc_4e83_adc1_d9fcf1395fc6.slice/crio-cd73effc27b7e4a6fcbd041954808de2971dea9fc8ed28e43cca97feab3fde2f WatchSource:0}: Error finding container cd73effc27b7e4a6fcbd041954808de2971dea9fc8ed28e43cca97feab3fde2f: Status 404 returned error can't find the container with id cd73effc27b7e4a6fcbd041954808de2971dea9fc8ed28e43cca97feab3fde2f Jul 26 15:44:37.866756 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:37.866723 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-5qlgd" event={"ID":"8fa87b24-e0fc-4e83-adc1-d9fcf1395fc6","Type":"ContainerStarted","Data":"cd73effc27b7e4a6fcbd041954808de2971dea9fc8ed28e43cca97feab3fde2f"} Jul 26 15:44:38.336316 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:38.336277 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:44:40.883641 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:40.883609 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-5qlgd" event={"ID":"8fa87b24-e0fc-4e83-adc1-d9fcf1395fc6","Type":"ContainerStarted","Data":"d69e83d7c72ad9797cce6e3c45e74f5fd03d017ced2594bb42785d01cf505381"} Jul 26 15:44:40.884131 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:40.883738 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-5qlgd" Jul 26 15:44:40.904567 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:40.904511 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-5qlgd" podStartSLOduration=1.609795404 podStartE2EDuration="3.904495007s" totalImagesPullingTime="2.294699603s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:44:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:44:40.902942386 +0000 UTC m=+403.642295715" watchObservedRunningTime="2026-07-26 15:44:40.904495007 +0000 UTC m=+403.643848317" Jul 26 15:44:41.135825 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:41.135744 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:44:41.775790 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:41.775746 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:44:42.961235 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:42.961196 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-q7jfk"] Jul 26 15:44:42.970462 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:42.970431 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-q7jfk" Jul 26 15:44:42.974299 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:42.974268 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-fgdws\"" Jul 26 15:44:42.974484 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:42.974371 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 26 15:44:42.975996 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:42.975971 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-q7jfk"] Jul 26 15:44:43.081321 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:43.081278 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srxjw\" (UniqueName: \"kubernetes.io/projected/da528638-84d1-40cf-9208-7929f925421a-kube-api-access-srxjw\") pod \"dns-operator-controller-manager-864dd78c4c-q7jfk\" (UID: \"da528638-84d1-40cf-9208-7929f925421a\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-q7jfk" Jul 26 15:44:43.181709 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:43.181675 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srxjw\" (UniqueName: \"kubernetes.io/projected/da528638-84d1-40cf-9208-7929f925421a-kube-api-access-srxjw\") pod \"dns-operator-controller-manager-864dd78c4c-q7jfk\" (UID: \"da528638-84d1-40cf-9208-7929f925421a\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-q7jfk" Jul 26 15:44:43.189555 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:43.189530 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-srxjw\" (UniqueName: \"kubernetes.io/projected/da528638-84d1-40cf-9208-7929f925421a-kube-api-access-srxjw\") pod \"dns-operator-controller-manager-864dd78c4c-q7jfk\" (UID: \"da528638-84d1-40cf-9208-7929f925421a\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-q7jfk" Jul 26 15:44:43.282563 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:43.282525 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-q7jfk" Jul 26 15:44:43.419384 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:43.419357 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-q7jfk"] Jul 26 15:44:43.421520 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:44:43.421495 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda528638_84d1_40cf_9208_7929f925421a.slice/crio-6b4fad55733f8c02f9fd4bdef7b3714a4b45533977600fd39862a08560ac235d WatchSource:0}: Error finding container 6b4fad55733f8c02f9fd4bdef7b3714a4b45533977600fd39862a08560ac235d: Status 404 returned error can't find the container with id 6b4fad55733f8c02f9fd4bdef7b3714a4b45533977600fd39862a08560ac235d Jul 26 15:44:43.524440 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:43.524404 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:44:43.902476 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:43.902440 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-q7jfk" event={"ID":"da528638-84d1-40cf-9208-7929f925421a","Type":"ContainerStarted","Data":"6b4fad55733f8c02f9fd4bdef7b3714a4b45533977600fd39862a08560ac235d"} Jul 26 15:44:44.011985 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:44.011948 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:44:45.915635 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:45.915598 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-q7jfk" event={"ID":"da528638-84d1-40cf-9208-7929f925421a","Type":"ContainerStarted","Data":"71c784db0fa75f657dbad56bae66f2a4a595a1f251820c32437bc93a07f9f03a"} Jul 26 15:44:45.916049 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:45.915658 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-q7jfk" Jul 26 15:44:45.937222 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:45.937173 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-q7jfk" podStartSLOduration=2.060312808 podStartE2EDuration="3.93715781s" totalImagesPullingTime="1.876845002s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:44:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:44:45.934672215 +0000 UTC m=+408.674025529" watchObservedRunningTime="2026-07-26 15:44:45.93715781 +0000 UTC m=+408.676511120" Jul 26 15:44:47.131479 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:47.131428 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:44:47.467849 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:47.467754 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vpmjr"] Jul 26 15:44:47.471554 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:47.471528 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vpmjr" Jul 26 15:44:47.473913 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:47.473891 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-7fjh9\"" Jul 26 15:44:47.485390 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:47.485369 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vpmjr"] Jul 26 15:44:47.621543 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:47.621505 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/769d0090-52e6-4543-95e0-ec93d477fd75-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-vpmjr\" (UID: \"769d0090-52e6-4543-95e0-ec93d477fd75\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vpmjr" Jul 26 15:44:47.621728 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:47.621625 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbg57\" (UniqueName: \"kubernetes.io/projected/769d0090-52e6-4543-95e0-ec93d477fd75-kube-api-access-nbg57\") pod \"kuadrant-operator-controller-manager-8684c9ff49-vpmjr\" (UID: \"769d0090-52e6-4543-95e0-ec93d477fd75\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vpmjr" Jul 26 15:44:47.722558 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:47.722446 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/769d0090-52e6-4543-95e0-ec93d477fd75-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-vpmjr\" (UID: \"769d0090-52e6-4543-95e0-ec93d477fd75\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vpmjr" Jul 26 15:44:47.722558 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:47.722556 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbg57\" (UniqueName: \"kubernetes.io/projected/769d0090-52e6-4543-95e0-ec93d477fd75-kube-api-access-nbg57\") pod \"kuadrant-operator-controller-manager-8684c9ff49-vpmjr\" (UID: \"769d0090-52e6-4543-95e0-ec93d477fd75\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vpmjr" Jul 26 15:44:47.722931 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:47.722906 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/769d0090-52e6-4543-95e0-ec93d477fd75-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-vpmjr\" (UID: \"769d0090-52e6-4543-95e0-ec93d477fd75\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vpmjr" Jul 26 15:44:47.734738 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:47.734710 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbg57\" (UniqueName: \"kubernetes.io/projected/769d0090-52e6-4543-95e0-ec93d477fd75-kube-api-access-nbg57\") pod \"kuadrant-operator-controller-manager-8684c9ff49-vpmjr\" (UID: \"769d0090-52e6-4543-95e0-ec93d477fd75\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vpmjr" Jul 26 15:44:47.781417 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:47.781362 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vpmjr" Jul 26 15:44:47.922355 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:47.922278 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vpmjr"] Jul 26 15:44:47.925018 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:44:47.924987 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod769d0090_52e6_4543_95e0_ec93d477fd75.slice/crio-7557e1c23676b1aca5c9ba50ca39b40599b606417fac28474cf38b08b7a05df2 WatchSource:0}: Error finding container 7557e1c23676b1aca5c9ba50ca39b40599b606417fac28474cf38b08b7a05df2: Status 404 returned error can't find the container with id 7557e1c23676b1aca5c9ba50ca39b40599b606417fac28474cf38b08b7a05df2 Jul 26 15:44:48.725761 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:48.725707 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:44:48.817078 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:48.816513 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:44:48.896190 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:48.896152 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:44:48.933781 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:48.933725 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vpmjr" event={"ID":"769d0090-52e6-4543-95e0-ec93d477fd75","Type":"ContainerStarted","Data":"7557e1c23676b1aca5c9ba50ca39b40599b606417fac28474cf38b08b7a05df2"} Jul 26 15:44:49.135654 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:49.135613 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:44:49.241838 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:49.241803 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:44:51.891375 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:51.891344 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-5qlgd" Jul 26 15:44:52.550050 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:52.550015 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:44:52.672913 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:52.672878 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:44:53.962466 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:53.962427 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vpmjr" event={"ID":"769d0090-52e6-4543-95e0-ec93d477fd75","Type":"ContainerStarted","Data":"b68e4131d6c40c8726df501dc8cce9aab1b2a3d632ae00bf21c97d58c4cb1f40"} Jul 26 15:44:53.962883 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:53.962611 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vpmjr" Jul 26 15:44:53.984705 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:53.984651 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vpmjr" podStartSLOduration=1.939231353 podStartE2EDuration="6.98463409s" totalImagesPullingTime="5.045402737s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:44:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:44:53.982468826 +0000 UTC m=+416.721822134" watchObservedRunningTime="2026-07-26 15:44:53.98463409 +0000 UTC m=+416.723987401" Jul 26 15:44:55.789631 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:55.789596 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:44:55.948222 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:55.948188 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:44:56.042193 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:56.042109 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:44:56.101451 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:56.101391 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:44:56.924366 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:56.924334 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-q7jfk" Jul 26 15:44:59.362528 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:59.362488 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-54449ffcf6-56rkx"] Jul 26 15:44:59.469522 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:59.469481 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:44:59.526345 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:59.526314 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:44:59.872676 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:59.872636 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:44:59.978595 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:44:59.978562 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:45:04.970172 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:04.970135 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vpmjr" Jul 26 15:45:06.664093 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:06.664057 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vpmjr"] Jul 26 15:45:06.664566 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:06.664234 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vpmjr" podUID="769d0090-52e6-4543-95e0-ec93d477fd75" containerName="manager" containerID="cri-o://b68e4131d6c40c8726df501dc8cce9aab1b2a3d632ae00bf21c97d58c4cb1f40" gracePeriod=2 Jul 26 15:45:06.675082 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:06.675055 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vpmjr"] Jul 26 15:45:06.696234 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:45:06.696198 2577 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"769d0090-52e6-4543-95e0-ec93d477fd75\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-8684c9ff49-vpmjr\": pods \"kuadrant-operator-controller-manager-8684c9ff49-vpmjr\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vpmjr" Jul 26 15:45:06.700593 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:06.700567 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2d66d"] Jul 26 15:45:06.701100 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:06.701086 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="769d0090-52e6-4543-95e0-ec93d477fd75" containerName="manager" Jul 26 15:45:06.701146 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:06.701104 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="769d0090-52e6-4543-95e0-ec93d477fd75" containerName="manager" Jul 26 15:45:06.701224 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:06.701214 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="769d0090-52e6-4543-95e0-ec93d477fd75" containerName="manager" Jul 26 15:45:06.704655 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:06.704633 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2d66d" Jul 26 15:45:06.706867 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:45:06.706833 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-vpmjr\" is forbidden: User \"system:node:ip-10-0-138-163.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-163.ec2.internal' and this object" podUID="769d0090-52e6-4543-95e0-ec93d477fd75" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vpmjr" Jul 26 15:45:06.720902 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:06.720869 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2d66d"] Jul 26 15:45:06.774751 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:06.774722 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b74a4de7-c2aa-4852-b366-225cc880e37e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-2d66d\" (UID: \"b74a4de7-c2aa-4852-b366-225cc880e37e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2d66d" Jul 26 15:45:06.774751 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:06.774753 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdvqq\" (UniqueName: \"kubernetes.io/projected/b74a4de7-c2aa-4852-b366-225cc880e37e-kube-api-access-qdvqq\") pod \"kuadrant-operator-controller-manager-8684c9ff49-2d66d\" (UID: \"b74a4de7-c2aa-4852-b366-225cc880e37e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2d66d" Jul 26 15:45:06.875976 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:06.875942 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b74a4de7-c2aa-4852-b366-225cc880e37e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-2d66d\" (UID: \"b74a4de7-c2aa-4852-b366-225cc880e37e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2d66d" Jul 26 15:45:06.876131 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:06.875982 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdvqq\" (UniqueName: \"kubernetes.io/projected/b74a4de7-c2aa-4852-b366-225cc880e37e-kube-api-access-qdvqq\") pod \"kuadrant-operator-controller-manager-8684c9ff49-2d66d\" (UID: \"b74a4de7-c2aa-4852-b366-225cc880e37e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2d66d" Jul 26 15:45:06.876456 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:06.876434 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b74a4de7-c2aa-4852-b366-225cc880e37e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-2d66d\" (UID: \"b74a4de7-c2aa-4852-b366-225cc880e37e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2d66d" Jul 26 15:45:06.888702 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:06.888675 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdvqq\" (UniqueName: \"kubernetes.io/projected/b74a4de7-c2aa-4852-b366-225cc880e37e-kube-api-access-qdvqq\") pod \"kuadrant-operator-controller-manager-8684c9ff49-2d66d\" (UID: \"b74a4de7-c2aa-4852-b366-225cc880e37e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2d66d" Jul 26 15:45:06.925924 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:06.925865 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vpmjr" Jul 26 15:45:06.928656 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:45:06.928632 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-vpmjr\" is forbidden: User \"system:node:ip-10-0-138-163.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-163.ec2.internal' and this object" podUID="769d0090-52e6-4543-95e0-ec93d477fd75" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vpmjr" Jul 26 15:45:06.976705 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:06.976676 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/769d0090-52e6-4543-95e0-ec93d477fd75-extensions-socket-volume\") pod \"769d0090-52e6-4543-95e0-ec93d477fd75\" (UID: \"769d0090-52e6-4543-95e0-ec93d477fd75\") " Jul 26 15:45:06.976848 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:06.976778 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nbg57\" (UniqueName: \"kubernetes.io/projected/769d0090-52e6-4543-95e0-ec93d477fd75-kube-api-access-nbg57\") pod \"769d0090-52e6-4543-95e0-ec93d477fd75\" (UID: \"769d0090-52e6-4543-95e0-ec93d477fd75\") " Jul 26 15:45:06.977117 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:06.977088 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/769d0090-52e6-4543-95e0-ec93d477fd75-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "769d0090-52e6-4543-95e0-ec93d477fd75" (UID: "769d0090-52e6-4543-95e0-ec93d477fd75"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:45:06.978842 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:06.978815 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/769d0090-52e6-4543-95e0-ec93d477fd75-kube-api-access-nbg57" (OuterVolumeSpecName: "kube-api-access-nbg57") pod "769d0090-52e6-4543-95e0-ec93d477fd75" (UID: "769d0090-52e6-4543-95e0-ec93d477fd75"). InnerVolumeSpecName "kube-api-access-nbg57". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 15:45:07.035922 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:07.035881 2577 generic.go:356] "Generic (PLEG): container finished" podID="769d0090-52e6-4543-95e0-ec93d477fd75" containerID="b68e4131d6c40c8726df501dc8cce9aab1b2a3d632ae00bf21c97d58c4cb1f40" exitCode=0 Jul 26 15:45:07.036100 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:07.035934 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vpmjr" Jul 26 15:45:07.036100 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:07.035965 2577 scope.go:117] "RemoveContainer" containerID="b68e4131d6c40c8726df501dc8cce9aab1b2a3d632ae00bf21c97d58c4cb1f40" Jul 26 15:45:07.038310 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:45:07.038278 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-vpmjr\" is forbidden: User \"system:node:ip-10-0-138-163.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-163.ec2.internal' and this object" podUID="769d0090-52e6-4543-95e0-ec93d477fd75" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vpmjr" Jul 26 15:45:07.046272 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:07.046256 2577 scope.go:117] "RemoveContainer" containerID="b68e4131d6c40c8726df501dc8cce9aab1b2a3d632ae00bf21c97d58c4cb1f40" Jul 26 15:45:07.046559 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:45:07.046540 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b68e4131d6c40c8726df501dc8cce9aab1b2a3d632ae00bf21c97d58c4cb1f40\": container with ID starting with b68e4131d6c40c8726df501dc8cce9aab1b2a3d632ae00bf21c97d58c4cb1f40 not found: ID does not exist" containerID="b68e4131d6c40c8726df501dc8cce9aab1b2a3d632ae00bf21c97d58c4cb1f40" Jul 26 15:45:07.046627 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:07.046567 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b68e4131d6c40c8726df501dc8cce9aab1b2a3d632ae00bf21c97d58c4cb1f40"} err="failed to get container status \"b68e4131d6c40c8726df501dc8cce9aab1b2a3d632ae00bf21c97d58c4cb1f40\": rpc error: code = NotFound desc = could not find container \"b68e4131d6c40c8726df501dc8cce9aab1b2a3d632ae00bf21c97d58c4cb1f40\": container with ID starting with b68e4131d6c40c8726df501dc8cce9aab1b2a3d632ae00bf21c97d58c4cb1f40 not found: ID does not exist" Jul 26 15:45:07.047633 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:07.047618 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2d66d" Jul 26 15:45:07.047727 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:45:07.047633 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-vpmjr\" is forbidden: User \"system:node:ip-10-0-138-163.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-163.ec2.internal' and this object" podUID="769d0090-52e6-4543-95e0-ec93d477fd75" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vpmjr" Jul 26 15:45:07.077998 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:07.077968 2577 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/769d0090-52e6-4543-95e0-ec93d477fd75-extensions-socket-volume\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 15:45:07.077998 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:07.077997 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nbg57\" (UniqueName: \"kubernetes.io/projected/769d0090-52e6-4543-95e0-ec93d477fd75-kube-api-access-nbg57\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 15:45:07.204317 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:07.204289 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2d66d"] Jul 26 15:45:07.206426 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:45:07.206386 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb74a4de7_c2aa_4852_b366_225cc880e37e.slice/crio-682a8b8d8dce3caed790c62289efeb7eb7162b93b8f7834707fdecd3ee9160f5 WatchSource:0}: Error finding container 682a8b8d8dce3caed790c62289efeb7eb7162b93b8f7834707fdecd3ee9160f5: Status 404 returned error can't find the container with id 682a8b8d8dce3caed790c62289efeb7eb7162b93b8f7834707fdecd3ee9160f5 Jul 26 15:45:07.786408 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:45:07.786365 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-vpmjr\" is forbidden: User \"system:node:ip-10-0-138-163.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-163.ec2.internal' and this object" podUID="769d0090-52e6-4543-95e0-ec93d477fd75" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vpmjr" Jul 26 15:45:07.787368 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:07.787345 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="769d0090-52e6-4543-95e0-ec93d477fd75" path="/var/lib/kubelet/pods/769d0090-52e6-4543-95e0-ec93d477fd75/volumes" Jul 26 15:45:08.043796 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:08.043707 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2d66d" event={"ID":"b74a4de7-c2aa-4852-b366-225cc880e37e","Type":"ContainerStarted","Data":"8cce1a7fbee2c8298e97c5f3ee5c7187b07d05092a9c647be0dceca3c30b8b42"} Jul 26 15:45:08.043796 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:08.043760 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2d66d" event={"ID":"b74a4de7-c2aa-4852-b366-225cc880e37e","Type":"ContainerStarted","Data":"682a8b8d8dce3caed790c62289efeb7eb7162b93b8f7834707fdecd3ee9160f5"} Jul 26 15:45:08.043983 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:08.043816 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2d66d" Jul 26 15:45:08.065168 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:08.065118 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2d66d" podStartSLOduration=2.065102292 podStartE2EDuration="2.065102292s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:45:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:45:08.063636523 +0000 UTC m=+430.802989827" watchObservedRunningTime="2026-07-26 15:45:08.065102292 +0000 UTC m=+430.804455602" Jul 26 15:45:08.918170 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:08.918112 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:45:09.016117 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:09.016083 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:45:10.377737 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:10.377696 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:45:10.525770 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:10.525733 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:45:14.303509 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:14.303476 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:45:14.405030 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:14.404992 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:45:17.303738 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:17.303678 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:45:17.404055 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:17.404016 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:45:19.051912 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:19.051878 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2d66d" Jul 26 15:45:21.267826 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:21.267728 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:45:21.366118 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:21.366078 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:45:24.388914 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:24.388861 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-54449ffcf6-56rkx" podUID="331ec89f-a6dd-459d-8582-be410b9176fe" containerName="console" containerID="cri-o://a489cf23ee712e1e617a52e2af3e9cd61484ae26811998dbf5ab2262ae6cf5f1" gracePeriod=15 Jul 26 15:45:24.665658 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:24.665591 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54449ffcf6-56rkx" Jul 26 15:45:24.719297 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:24.719263 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/331ec89f-a6dd-459d-8582-be410b9176fe-console-config\") pod \"331ec89f-a6dd-459d-8582-be410b9176fe\" (UID: \"331ec89f-a6dd-459d-8582-be410b9176fe\") " Jul 26 15:45:24.719297 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:24.719297 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/331ec89f-a6dd-459d-8582-be410b9176fe-service-ca\") pod \"331ec89f-a6dd-459d-8582-be410b9176fe\" (UID: \"331ec89f-a6dd-459d-8582-be410b9176fe\") " Jul 26 15:45:24.719569 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:24.719329 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nprcd\" (UniqueName: \"kubernetes.io/projected/331ec89f-a6dd-459d-8582-be410b9176fe-kube-api-access-nprcd\") pod \"331ec89f-a6dd-459d-8582-be410b9176fe\" (UID: \"331ec89f-a6dd-459d-8582-be410b9176fe\") " Jul 26 15:45:24.719569 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:24.719367 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/331ec89f-a6dd-459d-8582-be410b9176fe-console-oauth-config\") pod \"331ec89f-a6dd-459d-8582-be410b9176fe\" (UID: \"331ec89f-a6dd-459d-8582-be410b9176fe\") " Jul 26 15:45:24.719569 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:24.719427 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/331ec89f-a6dd-459d-8582-be410b9176fe-oauth-serving-cert\") pod \"331ec89f-a6dd-459d-8582-be410b9176fe\" (UID: \"331ec89f-a6dd-459d-8582-be410b9176fe\") " Jul 26 15:45:24.719569 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:24.719467 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/331ec89f-a6dd-459d-8582-be410b9176fe-console-serving-cert\") pod \"331ec89f-a6dd-459d-8582-be410b9176fe\" (UID: \"331ec89f-a6dd-459d-8582-be410b9176fe\") " Jul 26 15:45:24.719569 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:24.719498 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/331ec89f-a6dd-459d-8582-be410b9176fe-trusted-ca-bundle\") pod \"331ec89f-a6dd-459d-8582-be410b9176fe\" (UID: \"331ec89f-a6dd-459d-8582-be410b9176fe\") " Jul 26 15:45:24.719831 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:24.719712 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/331ec89f-a6dd-459d-8582-be410b9176fe-console-config" (OuterVolumeSpecName: "console-config") pod "331ec89f-a6dd-459d-8582-be410b9176fe" (UID: "331ec89f-a6dd-459d-8582-be410b9176fe"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 15:45:24.719955 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:24.719927 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/331ec89f-a6dd-459d-8582-be410b9176fe-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "331ec89f-a6dd-459d-8582-be410b9176fe" (UID: "331ec89f-a6dd-459d-8582-be410b9176fe"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 15:45:24.719997 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:24.719983 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/331ec89f-a6dd-459d-8582-be410b9176fe-service-ca" (OuterVolumeSpecName: "service-ca") pod "331ec89f-a6dd-459d-8582-be410b9176fe" (UID: "331ec89f-a6dd-459d-8582-be410b9176fe"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 15:45:24.720126 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:24.720098 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/331ec89f-a6dd-459d-8582-be410b9176fe-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "331ec89f-a6dd-459d-8582-be410b9176fe" (UID: "331ec89f-a6dd-459d-8582-be410b9176fe"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 15:45:24.721581 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:24.721554 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/331ec89f-a6dd-459d-8582-be410b9176fe-kube-api-access-nprcd" (OuterVolumeSpecName: "kube-api-access-nprcd") pod "331ec89f-a6dd-459d-8582-be410b9176fe" (UID: "331ec89f-a6dd-459d-8582-be410b9176fe"). InnerVolumeSpecName "kube-api-access-nprcd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 15:45:24.721675 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:24.721645 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/331ec89f-a6dd-459d-8582-be410b9176fe-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "331ec89f-a6dd-459d-8582-be410b9176fe" (UID: "331ec89f-a6dd-459d-8582-be410b9176fe"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 26 15:45:24.721675 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:24.721662 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/331ec89f-a6dd-459d-8582-be410b9176fe-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "331ec89f-a6dd-459d-8582-be410b9176fe" (UID: "331ec89f-a6dd-459d-8582-be410b9176fe"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 26 15:45:24.820642 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:24.820604 2577 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/331ec89f-a6dd-459d-8582-be410b9176fe-console-config\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 15:45:24.820642 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:24.820635 2577 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/331ec89f-a6dd-459d-8582-be410b9176fe-service-ca\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 15:45:24.820642 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:24.820644 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nprcd\" (UniqueName: \"kubernetes.io/projected/331ec89f-a6dd-459d-8582-be410b9176fe-kube-api-access-nprcd\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 15:45:24.820867 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:24.820654 2577 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/331ec89f-a6dd-459d-8582-be410b9176fe-console-oauth-config\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 15:45:24.820867 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:24.820665 2577 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/331ec89f-a6dd-459d-8582-be410b9176fe-oauth-serving-cert\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 15:45:24.820867 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:24.820673 2577 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/331ec89f-a6dd-459d-8582-be410b9176fe-console-serving-cert\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 15:45:24.820867 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:24.820680 2577 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/331ec89f-a6dd-459d-8582-be410b9176fe-trusted-ca-bundle\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 15:45:25.134101 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:25.134065 2577 generic.go:356] "Generic (PLEG): container finished" podID="331ec89f-a6dd-459d-8582-be410b9176fe" containerID="a489cf23ee712e1e617a52e2af3e9cd61484ae26811998dbf5ab2262ae6cf5f1" exitCode=0 Jul 26 15:45:25.134307 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:25.134109 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54449ffcf6-56rkx" Jul 26 15:45:25.134307 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:25.134156 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54449ffcf6-56rkx" event={"ID":"331ec89f-a6dd-459d-8582-be410b9176fe","Type":"ContainerDied","Data":"a489cf23ee712e1e617a52e2af3e9cd61484ae26811998dbf5ab2262ae6cf5f1"} Jul 26 15:45:25.134307 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:25.134210 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54449ffcf6-56rkx" event={"ID":"331ec89f-a6dd-459d-8582-be410b9176fe","Type":"ContainerDied","Data":"ffff0cb0d7a5de8aacc7624fe75184390e7b84b81c6fc3535bfc192b750775c8"} Jul 26 15:45:25.134307 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:25.134238 2577 scope.go:117] "RemoveContainer" containerID="a489cf23ee712e1e617a52e2af3e9cd61484ae26811998dbf5ab2262ae6cf5f1" Jul 26 15:45:25.143942 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:25.143926 2577 scope.go:117] "RemoveContainer" containerID="a489cf23ee712e1e617a52e2af3e9cd61484ae26811998dbf5ab2262ae6cf5f1" Jul 26 15:45:25.144191 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:45:25.144172 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a489cf23ee712e1e617a52e2af3e9cd61484ae26811998dbf5ab2262ae6cf5f1\": container with ID starting with a489cf23ee712e1e617a52e2af3e9cd61484ae26811998dbf5ab2262ae6cf5f1 not found: ID does not exist" containerID="a489cf23ee712e1e617a52e2af3e9cd61484ae26811998dbf5ab2262ae6cf5f1" Jul 26 15:45:25.144240 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:25.144199 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a489cf23ee712e1e617a52e2af3e9cd61484ae26811998dbf5ab2262ae6cf5f1"} err="failed to get container status \"a489cf23ee712e1e617a52e2af3e9cd61484ae26811998dbf5ab2262ae6cf5f1\": rpc error: code = NotFound desc = could not find container \"a489cf23ee712e1e617a52e2af3e9cd61484ae26811998dbf5ab2262ae6cf5f1\": container with ID starting with a489cf23ee712e1e617a52e2af3e9cd61484ae26811998dbf5ab2262ae6cf5f1 not found: ID does not exist" Jul 26 15:45:25.155319 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:25.155288 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-54449ffcf6-56rkx"] Jul 26 15:45:25.158895 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:25.158872 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-54449ffcf6-56rkx"] Jul 26 15:45:25.787291 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:25.787255 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="331ec89f-a6dd-459d-8582-be410b9176fe" path="/var/lib/kubelet/pods/331ec89f-a6dd-459d-8582-be410b9176fe/volumes" Jul 26 15:45:33.494383 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:33.494352 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2d66d"] Jul 26 15:45:33.494816 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:33.494613 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2d66d" podUID="b74a4de7-c2aa-4852-b366-225cc880e37e" containerName="manager" containerID="cri-o://8cce1a7fbee2c8298e97c5f3ee5c7187b07d05092a9c647be0dceca3c30b8b42" gracePeriod=10 Jul 26 15:45:33.685645 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:33.685612 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dccql"] Jul 26 15:45:33.686112 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:33.686084 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="331ec89f-a6dd-459d-8582-be410b9176fe" containerName="console" Jul 26 15:45:33.686112 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:33.686106 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="331ec89f-a6dd-459d-8582-be410b9176fe" containerName="console" Jul 26 15:45:33.686275 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:33.686189 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="331ec89f-a6dd-459d-8582-be410b9176fe" containerName="console" Jul 26 15:45:33.688241 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:33.688217 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dccql" Jul 26 15:45:33.703938 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:33.703911 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dccql"] Jul 26 15:45:33.749635 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:33.749571 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2d66d" Jul 26 15:45:33.797543 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:33.797511 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qdvqq\" (UniqueName: \"kubernetes.io/projected/b74a4de7-c2aa-4852-b366-225cc880e37e-kube-api-access-qdvqq\") pod \"b74a4de7-c2aa-4852-b366-225cc880e37e\" (UID: \"b74a4de7-c2aa-4852-b366-225cc880e37e\") " Jul 26 15:45:33.797730 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:33.797583 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b74a4de7-c2aa-4852-b366-225cc880e37e-extensions-socket-volume\") pod \"b74a4de7-c2aa-4852-b366-225cc880e37e\" (UID: \"b74a4de7-c2aa-4852-b366-225cc880e37e\") " Jul 26 15:45:33.797804 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:33.797731 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4f828c8f-f737-459b-81de-c844473c9ae7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-dccql\" (UID: \"4f828c8f-f737-459b-81de-c844473c9ae7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dccql" Jul 26 15:45:33.797925 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:33.797896 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tkg5\" (UniqueName: \"kubernetes.io/projected/4f828c8f-f737-459b-81de-c844473c9ae7-kube-api-access-2tkg5\") pod \"kuadrant-operator-controller-manager-86878c769f-dccql\" (UID: \"4f828c8f-f737-459b-81de-c844473c9ae7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dccql" Jul 26 15:45:33.797925 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:33.797918 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b74a4de7-c2aa-4852-b366-225cc880e37e-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "b74a4de7-c2aa-4852-b366-225cc880e37e" (UID: "b74a4de7-c2aa-4852-b366-225cc880e37e"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:45:33.799655 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:33.799635 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b74a4de7-c2aa-4852-b366-225cc880e37e-kube-api-access-qdvqq" (OuterVolumeSpecName: "kube-api-access-qdvqq") pod "b74a4de7-c2aa-4852-b366-225cc880e37e" (UID: "b74a4de7-c2aa-4852-b366-225cc880e37e"). InnerVolumeSpecName "kube-api-access-qdvqq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 15:45:33.899193 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:33.899156 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tkg5\" (UniqueName: \"kubernetes.io/projected/4f828c8f-f737-459b-81de-c844473c9ae7-kube-api-access-2tkg5\") pod \"kuadrant-operator-controller-manager-86878c769f-dccql\" (UID: \"4f828c8f-f737-459b-81de-c844473c9ae7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dccql" Jul 26 15:45:33.899359 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:33.899211 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4f828c8f-f737-459b-81de-c844473c9ae7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-dccql\" (UID: \"4f828c8f-f737-459b-81de-c844473c9ae7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dccql" Jul 26 15:45:33.899359 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:33.899240 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdvqq\" (UniqueName: \"kubernetes.io/projected/b74a4de7-c2aa-4852-b366-225cc880e37e-kube-api-access-qdvqq\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 15:45:33.899359 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:33.899250 2577 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b74a4de7-c2aa-4852-b366-225cc880e37e-extensions-socket-volume\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 15:45:33.899584 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:33.899568 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4f828c8f-f737-459b-81de-c844473c9ae7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-dccql\" (UID: \"4f828c8f-f737-459b-81de-c844473c9ae7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dccql" Jul 26 15:45:33.913661 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:33.913633 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tkg5\" (UniqueName: \"kubernetes.io/projected/4f828c8f-f737-459b-81de-c844473c9ae7-kube-api-access-2tkg5\") pod \"kuadrant-operator-controller-manager-86878c769f-dccql\" (UID: \"4f828c8f-f737-459b-81de-c844473c9ae7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dccql" Jul 26 15:45:34.000082 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:33.999985 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dccql" Jul 26 15:45:34.130960 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:34.130933 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dccql"] Jul 26 15:45:34.133159 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:45:34.133135 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f828c8f_f737_459b_81de_c844473c9ae7.slice/crio-54b1c398655621d4d632280ee6c7593250c1396308fb86f35c83fd86e1e45d92 WatchSource:0}: Error finding container 54b1c398655621d4d632280ee6c7593250c1396308fb86f35c83fd86e1e45d92: Status 404 returned error can't find the container with id 54b1c398655621d4d632280ee6c7593250c1396308fb86f35c83fd86e1e45d92 Jul 26 15:45:34.187222 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:34.187198 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dccql" event={"ID":"4f828c8f-f737-459b-81de-c844473c9ae7","Type":"ContainerStarted","Data":"54b1c398655621d4d632280ee6c7593250c1396308fb86f35c83fd86e1e45d92"} Jul 26 15:45:34.188566 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:34.188491 2577 generic.go:356] "Generic (PLEG): container finished" podID="b74a4de7-c2aa-4852-b366-225cc880e37e" containerID="8cce1a7fbee2c8298e97c5f3ee5c7187b07d05092a9c647be0dceca3c30b8b42" exitCode=0 Jul 26 15:45:34.188566 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:34.188531 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2d66d" Jul 26 15:45:34.188566 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:34.188543 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2d66d" event={"ID":"b74a4de7-c2aa-4852-b366-225cc880e37e","Type":"ContainerDied","Data":"8cce1a7fbee2c8298e97c5f3ee5c7187b07d05092a9c647be0dceca3c30b8b42"} Jul 26 15:45:34.188748 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:34.188587 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2d66d" event={"ID":"b74a4de7-c2aa-4852-b366-225cc880e37e","Type":"ContainerDied","Data":"682a8b8d8dce3caed790c62289efeb7eb7162b93b8f7834707fdecd3ee9160f5"} Jul 26 15:45:34.188748 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:34.188616 2577 scope.go:117] "RemoveContainer" containerID="8cce1a7fbee2c8298e97c5f3ee5c7187b07d05092a9c647be0dceca3c30b8b42" Jul 26 15:45:34.197921 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:34.197899 2577 scope.go:117] "RemoveContainer" containerID="8cce1a7fbee2c8298e97c5f3ee5c7187b07d05092a9c647be0dceca3c30b8b42" Jul 26 15:45:34.198211 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:45:34.198184 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8cce1a7fbee2c8298e97c5f3ee5c7187b07d05092a9c647be0dceca3c30b8b42\": container with ID starting with 8cce1a7fbee2c8298e97c5f3ee5c7187b07d05092a9c647be0dceca3c30b8b42 not found: ID does not exist" containerID="8cce1a7fbee2c8298e97c5f3ee5c7187b07d05092a9c647be0dceca3c30b8b42" Jul 26 15:45:34.198304 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:34.198216 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cce1a7fbee2c8298e97c5f3ee5c7187b07d05092a9c647be0dceca3c30b8b42"} err="failed to get container status \"8cce1a7fbee2c8298e97c5f3ee5c7187b07d05092a9c647be0dceca3c30b8b42\": rpc error: code = NotFound desc = could not find container \"8cce1a7fbee2c8298e97c5f3ee5c7187b07d05092a9c647be0dceca3c30b8b42\": container with ID starting with 8cce1a7fbee2c8298e97c5f3ee5c7187b07d05092a9c647be0dceca3c30b8b42 not found: ID does not exist" Jul 26 15:45:34.213720 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:34.213691 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2d66d"] Jul 26 15:45:34.219053 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:34.219028 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2d66d"] Jul 26 15:45:35.195271 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:35.195222 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dccql" event={"ID":"4f828c8f-f737-459b-81de-c844473c9ae7","Type":"ContainerStarted","Data":"9813e759a5a05cbe3500b50b10f5593ebaa8989a02998c23f6ceef24aa69ade8"} Jul 26 15:45:35.195701 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:35.195390 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dccql" Jul 26 15:45:35.217564 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:35.217503 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dccql" podStartSLOduration=2.21748693 podStartE2EDuration="2.21748693s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:45:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:45:35.214435117 +0000 UTC m=+457.953788427" watchObservedRunningTime="2026-07-26 15:45:35.21748693 +0000 UTC m=+457.956840239" Jul 26 15:45:35.787358 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:35.787328 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b74a4de7-c2aa-4852-b366-225cc880e37e" path="/var/lib/kubelet/pods/b74a4de7-c2aa-4852-b366-225cc880e37e/volumes" Jul 26 15:45:46.203837 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:46.203796 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dccql" Jul 26 15:45:49.806053 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:49.806022 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-596h4"] Jul 26 15:45:49.806478 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:49.806367 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b74a4de7-c2aa-4852-b366-225cc880e37e" containerName="manager" Jul 26 15:45:49.806478 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:49.806379 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b74a4de7-c2aa-4852-b366-225cc880e37e" containerName="manager" Jul 26 15:45:49.806478 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:49.806467 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="b74a4de7-c2aa-4852-b366-225cc880e37e" containerName="manager" Jul 26 15:45:49.810914 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:49.810892 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-596h4" Jul 26 15:45:49.813292 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:49.813270 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-bpv4m\"" Jul 26 15:45:49.820135 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:49.820108 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-596h4"] Jul 26 15:45:49.832707 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:49.832677 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:45:49.931509 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:49.931474 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34f385df-2d11-4b82-ad20-404ee0104f33-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-596h4\" (UID: \"34f385df-2d11-4b82-ad20-404ee0104f33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-596h4" Jul 26 15:45:49.931675 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:49.931541 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34f385df-2d11-4b82-ad20-404ee0104f33-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-596h4\" (UID: \"34f385df-2d11-4b82-ad20-404ee0104f33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-596h4" Jul 26 15:45:49.931675 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:49.931573 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hns48\" (UniqueName: \"kubernetes.io/projected/34f385df-2d11-4b82-ad20-404ee0104f33-kube-api-access-hns48\") pod \"maas-default-gateway-openshift-default-78bb78ff78-596h4\" (UID: \"34f385df-2d11-4b82-ad20-404ee0104f33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-596h4" Jul 26 15:45:49.931675 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:49.931594 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34f385df-2d11-4b82-ad20-404ee0104f33-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-596h4\" (UID: \"34f385df-2d11-4b82-ad20-404ee0104f33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-596h4" Jul 26 15:45:49.931675 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:49.931619 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34f385df-2d11-4b82-ad20-404ee0104f33-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-596h4\" (UID: \"34f385df-2d11-4b82-ad20-404ee0104f33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-596h4" Jul 26 15:45:49.931818 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:49.931708 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34f385df-2d11-4b82-ad20-404ee0104f33-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-596h4\" (UID: \"34f385df-2d11-4b82-ad20-404ee0104f33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-596h4" Jul 26 15:45:49.931818 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:49.931741 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34f385df-2d11-4b82-ad20-404ee0104f33-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-596h4\" (UID: \"34f385df-2d11-4b82-ad20-404ee0104f33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-596h4" Jul 26 15:45:49.931818 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:49.931788 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34f385df-2d11-4b82-ad20-404ee0104f33-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-596h4\" (UID: \"34f385df-2d11-4b82-ad20-404ee0104f33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-596h4" Jul 26 15:45:49.931818 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:49.931808 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34f385df-2d11-4b82-ad20-404ee0104f33-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-596h4\" (UID: \"34f385df-2d11-4b82-ad20-404ee0104f33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-596h4" Jul 26 15:45:50.032737 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:50.032706 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34f385df-2d11-4b82-ad20-404ee0104f33-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-596h4\" (UID: \"34f385df-2d11-4b82-ad20-404ee0104f33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-596h4" Jul 26 15:45:50.032905 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:50.032744 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34f385df-2d11-4b82-ad20-404ee0104f33-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-596h4\" (UID: \"34f385df-2d11-4b82-ad20-404ee0104f33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-596h4" Jul 26 15:45:50.032905 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:50.032766 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hns48\" (UniqueName: \"kubernetes.io/projected/34f385df-2d11-4b82-ad20-404ee0104f33-kube-api-access-hns48\") pod \"maas-default-gateway-openshift-default-78bb78ff78-596h4\" (UID: \"34f385df-2d11-4b82-ad20-404ee0104f33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-596h4" Jul 26 15:45:50.032905 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:50.032878 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34f385df-2d11-4b82-ad20-404ee0104f33-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-596h4\" (UID: \"34f385df-2d11-4b82-ad20-404ee0104f33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-596h4" Jul 26 15:45:50.033028 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:50.032917 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34f385df-2d11-4b82-ad20-404ee0104f33-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-596h4\" (UID: \"34f385df-2d11-4b82-ad20-404ee0104f33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-596h4" Jul 26 15:45:50.033095 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:50.033077 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34f385df-2d11-4b82-ad20-404ee0104f33-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-596h4\" (UID: \"34f385df-2d11-4b82-ad20-404ee0104f33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-596h4" Jul 26 15:45:50.033142 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:50.033128 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34f385df-2d11-4b82-ad20-404ee0104f33-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-596h4\" (UID: \"34f385df-2d11-4b82-ad20-404ee0104f33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-596h4" Jul 26 15:45:50.033200 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:50.033187 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34f385df-2d11-4b82-ad20-404ee0104f33-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-596h4\" (UID: \"34f385df-2d11-4b82-ad20-404ee0104f33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-596h4" Jul 26 15:45:50.033254 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:50.033216 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34f385df-2d11-4b82-ad20-404ee0104f33-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-596h4\" (UID: \"34f385df-2d11-4b82-ad20-404ee0104f33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-596h4" Jul 26 15:45:50.033482 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:50.033457 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34f385df-2d11-4b82-ad20-404ee0104f33-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-596h4\" (UID: \"34f385df-2d11-4b82-ad20-404ee0104f33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-596h4" Jul 26 15:45:50.033602 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:50.033504 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34f385df-2d11-4b82-ad20-404ee0104f33-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-596h4\" (UID: \"34f385df-2d11-4b82-ad20-404ee0104f33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-596h4" Jul 26 15:45:50.033602 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:50.033526 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34f385df-2d11-4b82-ad20-404ee0104f33-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-596h4\" (UID: \"34f385df-2d11-4b82-ad20-404ee0104f33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-596h4" Jul 26 15:45:50.033602 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:50.033559 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34f385df-2d11-4b82-ad20-404ee0104f33-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-596h4\" (UID: \"34f385df-2d11-4b82-ad20-404ee0104f33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-596h4" Jul 26 15:45:50.033602 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:50.033593 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34f385df-2d11-4b82-ad20-404ee0104f33-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-596h4\" (UID: \"34f385df-2d11-4b82-ad20-404ee0104f33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-596h4" Jul 26 15:45:50.035251 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:50.035222 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34f385df-2d11-4b82-ad20-404ee0104f33-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-596h4\" (UID: \"34f385df-2d11-4b82-ad20-404ee0104f33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-596h4" Jul 26 15:45:50.035553 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:50.035537 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34f385df-2d11-4b82-ad20-404ee0104f33-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-596h4\" (UID: \"34f385df-2d11-4b82-ad20-404ee0104f33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-596h4" Jul 26 15:45:50.040654 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:50.040625 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hns48\" (UniqueName: \"kubernetes.io/projected/34f385df-2d11-4b82-ad20-404ee0104f33-kube-api-access-hns48\") pod \"maas-default-gateway-openshift-default-78bb78ff78-596h4\" (UID: \"34f385df-2d11-4b82-ad20-404ee0104f33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-596h4" Jul 26 15:45:50.040750 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:50.040682 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34f385df-2d11-4b82-ad20-404ee0104f33-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-596h4\" (UID: \"34f385df-2d11-4b82-ad20-404ee0104f33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-596h4" Jul 26 15:45:50.052196 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:50.052172 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:45:50.122376 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:50.122293 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-596h4" Jul 26 15:45:50.261873 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:50.261847 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-596h4"] Jul 26 15:45:50.263579 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:45:50.263553 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34f385df_2d11_4b82_ad20_404ee0104f33.slice/crio-5bddc5625182b2c209f0f8b07683e7bfc13683b4c2e2b2f111ca323fb51dfb57 WatchSource:0}: Error finding container 5bddc5625182b2c209f0f8b07683e7bfc13683b4c2e2b2f111ca323fb51dfb57: Status 404 returned error can't find the container with id 5bddc5625182b2c209f0f8b07683e7bfc13683b4c2e2b2f111ca323fb51dfb57 Jul 26 15:45:50.265672 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:50.265645 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 15:45:50.265740 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:50.265704 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 15:45:50.265740 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:50.265730 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 15:45:50.278549 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:50.278520 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-596h4" event={"ID":"34f385df-2d11-4b82-ad20-404ee0104f33","Type":"ContainerStarted","Data":"5bddc5625182b2c209f0f8b07683e7bfc13683b4c2e2b2f111ca323fb51dfb57"} Jul 26 15:45:51.286183 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:51.286145 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-596h4" event={"ID":"34f385df-2d11-4b82-ad20-404ee0104f33","Type":"ContainerStarted","Data":"8a898a8f8451fc4452a97dd48a3be1955961d5d3462d5f6bde04e65474cccf73"} Jul 26 15:45:51.306301 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:51.306251 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-596h4" podStartSLOduration=2.306236438 podStartE2EDuration="2.306236438s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:45:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:45:51.304359973 +0000 UTC m=+474.043713306" watchObservedRunningTime="2026-07-26 15:45:51.306236438 +0000 UTC m=+474.045589824" Jul 26 15:45:52.123433 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:52.123375 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-596h4" Jul 26 15:45:52.127849 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:52.127824 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-596h4" Jul 26 15:45:52.291452 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:52.291414 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-596h4" Jul 26 15:45:52.292508 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:52.292490 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-596h4" Jul 26 15:45:52.869453 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:52.869416 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:45:52.991368 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:52.991332 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:45:53.047527 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:53.047489 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:45:53.693927 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:53.693895 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-f9pwz"] Jul 26 15:45:53.702233 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:53.702203 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-f9pwz" Jul 26 15:45:53.705109 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:53.704914 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-496wt\"" Jul 26 15:45:53.707834 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:53.707811 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-f9pwz"] Jul 26 15:45:53.730508 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:53.730470 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:45:53.809966 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:53.809930 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:45:53.864038 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:53.864006 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vq6nc\" (UniqueName: \"kubernetes.io/projected/da8d1b23-2ca3-4863-b94f-56ca3263977f-kube-api-access-vq6nc\") pod \"authorino-58cd955675-f9pwz\" (UID: \"da8d1b23-2ca3-4863-b94f-56ca3263977f\") " pod="kuadrant-system/authorino-58cd955675-f9pwz" Jul 26 15:45:53.964657 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:53.964559 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vq6nc\" (UniqueName: \"kubernetes.io/projected/da8d1b23-2ca3-4863-b94f-56ca3263977f-kube-api-access-vq6nc\") pod \"authorino-58cd955675-f9pwz\" (UID: \"da8d1b23-2ca3-4863-b94f-56ca3263977f\") " pod="kuadrant-system/authorino-58cd955675-f9pwz" Jul 26 15:45:53.972240 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:53.972208 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vq6nc\" (UniqueName: \"kubernetes.io/projected/da8d1b23-2ca3-4863-b94f-56ca3263977f-kube-api-access-vq6nc\") pod \"authorino-58cd955675-f9pwz\" (UID: \"da8d1b23-2ca3-4863-b94f-56ca3263977f\") " pod="kuadrant-system/authorino-58cd955675-f9pwz" Jul 26 15:45:54.013978 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:54.013953 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-f9pwz" Jul 26 15:45:54.143757 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:54.143725 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-f9pwz"] Jul 26 15:45:54.145824 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:45:54.145784 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podda8d1b23_2ca3_4863_b94f_56ca3263977f.slice/crio-8b6dbef19216e18c82cc494fca3866703a5e816972f4088a0195510d84c1a1a5 WatchSource:0}: Error finding container 8b6dbef19216e18c82cc494fca3866703a5e816972f4088a0195510d84c1a1a5: Status 404 returned error can't find the container with id 8b6dbef19216e18c82cc494fca3866703a5e816972f4088a0195510d84c1a1a5 Jul 26 15:45:54.303236 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:54.303202 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-f9pwz" event={"ID":"da8d1b23-2ca3-4863-b94f-56ca3263977f","Type":"ContainerStarted","Data":"8b6dbef19216e18c82cc494fca3866703a5e816972f4088a0195510d84c1a1a5"} Jul 26 15:45:57.324719 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:57.324630 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-f9pwz" event={"ID":"da8d1b23-2ca3-4863-b94f-56ca3263977f","Type":"ContainerStarted","Data":"5183e6c3cf3f4c77845a0708656ef49fb0b88e5f449d3e83d1df74321a3a4979"} Jul 26 15:45:57.338629 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:57.338581 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-f9pwz" podStartSLOduration=1.448737864 podStartE2EDuration="4.338564234s" totalImagesPullingTime="2.88982637s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:45:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:45:57.337899716 +0000 UTC m=+480.077253022" watchObservedRunningTime="2026-07-26 15:45:57.338564234 +0000 UTC m=+480.077917549" Jul 26 15:45:58.654704 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:58.654670 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-f9pwz"] Jul 26 15:45:59.335421 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:59.335362 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-f9pwz" podUID="da8d1b23-2ca3-4863-b94f-56ca3263977f" containerName="authorino" containerID="cri-o://5183e6c3cf3f4c77845a0708656ef49fb0b88e5f449d3e83d1df74321a3a4979" gracePeriod=30 Jul 26 15:45:59.583930 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:59.583906 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-f9pwz" Jul 26 15:45:59.716470 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:59.716355 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vq6nc\" (UniqueName: \"kubernetes.io/projected/da8d1b23-2ca3-4863-b94f-56ca3263977f-kube-api-access-vq6nc\") pod \"da8d1b23-2ca3-4863-b94f-56ca3263977f\" (UID: \"da8d1b23-2ca3-4863-b94f-56ca3263977f\") " Jul 26 15:45:59.718432 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:59.718390 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da8d1b23-2ca3-4863-b94f-56ca3263977f-kube-api-access-vq6nc" (OuterVolumeSpecName: "kube-api-access-vq6nc") pod "da8d1b23-2ca3-4863-b94f-56ca3263977f" (UID: "da8d1b23-2ca3-4863-b94f-56ca3263977f"). InnerVolumeSpecName "kube-api-access-vq6nc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 15:45:59.817361 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:45:59.817330 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vq6nc\" (UniqueName: \"kubernetes.io/projected/da8d1b23-2ca3-4863-b94f-56ca3263977f-kube-api-access-vq6nc\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 15:46:00.341373 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:46:00.341337 2577 generic.go:356] "Generic (PLEG): container finished" podID="da8d1b23-2ca3-4863-b94f-56ca3263977f" containerID="5183e6c3cf3f4c77845a0708656ef49fb0b88e5f449d3e83d1df74321a3a4979" exitCode=0 Jul 26 15:46:00.341587 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:46:00.341381 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-f9pwz" Jul 26 15:46:00.341587 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:46:00.341422 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-f9pwz" event={"ID":"da8d1b23-2ca3-4863-b94f-56ca3263977f","Type":"ContainerDied","Data":"5183e6c3cf3f4c77845a0708656ef49fb0b88e5f449d3e83d1df74321a3a4979"} Jul 26 15:46:00.341587 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:46:00.341466 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-f9pwz" event={"ID":"da8d1b23-2ca3-4863-b94f-56ca3263977f","Type":"ContainerDied","Data":"8b6dbef19216e18c82cc494fca3866703a5e816972f4088a0195510d84c1a1a5"} Jul 26 15:46:00.341587 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:46:00.341483 2577 scope.go:117] "RemoveContainer" containerID="5183e6c3cf3f4c77845a0708656ef49fb0b88e5f449d3e83d1df74321a3a4979" Jul 26 15:46:00.350104 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:46:00.350081 2577 scope.go:117] "RemoveContainer" containerID="5183e6c3cf3f4c77845a0708656ef49fb0b88e5f449d3e83d1df74321a3a4979" Jul 26 15:46:00.350368 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:46:00.350348 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5183e6c3cf3f4c77845a0708656ef49fb0b88e5f449d3e83d1df74321a3a4979\": container with ID starting with 5183e6c3cf3f4c77845a0708656ef49fb0b88e5f449d3e83d1df74321a3a4979 not found: ID does not exist" containerID="5183e6c3cf3f4c77845a0708656ef49fb0b88e5f449d3e83d1df74321a3a4979" Jul 26 15:46:00.350435 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:46:00.350377 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5183e6c3cf3f4c77845a0708656ef49fb0b88e5f449d3e83d1df74321a3a4979"} err="failed to get container status \"5183e6c3cf3f4c77845a0708656ef49fb0b88e5f449d3e83d1df74321a3a4979\": rpc error: code = NotFound desc = could not find container \"5183e6c3cf3f4c77845a0708656ef49fb0b88e5f449d3e83d1df74321a3a4979\": container with ID starting with 5183e6c3cf3f4c77845a0708656ef49fb0b88e5f449d3e83d1df74321a3a4979 not found: ID does not exist" Jul 26 15:46:00.357622 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:46:00.357601 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-f9pwz"] Jul 26 15:46:00.359625 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:46:00.359606 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-f9pwz"] Jul 26 15:46:01.786689 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:46:01.786653 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da8d1b23-2ca3-4863-b94f-56ca3263977f" path="/var/lib/kubelet/pods/da8d1b23-2ca3-4863-b94f-56ca3263977f/volumes" Jul 26 15:46:13.428199 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:46:13.428164 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:46:13.527062 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:46:13.527026 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:46:13.895272 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:46:13.895237 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:46:26.347413 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:46:26.347369 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:46:26.567175 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:46:26.567145 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-v5t2q"] Jul 26 15:46:26.567518 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:46:26.567505 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da8d1b23-2ca3-4863-b94f-56ca3263977f" containerName="authorino" Jul 26 15:46:26.567584 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:46:26.567520 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="da8d1b23-2ca3-4863-b94f-56ca3263977f" containerName="authorino" Jul 26 15:46:26.567584 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:46:26.567583 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="da8d1b23-2ca3-4863-b94f-56ca3263977f" containerName="authorino" Jul 26 15:46:26.570277 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:46:26.570259 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-v5t2q" Jul 26 15:46:26.572867 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:46:26.572844 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-496wt\"" Jul 26 15:46:26.577010 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:46:26.576981 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-v5t2q"] Jul 26 15:46:26.596235 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:46:26.596199 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:46:26.638223 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:46:26.638130 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l52h9\" (UniqueName: \"kubernetes.io/projected/154b7f07-8b07-466c-b6c3-38978025a351-kube-api-access-l52h9\") pod \"authorino-5d4b67c446-v5t2q\" (UID: \"154b7f07-8b07-466c-b6c3-38978025a351\") " pod="kuadrant-system/authorino-5d4b67c446-v5t2q" Jul 26 15:46:26.739582 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:46:26.739549 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l52h9\" (UniqueName: \"kubernetes.io/projected/154b7f07-8b07-466c-b6c3-38978025a351-kube-api-access-l52h9\") pod \"authorino-5d4b67c446-v5t2q\" (UID: \"154b7f07-8b07-466c-b6c3-38978025a351\") " pod="kuadrant-system/authorino-5d4b67c446-v5t2q" Jul 26 15:46:26.747949 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:46:26.747916 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l52h9\" (UniqueName: \"kubernetes.io/projected/154b7f07-8b07-466c-b6c3-38978025a351-kube-api-access-l52h9\") pod \"authorino-5d4b67c446-v5t2q\" (UID: \"154b7f07-8b07-466c-b6c3-38978025a351\") " pod="kuadrant-system/authorino-5d4b67c446-v5t2q" Jul 26 15:46:26.798899 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:46:26.798867 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-v5t2q"] Jul 26 15:46:26.799120 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:46:26.799109 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-v5t2q" Jul 26 15:46:26.825854 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:46:26.825826 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-78687c996-8sk4w"] Jul 26 15:46:26.828738 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:46:26.828716 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-78687c996-8sk4w" Jul 26 15:46:26.836082 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:46:26.836053 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-78687c996-8sk4w"] Jul 26 15:46:26.939160 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:46:26.939136 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-v5t2q"] Jul 26 15:46:26.941031 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:46:26.940999 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod154b7f07_8b07_466c_b6c3_38978025a351.slice/crio-f1b7a6e943e984a16e2756959145523cc6bc6c985597d9f43b20ca47094a296e WatchSource:0}: Error finding container f1b7a6e943e984a16e2756959145523cc6bc6c985597d9f43b20ca47094a296e: Status 404 returned error can't find the container with id f1b7a6e943e984a16e2756959145523cc6bc6c985597d9f43b20ca47094a296e Jul 26 15:46:26.941153 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:46:26.941056 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nw45g\" (UniqueName: \"kubernetes.io/projected/5ff70bce-6aa3-4c1f-88de-cd24ad5703e8-kube-api-access-nw45g\") pod \"authorino-78687c996-8sk4w\" (UID: \"5ff70bce-6aa3-4c1f-88de-cd24ad5703e8\") " pod="kuadrant-system/authorino-78687c996-8sk4w" Jul 26 15:46:27.042298 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:46:27.042260 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nw45g\" (UniqueName: \"kubernetes.io/projected/5ff70bce-6aa3-4c1f-88de-cd24ad5703e8-kube-api-access-nw45g\") pod \"authorino-78687c996-8sk4w\" (UID: \"5ff70bce-6aa3-4c1f-88de-cd24ad5703e8\") " pod="kuadrant-system/authorino-78687c996-8sk4w" Jul 26 15:46:27.043644 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:46:27.043624 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-78687c996-8sk4w"] Jul 26 15:46:27.043884 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:46:27.043864 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-nw45g], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-78687c996-8sk4w" podUID="5ff70bce-6aa3-4c1f-88de-cd24ad5703e8" Jul 26 15:46:27.057861 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:46:27.057829 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nw45g\" (UniqueName: \"kubernetes.io/projected/5ff70bce-6aa3-4c1f-88de-cd24ad5703e8-kube-api-access-nw45g\") pod \"authorino-78687c996-8sk4w\" (UID: \"5ff70bce-6aa3-4c1f-88de-cd24ad5703e8\") " pod="kuadrant-system/authorino-78687c996-8sk4w" Jul 26 15:46:27.070555 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:46:27.070523 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-776bd6465b-ks6fg"] Jul 26 15:46:27.073148 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:46:27.073128 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-776bd6465b-ks6fg" Jul 26 15:46:27.076115 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:46:27.076084 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 26 15:46:27.083365 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:46:27.083334 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-776bd6465b-ks6fg"] Jul 26 15:46:27.143007 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:46:27.142927 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/690378ba-7cb5-4774-bf6f-2c8109f38767-tls-cert\") pod \"authorino-776bd6465b-ks6fg\" (UID: \"690378ba-7cb5-4774-bf6f-2c8109f38767\") " pod="kuadrant-system/authorino-776bd6465b-ks6fg" Jul 26 15:46:27.143007 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:46:27.142971 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gr9hd\" (UniqueName: \"kubernetes.io/projected/690378ba-7cb5-4774-bf6f-2c8109f38767-kube-api-access-gr9hd\") pod \"authorino-776bd6465b-ks6fg\" (UID: \"690378ba-7cb5-4774-bf6f-2c8109f38767\") " pod="kuadrant-system/authorino-776bd6465b-ks6fg" Jul 26 15:46:27.243629 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:46:27.243600 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/690378ba-7cb5-4774-bf6f-2c8109f38767-tls-cert\") pod \"authorino-776bd6465b-ks6fg\" (UID: \"690378ba-7cb5-4774-bf6f-2c8109f38767\") " pod="kuadrant-system/authorino-776bd6465b-ks6fg" Jul 26 15:46:27.243751 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:46:27.243638 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gr9hd\" (UniqueName: \"kubernetes.io/projected/690378ba-7cb5-4774-bf6f-2c8109f38767-kube-api-access-gr9hd\") pod \"authorino-776bd6465b-ks6fg\" (UID: \"690378ba-7cb5-4774-bf6f-2c8109f38767\") " pod="kuadrant-system/authorino-776bd6465b-ks6fg" Jul 26 15:46:27.246049 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:46:27.246025 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/690378ba-7cb5-4774-bf6f-2c8109f38767-tls-cert\") pod \"authorino-776bd6465b-ks6fg\" (UID: \"690378ba-7cb5-4774-bf6f-2c8109f38767\") " pod="kuadrant-system/authorino-776bd6465b-ks6fg" Jul 26 15:46:27.251833 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:46:27.251813 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gr9hd\" (UniqueName: \"kubernetes.io/projected/690378ba-7cb5-4774-bf6f-2c8109f38767-kube-api-access-gr9hd\") pod \"authorino-776bd6465b-ks6fg\" (UID: \"690378ba-7cb5-4774-bf6f-2c8109f38767\") " pod="kuadrant-system/authorino-776bd6465b-ks6fg" Jul 26 15:46:27.390354 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:46:27.390321 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-776bd6465b-ks6fg" Jul 26 15:46:27.490000 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:46:27.489967 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-v5t2q" event={"ID":"154b7f07-8b07-466c-b6c3-38978025a351","Type":"ContainerStarted","Data":"5d01a04d11032f3c6aa129535a7d850a467dd4c1d483042ad5bfceb496c4312d"} Jul 26 15:46:27.490178 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:46:27.490015 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-v5t2q" event={"ID":"154b7f07-8b07-466c-b6c3-38978025a351","Type":"ContainerStarted","Data":"f1b7a6e943e984a16e2756959145523cc6bc6c985597d9f43b20ca47094a296e"} Jul 26 15:46:27.490178 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:46:27.489987 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5d4b67c446-v5t2q" podUID="154b7f07-8b07-466c-b6c3-38978025a351" containerName="authorino" containerID="cri-o://5d01a04d11032f3c6aa129535a7d850a467dd4c1d483042ad5bfceb496c4312d" gracePeriod=30 Jul 26 15:46:27.490178 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:46:27.490094 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-78687c996-8sk4w" Jul 26 15:46:27.501636 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:46:27.501609 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-78687c996-8sk4w" Jul 26 15:46:27.504384 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:46:27.504334 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5d4b67c446-v5t2q" podStartSLOduration=1.202437956 podStartE2EDuration="1.504315547s" totalImagesPullingTime="301.877591ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:46:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:46:27.504012423 +0000 UTC m=+510.243365760" watchObservedRunningTime="2026-07-26 15:46:27.504315547 +0000 UTC m=+510.243668859" Jul 26 15:46:27.521841 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:46:27.521820 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-776bd6465b-ks6fg"] Jul 26 15:46:27.544215 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:46:27.544188 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod690378ba_7cb5_4774_bf6f_2c8109f38767.slice/crio-e18d7bb579e30574cb108d0c29c26fe40c2425e6e0ebdb0e97dcb59625d43098 WatchSource:0}: Error finding container e18d7bb579e30574cb108d0c29c26fe40c2425e6e0ebdb0e97dcb59625d43098: Status 404 returned error can't find the container with id e18d7bb579e30574cb108d0c29c26fe40c2425e6e0ebdb0e97dcb59625d43098 Jul 26 15:46:27.547572 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:46:27.547551 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nw45g\" (UniqueName: \"kubernetes.io/projected/5ff70bce-6aa3-4c1f-88de-cd24ad5703e8-kube-api-access-nw45g\") pod \"5ff70bce-6aa3-4c1f-88de-cd24ad5703e8\" (UID: \"5ff70bce-6aa3-4c1f-88de-cd24ad5703e8\") " Jul 26 15:46:27.549536 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:46:27.549506 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ff70bce-6aa3-4c1f-88de-cd24ad5703e8-kube-api-access-nw45g" (OuterVolumeSpecName: "kube-api-access-nw45g") pod "5ff70bce-6aa3-4c1f-88de-cd24ad5703e8" (UID: "5ff70bce-6aa3-4c1f-88de-cd24ad5703e8"). InnerVolumeSpecName "kube-api-access-nw45g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 15:46:27.648312 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:46:27.648281 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nw45g\" (UniqueName: \"kubernetes.io/projected/5ff70bce-6aa3-4c1f-88de-cd24ad5703e8-kube-api-access-nw45g\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 15:46:27.730871 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:46:27.730845 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-v5t2q" Jul 26 15:46:27.749211 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:46:27.749164 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l52h9\" (UniqueName: \"kubernetes.io/projected/154b7f07-8b07-466c-b6c3-38978025a351-kube-api-access-l52h9\") pod \"154b7f07-8b07-466c-b6c3-38978025a351\" (UID: \"154b7f07-8b07-466c-b6c3-38978025a351\") " Jul 26 15:46:27.752460 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:46:27.752382 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/154b7f07-8b07-466c-b6c3-38978025a351-kube-api-access-l52h9" (OuterVolumeSpecName: "kube-api-access-l52h9") pod "154b7f07-8b07-466c-b6c3-38978025a351" (UID: "154b7f07-8b07-466c-b6c3-38978025a351"). InnerVolumeSpecName "kube-api-access-l52h9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 15:46:27.850851 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:46:27.850815 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l52h9\" (UniqueName: \"kubernetes.io/projected/154b7f07-8b07-466c-b6c3-38978025a351-kube-api-access-l52h9\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 15:46:28.497020 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:46:28.496981 2577 generic.go:356] "Generic (PLEG): container finished" podID="154b7f07-8b07-466c-b6c3-38978025a351" containerID="5d01a04d11032f3c6aa129535a7d850a467dd4c1d483042ad5bfceb496c4312d" exitCode=0 Jul 26 15:46:28.497020 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:46:28.497020 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-v5t2q" Jul 26 15:46:28.497562 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:46:28.497069 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-v5t2q" event={"ID":"154b7f07-8b07-466c-b6c3-38978025a351","Type":"ContainerDied","Data":"5d01a04d11032f3c6aa129535a7d850a467dd4c1d483042ad5bfceb496c4312d"} Jul 26 15:46:28.497562 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:46:28.497111 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-v5t2q" event={"ID":"154b7f07-8b07-466c-b6c3-38978025a351","Type":"ContainerDied","Data":"f1b7a6e943e984a16e2756959145523cc6bc6c985597d9f43b20ca47094a296e"} Jul 26 15:46:28.497562 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:46:28.497134 2577 scope.go:117] "RemoveContainer" containerID="5d01a04d11032f3c6aa129535a7d850a467dd4c1d483042ad5bfceb496c4312d" Jul 26 15:46:28.498951 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:46:28.498915 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-78687c996-8sk4w" Jul 26 15:46:28.498951 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:46:28.498932 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-776bd6465b-ks6fg" event={"ID":"690378ba-7cb5-4774-bf6f-2c8109f38767","Type":"ContainerStarted","Data":"3ce1e23145166415ee10b8d1f30eaf8f1e2892fb1d5b2a96660ec093efb2e5cc"} Jul 26 15:46:28.499153 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:46:28.498971 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-776bd6465b-ks6fg" event={"ID":"690378ba-7cb5-4774-bf6f-2c8109f38767","Type":"ContainerStarted","Data":"e18d7bb579e30574cb108d0c29c26fe40c2425e6e0ebdb0e97dcb59625d43098"} Jul 26 15:46:28.506110 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:46:28.506094 2577 scope.go:117] "RemoveContainer" containerID="5d01a04d11032f3c6aa129535a7d850a467dd4c1d483042ad5bfceb496c4312d" Jul 26 15:46:28.506365 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:46:28.506344 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d01a04d11032f3c6aa129535a7d850a467dd4c1d483042ad5bfceb496c4312d\": container with ID starting with 5d01a04d11032f3c6aa129535a7d850a467dd4c1d483042ad5bfceb496c4312d not found: ID does not exist" containerID="5d01a04d11032f3c6aa129535a7d850a467dd4c1d483042ad5bfceb496c4312d" Jul 26 15:46:28.506460 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:46:28.506374 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d01a04d11032f3c6aa129535a7d850a467dd4c1d483042ad5bfceb496c4312d"} err="failed to get container status \"5d01a04d11032f3c6aa129535a7d850a467dd4c1d483042ad5bfceb496c4312d\": rpc error: code = NotFound desc = could not find container \"5d01a04d11032f3c6aa129535a7d850a467dd4c1d483042ad5bfceb496c4312d\": container with ID starting with 5d01a04d11032f3c6aa129535a7d850a467dd4c1d483042ad5bfceb496c4312d not found: ID does not exist" Jul 26 15:46:28.515665 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:46:28.515621 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-776bd6465b-ks6fg" podStartSLOduration=1.182396131 podStartE2EDuration="1.515607567s" totalImagesPullingTime="333.211436ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:46:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:46:28.514632191 +0000 UTC m=+511.253985504" watchObservedRunningTime="2026-07-26 15:46:28.515607567 +0000 UTC m=+511.254960876" Jul 26 15:46:28.550519 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:46:28.550488 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-78687c996-8sk4w"] Jul 26 15:46:28.559754 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:46:28.559726 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-78687c996-8sk4w"] Jul 26 15:46:28.575702 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:46:28.575668 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-v5t2q"] Jul 26 15:46:28.579674 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:46:28.579649 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-v5t2q"] Jul 26 15:46:29.786838 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:46:29.786805 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="154b7f07-8b07-466c-b6c3-38978025a351" path="/var/lib/kubelet/pods/154b7f07-8b07-466c-b6c3-38978025a351/volumes" Jul 26 15:46:29.787192 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:46:29.787141 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5ff70bce-6aa3-4c1f-88de-cd24ad5703e8" path="/var/lib/kubelet/pods/5ff70bce-6aa3-4c1f-88de-cd24ad5703e8/volumes" Jul 26 15:46:33.079333 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:46:33.079293 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:46:33.151621 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:46:33.151559 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:46:33.206144 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:46:33.206109 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:46:33.561859 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:46:33.561823 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:46:36.889089 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:46:36.889053 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:46:36.986496 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:46:36.986462 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:46:46.022063 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:46:46.022022 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:46:52.881377 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:46:52.881281 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:46:52.935741 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:46:52.935705 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:47:09.588929 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:47:09.588891 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:47:09.681764 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:47:09.681714 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:47:10.099623 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:47:10.099590 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:47:10.253105 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:47:10.253070 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:47:10.875212 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:47:10.875178 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:47:10.973193 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:47:10.973153 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:47:18.769315 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:47:18.769277 2577 generic.go:356] "Generic (PLEG): container finished" podID="34f385df-2d11-4b82-ad20-404ee0104f33" containerID="8a898a8f8451fc4452a97dd48a3be1955961d5d3462d5f6bde04e65474cccf73" exitCode=0 Jul 26 15:47:18.769748 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:47:18.769323 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-596h4" event={"ID":"34f385df-2d11-4b82-ad20-404ee0104f33","Type":"ContainerDied","Data":"8a898a8f8451fc4452a97dd48a3be1955961d5d3462d5f6bde04e65474cccf73"} Jul 26 15:47:18.769748 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:47:18.769736 2577 scope.go:117] "RemoveContainer" containerID="8a898a8f8451fc4452a97dd48a3be1955961d5d3462d5f6bde04e65474cccf73" Jul 26 15:47:18.770762 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:47:18.770725 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 15:47:18.770858 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:47:18.770808 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 15:47:18.770898 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:47:18.770861 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 15:47:19.776592 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:47:19.776555 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-596h4" event={"ID":"34f385df-2d11-4b82-ad20-404ee0104f33","Type":"ContainerStarted","Data":"a4102448922788fddc12e711285cbefc8e82419fe868c41fb3345b673d2d9812"} Jul 26 15:47:20.122688 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:47:20.122594 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-596h4" Jul 26 15:47:20.124006 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:47:20.123976 2577 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-596h4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Jul 26 15:47:20.124131 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:47:20.124029 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-596h4" podUID="34f385df-2d11-4b82-ad20-404ee0104f33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Jul 26 15:47:20.206806 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:47:20.206743 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:47:20.305018 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:47:20.304971 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:47:21.123750 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:47:21.123715 2577 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-596h4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Jul 26 15:47:21.124145 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:47:21.123783 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-596h4" podUID="34f385df-2d11-4b82-ad20-404ee0104f33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Jul 26 15:47:22.018953 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:47:22.018916 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:47:22.122918 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:47:22.122879 2577 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-596h4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Jul 26 15:47:22.123088 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:47:22.122938 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-596h4" podUID="34f385df-2d11-4b82-ad20-404ee0104f33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Jul 26 15:47:22.291926 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:47:22.291892 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-596h4" Jul 26 15:47:23.123016 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:47:23.122974 2577 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-596h4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Jul 26 15:47:23.123202 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:47:23.123033 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-596h4" podUID="34f385df-2d11-4b82-ad20-404ee0104f33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Jul 26 15:47:24.127356 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:47:24.127323 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-596h4" Jul 26 15:47:24.128277 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:47:24.128258 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-596h4" Jul 26 15:47:28.271827 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:47:28.271793 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-knsvs"] Jul 26 15:47:28.272238 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:47:28.272154 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="154b7f07-8b07-466c-b6c3-38978025a351" containerName="authorino" Jul 26 15:47:28.272238 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:47:28.272166 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="154b7f07-8b07-466c-b6c3-38978025a351" containerName="authorino" Jul 26 15:47:28.272238 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:47:28.272237 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="154b7f07-8b07-466c-b6c3-38978025a351" containerName="authorino" Jul 26 15:47:28.275579 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:47:28.275563 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-knsvs" Jul 26 15:47:28.279554 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:47:28.279529 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 26 15:47:28.279554 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:47:28.279529 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 26 15:47:28.279754 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:47:28.279581 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-6w9zr\"" Jul 26 15:47:28.279754 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:47:28.279587 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Jul 26 15:47:28.286163 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:47:28.286139 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-knsvs"] Jul 26 15:47:28.307146 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:47:28.307107 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:47:28.369899 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:47:28.369863 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92d710d9-8053-4983-aa56-06536dc79041-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-knsvs\" (UID: \"92d710d9-8053-4983-aa56-06536dc79041\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-knsvs" Jul 26 15:47:28.370090 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:47:28.369902 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6z696\" (UniqueName: \"kubernetes.io/projected/92d710d9-8053-4983-aa56-06536dc79041-kube-api-access-6z696\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-knsvs\" (UID: \"92d710d9-8053-4983-aa56-06536dc79041\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-knsvs" Jul 26 15:47:28.370090 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:47:28.369972 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92d710d9-8053-4983-aa56-06536dc79041-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-knsvs\" (UID: \"92d710d9-8053-4983-aa56-06536dc79041\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-knsvs" Jul 26 15:47:28.370090 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:47:28.370014 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92d710d9-8053-4983-aa56-06536dc79041-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-knsvs\" (UID: \"92d710d9-8053-4983-aa56-06536dc79041\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-knsvs" Jul 26 15:47:28.370090 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:47:28.370040 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92d710d9-8053-4983-aa56-06536dc79041-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-knsvs\" (UID: \"92d710d9-8053-4983-aa56-06536dc79041\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-knsvs" Jul 26 15:47:28.370090 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:47:28.370072 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92d710d9-8053-4983-aa56-06536dc79041-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-knsvs\" (UID: \"92d710d9-8053-4983-aa56-06536dc79041\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-knsvs" Jul 26 15:47:28.470881 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:47:28.470851 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92d710d9-8053-4983-aa56-06536dc79041-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-knsvs\" (UID: \"92d710d9-8053-4983-aa56-06536dc79041\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-knsvs" Jul 26 15:47:28.471163 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:47:28.470898 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92d710d9-8053-4983-aa56-06536dc79041-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-knsvs\" (UID: \"92d710d9-8053-4983-aa56-06536dc79041\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-knsvs" Jul 26 15:47:28.471163 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:47:28.470922 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92d710d9-8053-4983-aa56-06536dc79041-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-knsvs\" (UID: \"92d710d9-8053-4983-aa56-06536dc79041\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-knsvs" Jul 26 15:47:28.471163 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:47:28.471045 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92d710d9-8053-4983-aa56-06536dc79041-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-knsvs\" (UID: \"92d710d9-8053-4983-aa56-06536dc79041\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-knsvs" Jul 26 15:47:28.471163 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:47:28.471112 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92d710d9-8053-4983-aa56-06536dc79041-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-knsvs\" (UID: \"92d710d9-8053-4983-aa56-06536dc79041\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-knsvs" Jul 26 15:47:28.471163 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:47:28.471147 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6z696\" (UniqueName: \"kubernetes.io/projected/92d710d9-8053-4983-aa56-06536dc79041-kube-api-access-6z696\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-knsvs\" (UID: \"92d710d9-8053-4983-aa56-06536dc79041\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-knsvs" Jul 26 15:47:28.471494 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:47:28.471325 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92d710d9-8053-4983-aa56-06536dc79041-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-knsvs\" (UID: \"92d710d9-8053-4983-aa56-06536dc79041\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-knsvs" Jul 26 15:47:28.471494 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:47:28.471418 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92d710d9-8053-4983-aa56-06536dc79041-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-knsvs\" (UID: \"92d710d9-8053-4983-aa56-06536dc79041\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-knsvs" Jul 26 15:47:28.471572 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:47:28.471517 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92d710d9-8053-4983-aa56-06536dc79041-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-knsvs\" (UID: \"92d710d9-8053-4983-aa56-06536dc79041\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-knsvs" Jul 26 15:47:28.473182 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:47:28.473160 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92d710d9-8053-4983-aa56-06536dc79041-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-knsvs\" (UID: \"92d710d9-8053-4983-aa56-06536dc79041\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-knsvs" Jul 26 15:47:28.473530 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:47:28.473513 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92d710d9-8053-4983-aa56-06536dc79041-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-knsvs\" (UID: \"92d710d9-8053-4983-aa56-06536dc79041\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-knsvs" Jul 26 15:47:28.478781 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:47:28.478760 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6z696\" (UniqueName: \"kubernetes.io/projected/92d710d9-8053-4983-aa56-06536dc79041-kube-api-access-6z696\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-knsvs\" (UID: \"92d710d9-8053-4983-aa56-06536dc79041\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-knsvs" Jul 26 15:47:28.588371 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:47:28.588333 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-knsvs" Jul 26 15:47:28.746655 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:47:28.746624 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-knsvs"] Jul 26 15:47:28.749421 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:47:28.749353 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92d710d9_8053_4983_aa56_06536dc79041.slice/crio-5a0ad25bc46d190d564c481a77732e1e92092c3fca4007f9476105f076a2a480 WatchSource:0}: Error finding container 5a0ad25bc46d190d564c481a77732e1e92092c3fca4007f9476105f076a2a480: Status 404 returned error can't find the container with id 5a0ad25bc46d190d564c481a77732e1e92092c3fca4007f9476105f076a2a480 Jul 26 15:47:28.835598 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:47:28.835562 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-knsvs" event={"ID":"92d710d9-8053-4983-aa56-06536dc79041","Type":"ContainerStarted","Data":"5a0ad25bc46d190d564c481a77732e1e92092c3fca4007f9476105f076a2a480"} Jul 26 15:47:30.848036 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:47:30.847998 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-knsvs" event={"ID":"92d710d9-8053-4983-aa56-06536dc79041","Type":"ContainerStarted","Data":"70ddead0144f9129388c4fd9a6fe656399b71e89ebc8ad3bad9224792fc66be9"} Jul 26 15:47:30.867255 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:47:30.867204 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-knsvs" podStartSLOduration=1.78779227 podStartE2EDuration="2.867187947s" totalImagesPullingTime="1.079395677s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:47:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:47:30.864111477 +0000 UTC m=+573.603464815" watchObservedRunningTime="2026-07-26 15:47:30.867187947 +0000 UTC m=+573.606541278" Jul 26 15:47:35.586830 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:47:35.586785 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:47:38.588942 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:47:38.588901 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-knsvs" Jul 26 15:47:38.589344 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:47:38.588970 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-knsvs" Jul 26 15:47:38.601354 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:47:38.601323 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-knsvs" Jul 26 15:47:38.901561 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:47:38.901486 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-knsvs" Jul 26 15:47:42.830534 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:47:42.830496 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:47:52.167780 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:47:52.167741 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-bjb5v"] Jul 26 15:47:52.171206 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:47:52.171186 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bjb5v" Jul 26 15:47:52.173468 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:47:52.173446 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Jul 26 15:47:52.185977 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:47:52.183725 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-bjb5v"] Jul 26 15:47:52.196191 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:47:52.196159 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:47:52.278185 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:47:52.278141 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2694dc63-b18b-42d3-9cab-8fef40fdbf33-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-bjb5v\" (UID: \"2694dc63-b18b-42d3-9cab-8fef40fdbf33\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bjb5v" Jul 26 15:47:52.278185 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:47:52.278195 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2694dc63-b18b-42d3-9cab-8fef40fdbf33-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-bjb5v\" (UID: \"2694dc63-b18b-42d3-9cab-8fef40fdbf33\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bjb5v" Jul 26 15:47:52.278453 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:47:52.278279 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2694dc63-b18b-42d3-9cab-8fef40fdbf33-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-bjb5v\" (UID: \"2694dc63-b18b-42d3-9cab-8fef40fdbf33\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bjb5v" Jul 26 15:47:52.278453 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:47:52.278318 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5g7vl\" (UniqueName: \"kubernetes.io/projected/2694dc63-b18b-42d3-9cab-8fef40fdbf33-kube-api-access-5g7vl\") pod \"e2e-trlp-test-simulated-kserve-598466548d-bjb5v\" (UID: \"2694dc63-b18b-42d3-9cab-8fef40fdbf33\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bjb5v" Jul 26 15:47:52.278453 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:47:52.278340 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2694dc63-b18b-42d3-9cab-8fef40fdbf33-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-bjb5v\" (UID: \"2694dc63-b18b-42d3-9cab-8fef40fdbf33\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bjb5v" Jul 26 15:47:52.278565 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:47:52.278453 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2694dc63-b18b-42d3-9cab-8fef40fdbf33-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-bjb5v\" (UID: \"2694dc63-b18b-42d3-9cab-8fef40fdbf33\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bjb5v" Jul 26 15:47:52.379630 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:47:52.379581 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5g7vl\" (UniqueName: \"kubernetes.io/projected/2694dc63-b18b-42d3-9cab-8fef40fdbf33-kube-api-access-5g7vl\") pod \"e2e-trlp-test-simulated-kserve-598466548d-bjb5v\" (UID: \"2694dc63-b18b-42d3-9cab-8fef40fdbf33\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bjb5v" Jul 26 15:47:52.379846 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:47:52.379641 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2694dc63-b18b-42d3-9cab-8fef40fdbf33-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-bjb5v\" (UID: \"2694dc63-b18b-42d3-9cab-8fef40fdbf33\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bjb5v" Jul 26 15:47:52.379846 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:47:52.379688 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2694dc63-b18b-42d3-9cab-8fef40fdbf33-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-bjb5v\" (UID: \"2694dc63-b18b-42d3-9cab-8fef40fdbf33\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bjb5v" Jul 26 15:47:52.379846 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:47:52.379749 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2694dc63-b18b-42d3-9cab-8fef40fdbf33-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-bjb5v\" (UID: \"2694dc63-b18b-42d3-9cab-8fef40fdbf33\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bjb5v" Jul 26 15:47:52.380019 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:47:52.379872 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2694dc63-b18b-42d3-9cab-8fef40fdbf33-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-bjb5v\" (UID: \"2694dc63-b18b-42d3-9cab-8fef40fdbf33\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bjb5v" Jul 26 15:47:52.380019 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:47:52.379946 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2694dc63-b18b-42d3-9cab-8fef40fdbf33-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-bjb5v\" (UID: \"2694dc63-b18b-42d3-9cab-8fef40fdbf33\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bjb5v" Jul 26 15:47:52.380200 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:47:52.380178 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2694dc63-b18b-42d3-9cab-8fef40fdbf33-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-bjb5v\" (UID: \"2694dc63-b18b-42d3-9cab-8fef40fdbf33\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bjb5v" Jul 26 15:47:52.380261 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:47:52.380187 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2694dc63-b18b-42d3-9cab-8fef40fdbf33-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-bjb5v\" (UID: \"2694dc63-b18b-42d3-9cab-8fef40fdbf33\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bjb5v" Jul 26 15:47:52.380300 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:47:52.380280 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2694dc63-b18b-42d3-9cab-8fef40fdbf33-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-bjb5v\" (UID: \"2694dc63-b18b-42d3-9cab-8fef40fdbf33\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bjb5v" Jul 26 15:47:52.382109 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:47:52.382087 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2694dc63-b18b-42d3-9cab-8fef40fdbf33-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-bjb5v\" (UID: \"2694dc63-b18b-42d3-9cab-8fef40fdbf33\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bjb5v" Jul 26 15:47:52.382330 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:47:52.382317 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2694dc63-b18b-42d3-9cab-8fef40fdbf33-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-bjb5v\" (UID: \"2694dc63-b18b-42d3-9cab-8fef40fdbf33\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bjb5v" Jul 26 15:47:52.387433 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:47:52.387412 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5g7vl\" (UniqueName: \"kubernetes.io/projected/2694dc63-b18b-42d3-9cab-8fef40fdbf33-kube-api-access-5g7vl\") pod \"e2e-trlp-test-simulated-kserve-598466548d-bjb5v\" (UID: \"2694dc63-b18b-42d3-9cab-8fef40fdbf33\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bjb5v" Jul 26 15:47:52.489231 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:47:52.489134 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bjb5v" Jul 26 15:47:52.628856 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:47:52.628822 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-bjb5v"] Jul 26 15:47:52.629964 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:47:52.629939 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2694dc63_b18b_42d3_9cab_8fef40fdbf33.slice/crio-a9a17a47c3be149169b5ab242dd6ec6afe6e34507218fa0b5453ea9d243c3a26 WatchSource:0}: Error finding container a9a17a47c3be149169b5ab242dd6ec6afe6e34507218fa0b5453ea9d243c3a26: Status 404 returned error can't find the container with id a9a17a47c3be149169b5ab242dd6ec6afe6e34507218fa0b5453ea9d243c3a26 Jul 26 15:47:52.970318 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:47:52.970277 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bjb5v" event={"ID":"2694dc63-b18b-42d3-9cab-8fef40fdbf33","Type":"ContainerStarted","Data":"58c28481ef80abbae0cbf4acb880bfa35d501802135879c667c0c5ba7be968ad"} Jul 26 15:47:52.970533 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:47:52.970334 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bjb5v" event={"ID":"2694dc63-b18b-42d3-9cab-8fef40fdbf33","Type":"ContainerStarted","Data":"a9a17a47c3be149169b5ab242dd6ec6afe6e34507218fa0b5453ea9d243c3a26"} Jul 26 15:47:52.990388 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:47:52.990332 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bjb5v" podStartSLOduration=0.815418968 podStartE2EDuration="990.314282ms" totalImagesPullingTime="174.895314ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:47:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:47:52.987296687 +0000 UTC m=+595.726650034" watchObservedRunningTime="2026-07-26 15:47:52.990314282 +0000 UTC m=+595.729667592" Jul 26 15:47:57.730938 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:47:57.730907 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-8wjqp_b6b8080d-0cee-4d0c-a05c-3e2f1b672a54/console-operator/1.log" Jul 26 15:47:57.735559 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:47:57.735534 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ztlp6_cbe2cd41-6ebc-4722-a813-57d83fa1d242/ovn-acl-logging/0.log" Jul 26 15:47:57.739247 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:47:57.739228 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-8wjqp_b6b8080d-0cee-4d0c-a05c-3e2f1b672a54/console-operator/1.log" Jul 26 15:47:57.743380 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:47:57.743360 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ztlp6_cbe2cd41-6ebc-4722-a813-57d83fa1d242/ovn-acl-logging/0.log" Jul 26 15:47:58.587076 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:47:58.587048 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:48:02.490118 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:48:02.490073 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bjb5v" Jul 26 15:48:02.490546 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:48:02.490141 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bjb5v" Jul 26 15:48:02.502654 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:48:02.502630 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bjb5v" Jul 26 15:48:03.041985 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:48:03.041951 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bjb5v" Jul 26 15:48:06.094558 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:48:06.094514 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:48:13.498667 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:48:13.498624 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:48:19.481046 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:48:19.481011 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:48:19.581608 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:48:19.581574 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:48:48.091336 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:48:48.091296 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:48:48.141125 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:48:48.141091 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:49:01.161731 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:49:01.161693 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-997bd858f-b6n8s"] Jul 26 15:49:01.165358 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:49:01.165334 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-997bd858f-b6n8s" Jul 26 15:49:01.170459 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:49:01.170429 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-997bd858f-b6n8s"] Jul 26 15:49:01.293260 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:49:01.293227 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9b08503b-3207-43e6-8fa2-1fc702a76fc5-tls-cert\") pod \"authorino-997bd858f-b6n8s\" (UID: \"9b08503b-3207-43e6-8fa2-1fc702a76fc5\") " pod="kuadrant-system/authorino-997bd858f-b6n8s" Jul 26 15:49:01.293464 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:49:01.293280 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pj7q9\" (UniqueName: \"kubernetes.io/projected/9b08503b-3207-43e6-8fa2-1fc702a76fc5-kube-api-access-pj7q9\") pod \"authorino-997bd858f-b6n8s\" (UID: \"9b08503b-3207-43e6-8fa2-1fc702a76fc5\") " pod="kuadrant-system/authorino-997bd858f-b6n8s" Jul 26 15:49:01.394351 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:49:01.394312 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pj7q9\" (UniqueName: \"kubernetes.io/projected/9b08503b-3207-43e6-8fa2-1fc702a76fc5-kube-api-access-pj7q9\") pod \"authorino-997bd858f-b6n8s\" (UID: \"9b08503b-3207-43e6-8fa2-1fc702a76fc5\") " pod="kuadrant-system/authorino-997bd858f-b6n8s" Jul 26 15:49:01.394544 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:49:01.394420 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9b08503b-3207-43e6-8fa2-1fc702a76fc5-tls-cert\") pod \"authorino-997bd858f-b6n8s\" (UID: \"9b08503b-3207-43e6-8fa2-1fc702a76fc5\") " pod="kuadrant-system/authorino-997bd858f-b6n8s" Jul 26 15:49:01.396761 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:49:01.396738 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9b08503b-3207-43e6-8fa2-1fc702a76fc5-tls-cert\") pod \"authorino-997bd858f-b6n8s\" (UID: \"9b08503b-3207-43e6-8fa2-1fc702a76fc5\") " pod="kuadrant-system/authorino-997bd858f-b6n8s" Jul 26 15:49:01.401645 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:49:01.401620 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pj7q9\" (UniqueName: \"kubernetes.io/projected/9b08503b-3207-43e6-8fa2-1fc702a76fc5-kube-api-access-pj7q9\") pod \"authorino-997bd858f-b6n8s\" (UID: \"9b08503b-3207-43e6-8fa2-1fc702a76fc5\") " pod="kuadrant-system/authorino-997bd858f-b6n8s" Jul 26 15:49:01.476002 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:49:01.475899 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-997bd858f-b6n8s" Jul 26 15:49:01.568380 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:49:01.568342 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:49:01.619067 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:49:01.619043 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-997bd858f-b6n8s"] Jul 26 15:49:01.621012 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:49:01.620985 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9b08503b_3207_43e6_8fa2_1fc702a76fc5.slice/crio-251e24862913ce90ea7817ed5f0b20a97a88f68664ebf745e1c74b3aca3a981d WatchSource:0}: Error finding container 251e24862913ce90ea7817ed5f0b20a97a88f68664ebf745e1c74b3aca3a981d: Status 404 returned error can't find the container with id 251e24862913ce90ea7817ed5f0b20a97a88f68664ebf745e1c74b3aca3a981d Jul 26 15:49:01.622273 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:49:01.622254 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 26 15:49:01.671275 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:49:01.671241 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:49:02.362363 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:49:02.362322 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-997bd858f-b6n8s" event={"ID":"9b08503b-3207-43e6-8fa2-1fc702a76fc5","Type":"ContainerStarted","Data":"dad2aa55bbdade8f10bb19c5e270f96e268efed3fe73b52f988d1b020af59dfe"} Jul 26 15:49:02.362766 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:49:02.362369 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-997bd858f-b6n8s" event={"ID":"9b08503b-3207-43e6-8fa2-1fc702a76fc5","Type":"ContainerStarted","Data":"251e24862913ce90ea7817ed5f0b20a97a88f68664ebf745e1c74b3aca3a981d"} Jul 26 15:49:02.381581 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:49:02.381527 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-997bd858f-b6n8s" podStartSLOduration=1.033164374 podStartE2EDuration="1.381509574s" totalImagesPullingTime="348.3452ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:49:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:49:02.380817161 +0000 UTC m=+665.120170483" watchObservedRunningTime="2026-07-26 15:49:02.381509574 +0000 UTC m=+665.120862884" Jul 26 15:49:02.411577 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:49:02.411535 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-776bd6465b-ks6fg"] Jul 26 15:49:02.412245 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:49:02.411977 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-776bd6465b-ks6fg" podUID="690378ba-7cb5-4774-bf6f-2c8109f38767" containerName="authorino" containerID="cri-o://3ce1e23145166415ee10b8d1f30eaf8f1e2892fb1d5b2a96660ec093efb2e5cc" gracePeriod=30 Jul 26 15:49:02.662321 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:49:02.662234 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:49:02.704544 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:49:02.704511 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-776bd6465b-ks6fg" Jul 26 15:49:02.753275 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:49:02.753227 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:49:02.809202 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:49:02.809170 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gr9hd\" (UniqueName: \"kubernetes.io/projected/690378ba-7cb5-4774-bf6f-2c8109f38767-kube-api-access-gr9hd\") pod \"690378ba-7cb5-4774-bf6f-2c8109f38767\" (UID: \"690378ba-7cb5-4774-bf6f-2c8109f38767\") " Jul 26 15:49:02.809375 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:49:02.809269 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/690378ba-7cb5-4774-bf6f-2c8109f38767-tls-cert\") pod \"690378ba-7cb5-4774-bf6f-2c8109f38767\" (UID: \"690378ba-7cb5-4774-bf6f-2c8109f38767\") " Jul 26 15:49:02.811382 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:49:02.811350 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/690378ba-7cb5-4774-bf6f-2c8109f38767-kube-api-access-gr9hd" (OuterVolumeSpecName: "kube-api-access-gr9hd") pod "690378ba-7cb5-4774-bf6f-2c8109f38767" (UID: "690378ba-7cb5-4774-bf6f-2c8109f38767"). InnerVolumeSpecName "kube-api-access-gr9hd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 15:49:02.821097 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:49:02.821055 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/690378ba-7cb5-4774-bf6f-2c8109f38767-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "690378ba-7cb5-4774-bf6f-2c8109f38767" (UID: "690378ba-7cb5-4774-bf6f-2c8109f38767"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 26 15:49:02.910262 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:49:02.910223 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gr9hd\" (UniqueName: \"kubernetes.io/projected/690378ba-7cb5-4774-bf6f-2c8109f38767-kube-api-access-gr9hd\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 15:49:02.910262 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:49:02.910257 2577 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/690378ba-7cb5-4774-bf6f-2c8109f38767-tls-cert\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 15:49:03.370755 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:49:03.370719 2577 generic.go:356] "Generic (PLEG): container finished" podID="690378ba-7cb5-4774-bf6f-2c8109f38767" containerID="3ce1e23145166415ee10b8d1f30eaf8f1e2892fb1d5b2a96660ec093efb2e5cc" exitCode=0 Jul 26 15:49:03.370755 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:49:03.370760 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-776bd6465b-ks6fg" Jul 26 15:49:03.371224 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:49:03.370798 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-776bd6465b-ks6fg" event={"ID":"690378ba-7cb5-4774-bf6f-2c8109f38767","Type":"ContainerDied","Data":"3ce1e23145166415ee10b8d1f30eaf8f1e2892fb1d5b2a96660ec093efb2e5cc"} Jul 26 15:49:03.371224 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:49:03.370844 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-776bd6465b-ks6fg" event={"ID":"690378ba-7cb5-4774-bf6f-2c8109f38767","Type":"ContainerDied","Data":"e18d7bb579e30574cb108d0c29c26fe40c2425e6e0ebdb0e97dcb59625d43098"} Jul 26 15:49:03.371224 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:49:03.370864 2577 scope.go:117] "RemoveContainer" containerID="3ce1e23145166415ee10b8d1f30eaf8f1e2892fb1d5b2a96660ec093efb2e5cc" Jul 26 15:49:03.380917 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:49:03.380899 2577 scope.go:117] "RemoveContainer" containerID="3ce1e23145166415ee10b8d1f30eaf8f1e2892fb1d5b2a96660ec093efb2e5cc" Jul 26 15:49:03.381200 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:49:03.381179 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ce1e23145166415ee10b8d1f30eaf8f1e2892fb1d5b2a96660ec093efb2e5cc\": container with ID starting with 3ce1e23145166415ee10b8d1f30eaf8f1e2892fb1d5b2a96660ec093efb2e5cc not found: ID does not exist" containerID="3ce1e23145166415ee10b8d1f30eaf8f1e2892fb1d5b2a96660ec093efb2e5cc" Jul 26 15:49:03.381302 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:49:03.381210 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ce1e23145166415ee10b8d1f30eaf8f1e2892fb1d5b2a96660ec093efb2e5cc"} err="failed to get container status \"3ce1e23145166415ee10b8d1f30eaf8f1e2892fb1d5b2a96660ec093efb2e5cc\": rpc error: code = NotFound desc = could not find container \"3ce1e23145166415ee10b8d1f30eaf8f1e2892fb1d5b2a96660ec093efb2e5cc\": container with ID starting with 3ce1e23145166415ee10b8d1f30eaf8f1e2892fb1d5b2a96660ec093efb2e5cc not found: ID does not exist" Jul 26 15:49:03.393573 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:49:03.393540 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-776bd6465b-ks6fg"] Jul 26 15:49:03.398286 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:49:03.398253 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-776bd6465b-ks6fg"] Jul 26 15:49:03.787179 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:49:03.787141 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="690378ba-7cb5-4774-bf6f-2c8109f38767" path="/var/lib/kubelet/pods/690378ba-7cb5-4774-bf6f-2c8109f38767/volumes" Jul 26 15:49:30.580223 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:49:30.580183 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:51:06.809471 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:51:06.809427 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:51:07.897228 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:51:07.897191 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:51:43.178589 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:51:43.178502 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:51:43.264820 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:51:43.264786 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:51:43.387551 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:51:43.387497 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:51:49.159350 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:51:49.159310 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:51:49.218122 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:51:49.218095 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:51:49.304072 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:51:49.304024 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:51:49.402525 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:51:49.402486 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:51:49.505451 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:51:49.505349 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:51:55.151372 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:51:55.151325 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:51:55.223275 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:51:55.223231 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:52:57.769321 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:52:57.769234 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-8wjqp_b6b8080d-0cee-4d0c-a05c-3e2f1b672a54/console-operator/1.log" Jul 26 15:52:57.773726 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:52:57.773695 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ztlp6_cbe2cd41-6ebc-4722-a813-57d83fa1d242/ovn-acl-logging/0.log" Jul 26 15:52:57.774684 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:52:57.774661 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-8wjqp_b6b8080d-0cee-4d0c-a05c-3e2f1b672a54/console-operator/1.log" Jul 26 15:52:57.778570 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:52:57.778550 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ztlp6_cbe2cd41-6ebc-4722-a813-57d83fa1d242/ovn-acl-logging/0.log" Jul 26 15:54:58.935788 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:54:58.935755 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:55:49.580903 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:55:49.580863 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dccql"] Jul 26 15:55:49.581480 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:55:49.581072 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dccql" podUID="4f828c8f-f737-459b-81de-c844473c9ae7" containerName="manager" containerID="cri-o://9813e759a5a05cbe3500b50b10f5593ebaa8989a02998c23f6ceef24aa69ade8" gracePeriod=10 Jul 26 15:55:53.337484 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:55:53.337457 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dccql" Jul 26 15:55:53.482289 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:55:53.482201 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2tkg5\" (UniqueName: \"kubernetes.io/projected/4f828c8f-f737-459b-81de-c844473c9ae7-kube-api-access-2tkg5\") pod \"4f828c8f-f737-459b-81de-c844473c9ae7\" (UID: \"4f828c8f-f737-459b-81de-c844473c9ae7\") " Jul 26 15:55:53.482289 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:55:53.482262 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4f828c8f-f737-459b-81de-c844473c9ae7-extensions-socket-volume\") pod \"4f828c8f-f737-459b-81de-c844473c9ae7\" (UID: \"4f828c8f-f737-459b-81de-c844473c9ae7\") " Jul 26 15:55:53.482677 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:55:53.482656 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f828c8f-f737-459b-81de-c844473c9ae7-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "4f828c8f-f737-459b-81de-c844473c9ae7" (UID: "4f828c8f-f737-459b-81de-c844473c9ae7"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:55:53.484295 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:55:53.484269 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f828c8f-f737-459b-81de-c844473c9ae7-kube-api-access-2tkg5" (OuterVolumeSpecName: "kube-api-access-2tkg5") pod "4f828c8f-f737-459b-81de-c844473c9ae7" (UID: "4f828c8f-f737-459b-81de-c844473c9ae7"). InnerVolumeSpecName "kube-api-access-2tkg5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 15:55:53.583642 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:55:53.583612 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2tkg5\" (UniqueName: \"kubernetes.io/projected/4f828c8f-f737-459b-81de-c844473c9ae7-kube-api-access-2tkg5\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 15:55:53.583642 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:55:53.583640 2577 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4f828c8f-f737-459b-81de-c844473c9ae7-extensions-socket-volume\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 15:55:53.650273 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:55:53.650233 2577 generic.go:356] "Generic (PLEG): container finished" podID="4f828c8f-f737-459b-81de-c844473c9ae7" containerID="9813e759a5a05cbe3500b50b10f5593ebaa8989a02998c23f6ceef24aa69ade8" exitCode=0 Jul 26 15:55:53.650273 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:55:53.650273 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dccql" Jul 26 15:55:53.650524 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:55:53.650320 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dccql" event={"ID":"4f828c8f-f737-459b-81de-c844473c9ae7","Type":"ContainerDied","Data":"9813e759a5a05cbe3500b50b10f5593ebaa8989a02998c23f6ceef24aa69ade8"} Jul 26 15:55:53.650524 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:55:53.650369 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dccql" event={"ID":"4f828c8f-f737-459b-81de-c844473c9ae7","Type":"ContainerDied","Data":"54b1c398655621d4d632280ee6c7593250c1396308fb86f35c83fd86e1e45d92"} Jul 26 15:55:53.650524 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:55:53.650416 2577 scope.go:117] "RemoveContainer" containerID="9813e759a5a05cbe3500b50b10f5593ebaa8989a02998c23f6ceef24aa69ade8" Jul 26 15:55:53.661148 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:55:53.661123 2577 scope.go:117] "RemoveContainer" containerID="9813e759a5a05cbe3500b50b10f5593ebaa8989a02998c23f6ceef24aa69ade8" Jul 26 15:55:53.661413 ip-10-0-138-163 kubenswrapper[2577]: E0726 15:55:53.661371 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9813e759a5a05cbe3500b50b10f5593ebaa8989a02998c23f6ceef24aa69ade8\": container with ID starting with 9813e759a5a05cbe3500b50b10f5593ebaa8989a02998c23f6ceef24aa69ade8 not found: ID does not exist" containerID="9813e759a5a05cbe3500b50b10f5593ebaa8989a02998c23f6ceef24aa69ade8" Jul 26 15:55:53.661485 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:55:53.661421 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9813e759a5a05cbe3500b50b10f5593ebaa8989a02998c23f6ceef24aa69ade8"} err="failed to get container status \"9813e759a5a05cbe3500b50b10f5593ebaa8989a02998c23f6ceef24aa69ade8\": rpc error: code = NotFound desc = could not find container \"9813e759a5a05cbe3500b50b10f5593ebaa8989a02998c23f6ceef24aa69ade8\": container with ID starting with 9813e759a5a05cbe3500b50b10f5593ebaa8989a02998c23f6ceef24aa69ade8 not found: ID does not exist" Jul 26 15:55:53.672422 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:55:53.672383 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dccql"] Jul 26 15:55:53.676045 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:55:53.676024 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dccql"] Jul 26 15:55:53.787738 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:55:53.787700 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4f828c8f-f737-459b-81de-c844473c9ae7" path="/var/lib/kubelet/pods/4f828c8f-f737-459b-81de-c844473c9ae7/volumes" Jul 26 15:56:50.689081 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:56:50.689043 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-cpvmq"] Jul 26 15:56:50.689583 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:56:50.689463 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f828c8f-f737-459b-81de-c844473c9ae7" containerName="manager" Jul 26 15:56:50.689583 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:56:50.689477 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f828c8f-f737-459b-81de-c844473c9ae7" containerName="manager" Jul 26 15:56:50.689583 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:56:50.689496 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="690378ba-7cb5-4774-bf6f-2c8109f38767" containerName="authorino" Jul 26 15:56:50.689583 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:56:50.689502 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="690378ba-7cb5-4774-bf6f-2c8109f38767" containerName="authorino" Jul 26 15:56:50.689583 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:56:50.689557 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="690378ba-7cb5-4774-bf6f-2c8109f38767" containerName="authorino" Jul 26 15:56:50.689583 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:56:50.689570 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f828c8f-f737-459b-81de-c844473c9ae7" containerName="manager" Jul 26 15:56:50.692691 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:56:50.692669 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-cpvmq" Jul 26 15:56:50.694957 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:56:50.694939 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-7fjh9\"" Jul 26 15:56:50.703305 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:56:50.703276 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-cpvmq"] Jul 26 15:56:50.764801 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:56:50.764758 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnd9d\" (UniqueName: \"kubernetes.io/projected/52da11a3-f37a-413e-b8c5-934a0aaf9526-kube-api-access-tnd9d\") pod \"kuadrant-operator-controller-manager-86878c769f-cpvmq\" (UID: \"52da11a3-f37a-413e-b8c5-934a0aaf9526\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-cpvmq" Jul 26 15:56:50.764982 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:56:50.764812 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/52da11a3-f37a-413e-b8c5-934a0aaf9526-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-cpvmq\" (UID: \"52da11a3-f37a-413e-b8c5-934a0aaf9526\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-cpvmq" Jul 26 15:56:50.865424 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:56:50.865361 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnd9d\" (UniqueName: \"kubernetes.io/projected/52da11a3-f37a-413e-b8c5-934a0aaf9526-kube-api-access-tnd9d\") pod \"kuadrant-operator-controller-manager-86878c769f-cpvmq\" (UID: \"52da11a3-f37a-413e-b8c5-934a0aaf9526\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-cpvmq" Jul 26 15:56:50.865424 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:56:50.865424 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/52da11a3-f37a-413e-b8c5-934a0aaf9526-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-cpvmq\" (UID: \"52da11a3-f37a-413e-b8c5-934a0aaf9526\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-cpvmq" Jul 26 15:56:50.865830 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:56:50.865811 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/52da11a3-f37a-413e-b8c5-934a0aaf9526-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-cpvmq\" (UID: \"52da11a3-f37a-413e-b8c5-934a0aaf9526\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-cpvmq" Jul 26 15:56:50.873228 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:56:50.873196 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnd9d\" (UniqueName: \"kubernetes.io/projected/52da11a3-f37a-413e-b8c5-934a0aaf9526-kube-api-access-tnd9d\") pod \"kuadrant-operator-controller-manager-86878c769f-cpvmq\" (UID: \"52da11a3-f37a-413e-b8c5-934a0aaf9526\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-cpvmq" Jul 26 15:56:51.004061 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:56:51.003975 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-cpvmq" Jul 26 15:56:51.146078 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:56:51.146049 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-cpvmq"] Jul 26 15:56:51.148743 ip-10-0-138-163 kubenswrapper[2577]: W0726 15:56:51.148700 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52da11a3_f37a_413e_b8c5_934a0aaf9526.slice/crio-f4c2a5c5b5a42982b811c01e1e5a48968607ac7bdab56c3942db9017d2e13000 WatchSource:0}: Error finding container f4c2a5c5b5a42982b811c01e1e5a48968607ac7bdab56c3942db9017d2e13000: Status 404 returned error can't find the container with id f4c2a5c5b5a42982b811c01e1e5a48968607ac7bdab56c3942db9017d2e13000 Jul 26 15:56:51.960853 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:56:51.960815 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-cpvmq" event={"ID":"52da11a3-f37a-413e-b8c5-934a0aaf9526","Type":"ContainerStarted","Data":"f3f9a72c95dfe81f5f0cc7d39f5202b0032fb728f32d13e0a93ed508085a9bdd"} Jul 26 15:56:51.961243 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:56:51.960877 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-cpvmq" event={"ID":"52da11a3-f37a-413e-b8c5-934a0aaf9526","Type":"ContainerStarted","Data":"f4c2a5c5b5a42982b811c01e1e5a48968607ac7bdab56c3942db9017d2e13000"} Jul 26 15:56:51.961243 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:56:51.960908 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-cpvmq" Jul 26 15:56:51.984178 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:56:51.984129 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-cpvmq" podStartSLOduration=1.984114445 podStartE2EDuration="1.984114445s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:56:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:56:51.983117749 +0000 UTC m=+1134.722471077" watchObservedRunningTime="2026-07-26 15:56:51.984114445 +0000 UTC m=+1134.723467757" Jul 26 15:57:02.969447 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:57:02.969386 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-cpvmq" Jul 26 15:57:31.639232 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:57:31.639196 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:57:57.807637 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:57:57.807608 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-8wjqp_b6b8080d-0cee-4d0c-a05c-3e2f1b672a54/console-operator/1.log" Jul 26 15:57:57.809958 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:57:57.809926 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-8wjqp_b6b8080d-0cee-4d0c-a05c-3e2f1b672a54/console-operator/1.log" Jul 26 15:57:57.811742 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:57:57.811722 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ztlp6_cbe2cd41-6ebc-4722-a813-57d83fa1d242/ovn-acl-logging/0.log" Jul 26 15:57:57.813889 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:57:57.813867 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ztlp6_cbe2cd41-6ebc-4722-a813-57d83fa1d242/ovn-acl-logging/0.log" Jul 26 15:58:12.924013 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:58:12.923978 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:59:31.201618 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:59:31.201586 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 15:59:43.209588 ip-10-0-138-163 kubenswrapper[2577]: I0726 15:59:43.209556 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:00:00.137559 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:00:00.137518 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29751360-vx2hl"] Jul 26 16:00:00.141073 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:00:00.141049 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29751360-vx2hl" Jul 26 16:00:00.143586 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:00:00.143564 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n79w9\"" Jul 26 16:00:00.148130 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:00:00.148104 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29751360-vx2hl"] Jul 26 16:00:00.152262 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:00:00.152237 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29751360-vx2hl" Jul 26 16:00:00.292058 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:00:00.292032 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29751360-vx2hl"] Jul 26 16:00:00.294422 ip-10-0-138-163 kubenswrapper[2577]: W0726 16:00:00.294369 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7e3b4ec_623e_435c_a990_b9c4f02652b1.slice/crio-4b9d481f1ea6c89d36190b8e761f3310953c0d7a052878e686cff2e944c7be16 WatchSource:0}: Error finding container 4b9d481f1ea6c89d36190b8e761f3310953c0d7a052878e686cff2e944c7be16: Status 404 returned error can't find the container with id 4b9d481f1ea6c89d36190b8e761f3310953c0d7a052878e686cff2e944c7be16 Jul 26 16:00:00.296386 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:00:00.296371 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 26 16:00:01.000479 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:00:01.000443 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29751360-vx2hl" event={"ID":"f7e3b4ec-623e-435c-a990-b9c4f02652b1","Type":"ContainerStarted","Data":"4b9d481f1ea6c89d36190b8e761f3310953c0d7a052878e686cff2e944c7be16"} Jul 26 16:00:03.013820 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:00:03.013784 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29751360-vx2hl" event={"ID":"f7e3b4ec-623e-435c-a990-b9c4f02652b1","Type":"ContainerStarted","Data":"178818602ccdb891751ffd57a11a54d58bd18b7b85e0c92c26190bf2a1995c9b"} Jul 26 16:00:03.029081 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:00:03.029028 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29751360-vx2hl" podStartSLOduration=0.765740347 podStartE2EDuration="3.02901073s" totalImagesPullingTime="2.263270383s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 16:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 16:00:03.026673722 +0000 UTC m=+1325.766027046" watchObservedRunningTime="2026-07-26 16:00:03.02901073 +0000 UTC m=+1325.768364040" Jul 26 16:00:04.021366 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:00:04.021280 2577 generic.go:356] "Generic (PLEG): container finished" podID="f7e3b4ec-623e-435c-a990-b9c4f02652b1" containerID="178818602ccdb891751ffd57a11a54d58bd18b7b85e0c92c26190bf2a1995c9b" exitCode=0 Jul 26 16:00:04.021744 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:00:04.021369 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29751360-vx2hl" event={"ID":"f7e3b4ec-623e-435c-a990-b9c4f02652b1","Type":"ContainerDied","Data":"178818602ccdb891751ffd57a11a54d58bd18b7b85e0c92c26190bf2a1995c9b"} Jul 26 16:00:05.167309 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:00:05.167282 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29751360-vx2hl" Jul 26 16:00:06.034901 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:00:06.034871 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29751360-vx2hl" Jul 26 16:00:06.035064 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:00:06.034929 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29751360-vx2hl" event={"ID":"f7e3b4ec-623e-435c-a990-b9c4f02652b1","Type":"ContainerDied","Data":"4b9d481f1ea6c89d36190b8e761f3310953c0d7a052878e686cff2e944c7be16"} Jul 26 16:00:06.035064 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:00:06.034993 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4b9d481f1ea6c89d36190b8e761f3310953c0d7a052878e686cff2e944c7be16" Jul 26 16:00:09.401369 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:00:09.401327 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:00:10.542346 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:00:10.542282 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:02:57.849132 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:02:57.849021 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-8wjqp_b6b8080d-0cee-4d0c-a05c-3e2f1b672a54/console-operator/1.log" Jul 26 16:02:57.853177 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:02:57.849890 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-8wjqp_b6b8080d-0cee-4d0c-a05c-3e2f1b672a54/console-operator/1.log" Jul 26 16:02:57.853177 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:02:57.853047 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ztlp6_cbe2cd41-6ebc-4722-a813-57d83fa1d242/ovn-acl-logging/0.log" Jul 26 16:02:57.854004 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:02:57.853981 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ztlp6_cbe2cd41-6ebc-4722-a813-57d83fa1d242/ovn-acl-logging/0.log" Jul 26 16:03:19.266275 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:03:19.266240 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:03:19.323139 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:03:19.323101 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:03:40.344898 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:03:40.344867 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:03:43.338357 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:03:43.338316 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:03:43.439280 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:03:43.439247 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:03:43.741380 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:03:43.741282 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:03:49.404063 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:03:49.404023 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:03:49.516726 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:03:49.516690 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:03:49.686473 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:03:49.686371 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:03:49.917006 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:03:49.916973 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-2b3f0867-openshift-default-6c4bd74488-hpp2l"] Jul 26 16:03:49.917545 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:03:49.917529 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7e3b4ec-623e-435c-a990-b9c4f02652b1" containerName="cleanup" Jul 26 16:03:49.917608 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:03:49.917549 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7e3b4ec-623e-435c-a990-b9c4f02652b1" containerName="cleanup" Jul 26 16:03:49.917670 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:03:49.917658 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7e3b4ec-623e-435c-a990-b9c4f02652b1" containerName="cleanup" Jul 26 16:03:49.921102 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:03:49.921084 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-2b3f0867-openshift-default-6c4bd74488-hpp2l" Jul 26 16:03:49.923359 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:03:49.923340 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-2b3f0867-openshift-default-dockercfg-g6h58\"" Jul 26 16:03:49.940960 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:03:49.940894 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-2b3f0867-openshift-default-6c4bd74488-hpp2l"] Jul 26 16:03:49.942751 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:03:49.942708 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:03:50.036319 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:03:50.036284 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d8e244f-8f0f-4b65-b21f-af312c5b9559-istio-envoy\") pod \"e2e-ait-2b3f0867-openshift-default-6c4bd74488-hpp2l\" (UID: \"5d8e244f-8f0f-4b65-b21f-af312c5b9559\") " pod="openshift-ingress/e2e-ait-2b3f0867-openshift-default-6c4bd74488-hpp2l" Jul 26 16:03:50.036526 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:03:50.036349 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d8e244f-8f0f-4b65-b21f-af312c5b9559-credential-socket\") pod \"e2e-ait-2b3f0867-openshift-default-6c4bd74488-hpp2l\" (UID: \"5d8e244f-8f0f-4b65-b21f-af312c5b9559\") " pod="openshift-ingress/e2e-ait-2b3f0867-openshift-default-6c4bd74488-hpp2l" Jul 26 16:03:50.036526 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:03:50.036419 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d8e244f-8f0f-4b65-b21f-af312c5b9559-istio-podinfo\") pod \"e2e-ait-2b3f0867-openshift-default-6c4bd74488-hpp2l\" (UID: \"5d8e244f-8f0f-4b65-b21f-af312c5b9559\") " pod="openshift-ingress/e2e-ait-2b3f0867-openshift-default-6c4bd74488-hpp2l" Jul 26 16:03:50.036526 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:03:50.036442 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d8e244f-8f0f-4b65-b21f-af312c5b9559-istiod-ca-cert\") pod \"e2e-ait-2b3f0867-openshift-default-6c4bd74488-hpp2l\" (UID: \"5d8e244f-8f0f-4b65-b21f-af312c5b9559\") " pod="openshift-ingress/e2e-ait-2b3f0867-openshift-default-6c4bd74488-hpp2l" Jul 26 16:03:50.036526 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:03:50.036465 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqzgh\" (UniqueName: \"kubernetes.io/projected/5d8e244f-8f0f-4b65-b21f-af312c5b9559-kube-api-access-vqzgh\") pod \"e2e-ait-2b3f0867-openshift-default-6c4bd74488-hpp2l\" (UID: \"5d8e244f-8f0f-4b65-b21f-af312c5b9559\") " pod="openshift-ingress/e2e-ait-2b3f0867-openshift-default-6c4bd74488-hpp2l" Jul 26 16:03:50.036673 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:03:50.036560 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d8e244f-8f0f-4b65-b21f-af312c5b9559-workload-socket\") pod \"e2e-ait-2b3f0867-openshift-default-6c4bd74488-hpp2l\" (UID: \"5d8e244f-8f0f-4b65-b21f-af312c5b9559\") " pod="openshift-ingress/e2e-ait-2b3f0867-openshift-default-6c4bd74488-hpp2l" Jul 26 16:03:50.036673 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:03:50.036593 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d8e244f-8f0f-4b65-b21f-af312c5b9559-istio-data\") pod \"e2e-ait-2b3f0867-openshift-default-6c4bd74488-hpp2l\" (UID: \"5d8e244f-8f0f-4b65-b21f-af312c5b9559\") " pod="openshift-ingress/e2e-ait-2b3f0867-openshift-default-6c4bd74488-hpp2l" Jul 26 16:03:50.036673 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:03:50.036626 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d8e244f-8f0f-4b65-b21f-af312c5b9559-workload-certs\") pod \"e2e-ait-2b3f0867-openshift-default-6c4bd74488-hpp2l\" (UID: \"5d8e244f-8f0f-4b65-b21f-af312c5b9559\") " pod="openshift-ingress/e2e-ait-2b3f0867-openshift-default-6c4bd74488-hpp2l" Jul 26 16:03:50.036768 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:03:50.036675 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d8e244f-8f0f-4b65-b21f-af312c5b9559-istio-token\") pod \"e2e-ait-2b3f0867-openshift-default-6c4bd74488-hpp2l\" (UID: \"5d8e244f-8f0f-4b65-b21f-af312c5b9559\") " pod="openshift-ingress/e2e-ait-2b3f0867-openshift-default-6c4bd74488-hpp2l" Jul 26 16:03:50.137486 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:03:50.137452 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d8e244f-8f0f-4b65-b21f-af312c5b9559-workload-socket\") pod \"e2e-ait-2b3f0867-openshift-default-6c4bd74488-hpp2l\" (UID: \"5d8e244f-8f0f-4b65-b21f-af312c5b9559\") " pod="openshift-ingress/e2e-ait-2b3f0867-openshift-default-6c4bd74488-hpp2l" Jul 26 16:03:50.137486 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:03:50.137485 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d8e244f-8f0f-4b65-b21f-af312c5b9559-istio-data\") pod \"e2e-ait-2b3f0867-openshift-default-6c4bd74488-hpp2l\" (UID: \"5d8e244f-8f0f-4b65-b21f-af312c5b9559\") " pod="openshift-ingress/e2e-ait-2b3f0867-openshift-default-6c4bd74488-hpp2l" Jul 26 16:03:50.137738 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:03:50.137513 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d8e244f-8f0f-4b65-b21f-af312c5b9559-workload-certs\") pod \"e2e-ait-2b3f0867-openshift-default-6c4bd74488-hpp2l\" (UID: \"5d8e244f-8f0f-4b65-b21f-af312c5b9559\") " pod="openshift-ingress/e2e-ait-2b3f0867-openshift-default-6c4bd74488-hpp2l" Jul 26 16:03:50.137738 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:03:50.137709 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d8e244f-8f0f-4b65-b21f-af312c5b9559-istio-token\") pod \"e2e-ait-2b3f0867-openshift-default-6c4bd74488-hpp2l\" (UID: \"5d8e244f-8f0f-4b65-b21f-af312c5b9559\") " pod="openshift-ingress/e2e-ait-2b3f0867-openshift-default-6c4bd74488-hpp2l" Jul 26 16:03:50.137850 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:03:50.137768 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d8e244f-8f0f-4b65-b21f-af312c5b9559-istio-envoy\") pod \"e2e-ait-2b3f0867-openshift-default-6c4bd74488-hpp2l\" (UID: \"5d8e244f-8f0f-4b65-b21f-af312c5b9559\") " pod="openshift-ingress/e2e-ait-2b3f0867-openshift-default-6c4bd74488-hpp2l" Jul 26 16:03:50.137850 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:03:50.137826 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d8e244f-8f0f-4b65-b21f-af312c5b9559-credential-socket\") pod \"e2e-ait-2b3f0867-openshift-default-6c4bd74488-hpp2l\" (UID: \"5d8e244f-8f0f-4b65-b21f-af312c5b9559\") " pod="openshift-ingress/e2e-ait-2b3f0867-openshift-default-6c4bd74488-hpp2l" Jul 26 16:03:50.137960 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:03:50.137895 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d8e244f-8f0f-4b65-b21f-af312c5b9559-istio-podinfo\") pod \"e2e-ait-2b3f0867-openshift-default-6c4bd74488-hpp2l\" (UID: \"5d8e244f-8f0f-4b65-b21f-af312c5b9559\") " pod="openshift-ingress/e2e-ait-2b3f0867-openshift-default-6c4bd74488-hpp2l" Jul 26 16:03:50.137960 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:03:50.137919 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d8e244f-8f0f-4b65-b21f-af312c5b9559-istiod-ca-cert\") pod \"e2e-ait-2b3f0867-openshift-default-6c4bd74488-hpp2l\" (UID: \"5d8e244f-8f0f-4b65-b21f-af312c5b9559\") " pod="openshift-ingress/e2e-ait-2b3f0867-openshift-default-6c4bd74488-hpp2l" Jul 26 16:03:50.137960 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:03:50.137945 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqzgh\" (UniqueName: \"kubernetes.io/projected/5d8e244f-8f0f-4b65-b21f-af312c5b9559-kube-api-access-vqzgh\") pod \"e2e-ait-2b3f0867-openshift-default-6c4bd74488-hpp2l\" (UID: \"5d8e244f-8f0f-4b65-b21f-af312c5b9559\") " pod="openshift-ingress/e2e-ait-2b3f0867-openshift-default-6c4bd74488-hpp2l" Jul 26 16:03:50.138213 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:03:50.138191 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d8e244f-8f0f-4b65-b21f-af312c5b9559-istio-data\") pod \"e2e-ait-2b3f0867-openshift-default-6c4bd74488-hpp2l\" (UID: \"5d8e244f-8f0f-4b65-b21f-af312c5b9559\") " pod="openshift-ingress/e2e-ait-2b3f0867-openshift-default-6c4bd74488-hpp2l" Jul 26 16:03:50.138483 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:03:50.138460 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d8e244f-8f0f-4b65-b21f-af312c5b9559-workload-certs\") pod \"e2e-ait-2b3f0867-openshift-default-6c4bd74488-hpp2l\" (UID: \"5d8e244f-8f0f-4b65-b21f-af312c5b9559\") " pod="openshift-ingress/e2e-ait-2b3f0867-openshift-default-6c4bd74488-hpp2l" Jul 26 16:03:50.138655 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:03:50.138504 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d8e244f-8f0f-4b65-b21f-af312c5b9559-workload-socket\") pod \"e2e-ait-2b3f0867-openshift-default-6c4bd74488-hpp2l\" (UID: \"5d8e244f-8f0f-4b65-b21f-af312c5b9559\") " pod="openshift-ingress/e2e-ait-2b3f0867-openshift-default-6c4bd74488-hpp2l" Jul 26 16:03:50.138655 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:03:50.138537 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d8e244f-8f0f-4b65-b21f-af312c5b9559-credential-socket\") pod \"e2e-ait-2b3f0867-openshift-default-6c4bd74488-hpp2l\" (UID: \"5d8e244f-8f0f-4b65-b21f-af312c5b9559\") " pod="openshift-ingress/e2e-ait-2b3f0867-openshift-default-6c4bd74488-hpp2l" Jul 26 16:03:50.139267 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:03:50.139238 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d8e244f-8f0f-4b65-b21f-af312c5b9559-istiod-ca-cert\") pod \"e2e-ait-2b3f0867-openshift-default-6c4bd74488-hpp2l\" (UID: \"5d8e244f-8f0f-4b65-b21f-af312c5b9559\") " pod="openshift-ingress/e2e-ait-2b3f0867-openshift-default-6c4bd74488-hpp2l" Jul 26 16:03:50.140587 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:03:50.140565 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d8e244f-8f0f-4b65-b21f-af312c5b9559-istio-envoy\") pod \"e2e-ait-2b3f0867-openshift-default-6c4bd74488-hpp2l\" (UID: \"5d8e244f-8f0f-4b65-b21f-af312c5b9559\") " pod="openshift-ingress/e2e-ait-2b3f0867-openshift-default-6c4bd74488-hpp2l" Jul 26 16:03:50.141323 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:03:50.141299 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d8e244f-8f0f-4b65-b21f-af312c5b9559-istio-podinfo\") pod \"e2e-ait-2b3f0867-openshift-default-6c4bd74488-hpp2l\" (UID: \"5d8e244f-8f0f-4b65-b21f-af312c5b9559\") " pod="openshift-ingress/e2e-ait-2b3f0867-openshift-default-6c4bd74488-hpp2l" Jul 26 16:03:50.145650 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:03:50.145631 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d8e244f-8f0f-4b65-b21f-af312c5b9559-istio-token\") pod \"e2e-ait-2b3f0867-openshift-default-6c4bd74488-hpp2l\" (UID: \"5d8e244f-8f0f-4b65-b21f-af312c5b9559\") " pod="openshift-ingress/e2e-ait-2b3f0867-openshift-default-6c4bd74488-hpp2l" Jul 26 16:03:50.145906 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:03:50.145886 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqzgh\" (UniqueName: \"kubernetes.io/projected/5d8e244f-8f0f-4b65-b21f-af312c5b9559-kube-api-access-vqzgh\") pod \"e2e-ait-2b3f0867-openshift-default-6c4bd74488-hpp2l\" (UID: \"5d8e244f-8f0f-4b65-b21f-af312c5b9559\") " pod="openshift-ingress/e2e-ait-2b3f0867-openshift-default-6c4bd74488-hpp2l" Jul 26 16:03:50.231913 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:03:50.231821 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-2b3f0867-openshift-default-6c4bd74488-hpp2l" Jul 26 16:03:50.377602 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:03:50.377575 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-2b3f0867-openshift-default-6c4bd74488-hpp2l"] Jul 26 16:03:50.379071 ip-10-0-138-163 kubenswrapper[2577]: W0726 16:03:50.379043 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d8e244f_8f0f_4b65_b21f_af312c5b9559.slice/crio-934c2022daf822f64ffc1bd2493499f915ffc745ab2feedd18ac120433673cbf WatchSource:0}: Error finding container 934c2022daf822f64ffc1bd2493499f915ffc745ab2feedd18ac120433673cbf: Status 404 returned error can't find the container with id 934c2022daf822f64ffc1bd2493499f915ffc745ab2feedd18ac120433673cbf Jul 26 16:03:50.381369 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:03:50.381291 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:03:50.381518 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:03:50.381410 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:03:50.381518 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:03:50.381463 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:03:50.824338 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:03:50.824299 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:03:51.295456 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:03:51.295422 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2b3f0867-openshift-default-6c4bd74488-hpp2l" event={"ID":"5d8e244f-8f0f-4b65-b21f-af312c5b9559","Type":"ContainerStarted","Data":"854951d80d1a50de6a0a80d8e22a140050484f3137105859315fb31439ad21fa"} Jul 26 16:03:51.295640 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:03:51.295475 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2b3f0867-openshift-default-6c4bd74488-hpp2l" event={"ID":"5d8e244f-8f0f-4b65-b21f-af312c5b9559","Type":"ContainerStarted","Data":"934c2022daf822f64ffc1bd2493499f915ffc745ab2feedd18ac120433673cbf"} Jul 26 16:03:51.316439 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:03:51.316353 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-2b3f0867-openshift-default-6c4bd74488-hpp2l" podStartSLOduration=2.316336412 podStartE2EDuration="2.316336412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 16:03:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 16:03:51.314624588 +0000 UTC m=+1554.053977921" watchObservedRunningTime="2026-07-26 16:03:51.316336412 +0000 UTC m=+1554.055689795" Jul 26 16:03:52.232221 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:03:52.232174 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-2b3f0867-openshift-default-6c4bd74488-hpp2l" Jul 26 16:03:52.237309 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:03:52.237284 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-2b3f0867-openshift-default-6c4bd74488-hpp2l" Jul 26 16:03:52.301362 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:03:52.301330 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-2b3f0867-openshift-default-6c4bd74488-hpp2l" Jul 26 16:03:52.302155 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:03:52.302135 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-2b3f0867-openshift-default-6c4bd74488-hpp2l" Jul 26 16:03:52.633344 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:03:52.633316 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:03:52.731308 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:03:52.731273 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:03:52.831112 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:03:52.831076 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:03:53.022100 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:03:53.022072 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:03:53.321452 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:03:53.321344 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:03:53.405453 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:03:53.405420 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:04:06.389671 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:04:06.389636 2577 generic.go:356] "Generic (PLEG): container finished" podID="5d8e244f-8f0f-4b65-b21f-af312c5b9559" containerID="854951d80d1a50de6a0a80d8e22a140050484f3137105859315fb31439ad21fa" exitCode=0 Jul 26 16:04:06.390128 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:04:06.389708 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2b3f0867-openshift-default-6c4bd74488-hpp2l" event={"ID":"5d8e244f-8f0f-4b65-b21f-af312c5b9559","Type":"ContainerDied","Data":"854951d80d1a50de6a0a80d8e22a140050484f3137105859315fb31439ad21fa"} Jul 26 16:04:06.390185 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:04:06.390135 2577 scope.go:117] "RemoveContainer" containerID="854951d80d1a50de6a0a80d8e22a140050484f3137105859315fb31439ad21fa" Jul 26 16:04:06.391143 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:04:06.391117 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:04:06.391215 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:04:06.391199 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:04:06.391256 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:04:06.391242 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:04:07.301410 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:04:07.301363 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-2b3f0867-openshift-default-6c4bd74488-hpp2l" Jul 26 16:04:07.396778 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:04:07.396748 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2b3f0867-openshift-default-6c4bd74488-hpp2l" event={"ID":"5d8e244f-8f0f-4b65-b21f-af312c5b9559","Type":"ContainerStarted","Data":"5336eb2ffff593830128b390d25271b4177e94647f41ca1de416b7160b477ab7"} Jul 26 16:04:08.232748 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:04:08.232711 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-2b3f0867-openshift-default-6c4bd74488-hpp2l" Jul 26 16:04:08.233724 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:04:08.233650 2577 patch_prober.go:28] interesting pod/e2e-ait-2b3f0867-openshift-default-6c4bd74488-hpp2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Jul 26 16:04:08.233816 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:04:08.233701 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-2b3f0867-openshift-default-6c4bd74488-hpp2l" podUID="5d8e244f-8f0f-4b65-b21f-af312c5b9559" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Jul 26 16:04:09.233238 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:04:09.233198 2577 patch_prober.go:28] interesting pod/e2e-ait-2b3f0867-openshift-default-6c4bd74488-hpp2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Jul 26 16:04:09.233900 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:04:09.233865 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-2b3f0867-openshift-default-6c4bd74488-hpp2l" podUID="5d8e244f-8f0f-4b65-b21f-af312c5b9559" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Jul 26 16:04:10.177033 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:04:10.176985 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:04:10.232360 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:04:10.232320 2577 patch_prober.go:28] interesting pod/e2e-ait-2b3f0867-openshift-default-6c4bd74488-hpp2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Jul 26 16:04:10.232681 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:04:10.232650 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-2b3f0867-openshift-default-6c4bd74488-hpp2l" podUID="5d8e244f-8f0f-4b65-b21f-af312c5b9559" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Jul 26 16:04:11.232320 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:04:11.232282 2577 patch_prober.go:28] interesting pod/e2e-ait-2b3f0867-openshift-default-6c4bd74488-hpp2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Jul 26 16:04:11.233656 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:04:11.233620 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-2b3f0867-openshift-default-6c4bd74488-hpp2l" podUID="5d8e244f-8f0f-4b65-b21f-af312c5b9559" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Jul 26 16:04:12.236945 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:04:12.236913 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-2b3f0867-openshift-default-6c4bd74488-hpp2l" Jul 26 16:04:12.237534 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:04:12.237296 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-2b3f0867-openshift-default-6c4bd74488-hpp2l" Jul 26 16:04:12.238195 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:04:12.238176 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-2b3f0867-openshift-default-6c4bd74488-hpp2l" Jul 26 16:04:17.940451 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:04:17.940412 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:04:28.585337 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:04:28.585299 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:04:28.652503 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:04:28.652468 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:04:28.711031 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:04:28.710997 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:04:33.234464 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:04:33.234421 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:04:33.370417 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:04:33.370374 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-2b3f0867-openshift-default-6c4bd74488-hpp2l"] Jul 26 16:04:33.370663 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:04:33.370630 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-2b3f0867-openshift-default-6c4bd74488-hpp2l" podUID="5d8e244f-8f0f-4b65-b21f-af312c5b9559" containerName="istio-proxy" containerID="cri-o://5336eb2ffff593830128b390d25271b4177e94647f41ca1de416b7160b477ab7" gracePeriod=30 Jul 26 16:04:33.390389 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:04:33.390359 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:04:33.667469 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:04:33.667437 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:04:33.895053 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:04:33.894994 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:04:33.988688 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:04:33.988599 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:04:34.089411 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:04:34.089374 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:04:34.356984 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:04:34.356955 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-28d9404a-gw-openshift-default-d478b57b-zfxcf"] Jul 26 16:04:34.359535 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:04:34.359516 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-28d9404a-gw-openshift-default-d478b57b-zfxcf" Jul 26 16:04:34.362002 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:04:34.361979 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-28d9404a-gw-openshift-default-dockercfg-tjrbk\"" Jul 26 16:04:34.372672 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:04:34.372647 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-28d9404a-gw-openshift-default-d478b57b-zfxcf"] Jul 26 16:04:34.386974 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:04:34.386941 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:04:34.406341 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:04:34.406305 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5cb1a79-829e-4ec6-a0db-0ab664abd86d-credential-socket\") pod \"e2e-migrate-28d9404a-gw-openshift-default-d478b57b-zfxcf\" (UID: \"f5cb1a79-829e-4ec6-a0db-0ab664abd86d\") " pod="openshift-ingress/e2e-migrate-28d9404a-gw-openshift-default-d478b57b-zfxcf" Jul 26 16:04:34.406341 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:04:34.406342 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5cb1a79-829e-4ec6-a0db-0ab664abd86d-istio-token\") pod \"e2e-migrate-28d9404a-gw-openshift-default-d478b57b-zfxcf\" (UID: \"f5cb1a79-829e-4ec6-a0db-0ab664abd86d\") " pod="openshift-ingress/e2e-migrate-28d9404a-gw-openshift-default-d478b57b-zfxcf" Jul 26 16:04:34.406558 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:04:34.406388 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lp5l7\" (UniqueName: \"kubernetes.io/projected/f5cb1a79-829e-4ec6-a0db-0ab664abd86d-kube-api-access-lp5l7\") pod \"e2e-migrate-28d9404a-gw-openshift-default-d478b57b-zfxcf\" (UID: \"f5cb1a79-829e-4ec6-a0db-0ab664abd86d\") " pod="openshift-ingress/e2e-migrate-28d9404a-gw-openshift-default-d478b57b-zfxcf" Jul 26 16:04:34.406558 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:04:34.406459 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5cb1a79-829e-4ec6-a0db-0ab664abd86d-istiod-ca-cert\") pod \"e2e-migrate-28d9404a-gw-openshift-default-d478b57b-zfxcf\" (UID: \"f5cb1a79-829e-4ec6-a0db-0ab664abd86d\") " pod="openshift-ingress/e2e-migrate-28d9404a-gw-openshift-default-d478b57b-zfxcf" Jul 26 16:04:34.406558 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:04:34.406483 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5cb1a79-829e-4ec6-a0db-0ab664abd86d-workload-socket\") pod \"e2e-migrate-28d9404a-gw-openshift-default-d478b57b-zfxcf\" (UID: \"f5cb1a79-829e-4ec6-a0db-0ab664abd86d\") " pod="openshift-ingress/e2e-migrate-28d9404a-gw-openshift-default-d478b57b-zfxcf" Jul 26 16:04:34.406558 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:04:34.406518 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5cb1a79-829e-4ec6-a0db-0ab664abd86d-workload-certs\") pod \"e2e-migrate-28d9404a-gw-openshift-default-d478b57b-zfxcf\" (UID: \"f5cb1a79-829e-4ec6-a0db-0ab664abd86d\") " pod="openshift-ingress/e2e-migrate-28d9404a-gw-openshift-default-d478b57b-zfxcf" Jul 26 16:04:34.406558 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:04:34.406555 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5cb1a79-829e-4ec6-a0db-0ab664abd86d-istio-podinfo\") pod \"e2e-migrate-28d9404a-gw-openshift-default-d478b57b-zfxcf\" (UID: \"f5cb1a79-829e-4ec6-a0db-0ab664abd86d\") " pod="openshift-ingress/e2e-migrate-28d9404a-gw-openshift-default-d478b57b-zfxcf" Jul 26 16:04:34.406729 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:04:34.406578 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5cb1a79-829e-4ec6-a0db-0ab664abd86d-istio-data\") pod \"e2e-migrate-28d9404a-gw-openshift-default-d478b57b-zfxcf\" (UID: \"f5cb1a79-829e-4ec6-a0db-0ab664abd86d\") " pod="openshift-ingress/e2e-migrate-28d9404a-gw-openshift-default-d478b57b-zfxcf" Jul 26 16:04:34.406729 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:04:34.406605 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5cb1a79-829e-4ec6-a0db-0ab664abd86d-istio-envoy\") pod \"e2e-migrate-28d9404a-gw-openshift-default-d478b57b-zfxcf\" (UID: \"f5cb1a79-829e-4ec6-a0db-0ab664abd86d\") " pod="openshift-ingress/e2e-migrate-28d9404a-gw-openshift-default-d478b57b-zfxcf" Jul 26 16:04:34.507633 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:04:34.507595 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lp5l7\" (UniqueName: \"kubernetes.io/projected/f5cb1a79-829e-4ec6-a0db-0ab664abd86d-kube-api-access-lp5l7\") pod \"e2e-migrate-28d9404a-gw-openshift-default-d478b57b-zfxcf\" (UID: \"f5cb1a79-829e-4ec6-a0db-0ab664abd86d\") " pod="openshift-ingress/e2e-migrate-28d9404a-gw-openshift-default-d478b57b-zfxcf" Jul 26 16:04:34.507832 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:04:34.507640 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5cb1a79-829e-4ec6-a0db-0ab664abd86d-istiod-ca-cert\") pod \"e2e-migrate-28d9404a-gw-openshift-default-d478b57b-zfxcf\" (UID: \"f5cb1a79-829e-4ec6-a0db-0ab664abd86d\") " pod="openshift-ingress/e2e-migrate-28d9404a-gw-openshift-default-d478b57b-zfxcf" Jul 26 16:04:34.507832 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:04:34.507660 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5cb1a79-829e-4ec6-a0db-0ab664abd86d-workload-socket\") pod \"e2e-migrate-28d9404a-gw-openshift-default-d478b57b-zfxcf\" (UID: \"f5cb1a79-829e-4ec6-a0db-0ab664abd86d\") " pod="openshift-ingress/e2e-migrate-28d9404a-gw-openshift-default-d478b57b-zfxcf" Jul 26 16:04:34.507832 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:04:34.507684 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5cb1a79-829e-4ec6-a0db-0ab664abd86d-workload-certs\") pod \"e2e-migrate-28d9404a-gw-openshift-default-d478b57b-zfxcf\" (UID: \"f5cb1a79-829e-4ec6-a0db-0ab664abd86d\") " pod="openshift-ingress/e2e-migrate-28d9404a-gw-openshift-default-d478b57b-zfxcf" Jul 26 16:04:34.507832 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:04:34.507717 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5cb1a79-829e-4ec6-a0db-0ab664abd86d-istio-podinfo\") pod \"e2e-migrate-28d9404a-gw-openshift-default-d478b57b-zfxcf\" (UID: \"f5cb1a79-829e-4ec6-a0db-0ab664abd86d\") " pod="openshift-ingress/e2e-migrate-28d9404a-gw-openshift-default-d478b57b-zfxcf" Jul 26 16:04:34.507832 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:04:34.507743 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5cb1a79-829e-4ec6-a0db-0ab664abd86d-istio-data\") pod \"e2e-migrate-28d9404a-gw-openshift-default-d478b57b-zfxcf\" (UID: \"f5cb1a79-829e-4ec6-a0db-0ab664abd86d\") " pod="openshift-ingress/e2e-migrate-28d9404a-gw-openshift-default-d478b57b-zfxcf" Jul 26 16:04:34.507832 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:04:34.507779 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5cb1a79-829e-4ec6-a0db-0ab664abd86d-istio-envoy\") pod \"e2e-migrate-28d9404a-gw-openshift-default-d478b57b-zfxcf\" (UID: \"f5cb1a79-829e-4ec6-a0db-0ab664abd86d\") " pod="openshift-ingress/e2e-migrate-28d9404a-gw-openshift-default-d478b57b-zfxcf" Jul 26 16:04:34.508183 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:04:34.507839 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5cb1a79-829e-4ec6-a0db-0ab664abd86d-credential-socket\") pod \"e2e-migrate-28d9404a-gw-openshift-default-d478b57b-zfxcf\" (UID: \"f5cb1a79-829e-4ec6-a0db-0ab664abd86d\") " pod="openshift-ingress/e2e-migrate-28d9404a-gw-openshift-default-d478b57b-zfxcf" Jul 26 16:04:34.508183 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:04:34.507863 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5cb1a79-829e-4ec6-a0db-0ab664abd86d-istio-token\") pod \"e2e-migrate-28d9404a-gw-openshift-default-d478b57b-zfxcf\" (UID: \"f5cb1a79-829e-4ec6-a0db-0ab664abd86d\") " pod="openshift-ingress/e2e-migrate-28d9404a-gw-openshift-default-d478b57b-zfxcf" Jul 26 16:04:34.508183 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:04:34.508092 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5cb1a79-829e-4ec6-a0db-0ab664abd86d-workload-socket\") pod \"e2e-migrate-28d9404a-gw-openshift-default-d478b57b-zfxcf\" (UID: \"f5cb1a79-829e-4ec6-a0db-0ab664abd86d\") " pod="openshift-ingress/e2e-migrate-28d9404a-gw-openshift-default-d478b57b-zfxcf" Jul 26 16:04:34.508327 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:04:34.508242 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5cb1a79-829e-4ec6-a0db-0ab664abd86d-workload-certs\") pod \"e2e-migrate-28d9404a-gw-openshift-default-d478b57b-zfxcf\" (UID: \"f5cb1a79-829e-4ec6-a0db-0ab664abd86d\") " pod="openshift-ingress/e2e-migrate-28d9404a-gw-openshift-default-d478b57b-zfxcf" Jul 26 16:04:34.508327 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:04:34.508303 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5cb1a79-829e-4ec6-a0db-0ab664abd86d-istio-data\") pod \"e2e-migrate-28d9404a-gw-openshift-default-d478b57b-zfxcf\" (UID: \"f5cb1a79-829e-4ec6-a0db-0ab664abd86d\") " pod="openshift-ingress/e2e-migrate-28d9404a-gw-openshift-default-d478b57b-zfxcf" Jul 26 16:04:34.508494 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:04:34.508472 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5cb1a79-829e-4ec6-a0db-0ab664abd86d-credential-socket\") pod \"e2e-migrate-28d9404a-gw-openshift-default-d478b57b-zfxcf\" (UID: \"f5cb1a79-829e-4ec6-a0db-0ab664abd86d\") " pod="openshift-ingress/e2e-migrate-28d9404a-gw-openshift-default-d478b57b-zfxcf" Jul 26 16:04:34.508619 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:04:34.508601 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5cb1a79-829e-4ec6-a0db-0ab664abd86d-istiod-ca-cert\") pod \"e2e-migrate-28d9404a-gw-openshift-default-d478b57b-zfxcf\" (UID: \"f5cb1a79-829e-4ec6-a0db-0ab664abd86d\") " pod="openshift-ingress/e2e-migrate-28d9404a-gw-openshift-default-d478b57b-zfxcf" Jul 26 16:04:34.510218 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:04:34.510192 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5cb1a79-829e-4ec6-a0db-0ab664abd86d-istio-envoy\") pod \"e2e-migrate-28d9404a-gw-openshift-default-d478b57b-zfxcf\" (UID: \"f5cb1a79-829e-4ec6-a0db-0ab664abd86d\") " pod="openshift-ingress/e2e-migrate-28d9404a-gw-openshift-default-d478b57b-zfxcf" Jul 26 16:04:34.510461 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:04:34.510443 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5cb1a79-829e-4ec6-a0db-0ab664abd86d-istio-podinfo\") pod \"e2e-migrate-28d9404a-gw-openshift-default-d478b57b-zfxcf\" (UID: \"f5cb1a79-829e-4ec6-a0db-0ab664abd86d\") " pod="openshift-ingress/e2e-migrate-28d9404a-gw-openshift-default-d478b57b-zfxcf" Jul 26 16:04:34.515568 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:04:34.515541 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5cb1a79-829e-4ec6-a0db-0ab664abd86d-istio-token\") pod \"e2e-migrate-28d9404a-gw-openshift-default-d478b57b-zfxcf\" (UID: \"f5cb1a79-829e-4ec6-a0db-0ab664abd86d\") " pod="openshift-ingress/e2e-migrate-28d9404a-gw-openshift-default-d478b57b-zfxcf" Jul 26 16:04:34.515669 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:04:34.515651 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lp5l7\" (UniqueName: \"kubernetes.io/projected/f5cb1a79-829e-4ec6-a0db-0ab664abd86d-kube-api-access-lp5l7\") pod \"e2e-migrate-28d9404a-gw-openshift-default-d478b57b-zfxcf\" (UID: \"f5cb1a79-829e-4ec6-a0db-0ab664abd86d\") " pod="openshift-ingress/e2e-migrate-28d9404a-gw-openshift-default-d478b57b-zfxcf" Jul 26 16:04:34.542607 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:04:34.542578 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:04:34.670554 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:04:34.670467 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-28d9404a-gw-openshift-default-d478b57b-zfxcf" Jul 26 16:04:34.809048 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:04:34.809019 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-28d9404a-gw-openshift-default-d478b57b-zfxcf"] Jul 26 16:04:34.810435 ip-10-0-138-163 kubenswrapper[2577]: W0726 16:04:34.810389 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5cb1a79_829e_4ec6_a0db_0ab664abd86d.slice/crio-45e96ff0cd028f9e9e9d59fc72a0304384dba0ca81fe6ad1eea394dc772c940e WatchSource:0}: Error finding container 45e96ff0cd028f9e9e9d59fc72a0304384dba0ca81fe6ad1eea394dc772c940e: Status 404 returned error can't find the container with id 45e96ff0cd028f9e9e9d59fc72a0304384dba0ca81fe6ad1eea394dc772c940e Jul 26 16:04:34.816431 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:04:34.813257 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:04:34.816431 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:04:34.813415 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:04:34.816431 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:04:34.813511 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:04:35.561465 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:04:35.561429 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-28d9404a-gw-openshift-default-d478b57b-zfxcf" event={"ID":"f5cb1a79-829e-4ec6-a0db-0ab664abd86d","Type":"ContainerStarted","Data":"1e9f322071e11f722ffd6ef822b96554316f99f863c4a45359e9099896b3d082"} Jul 26 16:04:35.561883 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:04:35.561512 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-28d9404a-gw-openshift-default-d478b57b-zfxcf" event={"ID":"f5cb1a79-829e-4ec6-a0db-0ab664abd86d","Type":"ContainerStarted","Data":"45e96ff0cd028f9e9e9d59fc72a0304384dba0ca81fe6ad1eea394dc772c940e"} Jul 26 16:04:35.582376 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:04:35.582322 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-28d9404a-gw-openshift-default-d478b57b-zfxcf" podStartSLOduration=1.582306037 podStartE2EDuration="1.582306037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 16:04:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 16:04:35.579898759 +0000 UTC m=+1598.319252082" watchObservedRunningTime="2026-07-26 16:04:35.582306037 +0000 UTC m=+1598.321659346" Jul 26 16:04:35.670922 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:04:35.670889 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-28d9404a-gw-openshift-default-d478b57b-zfxcf" Jul 26 16:04:35.675527 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:04:35.675502 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-28d9404a-gw-openshift-default-d478b57b-zfxcf" Jul 26 16:04:36.567467 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:04:36.567434 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-28d9404a-gw-openshift-default-d478b57b-zfxcf" Jul 26 16:04:36.568310 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:04:36.568291 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-28d9404a-gw-openshift-default-d478b57b-zfxcf" Jul 26 16:04:37.031149 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:04:37.031113 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:04:37.132120 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:04:37.132077 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:04:37.634330 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:04:37.634286 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:04:37.729600 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:04:37.729561 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:04:37.982000 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:04:37.981906 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:04:38.580594 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:04:38.580565 2577 generic.go:356] "Generic (PLEG): container finished" podID="5d8e244f-8f0f-4b65-b21f-af312c5b9559" containerID="5336eb2ffff593830128b390d25271b4177e94647f41ca1de416b7160b477ab7" exitCode=0 Jul 26 16:04:38.580758 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:04:38.580642 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2b3f0867-openshift-default-6c4bd74488-hpp2l" event={"ID":"5d8e244f-8f0f-4b65-b21f-af312c5b9559","Type":"ContainerDied","Data":"5336eb2ffff593830128b390d25271b4177e94647f41ca1de416b7160b477ab7"} Jul 26 16:04:38.580758 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:04:38.580711 2577 scope.go:117] "RemoveContainer" containerID="854951d80d1a50de6a0a80d8e22a140050484f3137105859315fb31439ad21fa" Jul 26 16:04:38.628389 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:04:38.628364 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-2b3f0867-openshift-default-6c4bd74488-hpp2l" Jul 26 16:04:38.746473 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:04:38.746362 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d8e244f-8f0f-4b65-b21f-af312c5b9559-credential-socket\") pod \"5d8e244f-8f0f-4b65-b21f-af312c5b9559\" (UID: \"5d8e244f-8f0f-4b65-b21f-af312c5b9559\") " Jul 26 16:04:38.746473 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:04:38.746427 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d8e244f-8f0f-4b65-b21f-af312c5b9559-workload-socket\") pod \"5d8e244f-8f0f-4b65-b21f-af312c5b9559\" (UID: \"5d8e244f-8f0f-4b65-b21f-af312c5b9559\") " Jul 26 16:04:38.746473 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:04:38.746477 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d8e244f-8f0f-4b65-b21f-af312c5b9559-istio-data\") pod \"5d8e244f-8f0f-4b65-b21f-af312c5b9559\" (UID: \"5d8e244f-8f0f-4b65-b21f-af312c5b9559\") " Jul 26 16:04:38.747033 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:04:38.746552 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d8e244f-8f0f-4b65-b21f-af312c5b9559-istio-envoy\") pod \"5d8e244f-8f0f-4b65-b21f-af312c5b9559\" (UID: \"5d8e244f-8f0f-4b65-b21f-af312c5b9559\") " Jul 26 16:04:38.747033 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:04:38.746572 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d8e244f-8f0f-4b65-b21f-af312c5b9559-istiod-ca-cert\") pod \"5d8e244f-8f0f-4b65-b21f-af312c5b9559\" (UID: \"5d8e244f-8f0f-4b65-b21f-af312c5b9559\") " Jul 26 16:04:38.747033 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:04:38.746607 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vqzgh\" (UniqueName: \"kubernetes.io/projected/5d8e244f-8f0f-4b65-b21f-af312c5b9559-kube-api-access-vqzgh\") pod \"5d8e244f-8f0f-4b65-b21f-af312c5b9559\" (UID: \"5d8e244f-8f0f-4b65-b21f-af312c5b9559\") " Jul 26 16:04:38.747033 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:04:38.746645 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d8e244f-8f0f-4b65-b21f-af312c5b9559-istio-podinfo\") pod \"5d8e244f-8f0f-4b65-b21f-af312c5b9559\" (UID: \"5d8e244f-8f0f-4b65-b21f-af312c5b9559\") " Jul 26 16:04:38.747033 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:04:38.746663 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d8e244f-8f0f-4b65-b21f-af312c5b9559-istio-token\") pod \"5d8e244f-8f0f-4b65-b21f-af312c5b9559\" (UID: \"5d8e244f-8f0f-4b65-b21f-af312c5b9559\") " Jul 26 16:04:38.747033 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:04:38.746693 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d8e244f-8f0f-4b65-b21f-af312c5b9559-workload-certs\") pod \"5d8e244f-8f0f-4b65-b21f-af312c5b9559\" (UID: \"5d8e244f-8f0f-4b65-b21f-af312c5b9559\") " Jul 26 16:04:38.747033 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:04:38.746678 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d8e244f-8f0f-4b65-b21f-af312c5b9559-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5d8e244f-8f0f-4b65-b21f-af312c5b9559" (UID: "5d8e244f-8f0f-4b65-b21f-af312c5b9559"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:04:38.747033 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:04:38.746893 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d8e244f-8f0f-4b65-b21f-af312c5b9559-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5d8e244f-8f0f-4b65-b21f-af312c5b9559" (UID: "5d8e244f-8f0f-4b65-b21f-af312c5b9559"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:04:38.747033 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:04:38.746963 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d8e244f-8f0f-4b65-b21f-af312c5b9559-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5d8e244f-8f0f-4b65-b21f-af312c5b9559" (UID: "5d8e244f-8f0f-4b65-b21f-af312c5b9559"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 16:04:38.747476 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:04:38.747050 2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d8e244f-8f0f-4b65-b21f-af312c5b9559-credential-socket\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 16:04:38.747476 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:04:38.747074 2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d8e244f-8f0f-4b65-b21f-af312c5b9559-workload-socket\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 16:04:38.747476 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:04:38.747282 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d8e244f-8f0f-4b65-b21f-af312c5b9559-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5d8e244f-8f0f-4b65-b21f-af312c5b9559" (UID: "5d8e244f-8f0f-4b65-b21f-af312c5b9559"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:04:38.747476 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:04:38.747387 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d8e244f-8f0f-4b65-b21f-af312c5b9559-istio-data" (OuterVolumeSpecName: "istio-data") pod "5d8e244f-8f0f-4b65-b21f-af312c5b9559" (UID: "5d8e244f-8f0f-4b65-b21f-af312c5b9559"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:04:38.748926 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:04:38.748904 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d8e244f-8f0f-4b65-b21f-af312c5b9559-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5d8e244f-8f0f-4b65-b21f-af312c5b9559" (UID: "5d8e244f-8f0f-4b65-b21f-af312c5b9559"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:04:38.749042 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:04:38.748982 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d8e244f-8f0f-4b65-b21f-af312c5b9559-kube-api-access-vqzgh" (OuterVolumeSpecName: "kube-api-access-vqzgh") pod "5d8e244f-8f0f-4b65-b21f-af312c5b9559" (UID: "5d8e244f-8f0f-4b65-b21f-af312c5b9559"). InnerVolumeSpecName "kube-api-access-vqzgh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 16:04:38.749137 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:04:38.749118 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d8e244f-8f0f-4b65-b21f-af312c5b9559-istio-token" (OuterVolumeSpecName: "istio-token") pod "5d8e244f-8f0f-4b65-b21f-af312c5b9559" (UID: "5d8e244f-8f0f-4b65-b21f-af312c5b9559"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 16:04:38.749182 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:04:38.749134 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5d8e244f-8f0f-4b65-b21f-af312c5b9559-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5d8e244f-8f0f-4b65-b21f-af312c5b9559" (UID: "5d8e244f-8f0f-4b65-b21f-af312c5b9559"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 26 16:04:38.847651 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:04:38.847616 2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d8e244f-8f0f-4b65-b21f-af312c5b9559-istio-data\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 16:04:38.847651 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:04:38.847645 2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d8e244f-8f0f-4b65-b21f-af312c5b9559-istio-envoy\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 16:04:38.847651 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:04:38.847655 2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d8e244f-8f0f-4b65-b21f-af312c5b9559-istiod-ca-cert\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 16:04:38.847873 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:04:38.847663 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vqzgh\" (UniqueName: \"kubernetes.io/projected/5d8e244f-8f0f-4b65-b21f-af312c5b9559-kube-api-access-vqzgh\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 16:04:38.847873 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:04:38.847673 2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d8e244f-8f0f-4b65-b21f-af312c5b9559-istio-podinfo\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 16:04:38.847873 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:04:38.847681 2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d8e244f-8f0f-4b65-b21f-af312c5b9559-istio-token\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 16:04:38.847873 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:04:38.847689 2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d8e244f-8f0f-4b65-b21f-af312c5b9559-workload-certs\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 16:04:39.588022 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:04:39.587982 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2b3f0867-openshift-default-6c4bd74488-hpp2l" event={"ID":"5d8e244f-8f0f-4b65-b21f-af312c5b9559","Type":"ContainerDied","Data":"934c2022daf822f64ffc1bd2493499f915ffc745ab2feedd18ac120433673cbf"} Jul 26 16:04:39.588022 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:04:39.588000 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-2b3f0867-openshift-default-6c4bd74488-hpp2l" Jul 26 16:04:39.588290 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:04:39.588047 2577 scope.go:117] "RemoveContainer" containerID="5336eb2ffff593830128b390d25271b4177e94647f41ca1de416b7160b477ab7" Jul 26 16:04:39.613268 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:04:39.613235 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-2b3f0867-openshift-default-6c4bd74488-hpp2l"] Jul 26 16:04:39.617201 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:04:39.617173 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-2b3f0867-openshift-default-6c4bd74488-hpp2l"] Jul 26 16:04:39.787058 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:04:39.787021 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5d8e244f-8f0f-4b65-b21f-af312c5b9559" path="/var/lib/kubelet/pods/5d8e244f-8f0f-4b65-b21f-af312c5b9559/volumes" Jul 26 16:04:50.652081 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:04:50.652044 2577 generic.go:356] "Generic (PLEG): container finished" podID="f5cb1a79-829e-4ec6-a0db-0ab664abd86d" containerID="1e9f322071e11f722ffd6ef822b96554316f99f863c4a45359e9099896b3d082" exitCode=0 Jul 26 16:04:50.652496 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:04:50.652100 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-28d9404a-gw-openshift-default-d478b57b-zfxcf" event={"ID":"f5cb1a79-829e-4ec6-a0db-0ab664abd86d","Type":"ContainerDied","Data":"1e9f322071e11f722ffd6ef822b96554316f99f863c4a45359e9099896b3d082"} Jul 26 16:04:50.652496 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:04:50.652491 2577 scope.go:117] "RemoveContainer" containerID="1e9f322071e11f722ffd6ef822b96554316f99f863c4a45359e9099896b3d082" Jul 26 16:04:50.653469 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:04:50.653439 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:04:50.653531 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:04:50.653518 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:04:50.653568 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:04:50.653552 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:04:51.567833 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:04:51.567792 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-28d9404a-gw-openshift-default-d478b57b-zfxcf" Jul 26 16:04:51.662571 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:04:51.662534 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-28d9404a-gw-openshift-default-d478b57b-zfxcf" event={"ID":"f5cb1a79-829e-4ec6-a0db-0ab664abd86d","Type":"ContainerStarted","Data":"c86b63459af56ad043da515da033985ab922770e4e4aed2441922b7cdf4eee85"} Jul 26 16:04:51.670964 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:04:51.670934 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-28d9404a-gw-openshift-default-d478b57b-zfxcf" Jul 26 16:04:51.672137 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:04:51.672111 2577 patch_prober.go:28] interesting pod/e2e-migrate-28d9404a-gw-openshift-default-d478b57b-zfxcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Jul 26 16:04:51.672267 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:04:51.672156 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-28d9404a-gw-openshift-default-d478b57b-zfxcf" podUID="f5cb1a79-829e-4ec6-a0db-0ab664abd86d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Jul 26 16:04:52.674462 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:04:52.674422 2577 patch_prober.go:28] interesting pod/e2e-migrate-28d9404a-gw-openshift-default-d478b57b-zfxcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Jul 26 16:04:52.674947 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:04:52.674520 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-28d9404a-gw-openshift-default-d478b57b-zfxcf" podUID="f5cb1a79-829e-4ec6-a0db-0ab664abd86d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Jul 26 16:04:53.671267 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:04:53.671228 2577 patch_prober.go:28] interesting pod/e2e-migrate-28d9404a-gw-openshift-default-d478b57b-zfxcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Jul 26 16:04:53.671480 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:04:53.671303 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-28d9404a-gw-openshift-default-d478b57b-zfxcf" podUID="f5cb1a79-829e-4ec6-a0db-0ab664abd86d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Jul 26 16:04:54.671660 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:04:54.671616 2577 patch_prober.go:28] interesting pod/e2e-migrate-28d9404a-gw-openshift-default-d478b57b-zfxcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Jul 26 16:04:54.672076 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:04:54.671687 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-28d9404a-gw-openshift-default-d478b57b-zfxcf" podUID="f5cb1a79-829e-4ec6-a0db-0ab664abd86d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Jul 26 16:04:55.675518 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:04:55.675485 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-28d9404a-gw-openshift-default-d478b57b-zfxcf" Jul 26 16:04:55.675903 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:04:55.675751 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-28d9404a-gw-openshift-default-d478b57b-zfxcf" Jul 26 16:04:55.676387 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:04:55.676366 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-28d9404a-gw-openshift-default-d478b57b-zfxcf" Jul 26 16:05:12.518775 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:05:12.518730 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:05:14.302251 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:05:14.302219 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:05:17.093351 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:05:17.093314 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:05:17.230496 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:05:17.230459 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-28d9404a-gw-openshift-default-d478b57b-zfxcf"] Jul 26 16:05:17.230783 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:05:17.230749 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-28d9404a-gw-openshift-default-d478b57b-zfxcf" podUID="f5cb1a79-829e-4ec6-a0db-0ab664abd86d" containerName="istio-proxy" containerID="cri-o://c86b63459af56ad043da515da033985ab922770e4e4aed2441922b7cdf4eee85" gracePeriod=30 Jul 26 16:05:17.253261 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:05:17.253227 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:05:17.488489 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:05:17.488363 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:05:17.695808 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:05:17.695776 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:05:17.981199 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:05:17.981164 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-9823a2a6-openshift-default-f94df46d8-689nx"] Jul 26 16:05:17.981605 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:05:17.981591 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d8e244f-8f0f-4b65-b21f-af312c5b9559" containerName="istio-proxy" Jul 26 16:05:17.981656 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:05:17.981607 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d8e244f-8f0f-4b65-b21f-af312c5b9559" containerName="istio-proxy" Jul 26 16:05:17.981656 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:05:17.981614 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d8e244f-8f0f-4b65-b21f-af312c5b9559" containerName="istio-proxy" Jul 26 16:05:17.981656 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:05:17.981619 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d8e244f-8f0f-4b65-b21f-af312c5b9559" containerName="istio-proxy" Jul 26 16:05:17.981761 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:05:17.981696 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d8e244f-8f0f-4b65-b21f-af312c5b9559" containerName="istio-proxy" Jul 26 16:05:17.981761 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:05:17.981706 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d8e244f-8f0f-4b65-b21f-af312c5b9559" containerName="istio-proxy" Jul 26 16:05:17.984985 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:05:17.984967 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9823a2a6-openshift-default-f94df46d8-689nx" Jul 26 16:05:17.987750 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:05:17.987728 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-9823a2a6-openshift-default-dockercfg-stzr7\"" Jul 26 16:05:17.998619 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:05:17.998594 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-9823a2a6-openshift-default-f94df46d8-689nx"] Jul 26 16:05:18.013236 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:05:18.013206 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:05:18.111660 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:05:18.111627 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjjhm\" (UniqueName: \"kubernetes.io/projected/c3565705-0b18-4d74-8fe7-72f9eee5f688-kube-api-access-bjjhm\") pod \"e2e-ait-9823a2a6-openshift-default-f94df46d8-689nx\" (UID: \"c3565705-0b18-4d74-8fe7-72f9eee5f688\") " pod="openshift-ingress/e2e-ait-9823a2a6-openshift-default-f94df46d8-689nx" Jul 26 16:05:18.112160 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:05:18.111670 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3565705-0b18-4d74-8fe7-72f9eee5f688-credential-socket\") pod \"e2e-ait-9823a2a6-openshift-default-f94df46d8-689nx\" (UID: \"c3565705-0b18-4d74-8fe7-72f9eee5f688\") " pod="openshift-ingress/e2e-ait-9823a2a6-openshift-default-f94df46d8-689nx" Jul 26 16:05:18.112160 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:05:18.111697 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3565705-0b18-4d74-8fe7-72f9eee5f688-istio-data\") pod \"e2e-ait-9823a2a6-openshift-default-f94df46d8-689nx\" (UID: \"c3565705-0b18-4d74-8fe7-72f9eee5f688\") " pod="openshift-ingress/e2e-ait-9823a2a6-openshift-default-f94df46d8-689nx" Jul 26 16:05:18.112160 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:05:18.111757 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3565705-0b18-4d74-8fe7-72f9eee5f688-istio-podinfo\") pod \"e2e-ait-9823a2a6-openshift-default-f94df46d8-689nx\" (UID: \"c3565705-0b18-4d74-8fe7-72f9eee5f688\") " pod="openshift-ingress/e2e-ait-9823a2a6-openshift-default-f94df46d8-689nx" Jul 26 16:05:18.112160 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:05:18.111802 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3565705-0b18-4d74-8fe7-72f9eee5f688-istiod-ca-cert\") pod \"e2e-ait-9823a2a6-openshift-default-f94df46d8-689nx\" (UID: \"c3565705-0b18-4d74-8fe7-72f9eee5f688\") " pod="openshift-ingress/e2e-ait-9823a2a6-openshift-default-f94df46d8-689nx" Jul 26 16:05:18.112160 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:05:18.111910 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3565705-0b18-4d74-8fe7-72f9eee5f688-workload-socket\") pod \"e2e-ait-9823a2a6-openshift-default-f94df46d8-689nx\" (UID: \"c3565705-0b18-4d74-8fe7-72f9eee5f688\") " pod="openshift-ingress/e2e-ait-9823a2a6-openshift-default-f94df46d8-689nx" Jul 26 16:05:18.112160 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:05:18.111943 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3565705-0b18-4d74-8fe7-72f9eee5f688-workload-certs\") pod \"e2e-ait-9823a2a6-openshift-default-f94df46d8-689nx\" (UID: \"c3565705-0b18-4d74-8fe7-72f9eee5f688\") " pod="openshift-ingress/e2e-ait-9823a2a6-openshift-default-f94df46d8-689nx" Jul 26 16:05:18.112160 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:05:18.112025 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3565705-0b18-4d74-8fe7-72f9eee5f688-istio-envoy\") pod \"e2e-ait-9823a2a6-openshift-default-f94df46d8-689nx\" (UID: \"c3565705-0b18-4d74-8fe7-72f9eee5f688\") " pod="openshift-ingress/e2e-ait-9823a2a6-openshift-default-f94df46d8-689nx" Jul 26 16:05:18.112160 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:05:18.112064 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3565705-0b18-4d74-8fe7-72f9eee5f688-istio-token\") pod \"e2e-ait-9823a2a6-openshift-default-f94df46d8-689nx\" (UID: \"c3565705-0b18-4d74-8fe7-72f9eee5f688\") " pod="openshift-ingress/e2e-ait-9823a2a6-openshift-default-f94df46d8-689nx" Jul 26 16:05:18.213427 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:05:18.213368 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3565705-0b18-4d74-8fe7-72f9eee5f688-workload-socket\") pod \"e2e-ait-9823a2a6-openshift-default-f94df46d8-689nx\" (UID: \"c3565705-0b18-4d74-8fe7-72f9eee5f688\") " pod="openshift-ingress/e2e-ait-9823a2a6-openshift-default-f94df46d8-689nx" Jul 26 16:05:18.213427 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:05:18.213429 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3565705-0b18-4d74-8fe7-72f9eee5f688-workload-certs\") pod \"e2e-ait-9823a2a6-openshift-default-f94df46d8-689nx\" (UID: \"c3565705-0b18-4d74-8fe7-72f9eee5f688\") " pod="openshift-ingress/e2e-ait-9823a2a6-openshift-default-f94df46d8-689nx" Jul 26 16:05:18.213671 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:05:18.213475 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3565705-0b18-4d74-8fe7-72f9eee5f688-istio-envoy\") pod \"e2e-ait-9823a2a6-openshift-default-f94df46d8-689nx\" (UID: \"c3565705-0b18-4d74-8fe7-72f9eee5f688\") " pod="openshift-ingress/e2e-ait-9823a2a6-openshift-default-f94df46d8-689nx" Jul 26 16:05:18.213671 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:05:18.213497 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3565705-0b18-4d74-8fe7-72f9eee5f688-istio-token\") pod \"e2e-ait-9823a2a6-openshift-default-f94df46d8-689nx\" (UID: \"c3565705-0b18-4d74-8fe7-72f9eee5f688\") " pod="openshift-ingress/e2e-ait-9823a2a6-openshift-default-f94df46d8-689nx" Jul 26 16:05:18.213671 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:05:18.213536 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjjhm\" (UniqueName: \"kubernetes.io/projected/c3565705-0b18-4d74-8fe7-72f9eee5f688-kube-api-access-bjjhm\") pod \"e2e-ait-9823a2a6-openshift-default-f94df46d8-689nx\" (UID: \"c3565705-0b18-4d74-8fe7-72f9eee5f688\") " pod="openshift-ingress/e2e-ait-9823a2a6-openshift-default-f94df46d8-689nx" Jul 26 16:05:18.213671 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:05:18.213567 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3565705-0b18-4d74-8fe7-72f9eee5f688-credential-socket\") pod \"e2e-ait-9823a2a6-openshift-default-f94df46d8-689nx\" (UID: \"c3565705-0b18-4d74-8fe7-72f9eee5f688\") " pod="openshift-ingress/e2e-ait-9823a2a6-openshift-default-f94df46d8-689nx" Jul 26 16:05:18.213671 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:05:18.213588 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3565705-0b18-4d74-8fe7-72f9eee5f688-istio-data\") pod \"e2e-ait-9823a2a6-openshift-default-f94df46d8-689nx\" (UID: \"c3565705-0b18-4d74-8fe7-72f9eee5f688\") " pod="openshift-ingress/e2e-ait-9823a2a6-openshift-default-f94df46d8-689nx" Jul 26 16:05:18.213671 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:05:18.213617 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3565705-0b18-4d74-8fe7-72f9eee5f688-istio-podinfo\") pod \"e2e-ait-9823a2a6-openshift-default-f94df46d8-689nx\" (UID: \"c3565705-0b18-4d74-8fe7-72f9eee5f688\") " pod="openshift-ingress/e2e-ait-9823a2a6-openshift-default-f94df46d8-689nx" Jul 26 16:05:18.213671 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:05:18.213641 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3565705-0b18-4d74-8fe7-72f9eee5f688-istiod-ca-cert\") pod \"e2e-ait-9823a2a6-openshift-default-f94df46d8-689nx\" (UID: \"c3565705-0b18-4d74-8fe7-72f9eee5f688\") " pod="openshift-ingress/e2e-ait-9823a2a6-openshift-default-f94df46d8-689nx" Jul 26 16:05:18.214095 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:05:18.213804 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3565705-0b18-4d74-8fe7-72f9eee5f688-workload-socket\") pod \"e2e-ait-9823a2a6-openshift-default-f94df46d8-689nx\" (UID: \"c3565705-0b18-4d74-8fe7-72f9eee5f688\") " pod="openshift-ingress/e2e-ait-9823a2a6-openshift-default-f94df46d8-689nx" Jul 26 16:05:18.214095 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:05:18.214000 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3565705-0b18-4d74-8fe7-72f9eee5f688-istio-data\") pod \"e2e-ait-9823a2a6-openshift-default-f94df46d8-689nx\" (UID: \"c3565705-0b18-4d74-8fe7-72f9eee5f688\") " pod="openshift-ingress/e2e-ait-9823a2a6-openshift-default-f94df46d8-689nx" Jul 26 16:05:18.214095 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:05:18.214036 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3565705-0b18-4d74-8fe7-72f9eee5f688-credential-socket\") pod \"e2e-ait-9823a2a6-openshift-default-f94df46d8-689nx\" (UID: \"c3565705-0b18-4d74-8fe7-72f9eee5f688\") " pod="openshift-ingress/e2e-ait-9823a2a6-openshift-default-f94df46d8-689nx" Jul 26 16:05:18.214386 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:05:18.214359 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3565705-0b18-4d74-8fe7-72f9eee5f688-workload-certs\") pod \"e2e-ait-9823a2a6-openshift-default-f94df46d8-689nx\" (UID: \"c3565705-0b18-4d74-8fe7-72f9eee5f688\") " pod="openshift-ingress/e2e-ait-9823a2a6-openshift-default-f94df46d8-689nx" Jul 26 16:05:18.214526 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:05:18.214462 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3565705-0b18-4d74-8fe7-72f9eee5f688-istiod-ca-cert\") pod \"e2e-ait-9823a2a6-openshift-default-f94df46d8-689nx\" (UID: \"c3565705-0b18-4d74-8fe7-72f9eee5f688\") " pod="openshift-ingress/e2e-ait-9823a2a6-openshift-default-f94df46d8-689nx" Jul 26 16:05:18.215887 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:05:18.215864 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3565705-0b18-4d74-8fe7-72f9eee5f688-istio-envoy\") pod \"e2e-ait-9823a2a6-openshift-default-f94df46d8-689nx\" (UID: \"c3565705-0b18-4d74-8fe7-72f9eee5f688\") " pod="openshift-ingress/e2e-ait-9823a2a6-openshift-default-f94df46d8-689nx" Jul 26 16:05:18.216101 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:05:18.216082 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3565705-0b18-4d74-8fe7-72f9eee5f688-istio-podinfo\") pod \"e2e-ait-9823a2a6-openshift-default-f94df46d8-689nx\" (UID: \"c3565705-0b18-4d74-8fe7-72f9eee5f688\") " pod="openshift-ingress/e2e-ait-9823a2a6-openshift-default-f94df46d8-689nx" Jul 26 16:05:18.221259 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:05:18.221225 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3565705-0b18-4d74-8fe7-72f9eee5f688-istio-token\") pod \"e2e-ait-9823a2a6-openshift-default-f94df46d8-689nx\" (UID: \"c3565705-0b18-4d74-8fe7-72f9eee5f688\") " pod="openshift-ingress/e2e-ait-9823a2a6-openshift-default-f94df46d8-689nx" Jul 26 16:05:18.221422 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:05:18.221380 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjjhm\" (UniqueName: \"kubernetes.io/projected/c3565705-0b18-4d74-8fe7-72f9eee5f688-kube-api-access-bjjhm\") pod \"e2e-ait-9823a2a6-openshift-default-f94df46d8-689nx\" (UID: \"c3565705-0b18-4d74-8fe7-72f9eee5f688\") " pod="openshift-ingress/e2e-ait-9823a2a6-openshift-default-f94df46d8-689nx" Jul 26 16:05:18.237275 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:05:18.237151 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:05:18.296079 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:05:18.296049 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9823a2a6-openshift-default-f94df46d8-689nx" Jul 26 16:05:18.437491 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:05:18.437467 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-9823a2a6-openshift-default-f94df46d8-689nx"] Jul 26 16:05:18.440328 ip-10-0-138-163 kubenswrapper[2577]: W0726 16:05:18.440299 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3565705_0b18_4d74_8fe7_72f9eee5f688.slice/crio-f70dae6ac79a74c5ff715abfaa788b0bc35725a0f6303c667a4bf8c9c2f422d9 WatchSource:0}: Error finding container f70dae6ac79a74c5ff715abfaa788b0bc35725a0f6303c667a4bf8c9c2f422d9: Status 404 returned error can't find the container with id f70dae6ac79a74c5ff715abfaa788b0bc35725a0f6303c667a4bf8c9c2f422d9 Jul 26 16:05:18.442499 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:05:18.442467 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:05:18.442603 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:05:18.442546 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:05:18.442603 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:05:18.442592 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:05:18.824737 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:05:18.824701 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9823a2a6-openshift-default-f94df46d8-689nx" event={"ID":"c3565705-0b18-4d74-8fe7-72f9eee5f688","Type":"ContainerStarted","Data":"29ca17846a5f2723db6f857d0047b127a885d77492e79d2f8d40d8ba1877eed1"} Jul 26 16:05:18.824921 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:05:18.824763 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9823a2a6-openshift-default-f94df46d8-689nx" event={"ID":"c3565705-0b18-4d74-8fe7-72f9eee5f688","Type":"ContainerStarted","Data":"f70dae6ac79a74c5ff715abfaa788b0bc35725a0f6303c667a4bf8c9c2f422d9"} Jul 26 16:05:18.845684 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:05:18.845628 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-9823a2a6-openshift-default-f94df46d8-689nx" podStartSLOduration=1.845610387 podStartE2EDuration="1.845610387s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 16:05:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 16:05:18.843944128 +0000 UTC m=+1641.583297465" watchObservedRunningTime="2026-07-26 16:05:18.845610387 +0000 UTC m=+1641.584963696" Jul 26 16:05:19.296500 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:05:19.296463 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-9823a2a6-openshift-default-f94df46d8-689nx" Jul 26 16:05:19.301521 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:05:19.301495 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-9823a2a6-openshift-default-f94df46d8-689nx" Jul 26 16:05:19.830030 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:05:19.830002 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-9823a2a6-openshift-default-f94df46d8-689nx" Jul 26 16:05:19.831167 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:05:19.831141 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-9823a2a6-openshift-default-f94df46d8-689nx" Jul 26 16:05:20.182752 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:05:20.182663 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:05:20.327191 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:05:20.327146 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:05:20.473282 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:05:20.473189 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:05:20.837195 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:05:20.837162 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:05:20.935021 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:05:20.934990 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:05:21.017155 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:05:21.017119 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:05:22.485353 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:05:22.485327 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-28d9404a-gw-openshift-default-d478b57b-zfxcf" Jul 26 16:05:22.552633 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:05:22.552601 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5cb1a79-829e-4ec6-a0db-0ab664abd86d-istiod-ca-cert\") pod \"f5cb1a79-829e-4ec6-a0db-0ab664abd86d\" (UID: \"f5cb1a79-829e-4ec6-a0db-0ab664abd86d\") " Jul 26 16:05:22.552633 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:05:22.552635 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5cb1a79-829e-4ec6-a0db-0ab664abd86d-workload-socket\") pod \"f5cb1a79-829e-4ec6-a0db-0ab664abd86d\" (UID: \"f5cb1a79-829e-4ec6-a0db-0ab664abd86d\") " Jul 26 16:05:22.552887 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:05:22.552652 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5cb1a79-829e-4ec6-a0db-0ab664abd86d-workload-certs\") pod \"f5cb1a79-829e-4ec6-a0db-0ab664abd86d\" (UID: \"f5cb1a79-829e-4ec6-a0db-0ab664abd86d\") " Jul 26 16:05:22.552887 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:05:22.552676 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5cb1a79-829e-4ec6-a0db-0ab664abd86d-istio-podinfo\") pod \"f5cb1a79-829e-4ec6-a0db-0ab664abd86d\" (UID: \"f5cb1a79-829e-4ec6-a0db-0ab664abd86d\") " Jul 26 16:05:22.552887 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:05:22.552700 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5cb1a79-829e-4ec6-a0db-0ab664abd86d-credential-socket\") pod \"f5cb1a79-829e-4ec6-a0db-0ab664abd86d\" (UID: \"f5cb1a79-829e-4ec6-a0db-0ab664abd86d\") " Jul 26 16:05:22.552887 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:05:22.552726 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5cb1a79-829e-4ec6-a0db-0ab664abd86d-istio-token\") pod \"f5cb1a79-829e-4ec6-a0db-0ab664abd86d\" (UID: \"f5cb1a79-829e-4ec6-a0db-0ab664abd86d\") " Jul 26 16:05:22.552887 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:05:22.552776 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5cb1a79-829e-4ec6-a0db-0ab664abd86d-istio-data\") pod \"f5cb1a79-829e-4ec6-a0db-0ab664abd86d\" (UID: \"f5cb1a79-829e-4ec6-a0db-0ab664abd86d\") " Jul 26 16:05:22.552887 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:05:22.552818 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lp5l7\" (UniqueName: \"kubernetes.io/projected/f5cb1a79-829e-4ec6-a0db-0ab664abd86d-kube-api-access-lp5l7\") pod \"f5cb1a79-829e-4ec6-a0db-0ab664abd86d\" (UID: \"f5cb1a79-829e-4ec6-a0db-0ab664abd86d\") " Jul 26 16:05:22.552887 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:05:22.552875 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5cb1a79-829e-4ec6-a0db-0ab664abd86d-istio-envoy\") pod \"f5cb1a79-829e-4ec6-a0db-0ab664abd86d\" (UID: \"f5cb1a79-829e-4ec6-a0db-0ab664abd86d\") " Jul 26 16:05:22.553274 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:05:22.552949 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5cb1a79-829e-4ec6-a0db-0ab664abd86d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f5cb1a79-829e-4ec6-a0db-0ab664abd86d" (UID: "f5cb1a79-829e-4ec6-a0db-0ab664abd86d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:05:22.553274 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:05:22.552968 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5cb1a79-829e-4ec6-a0db-0ab664abd86d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f5cb1a79-829e-4ec6-a0db-0ab664abd86d" (UID: "f5cb1a79-829e-4ec6-a0db-0ab664abd86d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:05:22.553274 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:05:22.552984 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5cb1a79-829e-4ec6-a0db-0ab664abd86d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f5cb1a79-829e-4ec6-a0db-0ab664abd86d" (UID: "f5cb1a79-829e-4ec6-a0db-0ab664abd86d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:05:22.553274 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:05:22.553082 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5cb1a79-829e-4ec6-a0db-0ab664abd86d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f5cb1a79-829e-4ec6-a0db-0ab664abd86d" (UID: "f5cb1a79-829e-4ec6-a0db-0ab664abd86d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 16:05:22.553517 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:05:22.553333 2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5cb1a79-829e-4ec6-a0db-0ab664abd86d-istiod-ca-cert\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 16:05:22.553517 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:05:22.553353 2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5cb1a79-829e-4ec6-a0db-0ab664abd86d-workload-socket\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 16:05:22.553517 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:05:22.553365 2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5cb1a79-829e-4ec6-a0db-0ab664abd86d-workload-certs\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 16:05:22.553517 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:05:22.553379 2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5cb1a79-829e-4ec6-a0db-0ab664abd86d-credential-socket\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 16:05:22.554049 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:05:22.554019 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5cb1a79-829e-4ec6-a0db-0ab664abd86d-istio-data" (OuterVolumeSpecName: "istio-data") pod "f5cb1a79-829e-4ec6-a0db-0ab664abd86d" (UID: "f5cb1a79-829e-4ec6-a0db-0ab664abd86d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:05:22.555127 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:05:22.555104 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5cb1a79-829e-4ec6-a0db-0ab664abd86d-istio-token" (OuterVolumeSpecName: "istio-token") pod "f5cb1a79-829e-4ec6-a0db-0ab664abd86d" (UID: "f5cb1a79-829e-4ec6-a0db-0ab664abd86d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 16:05:22.555213 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:05:22.555143 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5cb1a79-829e-4ec6-a0db-0ab664abd86d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f5cb1a79-829e-4ec6-a0db-0ab664abd86d" (UID: "f5cb1a79-829e-4ec6-a0db-0ab664abd86d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:05:22.555430 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:05:22.555387 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f5cb1a79-829e-4ec6-a0db-0ab664abd86d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f5cb1a79-829e-4ec6-a0db-0ab664abd86d" (UID: "f5cb1a79-829e-4ec6-a0db-0ab664abd86d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 26 16:05:22.555430 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:05:22.555392 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5cb1a79-829e-4ec6-a0db-0ab664abd86d-kube-api-access-lp5l7" (OuterVolumeSpecName: "kube-api-access-lp5l7") pod "f5cb1a79-829e-4ec6-a0db-0ab664abd86d" (UID: "f5cb1a79-829e-4ec6-a0db-0ab664abd86d"). InnerVolumeSpecName "kube-api-access-lp5l7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 16:05:22.654017 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:05:22.653981 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lp5l7\" (UniqueName: \"kubernetes.io/projected/f5cb1a79-829e-4ec6-a0db-0ab664abd86d-kube-api-access-lp5l7\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 16:05:22.654017 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:05:22.654010 2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5cb1a79-829e-4ec6-a0db-0ab664abd86d-istio-envoy\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 16:05:22.654017 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:05:22.654020 2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5cb1a79-829e-4ec6-a0db-0ab664abd86d-istio-podinfo\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 16:05:22.654017 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:05:22.654029 2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5cb1a79-829e-4ec6-a0db-0ab664abd86d-istio-token\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 16:05:22.654281 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:05:22.654037 2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5cb1a79-829e-4ec6-a0db-0ab664abd86d-istio-data\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 16:05:22.848692 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:05:22.848613 2577 generic.go:356] "Generic (PLEG): container finished" podID="f5cb1a79-829e-4ec6-a0db-0ab664abd86d" containerID="c86b63459af56ad043da515da033985ab922770e4e4aed2441922b7cdf4eee85" exitCode=0 Jul 26 16:05:22.848692 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:05:22.848663 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-28d9404a-gw-openshift-default-d478b57b-zfxcf" Jul 26 16:05:22.848881 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:05:22.848695 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-28d9404a-gw-openshift-default-d478b57b-zfxcf" event={"ID":"f5cb1a79-829e-4ec6-a0db-0ab664abd86d","Type":"ContainerDied","Data":"c86b63459af56ad043da515da033985ab922770e4e4aed2441922b7cdf4eee85"} Jul 26 16:05:22.848881 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:05:22.848748 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-28d9404a-gw-openshift-default-d478b57b-zfxcf" event={"ID":"f5cb1a79-829e-4ec6-a0db-0ab664abd86d","Type":"ContainerDied","Data":"45e96ff0cd028f9e9e9d59fc72a0304384dba0ca81fe6ad1eea394dc772c940e"} Jul 26 16:05:22.848881 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:05:22.848771 2577 scope.go:117] "RemoveContainer" containerID="c86b63459af56ad043da515da033985ab922770e4e4aed2441922b7cdf4eee85" Jul 26 16:05:22.858321 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:05:22.858306 2577 scope.go:117] "RemoveContainer" containerID="1e9f322071e11f722ffd6ef822b96554316f99f863c4a45359e9099896b3d082" Jul 26 16:05:22.866682 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:05:22.866662 2577 scope.go:117] "RemoveContainer" containerID="c86b63459af56ad043da515da033985ab922770e4e4aed2441922b7cdf4eee85" Jul 26 16:05:22.866934 ip-10-0-138-163 kubenswrapper[2577]: E0726 16:05:22.866912 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c86b63459af56ad043da515da033985ab922770e4e4aed2441922b7cdf4eee85\": container with ID starting with c86b63459af56ad043da515da033985ab922770e4e4aed2441922b7cdf4eee85 not found: ID does not exist" containerID="c86b63459af56ad043da515da033985ab922770e4e4aed2441922b7cdf4eee85" Jul 26 16:05:22.866999 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:05:22.866942 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c86b63459af56ad043da515da033985ab922770e4e4aed2441922b7cdf4eee85"} err="failed to get container status \"c86b63459af56ad043da515da033985ab922770e4e4aed2441922b7cdf4eee85\": rpc error: code = NotFound desc = could not find container \"c86b63459af56ad043da515da033985ab922770e4e4aed2441922b7cdf4eee85\": container with ID starting with c86b63459af56ad043da515da033985ab922770e4e4aed2441922b7cdf4eee85 not found: ID does not exist" Jul 26 16:05:22.866999 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:05:22.866958 2577 scope.go:117] "RemoveContainer" containerID="1e9f322071e11f722ffd6ef822b96554316f99f863c4a45359e9099896b3d082" Jul 26 16:05:22.867205 ip-10-0-138-163 kubenswrapper[2577]: E0726 16:05:22.867188 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e9f322071e11f722ffd6ef822b96554316f99f863c4a45359e9099896b3d082\": container with ID starting with 1e9f322071e11f722ffd6ef822b96554316f99f863c4a45359e9099896b3d082 not found: ID does not exist" containerID="1e9f322071e11f722ffd6ef822b96554316f99f863c4a45359e9099896b3d082" Jul 26 16:05:22.867250 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:05:22.867210 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e9f322071e11f722ffd6ef822b96554316f99f863c4a45359e9099896b3d082"} err="failed to get container status \"1e9f322071e11f722ffd6ef822b96554316f99f863c4a45359e9099896b3d082\": rpc error: code = NotFound desc = could not find container \"1e9f322071e11f722ffd6ef822b96554316f99f863c4a45359e9099896b3d082\": container with ID starting with 1e9f322071e11f722ffd6ef822b96554316f99f863c4a45359e9099896b3d082 not found: ID does not exist" Jul 26 16:05:22.870845 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:05:22.870824 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-28d9404a-gw-openshift-default-d478b57b-zfxcf"] Jul 26 16:05:22.876611 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:05:22.876589 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-28d9404a-gw-openshift-default-d478b57b-zfxcf"] Jul 26 16:05:23.787320 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:05:23.787285 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f5cb1a79-829e-4ec6-a0db-0ab664abd86d" path="/var/lib/kubelet/pods/f5cb1a79-829e-4ec6-a0db-0ab664abd86d/volumes" Jul 26 16:05:26.294930 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:05:26.294891 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:05:26.427983 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:05:26.427952 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:05:34.831906 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:05:34.831868 2577 patch_prober.go:28] interesting pod/e2e-ait-9823a2a6-openshift-default-f94df46d8-689nx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Jul 26 16:05:34.832377 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:05:34.831940 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-9823a2a6-openshift-default-f94df46d8-689nx" podUID="c3565705-0b18-4d74-8fe7-72f9eee5f688" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Jul 26 16:05:34.918359 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:05:34.918326 2577 generic.go:356] "Generic (PLEG): container finished" podID="c3565705-0b18-4d74-8fe7-72f9eee5f688" containerID="29ca17846a5f2723db6f857d0047b127a885d77492e79d2f8d40d8ba1877eed1" exitCode=0 Jul 26 16:05:34.918531 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:05:34.918414 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9823a2a6-openshift-default-f94df46d8-689nx" event={"ID":"c3565705-0b18-4d74-8fe7-72f9eee5f688","Type":"ContainerDied","Data":"29ca17846a5f2723db6f857d0047b127a885d77492e79d2f8d40d8ba1877eed1"} Jul 26 16:05:34.918828 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:05:34.918813 2577 scope.go:117] "RemoveContainer" containerID="29ca17846a5f2723db6f857d0047b127a885d77492e79d2f8d40d8ba1877eed1" Jul 26 16:05:34.919797 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:05:34.919769 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:05:34.919910 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:05:34.919836 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:05:34.919910 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:05:34.919865 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:05:35.925518 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:05:35.925487 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9823a2a6-openshift-default-f94df46d8-689nx" event={"ID":"c3565705-0b18-4d74-8fe7-72f9eee5f688","Type":"ContainerStarted","Data":"0403a4b8b9b1d41c615409d96039bdb49f35f848dc2b6f5e969ea93384dff12e"} Jul 26 16:05:36.296580 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:05:36.296543 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-9823a2a6-openshift-default-f94df46d8-689nx" Jul 26 16:05:36.296782 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:05:36.296762 2577 patch_prober.go:28] interesting pod/e2e-ait-9823a2a6-openshift-default-f94df46d8-689nx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Jul 26 16:05:36.296836 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:05:36.296818 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9823a2a6-openshift-default-f94df46d8-689nx" podUID="c3565705-0b18-4d74-8fe7-72f9eee5f688" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Jul 26 16:05:37.297460 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:05:37.297425 2577 patch_prober.go:28] interesting pod/e2e-ait-9823a2a6-openshift-default-f94df46d8-689nx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Jul 26 16:05:37.297857 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:05:37.297486 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9823a2a6-openshift-default-f94df46d8-689nx" podUID="c3565705-0b18-4d74-8fe7-72f9eee5f688" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Jul 26 16:05:38.296813 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:05:38.296776 2577 patch_prober.go:28] interesting pod/e2e-ait-9823a2a6-openshift-default-f94df46d8-689nx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Jul 26 16:05:38.296990 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:05:38.296828 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9823a2a6-openshift-default-f94df46d8-689nx" podUID="c3565705-0b18-4d74-8fe7-72f9eee5f688" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Jul 26 16:05:39.297033 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:05:39.296999 2577 patch_prober.go:28] interesting pod/e2e-ait-9823a2a6-openshift-default-f94df46d8-689nx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Jul 26 16:05:39.297463 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:05:39.297056 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9823a2a6-openshift-default-f94df46d8-689nx" podUID="c3565705-0b18-4d74-8fe7-72f9eee5f688" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Jul 26 16:05:40.301069 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:05:40.301033 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-9823a2a6-openshift-default-f94df46d8-689nx" Jul 26 16:05:40.301574 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:05:40.301423 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-9823a2a6-openshift-default-f94df46d8-689nx" Jul 26 16:05:40.302018 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:05:40.302001 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-9823a2a6-openshift-default-f94df46d8-689nx" Jul 26 16:05:56.325559 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:05:56.325523 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:05:56.425889 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:05:56.425855 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:05:56.532072 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:05:56.532036 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:06:00.977375 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:00.977339 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:06:02.097627 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:02.097591 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-9823a2a6-openshift-default-f94df46d8-689nx"] Jul 26 16:06:02.098110 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:02.097864 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-9823a2a6-openshift-default-f94df46d8-689nx" podUID="c3565705-0b18-4d74-8fe7-72f9eee5f688" containerName="istio-proxy" containerID="cri-o://0403a4b8b9b1d41c615409d96039bdb49f35f848dc2b6f5e969ea93384dff12e" gracePeriod=30 Jul 26 16:06:02.113845 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:02.113808 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:06:02.326478 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:02.326447 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:06:02.542840 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:02.542807 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:06:02.806009 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:02.805928 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-c17951b0-openshift-default-d64755cfb-gk565"] Jul 26 16:06:02.806493 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:02.806477 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5cb1a79-829e-4ec6-a0db-0ab664abd86d" containerName="istio-proxy" Jul 26 16:06:02.806555 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:02.806495 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5cb1a79-829e-4ec6-a0db-0ab664abd86d" containerName="istio-proxy" Jul 26 16:06:02.806555 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:02.806508 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5cb1a79-829e-4ec6-a0db-0ab664abd86d" containerName="istio-proxy" Jul 26 16:06:02.806555 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:02.806513 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5cb1a79-829e-4ec6-a0db-0ab664abd86d" containerName="istio-proxy" Jul 26 16:06:02.806653 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:02.806589 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5cb1a79-829e-4ec6-a0db-0ab664abd86d" containerName="istio-proxy" Jul 26 16:06:02.809753 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:02.809737 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-c17951b0-openshift-default-d64755cfb-gk565" Jul 26 16:06:02.812136 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:02.812117 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-c17951b0-openshift-default-dockercfg-76wn7\"" Jul 26 16:06:02.820243 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:02.820223 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-c17951b0-openshift-default-d64755cfb-gk565"] Jul 26 16:06:02.889136 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:02.889101 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:06:02.925532 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:02.925497 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6t5h\" (UniqueName: \"kubernetes.io/projected/4089403a-fc89-4e1c-ab2f-1e429c0467bd-kube-api-access-l6t5h\") pod \"e2e-derive-c17951b0-openshift-default-d64755cfb-gk565\" (UID: \"4089403a-fc89-4e1c-ab2f-1e429c0467bd\") " pod="openshift-ingress/e2e-derive-c17951b0-openshift-default-d64755cfb-gk565" Jul 26 16:06:02.925751 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:02.925541 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4089403a-fc89-4e1c-ab2f-1e429c0467bd-workload-certs\") pod \"e2e-derive-c17951b0-openshift-default-d64755cfb-gk565\" (UID: \"4089403a-fc89-4e1c-ab2f-1e429c0467bd\") " pod="openshift-ingress/e2e-derive-c17951b0-openshift-default-d64755cfb-gk565" Jul 26 16:06:02.925751 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:02.925562 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4089403a-fc89-4e1c-ab2f-1e429c0467bd-istio-token\") pod \"e2e-derive-c17951b0-openshift-default-d64755cfb-gk565\" (UID: \"4089403a-fc89-4e1c-ab2f-1e429c0467bd\") " pod="openshift-ingress/e2e-derive-c17951b0-openshift-default-d64755cfb-gk565" Jul 26 16:06:02.925751 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:02.925619 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4089403a-fc89-4e1c-ab2f-1e429c0467bd-istiod-ca-cert\") pod \"e2e-derive-c17951b0-openshift-default-d64755cfb-gk565\" (UID: \"4089403a-fc89-4e1c-ab2f-1e429c0467bd\") " pod="openshift-ingress/e2e-derive-c17951b0-openshift-default-d64755cfb-gk565" Jul 26 16:06:02.925751 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:02.925671 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4089403a-fc89-4e1c-ab2f-1e429c0467bd-istio-data\") pod \"e2e-derive-c17951b0-openshift-default-d64755cfb-gk565\" (UID: \"4089403a-fc89-4e1c-ab2f-1e429c0467bd\") " pod="openshift-ingress/e2e-derive-c17951b0-openshift-default-d64755cfb-gk565" Jul 26 16:06:02.925751 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:02.925706 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4089403a-fc89-4e1c-ab2f-1e429c0467bd-workload-socket\") pod \"e2e-derive-c17951b0-openshift-default-d64755cfb-gk565\" (UID: \"4089403a-fc89-4e1c-ab2f-1e429c0467bd\") " pod="openshift-ingress/e2e-derive-c17951b0-openshift-default-d64755cfb-gk565" Jul 26 16:06:02.926011 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:02.925777 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4089403a-fc89-4e1c-ab2f-1e429c0467bd-credential-socket\") pod \"e2e-derive-c17951b0-openshift-default-d64755cfb-gk565\" (UID: \"4089403a-fc89-4e1c-ab2f-1e429c0467bd\") " pod="openshift-ingress/e2e-derive-c17951b0-openshift-default-d64755cfb-gk565" Jul 26 16:06:02.926011 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:02.925866 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4089403a-fc89-4e1c-ab2f-1e429c0467bd-istio-envoy\") pod \"e2e-derive-c17951b0-openshift-default-d64755cfb-gk565\" (UID: \"4089403a-fc89-4e1c-ab2f-1e429c0467bd\") " pod="openshift-ingress/e2e-derive-c17951b0-openshift-default-d64755cfb-gk565" Jul 26 16:06:02.926011 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:02.925894 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4089403a-fc89-4e1c-ab2f-1e429c0467bd-istio-podinfo\") pod \"e2e-derive-c17951b0-openshift-default-d64755cfb-gk565\" (UID: \"4089403a-fc89-4e1c-ab2f-1e429c0467bd\") " pod="openshift-ingress/e2e-derive-c17951b0-openshift-default-d64755cfb-gk565" Jul 26 16:06:02.990640 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:02.990608 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:06:03.027016 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:03.026982 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4089403a-fc89-4e1c-ab2f-1e429c0467bd-istio-data\") pod \"e2e-derive-c17951b0-openshift-default-d64755cfb-gk565\" (UID: \"4089403a-fc89-4e1c-ab2f-1e429c0467bd\") " pod="openshift-ingress/e2e-derive-c17951b0-openshift-default-d64755cfb-gk565" Jul 26 16:06:03.027016 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:03.027018 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4089403a-fc89-4e1c-ab2f-1e429c0467bd-workload-socket\") pod \"e2e-derive-c17951b0-openshift-default-d64755cfb-gk565\" (UID: \"4089403a-fc89-4e1c-ab2f-1e429c0467bd\") " pod="openshift-ingress/e2e-derive-c17951b0-openshift-default-d64755cfb-gk565" Jul 26 16:06:03.027241 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:03.027148 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4089403a-fc89-4e1c-ab2f-1e429c0467bd-credential-socket\") pod \"e2e-derive-c17951b0-openshift-default-d64755cfb-gk565\" (UID: \"4089403a-fc89-4e1c-ab2f-1e429c0467bd\") " pod="openshift-ingress/e2e-derive-c17951b0-openshift-default-d64755cfb-gk565" Jul 26 16:06:03.027241 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:03.027208 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4089403a-fc89-4e1c-ab2f-1e429c0467bd-istio-envoy\") pod \"e2e-derive-c17951b0-openshift-default-d64755cfb-gk565\" (UID: \"4089403a-fc89-4e1c-ab2f-1e429c0467bd\") " pod="openshift-ingress/e2e-derive-c17951b0-openshift-default-d64755cfb-gk565" Jul 26 16:06:03.027241 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:03.027230 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4089403a-fc89-4e1c-ab2f-1e429c0467bd-istio-podinfo\") pod \"e2e-derive-c17951b0-openshift-default-d64755cfb-gk565\" (UID: \"4089403a-fc89-4e1c-ab2f-1e429c0467bd\") " pod="openshift-ingress/e2e-derive-c17951b0-openshift-default-d64755cfb-gk565" Jul 26 16:06:03.027377 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:03.027278 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6t5h\" (UniqueName: \"kubernetes.io/projected/4089403a-fc89-4e1c-ab2f-1e429c0467bd-kube-api-access-l6t5h\") pod \"e2e-derive-c17951b0-openshift-default-d64755cfb-gk565\" (UID: \"4089403a-fc89-4e1c-ab2f-1e429c0467bd\") " pod="openshift-ingress/e2e-derive-c17951b0-openshift-default-d64755cfb-gk565" Jul 26 16:06:03.027377 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:03.027301 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4089403a-fc89-4e1c-ab2f-1e429c0467bd-workload-certs\") pod \"e2e-derive-c17951b0-openshift-default-d64755cfb-gk565\" (UID: \"4089403a-fc89-4e1c-ab2f-1e429c0467bd\") " pod="openshift-ingress/e2e-derive-c17951b0-openshift-default-d64755cfb-gk565" Jul 26 16:06:03.027377 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:03.027324 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4089403a-fc89-4e1c-ab2f-1e429c0467bd-istio-token\") pod \"e2e-derive-c17951b0-openshift-default-d64755cfb-gk565\" (UID: \"4089403a-fc89-4e1c-ab2f-1e429c0467bd\") " pod="openshift-ingress/e2e-derive-c17951b0-openshift-default-d64755cfb-gk565" Jul 26 16:06:03.027377 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:03.027354 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4089403a-fc89-4e1c-ab2f-1e429c0467bd-istiod-ca-cert\") pod \"e2e-derive-c17951b0-openshift-default-d64755cfb-gk565\" (UID: \"4089403a-fc89-4e1c-ab2f-1e429c0467bd\") " pod="openshift-ingress/e2e-derive-c17951b0-openshift-default-d64755cfb-gk565" Jul 26 16:06:03.027611 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:03.027451 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4089403a-fc89-4e1c-ab2f-1e429c0467bd-istio-data\") pod \"e2e-derive-c17951b0-openshift-default-d64755cfb-gk565\" (UID: \"4089403a-fc89-4e1c-ab2f-1e429c0467bd\") " pod="openshift-ingress/e2e-derive-c17951b0-openshift-default-d64755cfb-gk565" Jul 26 16:06:03.027611 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:03.027501 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4089403a-fc89-4e1c-ab2f-1e429c0467bd-workload-socket\") pod \"e2e-derive-c17951b0-openshift-default-d64755cfb-gk565\" (UID: \"4089403a-fc89-4e1c-ab2f-1e429c0467bd\") " pod="openshift-ingress/e2e-derive-c17951b0-openshift-default-d64755cfb-gk565" Jul 26 16:06:03.027611 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:03.027603 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4089403a-fc89-4e1c-ab2f-1e429c0467bd-credential-socket\") pod \"e2e-derive-c17951b0-openshift-default-d64755cfb-gk565\" (UID: \"4089403a-fc89-4e1c-ab2f-1e429c0467bd\") " pod="openshift-ingress/e2e-derive-c17951b0-openshift-default-d64755cfb-gk565" Jul 26 16:06:03.027882 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:03.027856 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4089403a-fc89-4e1c-ab2f-1e429c0467bd-workload-certs\") pod \"e2e-derive-c17951b0-openshift-default-d64755cfb-gk565\" (UID: \"4089403a-fc89-4e1c-ab2f-1e429c0467bd\") " pod="openshift-ingress/e2e-derive-c17951b0-openshift-default-d64755cfb-gk565" Jul 26 16:06:03.028164 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:03.028139 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4089403a-fc89-4e1c-ab2f-1e429c0467bd-istiod-ca-cert\") pod \"e2e-derive-c17951b0-openshift-default-d64755cfb-gk565\" (UID: \"4089403a-fc89-4e1c-ab2f-1e429c0467bd\") " pod="openshift-ingress/e2e-derive-c17951b0-openshift-default-d64755cfb-gk565" Jul 26 16:06:03.029760 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:03.029724 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4089403a-fc89-4e1c-ab2f-1e429c0467bd-istio-envoy\") pod \"e2e-derive-c17951b0-openshift-default-d64755cfb-gk565\" (UID: \"4089403a-fc89-4e1c-ab2f-1e429c0467bd\") " pod="openshift-ingress/e2e-derive-c17951b0-openshift-default-d64755cfb-gk565" Jul 26 16:06:03.030013 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:03.029991 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4089403a-fc89-4e1c-ab2f-1e429c0467bd-istio-podinfo\") pod \"e2e-derive-c17951b0-openshift-default-d64755cfb-gk565\" (UID: \"4089403a-fc89-4e1c-ab2f-1e429c0467bd\") " pod="openshift-ingress/e2e-derive-c17951b0-openshift-default-d64755cfb-gk565" Jul 26 16:06:03.036083 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:03.036046 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4089403a-fc89-4e1c-ab2f-1e429c0467bd-istio-token\") pod \"e2e-derive-c17951b0-openshift-default-d64755cfb-gk565\" (UID: \"4089403a-fc89-4e1c-ab2f-1e429c0467bd\") " pod="openshift-ingress/e2e-derive-c17951b0-openshift-default-d64755cfb-gk565" Jul 26 16:06:03.036622 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:03.036599 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6t5h\" (UniqueName: \"kubernetes.io/projected/4089403a-fc89-4e1c-ab2f-1e429c0467bd-kube-api-access-l6t5h\") pod \"e2e-derive-c17951b0-openshift-default-d64755cfb-gk565\" (UID: \"4089403a-fc89-4e1c-ab2f-1e429c0467bd\") " pod="openshift-ingress/e2e-derive-c17951b0-openshift-default-d64755cfb-gk565" Jul 26 16:06:03.121299 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:03.121208 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-c17951b0-openshift-default-d64755cfb-gk565" Jul 26 16:06:03.273985 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:03.273952 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-c17951b0-openshift-default-d64755cfb-gk565"] Jul 26 16:06:03.275702 ip-10-0-138-163 kubenswrapper[2577]: W0726 16:06:03.275677 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4089403a_fc89_4e1c_ab2f_1e429c0467bd.slice/crio-18402f66d8b8a6495404e160bde14ff3e256f332640ad16f5aa397c1b45cec23 WatchSource:0}: Error finding container 18402f66d8b8a6495404e160bde14ff3e256f332640ad16f5aa397c1b45cec23: Status 404 returned error can't find the container with id 18402f66d8b8a6495404e160bde14ff3e256f332640ad16f5aa397c1b45cec23 Jul 26 16:06:03.278278 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:03.278243 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:06:03.278367 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:03.278309 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:06:03.278367 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:03.278339 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:06:04.088417 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:04.088335 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-c17951b0-openshift-default-d64755cfb-gk565" event={"ID":"4089403a-fc89-4e1c-ab2f-1e429c0467bd","Type":"ContainerStarted","Data":"653d8efa2fd46da914c3258c4069eac44794ae94ea5f9732d8fba0b9dcef8304"} Jul 26 16:06:04.088611 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:04.088438 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-c17951b0-openshift-default-d64755cfb-gk565" event={"ID":"4089403a-fc89-4e1c-ab2f-1e429c0467bd","Type":"ContainerStarted","Data":"18402f66d8b8a6495404e160bde14ff3e256f332640ad16f5aa397c1b45cec23"} Jul 26 16:06:04.110223 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:04.110170 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-c17951b0-openshift-default-d64755cfb-gk565" podStartSLOduration=2.11015163 podStartE2EDuration="2.11015163s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 16:06:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 16:06:04.10849879 +0000 UTC m=+1686.847852119" watchObservedRunningTime="2026-07-26 16:06:04.11015163 +0000 UTC m=+1686.849504940" Jul 26 16:06:04.121926 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:04.121892 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-c17951b0-openshift-default-d64755cfb-gk565" Jul 26 16:06:04.126871 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:04.126849 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-c17951b0-openshift-default-d64755cfb-gk565" Jul 26 16:06:05.093769 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:05.093730 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-c17951b0-openshift-default-d64755cfb-gk565" Jul 26 16:06:05.094833 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:05.094814 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-c17951b0-openshift-default-d64755cfb-gk565" Jul 26 16:06:05.571103 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:05.571065 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:06:05.672799 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:05.672762 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:06:05.816125 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:05.816085 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:06:05.906593 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:05.906499 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:06:06.374056 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:06.374015 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:06:06.469726 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:06.469686 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:06:07.361352 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:07.361288 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9823a2a6-openshift-default-f94df46d8-689nx" Jul 26 16:06:07.470630 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:07.470603 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3565705-0b18-4d74-8fe7-72f9eee5f688-istio-token\") pod \"c3565705-0b18-4d74-8fe7-72f9eee5f688\" (UID: \"c3565705-0b18-4d74-8fe7-72f9eee5f688\") " Jul 26 16:06:07.470630 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:07.470639 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3565705-0b18-4d74-8fe7-72f9eee5f688-credential-socket\") pod \"c3565705-0b18-4d74-8fe7-72f9eee5f688\" (UID: \"c3565705-0b18-4d74-8fe7-72f9eee5f688\") " Jul 26 16:06:07.470849 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:07.470680 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3565705-0b18-4d74-8fe7-72f9eee5f688-workload-socket\") pod \"c3565705-0b18-4d74-8fe7-72f9eee5f688\" (UID: \"c3565705-0b18-4d74-8fe7-72f9eee5f688\") " Jul 26 16:06:07.470849 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:07.470717 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3565705-0b18-4d74-8fe7-72f9eee5f688-istio-envoy\") pod \"c3565705-0b18-4d74-8fe7-72f9eee5f688\" (UID: \"c3565705-0b18-4d74-8fe7-72f9eee5f688\") " Jul 26 16:06:07.470849 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:07.470743 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3565705-0b18-4d74-8fe7-72f9eee5f688-istio-data\") pod \"c3565705-0b18-4d74-8fe7-72f9eee5f688\" (UID: \"c3565705-0b18-4d74-8fe7-72f9eee5f688\") " Jul 26 16:06:07.470849 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:07.470768 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3565705-0b18-4d74-8fe7-72f9eee5f688-workload-certs\") pod \"c3565705-0b18-4d74-8fe7-72f9eee5f688\" (UID: \"c3565705-0b18-4d74-8fe7-72f9eee5f688\") " Jul 26 16:06:07.470849 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:07.470843 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3565705-0b18-4d74-8fe7-72f9eee5f688-istio-podinfo\") pod \"c3565705-0b18-4d74-8fe7-72f9eee5f688\" (UID: \"c3565705-0b18-4d74-8fe7-72f9eee5f688\") " Jul 26 16:06:07.471098 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:07.470890 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bjjhm\" (UniqueName: \"kubernetes.io/projected/c3565705-0b18-4d74-8fe7-72f9eee5f688-kube-api-access-bjjhm\") pod \"c3565705-0b18-4d74-8fe7-72f9eee5f688\" (UID: \"c3565705-0b18-4d74-8fe7-72f9eee5f688\") " Jul 26 16:06:07.471098 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:07.470915 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3565705-0b18-4d74-8fe7-72f9eee5f688-istiod-ca-cert\") pod \"c3565705-0b18-4d74-8fe7-72f9eee5f688\" (UID: \"c3565705-0b18-4d74-8fe7-72f9eee5f688\") " Jul 26 16:06:07.471098 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:07.470931 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3565705-0b18-4d74-8fe7-72f9eee5f688-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c3565705-0b18-4d74-8fe7-72f9eee5f688" (UID: "c3565705-0b18-4d74-8fe7-72f9eee5f688"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:06:07.471260 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:07.471126 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3565705-0b18-4d74-8fe7-72f9eee5f688-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c3565705-0b18-4d74-8fe7-72f9eee5f688" (UID: "c3565705-0b18-4d74-8fe7-72f9eee5f688"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:06:07.471260 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:07.471148 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3565705-0b18-4d74-8fe7-72f9eee5f688-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c3565705-0b18-4d74-8fe7-72f9eee5f688" (UID: "c3565705-0b18-4d74-8fe7-72f9eee5f688"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:06:07.471260 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:07.471240 2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3565705-0b18-4d74-8fe7-72f9eee5f688-workload-socket\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 16:06:07.471451 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:07.471291 2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3565705-0b18-4d74-8fe7-72f9eee5f688-workload-certs\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 16:06:07.471451 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:07.471312 2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3565705-0b18-4d74-8fe7-72f9eee5f688-credential-socket\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 16:06:07.471545 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:07.471462 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3565705-0b18-4d74-8fe7-72f9eee5f688-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c3565705-0b18-4d74-8fe7-72f9eee5f688" (UID: "c3565705-0b18-4d74-8fe7-72f9eee5f688"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 16:06:07.471928 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:07.471905 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3565705-0b18-4d74-8fe7-72f9eee5f688-istio-data" (OuterVolumeSpecName: "istio-data") pod "c3565705-0b18-4d74-8fe7-72f9eee5f688" (UID: "c3565705-0b18-4d74-8fe7-72f9eee5f688"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:06:07.473071 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:07.473044 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c3565705-0b18-4d74-8fe7-72f9eee5f688-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c3565705-0b18-4d74-8fe7-72f9eee5f688" (UID: "c3565705-0b18-4d74-8fe7-72f9eee5f688"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 26 16:06:07.473195 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:07.473112 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3565705-0b18-4d74-8fe7-72f9eee5f688-kube-api-access-bjjhm" (OuterVolumeSpecName: "kube-api-access-bjjhm") pod "c3565705-0b18-4d74-8fe7-72f9eee5f688" (UID: "c3565705-0b18-4d74-8fe7-72f9eee5f688"). InnerVolumeSpecName "kube-api-access-bjjhm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 16:06:07.473380 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:07.473358 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3565705-0b18-4d74-8fe7-72f9eee5f688-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c3565705-0b18-4d74-8fe7-72f9eee5f688" (UID: "c3565705-0b18-4d74-8fe7-72f9eee5f688"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:06:07.473536 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:07.473421 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3565705-0b18-4d74-8fe7-72f9eee5f688-istio-token" (OuterVolumeSpecName: "istio-token") pod "c3565705-0b18-4d74-8fe7-72f9eee5f688" (UID: "c3565705-0b18-4d74-8fe7-72f9eee5f688"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 16:06:07.571806 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:07.571772 2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3565705-0b18-4d74-8fe7-72f9eee5f688-istio-envoy\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 16:06:07.571806 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:07.571800 2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3565705-0b18-4d74-8fe7-72f9eee5f688-istio-data\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 16:06:07.571806 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:07.571809 2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3565705-0b18-4d74-8fe7-72f9eee5f688-istio-podinfo\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 16:06:07.572039 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:07.571817 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjjhm\" (UniqueName: \"kubernetes.io/projected/c3565705-0b18-4d74-8fe7-72f9eee5f688-kube-api-access-bjjhm\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 16:06:07.572039 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:07.571827 2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3565705-0b18-4d74-8fe7-72f9eee5f688-istiod-ca-cert\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 16:06:07.572039 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:07.571835 2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3565705-0b18-4d74-8fe7-72f9eee5f688-istio-token\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 16:06:08.111110 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:08.111024 2577 generic.go:356] "Generic (PLEG): container finished" podID="c3565705-0b18-4d74-8fe7-72f9eee5f688" containerID="0403a4b8b9b1d41c615409d96039bdb49f35f848dc2b6f5e969ea93384dff12e" exitCode=0 Jul 26 16:06:08.111110 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:08.111074 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9823a2a6-openshift-default-f94df46d8-689nx" Jul 26 16:06:08.111341 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:08.111118 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9823a2a6-openshift-default-f94df46d8-689nx" event={"ID":"c3565705-0b18-4d74-8fe7-72f9eee5f688","Type":"ContainerDied","Data":"0403a4b8b9b1d41c615409d96039bdb49f35f848dc2b6f5e969ea93384dff12e"} Jul 26 16:06:08.111341 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:08.111177 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9823a2a6-openshift-default-f94df46d8-689nx" event={"ID":"c3565705-0b18-4d74-8fe7-72f9eee5f688","Type":"ContainerDied","Data":"f70dae6ac79a74c5ff715abfaa788b0bc35725a0f6303c667a4bf8c9c2f422d9"} Jul 26 16:06:08.111341 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:08.111206 2577 scope.go:117] "RemoveContainer" containerID="0403a4b8b9b1d41c615409d96039bdb49f35f848dc2b6f5e969ea93384dff12e" Jul 26 16:06:08.120440 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:08.120423 2577 scope.go:117] "RemoveContainer" containerID="29ca17846a5f2723db6f857d0047b127a885d77492e79d2f8d40d8ba1877eed1" Jul 26 16:06:08.128012 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:08.127983 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-9823a2a6-openshift-default-f94df46d8-689nx"] Jul 26 16:06:08.129145 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:08.129127 2577 scope.go:117] "RemoveContainer" containerID="0403a4b8b9b1d41c615409d96039bdb49f35f848dc2b6f5e969ea93384dff12e" Jul 26 16:06:08.129465 ip-10-0-138-163 kubenswrapper[2577]: E0726 16:06:08.129445 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0403a4b8b9b1d41c615409d96039bdb49f35f848dc2b6f5e969ea93384dff12e\": container with ID starting with 0403a4b8b9b1d41c615409d96039bdb49f35f848dc2b6f5e969ea93384dff12e not found: ID does not exist" containerID="0403a4b8b9b1d41c615409d96039bdb49f35f848dc2b6f5e969ea93384dff12e" Jul 26 16:06:08.129557 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:08.129478 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0403a4b8b9b1d41c615409d96039bdb49f35f848dc2b6f5e969ea93384dff12e"} err="failed to get container status \"0403a4b8b9b1d41c615409d96039bdb49f35f848dc2b6f5e969ea93384dff12e\": rpc error: code = NotFound desc = could not find container \"0403a4b8b9b1d41c615409d96039bdb49f35f848dc2b6f5e969ea93384dff12e\": container with ID starting with 0403a4b8b9b1d41c615409d96039bdb49f35f848dc2b6f5e969ea93384dff12e not found: ID does not exist" Jul 26 16:06:08.129557 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:08.129507 2577 scope.go:117] "RemoveContainer" containerID="29ca17846a5f2723db6f857d0047b127a885d77492e79d2f8d40d8ba1877eed1" Jul 26 16:06:08.129788 ip-10-0-138-163 kubenswrapper[2577]: E0726 16:06:08.129768 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29ca17846a5f2723db6f857d0047b127a885d77492e79d2f8d40d8ba1877eed1\": container with ID starting with 29ca17846a5f2723db6f857d0047b127a885d77492e79d2f8d40d8ba1877eed1 not found: ID does not exist" containerID="29ca17846a5f2723db6f857d0047b127a885d77492e79d2f8d40d8ba1877eed1" Jul 26 16:06:08.129829 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:08.129794 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29ca17846a5f2723db6f857d0047b127a885d77492e79d2f8d40d8ba1877eed1"} err="failed to get container status \"29ca17846a5f2723db6f857d0047b127a885d77492e79d2f8d40d8ba1877eed1\": rpc error: code = NotFound desc = could not find container \"29ca17846a5f2723db6f857d0047b127a885d77492e79d2f8d40d8ba1877eed1\": container with ID starting with 29ca17846a5f2723db6f857d0047b127a885d77492e79d2f8d40d8ba1877eed1 not found: ID does not exist" Jul 26 16:06:08.134836 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:08.134814 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-9823a2a6-openshift-default-f94df46d8-689nx"] Jul 26 16:06:09.787339 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:09.787295 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c3565705-0b18-4d74-8fe7-72f9eee5f688" path="/var/lib/kubelet/pods/c3565705-0b18-4d74-8fe7-72f9eee5f688/volumes" Jul 26 16:06:19.187326 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:19.187294 2577 generic.go:356] "Generic (PLEG): container finished" podID="4089403a-fc89-4e1c-ab2f-1e429c0467bd" containerID="653d8efa2fd46da914c3258c4069eac44794ae94ea5f9732d8fba0b9dcef8304" exitCode=0 Jul 26 16:06:19.187805 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:19.187375 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-c17951b0-openshift-default-d64755cfb-gk565" event={"ID":"4089403a-fc89-4e1c-ab2f-1e429c0467bd","Type":"ContainerDied","Data":"653d8efa2fd46da914c3258c4069eac44794ae94ea5f9732d8fba0b9dcef8304"} Jul 26 16:06:19.187805 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:19.187765 2577 scope.go:117] "RemoveContainer" containerID="653d8efa2fd46da914c3258c4069eac44794ae94ea5f9732d8fba0b9dcef8304" Jul 26 16:06:19.188885 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:19.188857 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:06:19.188958 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:19.188925 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:06:19.188996 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:19.188961 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:06:20.094206 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:20.094124 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-c17951b0-openshift-default-d64755cfb-gk565" Jul 26 16:06:20.194903 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:20.194870 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-c17951b0-openshift-default-d64755cfb-gk565" event={"ID":"4089403a-fc89-4e1c-ab2f-1e429c0467bd","Type":"ContainerStarted","Data":"f76d6ebf878bd6e935782931cd1f4ede8d7b9dd3d8fec667d7642cf3a53a7be1"} Jul 26 16:06:21.122176 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:21.122134 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-c17951b0-openshift-default-d64755cfb-gk565" Jul 26 16:06:21.123444 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:21.123417 2577 patch_prober.go:28] interesting pod/e2e-derive-c17951b0-openshift-default-d64755cfb-gk565 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Jul 26 16:06:21.123568 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:21.123474 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-c17951b0-openshift-default-d64755cfb-gk565" podUID="4089403a-fc89-4e1c-ab2f-1e429c0467bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Jul 26 16:06:22.121907 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:22.121870 2577 patch_prober.go:28] interesting pod/e2e-derive-c17951b0-openshift-default-d64755cfb-gk565 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Jul 26 16:06:22.122328 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:22.121929 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-c17951b0-openshift-default-d64755cfb-gk565" podUID="4089403a-fc89-4e1c-ab2f-1e429c0467bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Jul 26 16:06:23.122539 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:23.122505 2577 patch_prober.go:28] interesting pod/e2e-derive-c17951b0-openshift-default-d64755cfb-gk565 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Jul 26 16:06:23.122994 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:23.122559 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-c17951b0-openshift-default-d64755cfb-gk565" podUID="4089403a-fc89-4e1c-ab2f-1e429c0467bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Jul 26 16:06:24.122507 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:24.122471 2577 patch_prober.go:28] interesting pod/e2e-derive-c17951b0-openshift-default-d64755cfb-gk565 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Jul 26 16:06:24.122679 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:24.122526 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-c17951b0-openshift-default-d64755cfb-gk565" podUID="4089403a-fc89-4e1c-ab2f-1e429c0467bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Jul 26 16:06:25.126002 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:25.125974 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-c17951b0-openshift-default-d64755cfb-gk565" Jul 26 16:06:25.126498 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:25.126230 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-c17951b0-openshift-default-d64755cfb-gk565" Jul 26 16:06:25.127015 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:25.126998 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-c17951b0-openshift-default-d64755cfb-gk565" Jul 26 16:06:40.746286 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:40.746252 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:06:40.802235 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:40.802206 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:06:40.859705 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:40.859648 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:06:45.409567 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:45.409536 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:06:45.542291 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:45.542245 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-c17951b0-openshift-default-d64755cfb-gk565"] Jul 26 16:06:45.542587 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:45.542547 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-c17951b0-openshift-default-d64755cfb-gk565" podUID="4089403a-fc89-4e1c-ab2f-1e429c0467bd" containerName="istio-proxy" containerID="cri-o://f76d6ebf878bd6e935782931cd1f4ede8d7b9dd3d8fec667d7642cf3a53a7be1" gracePeriod=30 Jul 26 16:06:45.563756 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:45.563704 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:06:45.787509 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:45.787469 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:06:46.214646 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:46.214549 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:06:46.313315 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:46.313278 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:06:46.416209 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:46.416178 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:06:47.778622 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:47.778590 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:06:48.830954 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:48.830919 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:06:50.796841 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:50.796812 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-c17951b0-openshift-default-d64755cfb-gk565" Jul 26 16:06:50.973209 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:50.973120 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4089403a-fc89-4e1c-ab2f-1e429c0467bd-istio-token\") pod \"4089403a-fc89-4e1c-ab2f-1e429c0467bd\" (UID: \"4089403a-fc89-4e1c-ab2f-1e429c0467bd\") " Jul 26 16:06:50.973209 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:50.973180 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4089403a-fc89-4e1c-ab2f-1e429c0467bd-workload-socket\") pod \"4089403a-fc89-4e1c-ab2f-1e429c0467bd\" (UID: \"4089403a-fc89-4e1c-ab2f-1e429c0467bd\") " Jul 26 16:06:50.973472 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:50.973223 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4089403a-fc89-4e1c-ab2f-1e429c0467bd-credential-socket\") pod \"4089403a-fc89-4e1c-ab2f-1e429c0467bd\" (UID: \"4089403a-fc89-4e1c-ab2f-1e429c0467bd\") " Jul 26 16:06:50.973472 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:50.973239 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4089403a-fc89-4e1c-ab2f-1e429c0467bd-istio-envoy\") pod \"4089403a-fc89-4e1c-ab2f-1e429c0467bd\" (UID: \"4089403a-fc89-4e1c-ab2f-1e429c0467bd\") " Jul 26 16:06:50.973472 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:50.973264 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4089403a-fc89-4e1c-ab2f-1e429c0467bd-istio-podinfo\") pod \"4089403a-fc89-4e1c-ab2f-1e429c0467bd\" (UID: \"4089403a-fc89-4e1c-ab2f-1e429c0467bd\") " Jul 26 16:06:50.973472 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:50.973299 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4089403a-fc89-4e1c-ab2f-1e429c0467bd-istiod-ca-cert\") pod \"4089403a-fc89-4e1c-ab2f-1e429c0467bd\" (UID: \"4089403a-fc89-4e1c-ab2f-1e429c0467bd\") " Jul 26 16:06:50.973472 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:50.973338 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l6t5h\" (UniqueName: \"kubernetes.io/projected/4089403a-fc89-4e1c-ab2f-1e429c0467bd-kube-api-access-l6t5h\") pod \"4089403a-fc89-4e1c-ab2f-1e429c0467bd\" (UID: \"4089403a-fc89-4e1c-ab2f-1e429c0467bd\") " Jul 26 16:06:50.973472 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:50.973369 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4089403a-fc89-4e1c-ab2f-1e429c0467bd-istio-data\") pod \"4089403a-fc89-4e1c-ab2f-1e429c0467bd\" (UID: \"4089403a-fc89-4e1c-ab2f-1e429c0467bd\") " Jul 26 16:06:50.973472 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:50.973443 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4089403a-fc89-4e1c-ab2f-1e429c0467bd-workload-certs\") pod \"4089403a-fc89-4e1c-ab2f-1e429c0467bd\" (UID: \"4089403a-fc89-4e1c-ab2f-1e429c0467bd\") " Jul 26 16:06:50.973860 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:50.973569 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4089403a-fc89-4e1c-ab2f-1e429c0467bd-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4089403a-fc89-4e1c-ab2f-1e429c0467bd" (UID: "4089403a-fc89-4e1c-ab2f-1e429c0467bd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:06:50.973860 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:50.973797 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4089403a-fc89-4e1c-ab2f-1e429c0467bd-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4089403a-fc89-4e1c-ab2f-1e429c0467bd" (UID: "4089403a-fc89-4e1c-ab2f-1e429c0467bd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:06:50.973860 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:50.973833 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4089403a-fc89-4e1c-ab2f-1e429c0467bd-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4089403a-fc89-4e1c-ab2f-1e429c0467bd" (UID: "4089403a-fc89-4e1c-ab2f-1e429c0467bd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:06:50.973860 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:50.973821 2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4089403a-fc89-4e1c-ab2f-1e429c0467bd-workload-socket\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 16:06:50.974074 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:50.973865 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4089403a-fc89-4e1c-ab2f-1e429c0467bd-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4089403a-fc89-4e1c-ab2f-1e429c0467bd" (UID: "4089403a-fc89-4e1c-ab2f-1e429c0467bd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 16:06:50.974564 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:50.974541 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4089403a-fc89-4e1c-ab2f-1e429c0467bd-istio-data" (OuterVolumeSpecName: "istio-data") pod "4089403a-fc89-4e1c-ab2f-1e429c0467bd" (UID: "4089403a-fc89-4e1c-ab2f-1e429c0467bd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:06:50.975527 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:50.975495 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4089403a-fc89-4e1c-ab2f-1e429c0467bd-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4089403a-fc89-4e1c-ab2f-1e429c0467bd" (UID: "4089403a-fc89-4e1c-ab2f-1e429c0467bd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 26 16:06:50.975637 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:50.975561 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4089403a-fc89-4e1c-ab2f-1e429c0467bd-kube-api-access-l6t5h" (OuterVolumeSpecName: "kube-api-access-l6t5h") pod "4089403a-fc89-4e1c-ab2f-1e429c0467bd" (UID: "4089403a-fc89-4e1c-ab2f-1e429c0467bd"). InnerVolumeSpecName "kube-api-access-l6t5h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 16:06:50.975817 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:50.975796 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4089403a-fc89-4e1c-ab2f-1e429c0467bd-istio-token" (OuterVolumeSpecName: "istio-token") pod "4089403a-fc89-4e1c-ab2f-1e429c0467bd" (UID: "4089403a-fc89-4e1c-ab2f-1e429c0467bd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 16:06:50.975817 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:50.975810 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4089403a-fc89-4e1c-ab2f-1e429c0467bd-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4089403a-fc89-4e1c-ab2f-1e429c0467bd" (UID: "4089403a-fc89-4e1c-ab2f-1e429c0467bd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:06:51.074421 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:51.074359 2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4089403a-fc89-4e1c-ab2f-1e429c0467bd-credential-socket\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 16:06:51.074421 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:51.074392 2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4089403a-fc89-4e1c-ab2f-1e429c0467bd-istio-envoy\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 16:06:51.074625 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:51.074433 2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4089403a-fc89-4e1c-ab2f-1e429c0467bd-istio-podinfo\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 16:06:51.074625 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:51.074446 2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4089403a-fc89-4e1c-ab2f-1e429c0467bd-istiod-ca-cert\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 16:06:51.074625 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:51.074459 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6t5h\" (UniqueName: \"kubernetes.io/projected/4089403a-fc89-4e1c-ab2f-1e429c0467bd-kube-api-access-l6t5h\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 16:06:51.074625 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:51.074471 2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4089403a-fc89-4e1c-ab2f-1e429c0467bd-istio-data\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 16:06:51.074625 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:51.074483 2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4089403a-fc89-4e1c-ab2f-1e429c0467bd-workload-certs\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 16:06:51.074625 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:51.074490 2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4089403a-fc89-4e1c-ab2f-1e429c0467bd-istio-token\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 16:06:51.368426 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:51.368377 2577 generic.go:356] "Generic (PLEG): container finished" podID="4089403a-fc89-4e1c-ab2f-1e429c0467bd" containerID="f76d6ebf878bd6e935782931cd1f4ede8d7b9dd3d8fec667d7642cf3a53a7be1" exitCode=0 Jul 26 16:06:51.368426 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:51.368437 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-c17951b0-openshift-default-d64755cfb-gk565" Jul 26 16:06:51.368702 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:51.368469 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-c17951b0-openshift-default-d64755cfb-gk565" event={"ID":"4089403a-fc89-4e1c-ab2f-1e429c0467bd","Type":"ContainerDied","Data":"f76d6ebf878bd6e935782931cd1f4ede8d7b9dd3d8fec667d7642cf3a53a7be1"} Jul 26 16:06:51.368702 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:51.368523 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-c17951b0-openshift-default-d64755cfb-gk565" event={"ID":"4089403a-fc89-4e1c-ab2f-1e429c0467bd","Type":"ContainerDied","Data":"18402f66d8b8a6495404e160bde14ff3e256f332640ad16f5aa397c1b45cec23"} Jul 26 16:06:51.368702 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:51.368537 2577 scope.go:117] "RemoveContainer" containerID="f76d6ebf878bd6e935782931cd1f4ede8d7b9dd3d8fec667d7642cf3a53a7be1" Jul 26 16:06:51.378416 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:51.378380 2577 scope.go:117] "RemoveContainer" containerID="653d8efa2fd46da914c3258c4069eac44794ae94ea5f9732d8fba0b9dcef8304" Jul 26 16:06:51.386643 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:51.386616 2577 scope.go:117] "RemoveContainer" containerID="f76d6ebf878bd6e935782931cd1f4ede8d7b9dd3d8fec667d7642cf3a53a7be1" Jul 26 16:06:51.386871 ip-10-0-138-163 kubenswrapper[2577]: E0726 16:06:51.386852 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f76d6ebf878bd6e935782931cd1f4ede8d7b9dd3d8fec667d7642cf3a53a7be1\": container with ID starting with f76d6ebf878bd6e935782931cd1f4ede8d7b9dd3d8fec667d7642cf3a53a7be1 not found: ID does not exist" containerID="f76d6ebf878bd6e935782931cd1f4ede8d7b9dd3d8fec667d7642cf3a53a7be1" Jul 26 16:06:51.386921 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:51.386879 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f76d6ebf878bd6e935782931cd1f4ede8d7b9dd3d8fec667d7642cf3a53a7be1"} err="failed to get container status \"f76d6ebf878bd6e935782931cd1f4ede8d7b9dd3d8fec667d7642cf3a53a7be1\": rpc error: code = NotFound desc = could not find container \"f76d6ebf878bd6e935782931cd1f4ede8d7b9dd3d8fec667d7642cf3a53a7be1\": container with ID starting with f76d6ebf878bd6e935782931cd1f4ede8d7b9dd3d8fec667d7642cf3a53a7be1 not found: ID does not exist" Jul 26 16:06:51.386921 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:51.386894 2577 scope.go:117] "RemoveContainer" containerID="653d8efa2fd46da914c3258c4069eac44794ae94ea5f9732d8fba0b9dcef8304" Jul 26 16:06:51.387151 ip-10-0-138-163 kubenswrapper[2577]: E0726 16:06:51.387132 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"653d8efa2fd46da914c3258c4069eac44794ae94ea5f9732d8fba0b9dcef8304\": container with ID starting with 653d8efa2fd46da914c3258c4069eac44794ae94ea5f9732d8fba0b9dcef8304 not found: ID does not exist" containerID="653d8efa2fd46da914c3258c4069eac44794ae94ea5f9732d8fba0b9dcef8304" Jul 26 16:06:51.387202 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:51.387156 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"653d8efa2fd46da914c3258c4069eac44794ae94ea5f9732d8fba0b9dcef8304"} err="failed to get container status \"653d8efa2fd46da914c3258c4069eac44794ae94ea5f9732d8fba0b9dcef8304\": rpc error: code = NotFound desc = could not find container \"653d8efa2fd46da914c3258c4069eac44794ae94ea5f9732d8fba0b9dcef8304\": container with ID starting with 653d8efa2fd46da914c3258c4069eac44794ae94ea5f9732d8fba0b9dcef8304 not found: ID does not exist" Jul 26 16:06:51.390654 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:51.390631 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-c17951b0-openshift-default-d64755cfb-gk565"] Jul 26 16:06:51.395375 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:51.395355 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-c17951b0-openshift-default-d64755cfb-gk565"] Jul 26 16:06:51.787940 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:51.787907 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4089403a-fc89-4e1c-ab2f-1e429c0467bd" path="/var/lib/kubelet/pods/4089403a-fc89-4e1c-ab2f-1e429c0467bd/volumes" Jul 26 16:06:53.615775 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:53.615740 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:06:53.675924 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:53.675894 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:06:53.747515 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:53.747481 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:06:53.848184 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:53.848146 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:06:53.948295 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:53.948192 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:06:54.273319 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:06:54.273290 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:07:20.212064 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:07:20.212024 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:07:26.529046 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:07:26.529003 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:07:26.640526 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:07:26.640487 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:07:26.729625 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:07:26.729553 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:07:42.552648 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:07:42.552608 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:07:48.553503 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:07:48.553456 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:07:48.614348 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:07:48.614316 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:07:48.689016 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:07:48.688965 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:07:48.788990 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:07:48.788961 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:07:57.883916 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:07:57.883886 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-8wjqp_b6b8080d-0cee-4d0c-a05c-3e2f1b672a54/console-operator/1.log" Jul 26 16:07:57.884806 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:07:57.884781 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-8wjqp_b6b8080d-0cee-4d0c-a05c-3e2f1b672a54/console-operator/1.log" Jul 26 16:07:57.887979 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:07:57.887956 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ztlp6_cbe2cd41-6ebc-4722-a813-57d83fa1d242/ovn-acl-logging/0.log" Jul 26 16:07:57.888550 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:07:57.888533 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ztlp6_cbe2cd41-6ebc-4722-a813-57d83fa1d242/ovn-acl-logging/0.log" Jul 26 16:07:59.283305 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:07:59.283266 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:08:00.097845 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:08:00.097810 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:08:06.597134 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:08:06.597098 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:08:06.705873 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:08:06.705838 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:08:06.797649 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:08:06.797614 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:08:06.919198 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:08:06.919098 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:08:07.696962 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:08:07.696928 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:08:07.794445 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:08:07.794378 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:08:07.898987 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:08:07.898952 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:08:09.476148 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:08:09.476117 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:08:15.376133 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:08:15.376097 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:08:15.436806 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:08:15.436767 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:08:16.077116 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:08:16.077081 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:08:22.412134 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:08:22.412096 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:08:22.513062 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:08:22.513012 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:08:22.651613 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:08:22.651574 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:08:22.921849 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:08:22.921817 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-7827eee1-openshift-default-679fcc4d84-m7hbx"] Jul 26 16:08:22.922320 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:08:22.922290 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3565705-0b18-4d74-8fe7-72f9eee5f688" containerName="istio-proxy" Jul 26 16:08:22.922320 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:08:22.922312 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3565705-0b18-4d74-8fe7-72f9eee5f688" containerName="istio-proxy" Jul 26 16:08:22.922528 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:08:22.922326 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3565705-0b18-4d74-8fe7-72f9eee5f688" containerName="istio-proxy" Jul 26 16:08:22.922528 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:08:22.922336 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3565705-0b18-4d74-8fe7-72f9eee5f688" containerName="istio-proxy" Jul 26 16:08:22.922528 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:08:22.922349 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4089403a-fc89-4e1c-ab2f-1e429c0467bd" containerName="istio-proxy" Jul 26 16:08:22.922528 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:08:22.922358 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="4089403a-fc89-4e1c-ab2f-1e429c0467bd" containerName="istio-proxy" Jul 26 16:08:22.922528 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:08:22.922365 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4089403a-fc89-4e1c-ab2f-1e429c0467bd" containerName="istio-proxy" Jul 26 16:08:22.922528 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:08:22.922373 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="4089403a-fc89-4e1c-ab2f-1e429c0467bd" containerName="istio-proxy" Jul 26 16:08:22.922528 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:08:22.922494 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5cb1a79-829e-4ec6-a0db-0ab664abd86d" containerName="istio-proxy" Jul 26 16:08:22.922528 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:08:22.922512 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="4089403a-fc89-4e1c-ab2f-1e429c0467bd" containerName="istio-proxy" Jul 26 16:08:22.922528 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:08:22.922522 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="4089403a-fc89-4e1c-ab2f-1e429c0467bd" containerName="istio-proxy" Jul 26 16:08:22.922528 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:08:22.922531 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3565705-0b18-4d74-8fe7-72f9eee5f688" containerName="istio-proxy" Jul 26 16:08:22.923011 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:08:22.922538 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3565705-0b18-4d74-8fe7-72f9eee5f688" containerName="istio-proxy" Jul 26 16:08:22.925763 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:08:22.925735 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7827eee1-openshift-default-679fcc4d84-m7hbx" Jul 26 16:08:22.928738 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:08:22.928709 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-7827eee1-openshift-default-dockercfg-m8pkj\"" Jul 26 16:08:22.953308 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:08:22.953270 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-7827eee1-openshift-default-679fcc4d84-m7hbx"] Jul 26 16:08:22.956891 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:08:22.956862 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:08:22.989274 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:08:22.989238 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71ae2e8f-a0cc-4fb8-9d36-756d1a077b64-workload-certs\") pod \"e2e-mt-7827eee1-openshift-default-679fcc4d84-m7hbx\" (UID: \"71ae2e8f-a0cc-4fb8-9d36-756d1a077b64\") " pod="openshift-ingress/e2e-mt-7827eee1-openshift-default-679fcc4d84-m7hbx" Jul 26 16:08:22.989478 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:08:22.989278 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71ae2e8f-a0cc-4fb8-9d36-756d1a077b64-credential-socket\") pod \"e2e-mt-7827eee1-openshift-default-679fcc4d84-m7hbx\" (UID: \"71ae2e8f-a0cc-4fb8-9d36-756d1a077b64\") " pod="openshift-ingress/e2e-mt-7827eee1-openshift-default-679fcc4d84-m7hbx" Jul 26 16:08:22.989478 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:08:22.989304 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71ae2e8f-a0cc-4fb8-9d36-756d1a077b64-istiod-ca-cert\") pod \"e2e-mt-7827eee1-openshift-default-679fcc4d84-m7hbx\" (UID: \"71ae2e8f-a0cc-4fb8-9d36-756d1a077b64\") " pod="openshift-ingress/e2e-mt-7827eee1-openshift-default-679fcc4d84-m7hbx" Jul 26 16:08:22.989478 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:08:22.989427 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71ae2e8f-a0cc-4fb8-9d36-756d1a077b64-istio-podinfo\") pod \"e2e-mt-7827eee1-openshift-default-679fcc4d84-m7hbx\" (UID: \"71ae2e8f-a0cc-4fb8-9d36-756d1a077b64\") " pod="openshift-ingress/e2e-mt-7827eee1-openshift-default-679fcc4d84-m7hbx" Jul 26 16:08:22.989670 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:08:22.989515 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71ae2e8f-a0cc-4fb8-9d36-756d1a077b64-istio-envoy\") pod \"e2e-mt-7827eee1-openshift-default-679fcc4d84-m7hbx\" (UID: \"71ae2e8f-a0cc-4fb8-9d36-756d1a077b64\") " pod="openshift-ingress/e2e-mt-7827eee1-openshift-default-679fcc4d84-m7hbx" Jul 26 16:08:22.989670 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:08:22.989549 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfzjt\" (UniqueName: \"kubernetes.io/projected/71ae2e8f-a0cc-4fb8-9d36-756d1a077b64-kube-api-access-wfzjt\") pod \"e2e-mt-7827eee1-openshift-default-679fcc4d84-m7hbx\" (UID: \"71ae2e8f-a0cc-4fb8-9d36-756d1a077b64\") " pod="openshift-ingress/e2e-mt-7827eee1-openshift-default-679fcc4d84-m7hbx" Jul 26 16:08:22.989670 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:08:22.989591 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71ae2e8f-a0cc-4fb8-9d36-756d1a077b64-istio-token\") pod \"e2e-mt-7827eee1-openshift-default-679fcc4d84-m7hbx\" (UID: \"71ae2e8f-a0cc-4fb8-9d36-756d1a077b64\") " pod="openshift-ingress/e2e-mt-7827eee1-openshift-default-679fcc4d84-m7hbx" Jul 26 16:08:22.989670 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:08:22.989640 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71ae2e8f-a0cc-4fb8-9d36-756d1a077b64-istio-data\") pod \"e2e-mt-7827eee1-openshift-default-679fcc4d84-m7hbx\" (UID: \"71ae2e8f-a0cc-4fb8-9d36-756d1a077b64\") " pod="openshift-ingress/e2e-mt-7827eee1-openshift-default-679fcc4d84-m7hbx" Jul 26 16:08:22.989851 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:08:22.989703 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71ae2e8f-a0cc-4fb8-9d36-756d1a077b64-workload-socket\") pod \"e2e-mt-7827eee1-openshift-default-679fcc4d84-m7hbx\" (UID: \"71ae2e8f-a0cc-4fb8-9d36-756d1a077b64\") " pod="openshift-ingress/e2e-mt-7827eee1-openshift-default-679fcc4d84-m7hbx" Jul 26 16:08:23.090973 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:08:23.090934 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71ae2e8f-a0cc-4fb8-9d36-756d1a077b64-workload-certs\") pod \"e2e-mt-7827eee1-openshift-default-679fcc4d84-m7hbx\" (UID: \"71ae2e8f-a0cc-4fb8-9d36-756d1a077b64\") " pod="openshift-ingress/e2e-mt-7827eee1-openshift-default-679fcc4d84-m7hbx" Jul 26 16:08:23.090973 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:08:23.090970 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71ae2e8f-a0cc-4fb8-9d36-756d1a077b64-credential-socket\") pod \"e2e-mt-7827eee1-openshift-default-679fcc4d84-m7hbx\" (UID: \"71ae2e8f-a0cc-4fb8-9d36-756d1a077b64\") " pod="openshift-ingress/e2e-mt-7827eee1-openshift-default-679fcc4d84-m7hbx" Jul 26 16:08:23.091224 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:08:23.090988 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71ae2e8f-a0cc-4fb8-9d36-756d1a077b64-istiod-ca-cert\") pod \"e2e-mt-7827eee1-openshift-default-679fcc4d84-m7hbx\" (UID: \"71ae2e8f-a0cc-4fb8-9d36-756d1a077b64\") " pod="openshift-ingress/e2e-mt-7827eee1-openshift-default-679fcc4d84-m7hbx" Jul 26 16:08:23.091224 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:08:23.091007 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71ae2e8f-a0cc-4fb8-9d36-756d1a077b64-istio-podinfo\") pod \"e2e-mt-7827eee1-openshift-default-679fcc4d84-m7hbx\" (UID: \"71ae2e8f-a0cc-4fb8-9d36-756d1a077b64\") " pod="openshift-ingress/e2e-mt-7827eee1-openshift-default-679fcc4d84-m7hbx" Jul 26 16:08:23.091224 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:08:23.091040 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71ae2e8f-a0cc-4fb8-9d36-756d1a077b64-istio-envoy\") pod \"e2e-mt-7827eee1-openshift-default-679fcc4d84-m7hbx\" (UID: \"71ae2e8f-a0cc-4fb8-9d36-756d1a077b64\") " pod="openshift-ingress/e2e-mt-7827eee1-openshift-default-679fcc4d84-m7hbx" Jul 26 16:08:23.091224 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:08:23.091070 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfzjt\" (UniqueName: \"kubernetes.io/projected/71ae2e8f-a0cc-4fb8-9d36-756d1a077b64-kube-api-access-wfzjt\") pod \"e2e-mt-7827eee1-openshift-default-679fcc4d84-m7hbx\" (UID: \"71ae2e8f-a0cc-4fb8-9d36-756d1a077b64\") " pod="openshift-ingress/e2e-mt-7827eee1-openshift-default-679fcc4d84-m7hbx" Jul 26 16:08:23.091224 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:08:23.091094 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71ae2e8f-a0cc-4fb8-9d36-756d1a077b64-istio-token\") pod \"e2e-mt-7827eee1-openshift-default-679fcc4d84-m7hbx\" (UID: \"71ae2e8f-a0cc-4fb8-9d36-756d1a077b64\") " pod="openshift-ingress/e2e-mt-7827eee1-openshift-default-679fcc4d84-m7hbx" Jul 26 16:08:23.091224 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:08:23.091129 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71ae2e8f-a0cc-4fb8-9d36-756d1a077b64-istio-data\") pod \"e2e-mt-7827eee1-openshift-default-679fcc4d84-m7hbx\" (UID: \"71ae2e8f-a0cc-4fb8-9d36-756d1a077b64\") " pod="openshift-ingress/e2e-mt-7827eee1-openshift-default-679fcc4d84-m7hbx" Jul 26 16:08:23.091224 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:08:23.091185 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71ae2e8f-a0cc-4fb8-9d36-756d1a077b64-workload-socket\") pod \"e2e-mt-7827eee1-openshift-default-679fcc4d84-m7hbx\" (UID: \"71ae2e8f-a0cc-4fb8-9d36-756d1a077b64\") " pod="openshift-ingress/e2e-mt-7827eee1-openshift-default-679fcc4d84-m7hbx" Jul 26 16:08:23.091687 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:08:23.091657 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71ae2e8f-a0cc-4fb8-9d36-756d1a077b64-credential-socket\") pod \"e2e-mt-7827eee1-openshift-default-679fcc4d84-m7hbx\" (UID: \"71ae2e8f-a0cc-4fb8-9d36-756d1a077b64\") " pod="openshift-ingress/e2e-mt-7827eee1-openshift-default-679fcc4d84-m7hbx" Jul 26 16:08:23.091794 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:08:23.091702 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71ae2e8f-a0cc-4fb8-9d36-756d1a077b64-istio-data\") pod \"e2e-mt-7827eee1-openshift-default-679fcc4d84-m7hbx\" (UID: \"71ae2e8f-a0cc-4fb8-9d36-756d1a077b64\") " pod="openshift-ingress/e2e-mt-7827eee1-openshift-default-679fcc4d84-m7hbx" Jul 26 16:08:23.091883 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:08:23.091851 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71ae2e8f-a0cc-4fb8-9d36-756d1a077b64-workload-certs\") pod \"e2e-mt-7827eee1-openshift-default-679fcc4d84-m7hbx\" (UID: \"71ae2e8f-a0cc-4fb8-9d36-756d1a077b64\") " pod="openshift-ingress/e2e-mt-7827eee1-openshift-default-679fcc4d84-m7hbx" Jul 26 16:08:23.091944 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:08:23.091913 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71ae2e8f-a0cc-4fb8-9d36-756d1a077b64-workload-socket\") pod \"e2e-mt-7827eee1-openshift-default-679fcc4d84-m7hbx\" (UID: \"71ae2e8f-a0cc-4fb8-9d36-756d1a077b64\") " pod="openshift-ingress/e2e-mt-7827eee1-openshift-default-679fcc4d84-m7hbx" Jul 26 16:08:23.092333 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:08:23.092311 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71ae2e8f-a0cc-4fb8-9d36-756d1a077b64-istiod-ca-cert\") pod \"e2e-mt-7827eee1-openshift-default-679fcc4d84-m7hbx\" (UID: \"71ae2e8f-a0cc-4fb8-9d36-756d1a077b64\") " pod="openshift-ingress/e2e-mt-7827eee1-openshift-default-679fcc4d84-m7hbx" Jul 26 16:08:23.093586 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:08:23.093553 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71ae2e8f-a0cc-4fb8-9d36-756d1a077b64-istio-envoy\") pod \"e2e-mt-7827eee1-openshift-default-679fcc4d84-m7hbx\" (UID: \"71ae2e8f-a0cc-4fb8-9d36-756d1a077b64\") " pod="openshift-ingress/e2e-mt-7827eee1-openshift-default-679fcc4d84-m7hbx" Jul 26 16:08:23.093693 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:08:23.093609 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71ae2e8f-a0cc-4fb8-9d36-756d1a077b64-istio-podinfo\") pod \"e2e-mt-7827eee1-openshift-default-679fcc4d84-m7hbx\" (UID: \"71ae2e8f-a0cc-4fb8-9d36-756d1a077b64\") " pod="openshift-ingress/e2e-mt-7827eee1-openshift-default-679fcc4d84-m7hbx" Jul 26 16:08:23.094846 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:08:23.094825 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:08:23.102207 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:08:23.102163 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71ae2e8f-a0cc-4fb8-9d36-756d1a077b64-istio-token\") pod \"e2e-mt-7827eee1-openshift-default-679fcc4d84-m7hbx\" (UID: \"71ae2e8f-a0cc-4fb8-9d36-756d1a077b64\") " pod="openshift-ingress/e2e-mt-7827eee1-openshift-default-679fcc4d84-m7hbx" Jul 26 16:08:23.105080 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:08:23.105059 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfzjt\" (UniqueName: \"kubernetes.io/projected/71ae2e8f-a0cc-4fb8-9d36-756d1a077b64-kube-api-access-wfzjt\") pod \"e2e-mt-7827eee1-openshift-default-679fcc4d84-m7hbx\" (UID: \"71ae2e8f-a0cc-4fb8-9d36-756d1a077b64\") " pod="openshift-ingress/e2e-mt-7827eee1-openshift-default-679fcc4d84-m7hbx" Jul 26 16:08:23.237753 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:08:23.237649 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7827eee1-openshift-default-679fcc4d84-m7hbx" Jul 26 16:08:23.393096 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:08:23.393061 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-7827eee1-openshift-default-679fcc4d84-m7hbx"] Jul 26 16:08:23.397228 ip-10-0-138-163 kubenswrapper[2577]: W0726 16:08:23.397201 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71ae2e8f_a0cc_4fb8_9d36_756d1a077b64.slice/crio-a8ec25846ed2a8c946a225847b61633b6166f32c765a4e28988920e1fa27a866 WatchSource:0}: Error finding container a8ec25846ed2a8c946a225847b61633b6166f32c765a4e28988920e1fa27a866: Status 404 returned error can't find the container with id a8ec25846ed2a8c946a225847b61633b6166f32c765a4e28988920e1fa27a866 Jul 26 16:08:23.399390 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:08:23.399361 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:08:23.399508 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:08:23.399448 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:08:23.399508 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:08:23.399480 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:08:23.881658 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:08:23.881613 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7827eee1-openshift-default-679fcc4d84-m7hbx" event={"ID":"71ae2e8f-a0cc-4fb8-9d36-756d1a077b64","Type":"ContainerStarted","Data":"0f08764299ca866909bf52d6b76588e0cedc96d238eb8fb76170ba2bb454e5f2"} Jul 26 16:08:23.882080 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:08:23.881687 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7827eee1-openshift-default-679fcc4d84-m7hbx" event={"ID":"71ae2e8f-a0cc-4fb8-9d36-756d1a077b64","Type":"ContainerStarted","Data":"a8ec25846ed2a8c946a225847b61633b6166f32c765a4e28988920e1fa27a866"} Jul 26 16:08:23.902322 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:08:23.902276 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-7827eee1-openshift-default-679fcc4d84-m7hbx" podStartSLOduration=1.9022614629999999 podStartE2EDuration="1.902261463s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 16:08:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 16:08:23.901104808 +0000 UTC m=+1826.640458120" watchObservedRunningTime="2026-07-26 16:08:23.902261463 +0000 UTC m=+1826.641614772" Jul 26 16:08:24.238546 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:08:24.238448 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-7827eee1-openshift-default-679fcc4d84-m7hbx" Jul 26 16:08:24.240192 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:08:24.240158 2577 patch_prober.go:28] interesting pod/e2e-mt-7827eee1-openshift-default-679fcc4d84-m7hbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Jul 26 16:08:24.240335 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:08:24.240266 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7827eee1-openshift-default-679fcc4d84-m7hbx" podUID="71ae2e8f-a0cc-4fb8-9d36-756d1a077b64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Jul 26 16:08:25.241704 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:08:25.241672 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-7827eee1-openshift-default-679fcc4d84-m7hbx" Jul 26 16:08:25.779535 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:08:25.779497 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:08:25.854819 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:08:25.854782 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:08:25.892617 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:08:25.892584 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-7827eee1-openshift-default-679fcc4d84-m7hbx" Jul 26 16:08:25.893465 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:08:25.893423 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-7827eee1-openshift-default-679fcc4d84-m7hbx" Jul 26 16:08:25.927392 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:08:25.927341 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:08:26.032139 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:08:26.032051 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:08:26.446072 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:08:26.445989 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:08:26.546491 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:08:26.546459 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:08:30.258642 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:08:30.258602 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:08:30.917248 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:08:30.917213 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:08:31.009870 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:08:31.009841 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:08:31.302970 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:08:31.302937 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:08:36.951287 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:08:36.951252 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:08:37.009678 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:08:37.009633 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:08:39.984330 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:08:39.984294 2577 generic.go:356] "Generic (PLEG): container finished" podID="71ae2e8f-a0cc-4fb8-9d36-756d1a077b64" containerID="0f08764299ca866909bf52d6b76588e0cedc96d238eb8fb76170ba2bb454e5f2" exitCode=0 Jul 26 16:08:39.984860 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:08:39.984366 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7827eee1-openshift-default-679fcc4d84-m7hbx" event={"ID":"71ae2e8f-a0cc-4fb8-9d36-756d1a077b64","Type":"ContainerDied","Data":"0f08764299ca866909bf52d6b76588e0cedc96d238eb8fb76170ba2bb454e5f2"} Jul 26 16:08:39.984860 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:08:39.984791 2577 scope.go:117] "RemoveContainer" containerID="0f08764299ca866909bf52d6b76588e0cedc96d238eb8fb76170ba2bb454e5f2" Jul 26 16:08:39.985787 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:08:39.985763 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:08:39.985899 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:08:39.985856 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:08:39.985899 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:08:39.985897 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:08:40.893078 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:08:40.893037 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-7827eee1-openshift-default-679fcc4d84-m7hbx" Jul 26 16:08:40.992741 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:08:40.992703 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7827eee1-openshift-default-679fcc4d84-m7hbx" event={"ID":"71ae2e8f-a0cc-4fb8-9d36-756d1a077b64","Type":"ContainerStarted","Data":"187176c722508b13e41a26c1a24ba5851718254c61a55cde0d29538f66618a1e"} Jul 26 16:08:41.239964 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:08:41.239865 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-7827eee1-openshift-default-679fcc4d84-m7hbx" Jul 26 16:08:41.240264 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:08:41.240242 2577 patch_prober.go:28] interesting pod/e2e-mt-7827eee1-openshift-default-679fcc4d84-m7hbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Jul 26 16:08:41.240460 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:08:41.240434 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7827eee1-openshift-default-679fcc4d84-m7hbx" podUID="71ae2e8f-a0cc-4fb8-9d36-756d1a077b64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Jul 26 16:08:42.239353 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:08:42.238863 2577 patch_prober.go:28] interesting pod/e2e-mt-7827eee1-openshift-default-679fcc4d84-m7hbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Jul 26 16:08:42.239353 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:08:42.238931 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7827eee1-openshift-default-679fcc4d84-m7hbx" podUID="71ae2e8f-a0cc-4fb8-9d36-756d1a077b64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Jul 26 16:08:43.238149 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:08:43.238099 2577 patch_prober.go:28] interesting pod/e2e-mt-7827eee1-openshift-default-679fcc4d84-m7hbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Jul 26 16:08:43.238373 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:08:43.238186 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7827eee1-openshift-default-679fcc4d84-m7hbx" podUID="71ae2e8f-a0cc-4fb8-9d36-756d1a077b64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Jul 26 16:08:44.238895 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:08:44.238853 2577 patch_prober.go:28] interesting pod/e2e-mt-7827eee1-openshift-default-679fcc4d84-m7hbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Jul 26 16:08:44.239523 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:08:44.239491 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7827eee1-openshift-default-679fcc4d84-m7hbx" podUID="71ae2e8f-a0cc-4fb8-9d36-756d1a077b64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Jul 26 16:08:45.241817 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:08:45.241781 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-7827eee1-openshift-default-679fcc4d84-m7hbx" Jul 26 16:08:45.242285 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:08:45.242260 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-7827eee1-openshift-default-679fcc4d84-m7hbx" Jul 26 16:08:45.242917 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:08:45.242897 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-7827eee1-openshift-default-679fcc4d84-m7hbx" Jul 26 16:08:47.801718 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:08:47.801684 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:08:47.901766 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:08:47.901730 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:08:52.374802 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:08:52.374764 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:08:52.739183 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:08:52.739100 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:08:58.675363 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:08:58.675330 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:08:58.777331 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:08:58.777295 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:08:58.919142 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:08:58.919107 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-7827eee1-openshift-default-679fcc4d84-m7hbx"] Jul 26 16:08:58.919412 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:08:58.919353 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-7827eee1-openshift-default-679fcc4d84-m7hbx" podUID="71ae2e8f-a0cc-4fb8-9d36-756d1a077b64" containerName="istio-proxy" containerID="cri-o://187176c722508b13e41a26c1a24ba5851718254c61a55cde0d29538f66618a1e" gracePeriod=30 Jul 26 16:08:58.945330 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:08:58.945223 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:08:59.177026 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:08:59.176987 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:08:59.317165 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:08:59.317134 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:08:59.481347 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:08:59.481311 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:08:59.601512 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:08:59.601413 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:09:00.244903 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:09:00.244871 2577 patch_prober.go:28] interesting pod/e2e-mt-7827eee1-openshift-default-679fcc4d84-m7hbx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 26 16:09:00.245313 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:09:00.244956 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-7827eee1-openshift-default-679fcc4d84-m7hbx" podUID="71ae2e8f-a0cc-4fb8-9d36-756d1a077b64" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 26 16:09:04.119577 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:09:04.119536 2577 generic.go:356] "Generic (PLEG): container finished" podID="71ae2e8f-a0cc-4fb8-9d36-756d1a077b64" containerID="187176c722508b13e41a26c1a24ba5851718254c61a55cde0d29538f66618a1e" exitCode=0 Jul 26 16:09:04.119972 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:09:04.119605 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7827eee1-openshift-default-679fcc4d84-m7hbx" event={"ID":"71ae2e8f-a0cc-4fb8-9d36-756d1a077b64","Type":"ContainerDied","Data":"187176c722508b13e41a26c1a24ba5851718254c61a55cde0d29538f66618a1e"} Jul 26 16:09:04.119972 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:09:04.119682 2577 scope.go:117] "RemoveContainer" containerID="0f08764299ca866909bf52d6b76588e0cedc96d238eb8fb76170ba2bb454e5f2" Jul 26 16:09:04.173133 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:09:04.173110 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7827eee1-openshift-default-679fcc4d84-m7hbx" Jul 26 16:09:04.280778 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:09:04.280751 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71ae2e8f-a0cc-4fb8-9d36-756d1a077b64-istio-podinfo\") pod \"71ae2e8f-a0cc-4fb8-9d36-756d1a077b64\" (UID: \"71ae2e8f-a0cc-4fb8-9d36-756d1a077b64\") " Jul 26 16:09:04.280926 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:09:04.280787 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71ae2e8f-a0cc-4fb8-9d36-756d1a077b64-workload-socket\") pod \"71ae2e8f-a0cc-4fb8-9d36-756d1a077b64\" (UID: \"71ae2e8f-a0cc-4fb8-9d36-756d1a077b64\") " Jul 26 16:09:04.280926 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:09:04.280851 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71ae2e8f-a0cc-4fb8-9d36-756d1a077b64-istiod-ca-cert\") pod \"71ae2e8f-a0cc-4fb8-9d36-756d1a077b64\" (UID: \"71ae2e8f-a0cc-4fb8-9d36-756d1a077b64\") " Jul 26 16:09:04.280926 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:09:04.280882 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71ae2e8f-a0cc-4fb8-9d36-756d1a077b64-credential-socket\") pod \"71ae2e8f-a0cc-4fb8-9d36-756d1a077b64\" (UID: \"71ae2e8f-a0cc-4fb8-9d36-756d1a077b64\") " Jul 26 16:09:04.280926 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:09:04.280906 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71ae2e8f-a0cc-4fb8-9d36-756d1a077b64-workload-certs\") pod \"71ae2e8f-a0cc-4fb8-9d36-756d1a077b64\" (UID: \"71ae2e8f-a0cc-4fb8-9d36-756d1a077b64\") " Jul 26 16:09:04.281175 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:09:04.280945 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71ae2e8f-a0cc-4fb8-9d36-756d1a077b64-istio-token\") pod \"71ae2e8f-a0cc-4fb8-9d36-756d1a077b64\" (UID: \"71ae2e8f-a0cc-4fb8-9d36-756d1a077b64\") " Jul 26 16:09:04.281175 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:09:04.280998 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71ae2e8f-a0cc-4fb8-9d36-756d1a077b64-istio-envoy\") pod \"71ae2e8f-a0cc-4fb8-9d36-756d1a077b64\" (UID: \"71ae2e8f-a0cc-4fb8-9d36-756d1a077b64\") " Jul 26 16:09:04.281175 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:09:04.281037 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wfzjt\" (UniqueName: \"kubernetes.io/projected/71ae2e8f-a0cc-4fb8-9d36-756d1a077b64-kube-api-access-wfzjt\") pod \"71ae2e8f-a0cc-4fb8-9d36-756d1a077b64\" (UID: \"71ae2e8f-a0cc-4fb8-9d36-756d1a077b64\") " Jul 26 16:09:04.281175 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:09:04.281062 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71ae2e8f-a0cc-4fb8-9d36-756d1a077b64-istio-data\") pod \"71ae2e8f-a0cc-4fb8-9d36-756d1a077b64\" (UID: \"71ae2e8f-a0cc-4fb8-9d36-756d1a077b64\") " Jul 26 16:09:04.281385 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:09:04.281188 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71ae2e8f-a0cc-4fb8-9d36-756d1a077b64-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "71ae2e8f-a0cc-4fb8-9d36-756d1a077b64" (UID: "71ae2e8f-a0cc-4fb8-9d36-756d1a077b64"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:09:04.281385 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:09:04.281209 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71ae2e8f-a0cc-4fb8-9d36-756d1a077b64-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "71ae2e8f-a0cc-4fb8-9d36-756d1a077b64" (UID: "71ae2e8f-a0cc-4fb8-9d36-756d1a077b64"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:09:04.281385 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:09:04.281298 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71ae2e8f-a0cc-4fb8-9d36-756d1a077b64-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "71ae2e8f-a0cc-4fb8-9d36-756d1a077b64" (UID: "71ae2e8f-a0cc-4fb8-9d36-756d1a077b64"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 16:09:04.281586 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:09:04.281520 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71ae2e8f-a0cc-4fb8-9d36-756d1a077b64-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "71ae2e8f-a0cc-4fb8-9d36-756d1a077b64" (UID: "71ae2e8f-a0cc-4fb8-9d36-756d1a077b64"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:09:04.281586 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:09:04.281529 2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71ae2e8f-a0cc-4fb8-9d36-756d1a077b64-istiod-ca-cert\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 16:09:04.281586 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:09:04.281564 2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71ae2e8f-a0cc-4fb8-9d36-756d1a077b64-credential-socket\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 16:09:04.281586 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:09:04.281580 2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71ae2e8f-a0cc-4fb8-9d36-756d1a077b64-workload-socket\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 16:09:04.281860 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:09:04.281843 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71ae2e8f-a0cc-4fb8-9d36-756d1a077b64-istio-data" (OuterVolumeSpecName: "istio-data") pod "71ae2e8f-a0cc-4fb8-9d36-756d1a077b64" (UID: "71ae2e8f-a0cc-4fb8-9d36-756d1a077b64"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:09:04.283467 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:09:04.283440 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71ae2e8f-a0cc-4fb8-9d36-756d1a077b64-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "71ae2e8f-a0cc-4fb8-9d36-756d1a077b64" (UID: "71ae2e8f-a0cc-4fb8-9d36-756d1a077b64"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:09:04.283572 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:09:04.283522 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/71ae2e8f-a0cc-4fb8-9d36-756d1a077b64-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "71ae2e8f-a0cc-4fb8-9d36-756d1a077b64" (UID: "71ae2e8f-a0cc-4fb8-9d36-756d1a077b64"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 26 16:09:04.283686 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:09:04.283666 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71ae2e8f-a0cc-4fb8-9d36-756d1a077b64-istio-token" (OuterVolumeSpecName: "istio-token") pod "71ae2e8f-a0cc-4fb8-9d36-756d1a077b64" (UID: "71ae2e8f-a0cc-4fb8-9d36-756d1a077b64"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 16:09:04.283741 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:09:04.283723 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71ae2e8f-a0cc-4fb8-9d36-756d1a077b64-kube-api-access-wfzjt" (OuterVolumeSpecName: "kube-api-access-wfzjt") pod "71ae2e8f-a0cc-4fb8-9d36-756d1a077b64" (UID: "71ae2e8f-a0cc-4fb8-9d36-756d1a077b64"). InnerVolumeSpecName "kube-api-access-wfzjt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 16:09:04.382975 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:09:04.382940 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wfzjt\" (UniqueName: \"kubernetes.io/projected/71ae2e8f-a0cc-4fb8-9d36-756d1a077b64-kube-api-access-wfzjt\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 16:09:04.382975 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:09:04.382968 2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71ae2e8f-a0cc-4fb8-9d36-756d1a077b64-istio-data\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 16:09:04.382975 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:09:04.382978 2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71ae2e8f-a0cc-4fb8-9d36-756d1a077b64-istio-podinfo\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 16:09:04.383204 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:09:04.382986 2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71ae2e8f-a0cc-4fb8-9d36-756d1a077b64-workload-certs\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 16:09:04.383204 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:09:04.382995 2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71ae2e8f-a0cc-4fb8-9d36-756d1a077b64-istio-token\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 16:09:04.383204 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:09:04.383002 2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71ae2e8f-a0cc-4fb8-9d36-756d1a077b64-istio-envoy\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 16:09:05.126199 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:09:05.126161 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7827eee1-openshift-default-679fcc4d84-m7hbx" event={"ID":"71ae2e8f-a0cc-4fb8-9d36-756d1a077b64","Type":"ContainerDied","Data":"a8ec25846ed2a8c946a225847b61633b6166f32c765a4e28988920e1fa27a866"} Jul 26 16:09:05.126649 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:09:05.126201 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7827eee1-openshift-default-679fcc4d84-m7hbx" Jul 26 16:09:05.126649 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:09:05.126229 2577 scope.go:117] "RemoveContainer" containerID="187176c722508b13e41a26c1a24ba5851718254c61a55cde0d29538f66618a1e" Jul 26 16:09:05.149256 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:09:05.149232 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-7827eee1-openshift-default-679fcc4d84-m7hbx"] Jul 26 16:09:05.153026 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:09:05.153001 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-7827eee1-openshift-default-679fcc4d84-m7hbx"] Jul 26 16:09:05.226126 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:09:05.226095 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:09:05.291849 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:09:05.291810 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:09:05.373099 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:09:05.373069 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:09:05.470881 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:09:05.470793 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:09:05.570899 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:09:05.570863 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:09:05.787500 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:09:05.787469 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="71ae2e8f-a0cc-4fb8-9d36-756d1a077b64" path="/var/lib/kubelet/pods/71ae2e8f-a0cc-4fb8-9d36-756d1a077b64/volumes" Jul 26 16:09:11.380202 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:09:11.380168 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:09:11.479105 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:09:11.479069 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:09:18.165547 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:09:18.165511 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:09:18.466813 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:09:18.466730 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-1a44779b-openshift-default-86db955c64-xc8l6"] Jul 26 16:09:18.467319 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:09:18.467300 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71ae2e8f-a0cc-4fb8-9d36-756d1a077b64" containerName="istio-proxy" Jul 26 16:09:18.467371 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:09:18.467321 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="71ae2e8f-a0cc-4fb8-9d36-756d1a077b64" containerName="istio-proxy" Jul 26 16:09:18.467423 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:09:18.467415 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="71ae2e8f-a0cc-4fb8-9d36-756d1a077b64" containerName="istio-proxy" Jul 26 16:09:18.471991 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:09:18.471975 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1a44779b-openshift-default-86db955c64-xc8l6" Jul 26 16:09:18.474546 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:09:18.474527 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-1a44779b-openshift-default-dockercfg-vvjjp\"" Jul 26 16:09:18.480928 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:09:18.480885 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-1a44779b-openshift-default-86db955c64-xc8l6"] Jul 26 16:09:18.496069 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:09:18.496041 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:09:18.598192 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:09:18.598155 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:09:18.610671 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:09:18.610639 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cec097ae-2435-4525-af6c-3efe7a725187-istio-podinfo\") pod \"e2e-mt-1a44779b-openshift-default-86db955c64-xc8l6\" (UID: \"cec097ae-2435-4525-af6c-3efe7a725187\") " pod="openshift-ingress/e2e-mt-1a44779b-openshift-default-86db955c64-xc8l6" Jul 26 16:09:18.610671 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:09:18.610671 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cec097ae-2435-4525-af6c-3efe7a725187-istio-data\") pod \"e2e-mt-1a44779b-openshift-default-86db955c64-xc8l6\" (UID: \"cec097ae-2435-4525-af6c-3efe7a725187\") " pod="openshift-ingress/e2e-mt-1a44779b-openshift-default-86db955c64-xc8l6" Jul 26 16:09:18.610880 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:09:18.610696 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cec097ae-2435-4525-af6c-3efe7a725187-istio-envoy\") pod \"e2e-mt-1a44779b-openshift-default-86db955c64-xc8l6\" (UID: \"cec097ae-2435-4525-af6c-3efe7a725187\") " pod="openshift-ingress/e2e-mt-1a44779b-openshift-default-86db955c64-xc8l6" Jul 26 16:09:18.610880 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:09:18.610714 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4422c\" (UniqueName: \"kubernetes.io/projected/cec097ae-2435-4525-af6c-3efe7a725187-kube-api-access-4422c\") pod \"e2e-mt-1a44779b-openshift-default-86db955c64-xc8l6\" (UID: \"cec097ae-2435-4525-af6c-3efe7a725187\") " pod="openshift-ingress/e2e-mt-1a44779b-openshift-default-86db955c64-xc8l6" Jul 26 16:09:18.610880 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:09:18.610733 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cec097ae-2435-4525-af6c-3efe7a725187-credential-socket\") pod \"e2e-mt-1a44779b-openshift-default-86db955c64-xc8l6\" (UID: \"cec097ae-2435-4525-af6c-3efe7a725187\") " pod="openshift-ingress/e2e-mt-1a44779b-openshift-default-86db955c64-xc8l6" Jul 26 16:09:18.610880 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:09:18.610795 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cec097ae-2435-4525-af6c-3efe7a725187-workload-certs\") pod \"e2e-mt-1a44779b-openshift-default-86db955c64-xc8l6\" (UID: \"cec097ae-2435-4525-af6c-3efe7a725187\") " pod="openshift-ingress/e2e-mt-1a44779b-openshift-default-86db955c64-xc8l6" Jul 26 16:09:18.610880 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:09:18.610860 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cec097ae-2435-4525-af6c-3efe7a725187-istio-token\") pod \"e2e-mt-1a44779b-openshift-default-86db955c64-xc8l6\" (UID: \"cec097ae-2435-4525-af6c-3efe7a725187\") " pod="openshift-ingress/e2e-mt-1a44779b-openshift-default-86db955c64-xc8l6" Jul 26 16:09:18.611063 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:09:18.610895 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cec097ae-2435-4525-af6c-3efe7a725187-workload-socket\") pod \"e2e-mt-1a44779b-openshift-default-86db955c64-xc8l6\" (UID: \"cec097ae-2435-4525-af6c-3efe7a725187\") " pod="openshift-ingress/e2e-mt-1a44779b-openshift-default-86db955c64-xc8l6" Jul 26 16:09:18.611063 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:09:18.610917 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cec097ae-2435-4525-af6c-3efe7a725187-istiod-ca-cert\") pod \"e2e-mt-1a44779b-openshift-default-86db955c64-xc8l6\" (UID: \"cec097ae-2435-4525-af6c-3efe7a725187\") " pod="openshift-ingress/e2e-mt-1a44779b-openshift-default-86db955c64-xc8l6" Jul 26 16:09:18.711566 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:09:18.711531 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cec097ae-2435-4525-af6c-3efe7a725187-istio-podinfo\") pod \"e2e-mt-1a44779b-openshift-default-86db955c64-xc8l6\" (UID: \"cec097ae-2435-4525-af6c-3efe7a725187\") " pod="openshift-ingress/e2e-mt-1a44779b-openshift-default-86db955c64-xc8l6" Jul 26 16:09:18.711566 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:09:18.711569 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cec097ae-2435-4525-af6c-3efe7a725187-istio-data\") pod \"e2e-mt-1a44779b-openshift-default-86db955c64-xc8l6\" (UID: \"cec097ae-2435-4525-af6c-3efe7a725187\") " pod="openshift-ingress/e2e-mt-1a44779b-openshift-default-86db955c64-xc8l6" Jul 26 16:09:18.711823 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:09:18.711604 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cec097ae-2435-4525-af6c-3efe7a725187-istio-envoy\") pod \"e2e-mt-1a44779b-openshift-default-86db955c64-xc8l6\" (UID: \"cec097ae-2435-4525-af6c-3efe7a725187\") " pod="openshift-ingress/e2e-mt-1a44779b-openshift-default-86db955c64-xc8l6" Jul 26 16:09:18.711823 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:09:18.711628 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4422c\" (UniqueName: \"kubernetes.io/projected/cec097ae-2435-4525-af6c-3efe7a725187-kube-api-access-4422c\") pod \"e2e-mt-1a44779b-openshift-default-86db955c64-xc8l6\" (UID: \"cec097ae-2435-4525-af6c-3efe7a725187\") " pod="openshift-ingress/e2e-mt-1a44779b-openshift-default-86db955c64-xc8l6" Jul 26 16:09:18.711823 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:09:18.711655 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cec097ae-2435-4525-af6c-3efe7a725187-credential-socket\") pod \"e2e-mt-1a44779b-openshift-default-86db955c64-xc8l6\" (UID: \"cec097ae-2435-4525-af6c-3efe7a725187\") " pod="openshift-ingress/e2e-mt-1a44779b-openshift-default-86db955c64-xc8l6" Jul 26 16:09:18.711823 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:09:18.711680 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cec097ae-2435-4525-af6c-3efe7a725187-workload-certs\") pod \"e2e-mt-1a44779b-openshift-default-86db955c64-xc8l6\" (UID: \"cec097ae-2435-4525-af6c-3efe7a725187\") " pod="openshift-ingress/e2e-mt-1a44779b-openshift-default-86db955c64-xc8l6" Jul 26 16:09:18.711823 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:09:18.711727 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cec097ae-2435-4525-af6c-3efe7a725187-istio-token\") pod \"e2e-mt-1a44779b-openshift-default-86db955c64-xc8l6\" (UID: \"cec097ae-2435-4525-af6c-3efe7a725187\") " pod="openshift-ingress/e2e-mt-1a44779b-openshift-default-86db955c64-xc8l6" Jul 26 16:09:18.711823 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:09:18.711763 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cec097ae-2435-4525-af6c-3efe7a725187-workload-socket\") pod \"e2e-mt-1a44779b-openshift-default-86db955c64-xc8l6\" (UID: \"cec097ae-2435-4525-af6c-3efe7a725187\") " pod="openshift-ingress/e2e-mt-1a44779b-openshift-default-86db955c64-xc8l6" Jul 26 16:09:18.711823 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:09:18.711793 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cec097ae-2435-4525-af6c-3efe7a725187-istiod-ca-cert\") pod \"e2e-mt-1a44779b-openshift-default-86db955c64-xc8l6\" (UID: \"cec097ae-2435-4525-af6c-3efe7a725187\") " pod="openshift-ingress/e2e-mt-1a44779b-openshift-default-86db955c64-xc8l6" Jul 26 16:09:18.712151 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:09:18.712045 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cec097ae-2435-4525-af6c-3efe7a725187-credential-socket\") pod \"e2e-mt-1a44779b-openshift-default-86db955c64-xc8l6\" (UID: \"cec097ae-2435-4525-af6c-3efe7a725187\") " pod="openshift-ingress/e2e-mt-1a44779b-openshift-default-86db955c64-xc8l6" Jul 26 16:09:18.712151 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:09:18.712070 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cec097ae-2435-4525-af6c-3efe7a725187-istio-data\") pod \"e2e-mt-1a44779b-openshift-default-86db955c64-xc8l6\" (UID: \"cec097ae-2435-4525-af6c-3efe7a725187\") " pod="openshift-ingress/e2e-mt-1a44779b-openshift-default-86db955c64-xc8l6" Jul 26 16:09:18.712328 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:09:18.712298 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cec097ae-2435-4525-af6c-3efe7a725187-workload-certs\") pod \"e2e-mt-1a44779b-openshift-default-86db955c64-xc8l6\" (UID: \"cec097ae-2435-4525-af6c-3efe7a725187\") " pod="openshift-ingress/e2e-mt-1a44779b-openshift-default-86db955c64-xc8l6" Jul 26 16:09:18.712328 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:09:18.712322 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cec097ae-2435-4525-af6c-3efe7a725187-workload-socket\") pod \"e2e-mt-1a44779b-openshift-default-86db955c64-xc8l6\" (UID: \"cec097ae-2435-4525-af6c-3efe7a725187\") " pod="openshift-ingress/e2e-mt-1a44779b-openshift-default-86db955c64-xc8l6" Jul 26 16:09:18.712690 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:09:18.712670 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cec097ae-2435-4525-af6c-3efe7a725187-istiod-ca-cert\") pod \"e2e-mt-1a44779b-openshift-default-86db955c64-xc8l6\" (UID: \"cec097ae-2435-4525-af6c-3efe7a725187\") " pod="openshift-ingress/e2e-mt-1a44779b-openshift-default-86db955c64-xc8l6" Jul 26 16:09:18.714011 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:09:18.713984 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cec097ae-2435-4525-af6c-3efe7a725187-istio-envoy\") pod \"e2e-mt-1a44779b-openshift-default-86db955c64-xc8l6\" (UID: \"cec097ae-2435-4525-af6c-3efe7a725187\") " pod="openshift-ingress/e2e-mt-1a44779b-openshift-default-86db955c64-xc8l6" Jul 26 16:09:18.714314 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:09:18.714295 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cec097ae-2435-4525-af6c-3efe7a725187-istio-podinfo\") pod \"e2e-mt-1a44779b-openshift-default-86db955c64-xc8l6\" (UID: \"cec097ae-2435-4525-af6c-3efe7a725187\") " pod="openshift-ingress/e2e-mt-1a44779b-openshift-default-86db955c64-xc8l6" Jul 26 16:09:18.719613 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:09:18.719547 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cec097ae-2435-4525-af6c-3efe7a725187-istio-token\") pod \"e2e-mt-1a44779b-openshift-default-86db955c64-xc8l6\" (UID: \"cec097ae-2435-4525-af6c-3efe7a725187\") " pod="openshift-ingress/e2e-mt-1a44779b-openshift-default-86db955c64-xc8l6" Jul 26 16:09:18.719721 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:09:18.719702 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4422c\" (UniqueName: \"kubernetes.io/projected/cec097ae-2435-4525-af6c-3efe7a725187-kube-api-access-4422c\") pod \"e2e-mt-1a44779b-openshift-default-86db955c64-xc8l6\" (UID: \"cec097ae-2435-4525-af6c-3efe7a725187\") " pod="openshift-ingress/e2e-mt-1a44779b-openshift-default-86db955c64-xc8l6" Jul 26 16:09:18.783963 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:09:18.783930 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1a44779b-openshift-default-86db955c64-xc8l6" Jul 26 16:09:18.927881 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:09:18.927858 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-1a44779b-openshift-default-86db955c64-xc8l6"] Jul 26 16:09:18.929778 ip-10-0-138-163 kubenswrapper[2577]: W0726 16:09:18.929755 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcec097ae_2435_4525_af6c_3efe7a725187.slice/crio-719677abc6ca40eb35f5a0b8dec62671c7bbef75a66a6ae13a002a82f5352e45 WatchSource:0}: Error finding container 719677abc6ca40eb35f5a0b8dec62671c7bbef75a66a6ae13a002a82f5352e45: Status 404 returned error can't find the container with id 719677abc6ca40eb35f5a0b8dec62671c7bbef75a66a6ae13a002a82f5352e45 Jul 26 16:09:18.932006 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:09:18.931977 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:09:18.932092 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:09:18.932039 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:09:18.932092 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:09:18.932067 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:09:19.203904 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:09:19.203869 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1a44779b-openshift-default-86db955c64-xc8l6" event={"ID":"cec097ae-2435-4525-af6c-3efe7a725187","Type":"ContainerStarted","Data":"4afb7154ca44a42ff1396bd76a85ac784078f3515c5cb216add337c66ca3cd7f"} Jul 26 16:09:19.204298 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:09:19.203921 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1a44779b-openshift-default-86db955c64-xc8l6" event={"ID":"cec097ae-2435-4525-af6c-3efe7a725187","Type":"ContainerStarted","Data":"719677abc6ca40eb35f5a0b8dec62671c7bbef75a66a6ae13a002a82f5352e45"} Jul 26 16:09:19.222105 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:09:19.222009 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-1a44779b-openshift-default-86db955c64-xc8l6" podStartSLOduration=1.221991871 podStartE2EDuration="1.221991871s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 16:09:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 16:09:19.221021496 +0000 UTC m=+1881.960374809" watchObservedRunningTime="2026-07-26 16:09:19.221991871 +0000 UTC m=+1881.961345181" Jul 26 16:09:19.788034 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:09:19.788004 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-1a44779b-openshift-default-86db955c64-xc8l6" Jul 26 16:09:19.789553 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:09:19.789529 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-1a44779b-openshift-default-86db955c64-xc8l6" Jul 26 16:09:20.209548 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:09:20.209431 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-1a44779b-openshift-default-86db955c64-xc8l6" Jul 26 16:09:20.210262 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:09:20.210243 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-1a44779b-openshift-default-86db955c64-xc8l6" Jul 26 16:09:21.138143 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:09:21.138107 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:09:21.209771 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:09:21.209736 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:09:21.285212 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:09:21.285181 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:09:21.454561 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:09:21.454467 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:09:21.793860 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:09:21.793831 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:09:21.897881 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:09:21.897850 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:09:25.626165 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:09:25.626118 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:09:34.296238 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:09:34.296199 2577 generic.go:356] "Generic (PLEG): container finished" podID="cec097ae-2435-4525-af6c-3efe7a725187" containerID="4afb7154ca44a42ff1396bd76a85ac784078f3515c5cb216add337c66ca3cd7f" exitCode=0 Jul 26 16:09:34.296790 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:09:34.296277 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1a44779b-openshift-default-86db955c64-xc8l6" event={"ID":"cec097ae-2435-4525-af6c-3efe7a725187","Type":"ContainerDied","Data":"4afb7154ca44a42ff1396bd76a85ac784078f3515c5cb216add337c66ca3cd7f"} Jul 26 16:09:34.296860 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:09:34.296831 2577 scope.go:117] "RemoveContainer" containerID="4afb7154ca44a42ff1396bd76a85ac784078f3515c5cb216add337c66ca3cd7f" Jul 26 16:09:34.297893 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:09:34.297867 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:09:34.297996 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:09:34.297932 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:09:34.297996 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:09:34.297965 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:09:35.209683 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:09:35.209645 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-1a44779b-openshift-default-86db955c64-xc8l6" Jul 26 16:09:35.304257 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:09:35.304219 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1a44779b-openshift-default-86db955c64-xc8l6" event={"ID":"cec097ae-2435-4525-af6c-3efe7a725187","Type":"ContainerStarted","Data":"38ed734fcc551439743f3d402b62420c4a051457ba416a2753cfd1f08b5985e2"} Jul 26 16:09:35.785725 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:09:35.785690 2577 patch_prober.go:28] interesting pod/e2e-mt-1a44779b-openshift-default-86db955c64-xc8l6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Jul 26 16:09:35.785949 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:09:35.785761 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1a44779b-openshift-default-86db955c64-xc8l6" podUID="cec097ae-2435-4525-af6c-3efe7a725187" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Jul 26 16:09:35.787723 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:09:35.787701 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-1a44779b-openshift-default-86db955c64-xc8l6" Jul 26 16:09:35.839388 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:09:35.839264 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:09:36.785167 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:09:36.785133 2577 patch_prober.go:28] interesting pod/e2e-mt-1a44779b-openshift-default-86db955c64-xc8l6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Jul 26 16:09:36.785645 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:09:36.785192 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1a44779b-openshift-default-86db955c64-xc8l6" podUID="cec097ae-2435-4525-af6c-3efe7a725187" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Jul 26 16:09:37.784645 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:09:37.784596 2577 patch_prober.go:28] interesting pod/e2e-mt-1a44779b-openshift-default-86db955c64-xc8l6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Jul 26 16:09:37.784856 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:09:37.784649 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1a44779b-openshift-default-86db955c64-xc8l6" podUID="cec097ae-2435-4525-af6c-3efe7a725187" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Jul 26 16:09:38.784333 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:09:38.784294 2577 patch_prober.go:28] interesting pod/e2e-mt-1a44779b-openshift-default-86db955c64-xc8l6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Jul 26 16:09:38.784750 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:09:38.784353 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1a44779b-openshift-default-86db955c64-xc8l6" podUID="cec097ae-2435-4525-af6c-3efe7a725187" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Jul 26 16:09:39.788224 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:09:39.788198 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-1a44779b-openshift-default-86db955c64-xc8l6" Jul 26 16:09:39.788641 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:09:39.788603 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-1a44779b-openshift-default-86db955c64-xc8l6" Jul 26 16:09:39.789230 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:09:39.789212 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-1a44779b-openshift-default-86db955c64-xc8l6" Jul 26 16:10:15.536746 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:10:15.536714 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:10:16.588023 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:10:16.587984 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:10:20.077702 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:10:20.077666 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:10:23.230894 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:10:23.230849 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:10:28.314208 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:10:28.314169 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:11:04.690615 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:11:04.690540 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:11:05.471670 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:11:05.471636 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:11:09.998020 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:11:09.997983 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:11:10.054847 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:11:10.054818 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:11:11.254496 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:11:11.254458 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:11:11.324184 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:11:11.324152 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:11:11.533665 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:11:11.533632 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-1a44779b-openshift-default-86db955c64-xc8l6"] Jul 26 16:11:11.533858 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:11:11.533827 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-1a44779b-openshift-default-86db955c64-xc8l6" podUID="cec097ae-2435-4525-af6c-3efe7a725187" containerName="istio-proxy" containerID="cri-o://38ed734fcc551439743f3d402b62420c4a051457ba416a2753cfd1f08b5985e2" gracePeriod=30 Jul 26 16:11:11.555165 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:11:11.555120 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:11:11.783387 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:11:11.783349 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:11:12.894952 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:11:12.894916 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:11:12.993118 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:11:12.993086 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:11:13.091879 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:11:13.091848 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:11:13.836013 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:11:13.835966 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:11:13.930420 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:11:13.930364 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:11:14.030222 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:11:14.030192 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:11:15.397326 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:11:15.397286 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:11:16.803901 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:11:16.803836 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1a44779b-openshift-default-86db955c64-xc8l6" Jul 26 16:11:16.861815 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:11:16.861782 2577 generic.go:356] "Generic (PLEG): container finished" podID="cec097ae-2435-4525-af6c-3efe7a725187" containerID="38ed734fcc551439743f3d402b62420c4a051457ba416a2753cfd1f08b5985e2" exitCode=0 Jul 26 16:11:16.862017 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:11:16.861826 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1a44779b-openshift-default-86db955c64-xc8l6" Jul 26 16:11:16.862017 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:11:16.861832 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1a44779b-openshift-default-86db955c64-xc8l6" event={"ID":"cec097ae-2435-4525-af6c-3efe7a725187","Type":"ContainerDied","Data":"38ed734fcc551439743f3d402b62420c4a051457ba416a2753cfd1f08b5985e2"} Jul 26 16:11:16.862017 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:11:16.861875 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1a44779b-openshift-default-86db955c64-xc8l6" event={"ID":"cec097ae-2435-4525-af6c-3efe7a725187","Type":"ContainerDied","Data":"719677abc6ca40eb35f5a0b8dec62671c7bbef75a66a6ae13a002a82f5352e45"} Jul 26 16:11:16.862017 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:11:16.861898 2577 scope.go:117] "RemoveContainer" containerID="38ed734fcc551439743f3d402b62420c4a051457ba416a2753cfd1f08b5985e2" Jul 26 16:11:16.867326 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:11:16.867305 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cec097ae-2435-4525-af6c-3efe7a725187-credential-socket\") pod \"cec097ae-2435-4525-af6c-3efe7a725187\" (UID: \"cec097ae-2435-4525-af6c-3efe7a725187\") " Jul 26 16:11:16.867626 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:11:16.867347 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cec097ae-2435-4525-af6c-3efe7a725187-istio-envoy\") pod \"cec097ae-2435-4525-af6c-3efe7a725187\" (UID: \"cec097ae-2435-4525-af6c-3efe7a725187\") " Jul 26 16:11:16.867626 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:11:16.867379 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cec097ae-2435-4525-af6c-3efe7a725187-workload-certs\") pod \"cec097ae-2435-4525-af6c-3efe7a725187\" (UID: \"cec097ae-2435-4525-af6c-3efe7a725187\") " Jul 26 16:11:16.867626 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:11:16.867437 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cec097ae-2435-4525-af6c-3efe7a725187-istiod-ca-cert\") pod \"cec097ae-2435-4525-af6c-3efe7a725187\" (UID: \"cec097ae-2435-4525-af6c-3efe7a725187\") " Jul 26 16:11:16.867626 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:11:16.867487 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4422c\" (UniqueName: \"kubernetes.io/projected/cec097ae-2435-4525-af6c-3efe7a725187-kube-api-access-4422c\") pod \"cec097ae-2435-4525-af6c-3efe7a725187\" (UID: \"cec097ae-2435-4525-af6c-3efe7a725187\") " Jul 26 16:11:16.867626 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:11:16.867515 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cec097ae-2435-4525-af6c-3efe7a725187-workload-socket\") pod \"cec097ae-2435-4525-af6c-3efe7a725187\" (UID: \"cec097ae-2435-4525-af6c-3efe7a725187\") " Jul 26 16:11:16.867626 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:11:16.867601 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cec097ae-2435-4525-af6c-3efe7a725187-istio-data\") pod \"cec097ae-2435-4525-af6c-3efe7a725187\" (UID: \"cec097ae-2435-4525-af6c-3efe7a725187\") " Jul 26 16:11:16.868279 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:11:16.867629 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cec097ae-2435-4525-af6c-3efe7a725187-istio-token\") pod \"cec097ae-2435-4525-af6c-3efe7a725187\" (UID: \"cec097ae-2435-4525-af6c-3efe7a725187\") " Jul 26 16:11:16.868279 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:11:16.867680 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cec097ae-2435-4525-af6c-3efe7a725187-istio-podinfo\") pod \"cec097ae-2435-4525-af6c-3efe7a725187\" (UID: \"cec097ae-2435-4525-af6c-3efe7a725187\") " Jul 26 16:11:16.868513 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:11:16.868488 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cec097ae-2435-4525-af6c-3efe7a725187-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "cec097ae-2435-4525-af6c-3efe7a725187" (UID: "cec097ae-2435-4525-af6c-3efe7a725187"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:11:16.869137 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:11:16.868877 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cec097ae-2435-4525-af6c-3efe7a725187-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "cec097ae-2435-4525-af6c-3efe7a725187" (UID: "cec097ae-2435-4525-af6c-3efe7a725187"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 16:11:16.869137 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:11:16.869123 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cec097ae-2435-4525-af6c-3efe7a725187-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "cec097ae-2435-4525-af6c-3efe7a725187" (UID: "cec097ae-2435-4525-af6c-3efe7a725187"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:11:16.869299 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:11:16.869181 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cec097ae-2435-4525-af6c-3efe7a725187-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "cec097ae-2435-4525-af6c-3efe7a725187" (UID: "cec097ae-2435-4525-af6c-3efe7a725187"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:11:16.869299 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:11:16.869190 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cec097ae-2435-4525-af6c-3efe7a725187-istio-data" (OuterVolumeSpecName: "istio-data") pod "cec097ae-2435-4525-af6c-3efe7a725187" (UID: "cec097ae-2435-4525-af6c-3efe7a725187"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:11:16.871513 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:11:16.871478 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cec097ae-2435-4525-af6c-3efe7a725187-istio-token" (OuterVolumeSpecName: "istio-token") pod "cec097ae-2435-4525-af6c-3efe7a725187" (UID: "cec097ae-2435-4525-af6c-3efe7a725187"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 16:11:16.871513 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:11:16.871502 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cec097ae-2435-4525-af6c-3efe7a725187-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "cec097ae-2435-4525-af6c-3efe7a725187" (UID: "cec097ae-2435-4525-af6c-3efe7a725187"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:11:16.871656 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:11:16.871542 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cec097ae-2435-4525-af6c-3efe7a725187-kube-api-access-4422c" (OuterVolumeSpecName: "kube-api-access-4422c") pod "cec097ae-2435-4525-af6c-3efe7a725187" (UID: "cec097ae-2435-4525-af6c-3efe7a725187"). InnerVolumeSpecName "kube-api-access-4422c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 16:11:16.871697 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:11:16.871660 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cec097ae-2435-4525-af6c-3efe7a725187-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "cec097ae-2435-4525-af6c-3efe7a725187" (UID: "cec097ae-2435-4525-af6c-3efe7a725187"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 26 16:11:16.874700 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:11:16.874652 2577 scope.go:117] "RemoveContainer" containerID="4afb7154ca44a42ff1396bd76a85ac784078f3515c5cb216add337c66ca3cd7f" Jul 26 16:11:16.898835 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:11:16.898812 2577 scope.go:117] "RemoveContainer" containerID="38ed734fcc551439743f3d402b62420c4a051457ba416a2753cfd1f08b5985e2" Jul 26 16:11:16.899180 ip-10-0-138-163 kubenswrapper[2577]: E0726 16:11:16.899153 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38ed734fcc551439743f3d402b62420c4a051457ba416a2753cfd1f08b5985e2\": container with ID starting with 38ed734fcc551439743f3d402b62420c4a051457ba416a2753cfd1f08b5985e2 not found: ID does not exist" containerID="38ed734fcc551439743f3d402b62420c4a051457ba416a2753cfd1f08b5985e2" Jul 26 16:11:16.899286 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:11:16.899188 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38ed734fcc551439743f3d402b62420c4a051457ba416a2753cfd1f08b5985e2"} err="failed to get container status \"38ed734fcc551439743f3d402b62420c4a051457ba416a2753cfd1f08b5985e2\": rpc error: code = NotFound desc = could not find container \"38ed734fcc551439743f3d402b62420c4a051457ba416a2753cfd1f08b5985e2\": container with ID starting with 38ed734fcc551439743f3d402b62420c4a051457ba416a2753cfd1f08b5985e2 not found: ID does not exist" Jul 26 16:11:16.899286 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:11:16.899206 2577 scope.go:117] "RemoveContainer" containerID="4afb7154ca44a42ff1396bd76a85ac784078f3515c5cb216add337c66ca3cd7f" Jul 26 16:11:16.899545 ip-10-0-138-163 kubenswrapper[2577]: E0726 16:11:16.899519 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4afb7154ca44a42ff1396bd76a85ac784078f3515c5cb216add337c66ca3cd7f\": container with ID starting with 4afb7154ca44a42ff1396bd76a85ac784078f3515c5cb216add337c66ca3cd7f not found: ID does not exist" containerID="4afb7154ca44a42ff1396bd76a85ac784078f3515c5cb216add337c66ca3cd7f" Jul 26 16:11:16.899597 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:11:16.899550 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4afb7154ca44a42ff1396bd76a85ac784078f3515c5cb216add337c66ca3cd7f"} err="failed to get container status \"4afb7154ca44a42ff1396bd76a85ac784078f3515c5cb216add337c66ca3cd7f\": rpc error: code = NotFound desc = could not find container \"4afb7154ca44a42ff1396bd76a85ac784078f3515c5cb216add337c66ca3cd7f\": container with ID starting with 4afb7154ca44a42ff1396bd76a85ac784078f3515c5cb216add337c66ca3cd7f not found: ID does not exist" Jul 26 16:11:16.969204 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:11:16.969051 2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cec097ae-2435-4525-af6c-3efe7a725187-istio-data\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 16:11:16.969204 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:11:16.969089 2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cec097ae-2435-4525-af6c-3efe7a725187-istio-token\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 16:11:16.969204 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:11:16.969102 2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cec097ae-2435-4525-af6c-3efe7a725187-istio-podinfo\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 16:11:16.969204 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:11:16.969116 2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cec097ae-2435-4525-af6c-3efe7a725187-credential-socket\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 16:11:16.969204 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:11:16.969129 2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cec097ae-2435-4525-af6c-3efe7a725187-istio-envoy\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 16:11:16.969204 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:11:16.969142 2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cec097ae-2435-4525-af6c-3efe7a725187-workload-certs\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 16:11:16.969204 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:11:16.969153 2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cec097ae-2435-4525-af6c-3efe7a725187-istiod-ca-cert\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 16:11:16.969204 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:11:16.969165 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4422c\" (UniqueName: \"kubernetes.io/projected/cec097ae-2435-4525-af6c-3efe7a725187-kube-api-access-4422c\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 16:11:16.969204 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:11:16.969178 2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cec097ae-2435-4525-af6c-3efe7a725187-workload-socket\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 16:11:17.187978 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:11:17.187944 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-1a44779b-openshift-default-86db955c64-xc8l6"] Jul 26 16:11:17.193559 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:11:17.193532 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-1a44779b-openshift-default-86db955c64-xc8l6"] Jul 26 16:11:17.793369 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:11:17.793336 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cec097ae-2435-4525-af6c-3efe7a725187" path="/var/lib/kubelet/pods/cec097ae-2435-4525-af6c-3efe7a725187/volumes" Jul 26 16:11:21.224972 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:11:21.224942 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:11:21.289144 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:11:21.289114 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:11:21.892009 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:11:21.891977 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:11:27.672759 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:11:27.672716 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:11:27.739367 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:11:27.739328 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:11:27.818046 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:11:27.818002 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:11:27.915407 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:11:27.915350 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:11:38.388440 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:11:38.388378 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:11:38.728810 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:11:38.728717 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:11:54.686080 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:11:54.686046 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:11:59.696912 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:11:59.696875 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:12:05.871731 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:05.871693 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:12:05.971979 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:05.971943 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:12:06.212132 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:06.212051 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:12:06.499503 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:06.499414 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-7c7176-openshift-default-568d6865b8-m9qkz"] Jul 26 16:12:06.499795 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:06.499783 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71ae2e8f-a0cc-4fb8-9d36-756d1a077b64" containerName="istio-proxy" Jul 26 16:12:06.499837 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:06.499796 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="71ae2e8f-a0cc-4fb8-9d36-756d1a077b64" containerName="istio-proxy" Jul 26 16:12:06.499837 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:06.499806 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cec097ae-2435-4525-af6c-3efe7a725187" containerName="istio-proxy" Jul 26 16:12:06.499837 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:06.499812 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="cec097ae-2435-4525-af6c-3efe7a725187" containerName="istio-proxy" Jul 26 16:12:06.499837 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:06.499836 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cec097ae-2435-4525-af6c-3efe7a725187" containerName="istio-proxy" Jul 26 16:12:06.499959 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:06.499842 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="cec097ae-2435-4525-af6c-3efe7a725187" containerName="istio-proxy" Jul 26 16:12:06.499959 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:06.499898 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="cec097ae-2435-4525-af6c-3efe7a725187" containerName="istio-proxy" Jul 26 16:12:06.499959 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:06.499905 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="71ae2e8f-a0cc-4fb8-9d36-756d1a077b64" containerName="istio-proxy" Jul 26 16:12:06.499959 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:06.499912 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="cec097ae-2435-4525-af6c-3efe7a725187" containerName="istio-proxy" Jul 26 16:12:06.502977 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:06.502960 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7c7176-openshift-default-568d6865b8-m9qkz" Jul 26 16:12:06.505367 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:06.505348 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-7c7176-openshift-default-dockercfg-qjnfj\"" Jul 26 16:12:06.515801 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:06.515777 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-7c7176-openshift-default-568d6865b8-m9qkz"] Jul 26 16:12:06.523447 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:06.523425 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:12:06.537170 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:06.537139 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5cb52e0-d049-45ad-ac1a-64f58ffeec11-istio-data\") pod \"e2e-inf-a-7c7176-openshift-default-568d6865b8-m9qkz\" (UID: \"e5cb52e0-d049-45ad-ac1a-64f58ffeec11\") " pod="openshift-ingress/e2e-inf-a-7c7176-openshift-default-568d6865b8-m9qkz" Jul 26 16:12:06.537350 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:06.537181 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5cb52e0-d049-45ad-ac1a-64f58ffeec11-istio-envoy\") pod \"e2e-inf-a-7c7176-openshift-default-568d6865b8-m9qkz\" (UID: \"e5cb52e0-d049-45ad-ac1a-64f58ffeec11\") " pod="openshift-ingress/e2e-inf-a-7c7176-openshift-default-568d6865b8-m9qkz" Jul 26 16:12:06.537350 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:06.537275 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5cb52e0-d049-45ad-ac1a-64f58ffeec11-credential-socket\") pod \"e2e-inf-a-7c7176-openshift-default-568d6865b8-m9qkz\" (UID: \"e5cb52e0-d049-45ad-ac1a-64f58ffeec11\") " pod="openshift-ingress/e2e-inf-a-7c7176-openshift-default-568d6865b8-m9qkz" Jul 26 16:12:06.537350 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:06.537342 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5cb52e0-d049-45ad-ac1a-64f58ffeec11-istio-token\") pod \"e2e-inf-a-7c7176-openshift-default-568d6865b8-m9qkz\" (UID: \"e5cb52e0-d049-45ad-ac1a-64f58ffeec11\") " pod="openshift-ingress/e2e-inf-a-7c7176-openshift-default-568d6865b8-m9qkz" Jul 26 16:12:06.537560 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:06.537366 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5cb52e0-d049-45ad-ac1a-64f58ffeec11-workload-certs\") pod \"e2e-inf-a-7c7176-openshift-default-568d6865b8-m9qkz\" (UID: \"e5cb52e0-d049-45ad-ac1a-64f58ffeec11\") " pod="openshift-ingress/e2e-inf-a-7c7176-openshift-default-568d6865b8-m9qkz" Jul 26 16:12:06.537560 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:06.537410 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8v9jq\" (UniqueName: \"kubernetes.io/projected/e5cb52e0-d049-45ad-ac1a-64f58ffeec11-kube-api-access-8v9jq\") pod \"e2e-inf-a-7c7176-openshift-default-568d6865b8-m9qkz\" (UID: \"e5cb52e0-d049-45ad-ac1a-64f58ffeec11\") " pod="openshift-ingress/e2e-inf-a-7c7176-openshift-default-568d6865b8-m9qkz" Jul 26 16:12:06.537560 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:06.537454 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5cb52e0-d049-45ad-ac1a-64f58ffeec11-istiod-ca-cert\") pod \"e2e-inf-a-7c7176-openshift-default-568d6865b8-m9qkz\" (UID: \"e5cb52e0-d049-45ad-ac1a-64f58ffeec11\") " pod="openshift-ingress/e2e-inf-a-7c7176-openshift-default-568d6865b8-m9qkz" Jul 26 16:12:06.537560 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:06.537501 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5cb52e0-d049-45ad-ac1a-64f58ffeec11-istio-podinfo\") pod \"e2e-inf-a-7c7176-openshift-default-568d6865b8-m9qkz\" (UID: \"e5cb52e0-d049-45ad-ac1a-64f58ffeec11\") " pod="openshift-ingress/e2e-inf-a-7c7176-openshift-default-568d6865b8-m9qkz" Jul 26 16:12:06.537560 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:06.537527 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5cb52e0-d049-45ad-ac1a-64f58ffeec11-workload-socket\") pod \"e2e-inf-a-7c7176-openshift-default-568d6865b8-m9qkz\" (UID: \"e5cb52e0-d049-45ad-ac1a-64f58ffeec11\") " pod="openshift-ingress/e2e-inf-a-7c7176-openshift-default-568d6865b8-m9qkz" Jul 26 16:12:06.626355 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:06.626316 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:12:06.638442 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:06.638392 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5cb52e0-d049-45ad-ac1a-64f58ffeec11-credential-socket\") pod \"e2e-inf-a-7c7176-openshift-default-568d6865b8-m9qkz\" (UID: \"e5cb52e0-d049-45ad-ac1a-64f58ffeec11\") " pod="openshift-ingress/e2e-inf-a-7c7176-openshift-default-568d6865b8-m9qkz" Jul 26 16:12:06.638620 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:06.638473 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5cb52e0-d049-45ad-ac1a-64f58ffeec11-istio-token\") pod \"e2e-inf-a-7c7176-openshift-default-568d6865b8-m9qkz\" (UID: \"e5cb52e0-d049-45ad-ac1a-64f58ffeec11\") " pod="openshift-ingress/e2e-inf-a-7c7176-openshift-default-568d6865b8-m9qkz" Jul 26 16:12:06.638620 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:06.638501 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5cb52e0-d049-45ad-ac1a-64f58ffeec11-workload-certs\") pod \"e2e-inf-a-7c7176-openshift-default-568d6865b8-m9qkz\" (UID: \"e5cb52e0-d049-45ad-ac1a-64f58ffeec11\") " pod="openshift-ingress/e2e-inf-a-7c7176-openshift-default-568d6865b8-m9qkz" Jul 26 16:12:06.638620 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:06.638538 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8v9jq\" (UniqueName: \"kubernetes.io/projected/e5cb52e0-d049-45ad-ac1a-64f58ffeec11-kube-api-access-8v9jq\") pod \"e2e-inf-a-7c7176-openshift-default-568d6865b8-m9qkz\" (UID: \"e5cb52e0-d049-45ad-ac1a-64f58ffeec11\") " pod="openshift-ingress/e2e-inf-a-7c7176-openshift-default-568d6865b8-m9qkz" Jul 26 16:12:06.638620 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:06.638568 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5cb52e0-d049-45ad-ac1a-64f58ffeec11-istiod-ca-cert\") pod \"e2e-inf-a-7c7176-openshift-default-568d6865b8-m9qkz\" (UID: \"e5cb52e0-d049-45ad-ac1a-64f58ffeec11\") " pod="openshift-ingress/e2e-inf-a-7c7176-openshift-default-568d6865b8-m9qkz" Jul 26 16:12:06.638620 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:06.638608 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5cb52e0-d049-45ad-ac1a-64f58ffeec11-istio-podinfo\") pod \"e2e-inf-a-7c7176-openshift-default-568d6865b8-m9qkz\" (UID: \"e5cb52e0-d049-45ad-ac1a-64f58ffeec11\") " pod="openshift-ingress/e2e-inf-a-7c7176-openshift-default-568d6865b8-m9qkz" Jul 26 16:12:06.638890 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:06.638638 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5cb52e0-d049-45ad-ac1a-64f58ffeec11-workload-socket\") pod \"e2e-inf-a-7c7176-openshift-default-568d6865b8-m9qkz\" (UID: \"e5cb52e0-d049-45ad-ac1a-64f58ffeec11\") " pod="openshift-ingress/e2e-inf-a-7c7176-openshift-default-568d6865b8-m9qkz" Jul 26 16:12:06.638890 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:06.638667 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5cb52e0-d049-45ad-ac1a-64f58ffeec11-istio-data\") pod \"e2e-inf-a-7c7176-openshift-default-568d6865b8-m9qkz\" (UID: \"e5cb52e0-d049-45ad-ac1a-64f58ffeec11\") " pod="openshift-ingress/e2e-inf-a-7c7176-openshift-default-568d6865b8-m9qkz" Jul 26 16:12:06.638890 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:06.638787 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5cb52e0-d049-45ad-ac1a-64f58ffeec11-istio-envoy\") pod \"e2e-inf-a-7c7176-openshift-default-568d6865b8-m9qkz\" (UID: \"e5cb52e0-d049-45ad-ac1a-64f58ffeec11\") " pod="openshift-ingress/e2e-inf-a-7c7176-openshift-default-568d6865b8-m9qkz" Jul 26 16:12:06.638890 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:06.638823 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5cb52e0-d049-45ad-ac1a-64f58ffeec11-credential-socket\") pod \"e2e-inf-a-7c7176-openshift-default-568d6865b8-m9qkz\" (UID: \"e5cb52e0-d049-45ad-ac1a-64f58ffeec11\") " pod="openshift-ingress/e2e-inf-a-7c7176-openshift-default-568d6865b8-m9qkz" Jul 26 16:12:06.639092 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:06.639044 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5cb52e0-d049-45ad-ac1a-64f58ffeec11-workload-socket\") pod \"e2e-inf-a-7c7176-openshift-default-568d6865b8-m9qkz\" (UID: \"e5cb52e0-d049-45ad-ac1a-64f58ffeec11\") " pod="openshift-ingress/e2e-inf-a-7c7176-openshift-default-568d6865b8-m9qkz" Jul 26 16:12:06.639152 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:06.639088 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5cb52e0-d049-45ad-ac1a-64f58ffeec11-workload-certs\") pod \"e2e-inf-a-7c7176-openshift-default-568d6865b8-m9qkz\" (UID: \"e5cb52e0-d049-45ad-ac1a-64f58ffeec11\") " pod="openshift-ingress/e2e-inf-a-7c7176-openshift-default-568d6865b8-m9qkz" Jul 26 16:12:06.639274 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:06.639250 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5cb52e0-d049-45ad-ac1a-64f58ffeec11-istio-data\") pod \"e2e-inf-a-7c7176-openshift-default-568d6865b8-m9qkz\" (UID: \"e5cb52e0-d049-45ad-ac1a-64f58ffeec11\") " pod="openshift-ingress/e2e-inf-a-7c7176-openshift-default-568d6865b8-m9qkz" Jul 26 16:12:06.639372 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:06.639354 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5cb52e0-d049-45ad-ac1a-64f58ffeec11-istiod-ca-cert\") pod \"e2e-inf-a-7c7176-openshift-default-568d6865b8-m9qkz\" (UID: \"e5cb52e0-d049-45ad-ac1a-64f58ffeec11\") " pod="openshift-ingress/e2e-inf-a-7c7176-openshift-default-568d6865b8-m9qkz" Jul 26 16:12:06.641246 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:06.641212 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5cb52e0-d049-45ad-ac1a-64f58ffeec11-istio-envoy\") pod \"e2e-inf-a-7c7176-openshift-default-568d6865b8-m9qkz\" (UID: \"e5cb52e0-d049-45ad-ac1a-64f58ffeec11\") " pod="openshift-ingress/e2e-inf-a-7c7176-openshift-default-568d6865b8-m9qkz" Jul 26 16:12:06.641606 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:06.641586 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5cb52e0-d049-45ad-ac1a-64f58ffeec11-istio-podinfo\") pod \"e2e-inf-a-7c7176-openshift-default-568d6865b8-m9qkz\" (UID: \"e5cb52e0-d049-45ad-ac1a-64f58ffeec11\") " pod="openshift-ingress/e2e-inf-a-7c7176-openshift-default-568d6865b8-m9qkz" Jul 26 16:12:06.647191 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:06.647167 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5cb52e0-d049-45ad-ac1a-64f58ffeec11-istio-token\") pod \"e2e-inf-a-7c7176-openshift-default-568d6865b8-m9qkz\" (UID: \"e5cb52e0-d049-45ad-ac1a-64f58ffeec11\") " pod="openshift-ingress/e2e-inf-a-7c7176-openshift-default-568d6865b8-m9qkz" Jul 26 16:12:06.649165 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:06.649145 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8v9jq\" (UniqueName: \"kubernetes.io/projected/e5cb52e0-d049-45ad-ac1a-64f58ffeec11-kube-api-access-8v9jq\") pod \"e2e-inf-a-7c7176-openshift-default-568d6865b8-m9qkz\" (UID: \"e5cb52e0-d049-45ad-ac1a-64f58ffeec11\") " pod="openshift-ingress/e2e-inf-a-7c7176-openshift-default-568d6865b8-m9qkz" Jul 26 16:12:06.813723 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:06.813676 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7c7176-openshift-default-568d6865b8-m9qkz" Jul 26 16:12:06.957576 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:06.957548 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-7c7176-openshift-default-568d6865b8-m9qkz"] Jul 26 16:12:06.959532 ip-10-0-138-163 kubenswrapper[2577]: W0726 16:12:06.959507 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5cb52e0_d049_45ad_ac1a_64f58ffeec11.slice/crio-47f0678e6d015daee5fe6d01236446bfc4a30ca4d1c2e72b56356766610eb5d7 WatchSource:0}: Error finding container 47f0678e6d015daee5fe6d01236446bfc4a30ca4d1c2e72b56356766610eb5d7: Status 404 returned error can't find the container with id 47f0678e6d015daee5fe6d01236446bfc4a30ca4d1c2e72b56356766610eb5d7 Jul 26 16:12:06.961966 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:06.961937 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:12:06.962039 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:06.961996 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:12:06.962039 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:06.962025 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:12:07.145051 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:07.144949 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7c7176-openshift-default-568d6865b8-m9qkz" event={"ID":"e5cb52e0-d049-45ad-ac1a-64f58ffeec11","Type":"ContainerStarted","Data":"af279cdbd19bbd53ed3f17671e05619fd8e27f11f40e62940cc430f0674df166"} Jul 26 16:12:07.145051 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:07.145022 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7c7176-openshift-default-568d6865b8-m9qkz" event={"ID":"e5cb52e0-d049-45ad-ac1a-64f58ffeec11","Type":"ContainerStarted","Data":"47f0678e6d015daee5fe6d01236446bfc4a30ca4d1c2e72b56356766610eb5d7"} Jul 26 16:12:07.179339 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:07.179275 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-7c7176-openshift-default-568d6865b8-m9qkz" podStartSLOduration=1.179253988 podStartE2EDuration="1.179253988s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 16:12:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 16:12:07.17511146 +0000 UTC m=+2049.914464770" watchObservedRunningTime="2026-07-26 16:12:07.179253988 +0000 UTC m=+2049.918607299" Jul 26 16:12:07.813990 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:07.813949 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-7c7176-openshift-default-568d6865b8-m9qkz" Jul 26 16:12:08.819024 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:08.818992 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-7c7176-openshift-default-568d6865b8-m9qkz" Jul 26 16:12:09.156388 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:09.156312 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-7c7176-openshift-default-568d6865b8-m9qkz" Jul 26 16:12:09.157742 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:09.157720 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-7c7176-openshift-default-568d6865b8-m9qkz" Jul 26 16:12:09.173447 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:09.173415 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:12:09.273435 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:09.273377 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:12:09.399941 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:09.399908 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:12:10.035548 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:10.035515 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:12:10.139650 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:10.139616 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:12:10.216494 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:10.216461 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:12:13.561724 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:13.561693 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:12:13.914434 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:13.914327 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:12:14.171663 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:14.171571 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-959cca-openshift-default-586f57c5c6-7v96k"] Jul 26 16:12:14.176038 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:14.175996 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-959cca-openshift-default-586f57c5c6-7v96k" Jul 26 16:12:14.179201 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:14.179170 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-959cca-openshift-default-dockercfg-nd8lk\"" Jul 26 16:12:14.188312 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:14.188284 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-959cca-openshift-default-586f57c5c6-7v96k"] Jul 26 16:12:14.213920 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:14.213891 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:12:14.306794 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:14.306757 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15a17353-4f84-47ca-a17e-4636c0e6c801-istio-podinfo\") pod \"e2e-inf-b-959cca-openshift-default-586f57c5c6-7v96k\" (UID: \"15a17353-4f84-47ca-a17e-4636c0e6c801\") " pod="openshift-ingress/e2e-inf-b-959cca-openshift-default-586f57c5c6-7v96k" Jul 26 16:12:14.306794 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:14.306792 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2l65p\" (UniqueName: \"kubernetes.io/projected/15a17353-4f84-47ca-a17e-4636c0e6c801-kube-api-access-2l65p\") pod \"e2e-inf-b-959cca-openshift-default-586f57c5c6-7v96k\" (UID: \"15a17353-4f84-47ca-a17e-4636c0e6c801\") " pod="openshift-ingress/e2e-inf-b-959cca-openshift-default-586f57c5c6-7v96k" Jul 26 16:12:14.307059 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:14.306815 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15a17353-4f84-47ca-a17e-4636c0e6c801-istiod-ca-cert\") pod \"e2e-inf-b-959cca-openshift-default-586f57c5c6-7v96k\" (UID: \"15a17353-4f84-47ca-a17e-4636c0e6c801\") " pod="openshift-ingress/e2e-inf-b-959cca-openshift-default-586f57c5c6-7v96k" Jul 26 16:12:14.307059 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:14.306835 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15a17353-4f84-47ca-a17e-4636c0e6c801-workload-socket\") pod \"e2e-inf-b-959cca-openshift-default-586f57c5c6-7v96k\" (UID: \"15a17353-4f84-47ca-a17e-4636c0e6c801\") " pod="openshift-ingress/e2e-inf-b-959cca-openshift-default-586f57c5c6-7v96k" Jul 26 16:12:14.307059 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:14.306912 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15a17353-4f84-47ca-a17e-4636c0e6c801-istio-data\") pod \"e2e-inf-b-959cca-openshift-default-586f57c5c6-7v96k\" (UID: \"15a17353-4f84-47ca-a17e-4636c0e6c801\") " pod="openshift-ingress/e2e-inf-b-959cca-openshift-default-586f57c5c6-7v96k" Jul 26 16:12:14.307059 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:14.306982 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15a17353-4f84-47ca-a17e-4636c0e6c801-workload-certs\") pod \"e2e-inf-b-959cca-openshift-default-586f57c5c6-7v96k\" (UID: \"15a17353-4f84-47ca-a17e-4636c0e6c801\") " pod="openshift-ingress/e2e-inf-b-959cca-openshift-default-586f57c5c6-7v96k" Jul 26 16:12:14.307059 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:14.307037 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15a17353-4f84-47ca-a17e-4636c0e6c801-istio-envoy\") pod \"e2e-inf-b-959cca-openshift-default-586f57c5c6-7v96k\" (UID: \"15a17353-4f84-47ca-a17e-4636c0e6c801\") " pod="openshift-ingress/e2e-inf-b-959cca-openshift-default-586f57c5c6-7v96k" Jul 26 16:12:14.307273 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:14.307068 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15a17353-4f84-47ca-a17e-4636c0e6c801-istio-token\") pod \"e2e-inf-b-959cca-openshift-default-586f57c5c6-7v96k\" (UID: \"15a17353-4f84-47ca-a17e-4636c0e6c801\") " pod="openshift-ingress/e2e-inf-b-959cca-openshift-default-586f57c5c6-7v96k" Jul 26 16:12:14.307273 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:14.307093 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15a17353-4f84-47ca-a17e-4636c0e6c801-credential-socket\") pod \"e2e-inf-b-959cca-openshift-default-586f57c5c6-7v96k\" (UID: \"15a17353-4f84-47ca-a17e-4636c0e6c801\") " pod="openshift-ingress/e2e-inf-b-959cca-openshift-default-586f57c5c6-7v96k" Jul 26 16:12:14.403276 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:14.403237 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:12:14.407987 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:14.407961 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15a17353-4f84-47ca-a17e-4636c0e6c801-istio-envoy\") pod \"e2e-inf-b-959cca-openshift-default-586f57c5c6-7v96k\" (UID: \"15a17353-4f84-47ca-a17e-4636c0e6c801\") " pod="openshift-ingress/e2e-inf-b-959cca-openshift-default-586f57c5c6-7v96k" Jul 26 16:12:14.408095 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:14.408013 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15a17353-4f84-47ca-a17e-4636c0e6c801-istio-token\") pod \"e2e-inf-b-959cca-openshift-default-586f57c5c6-7v96k\" (UID: \"15a17353-4f84-47ca-a17e-4636c0e6c801\") " pod="openshift-ingress/e2e-inf-b-959cca-openshift-default-586f57c5c6-7v96k" Jul 26 16:12:14.408095 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:14.408047 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15a17353-4f84-47ca-a17e-4636c0e6c801-credential-socket\") pod \"e2e-inf-b-959cca-openshift-default-586f57c5c6-7v96k\" (UID: \"15a17353-4f84-47ca-a17e-4636c0e6c801\") " pod="openshift-ingress/e2e-inf-b-959cca-openshift-default-586f57c5c6-7v96k" Jul 26 16:12:14.408179 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:14.408103 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15a17353-4f84-47ca-a17e-4636c0e6c801-istio-podinfo\") pod \"e2e-inf-b-959cca-openshift-default-586f57c5c6-7v96k\" (UID: \"15a17353-4f84-47ca-a17e-4636c0e6c801\") " pod="openshift-ingress/e2e-inf-b-959cca-openshift-default-586f57c5c6-7v96k" Jul 26 16:12:14.408179 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:14.408130 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2l65p\" (UniqueName: \"kubernetes.io/projected/15a17353-4f84-47ca-a17e-4636c0e6c801-kube-api-access-2l65p\") pod \"e2e-inf-b-959cca-openshift-default-586f57c5c6-7v96k\" (UID: \"15a17353-4f84-47ca-a17e-4636c0e6c801\") " pod="openshift-ingress/e2e-inf-b-959cca-openshift-default-586f57c5c6-7v96k" Jul 26 16:12:14.408179 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:14.408170 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15a17353-4f84-47ca-a17e-4636c0e6c801-istiod-ca-cert\") pod \"e2e-inf-b-959cca-openshift-default-586f57c5c6-7v96k\" (UID: \"15a17353-4f84-47ca-a17e-4636c0e6c801\") " pod="openshift-ingress/e2e-inf-b-959cca-openshift-default-586f57c5c6-7v96k" Jul 26 16:12:14.408538 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:14.408506 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15a17353-4f84-47ca-a17e-4636c0e6c801-credential-socket\") pod \"e2e-inf-b-959cca-openshift-default-586f57c5c6-7v96k\" (UID: \"15a17353-4f84-47ca-a17e-4636c0e6c801\") " pod="openshift-ingress/e2e-inf-b-959cca-openshift-default-586f57c5c6-7v96k" Jul 26 16:12:14.408538 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:14.408205 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15a17353-4f84-47ca-a17e-4636c0e6c801-workload-socket\") pod \"e2e-inf-b-959cca-openshift-default-586f57c5c6-7v96k\" (UID: \"15a17353-4f84-47ca-a17e-4636c0e6c801\") " pod="openshift-ingress/e2e-inf-b-959cca-openshift-default-586f57c5c6-7v96k" Jul 26 16:12:14.408728 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:14.408612 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15a17353-4f84-47ca-a17e-4636c0e6c801-istio-data\") pod \"e2e-inf-b-959cca-openshift-default-586f57c5c6-7v96k\" (UID: \"15a17353-4f84-47ca-a17e-4636c0e6c801\") " pod="openshift-ingress/e2e-inf-b-959cca-openshift-default-586f57c5c6-7v96k" Jul 26 16:12:14.408728 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:14.408666 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15a17353-4f84-47ca-a17e-4636c0e6c801-workload-socket\") pod \"e2e-inf-b-959cca-openshift-default-586f57c5c6-7v96k\" (UID: \"15a17353-4f84-47ca-a17e-4636c0e6c801\") " pod="openshift-ingress/e2e-inf-b-959cca-openshift-default-586f57c5c6-7v96k" Jul 26 16:12:14.408728 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:14.408693 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15a17353-4f84-47ca-a17e-4636c0e6c801-workload-certs\") pod \"e2e-inf-b-959cca-openshift-default-586f57c5c6-7v96k\" (UID: \"15a17353-4f84-47ca-a17e-4636c0e6c801\") " pod="openshift-ingress/e2e-inf-b-959cca-openshift-default-586f57c5c6-7v96k" Jul 26 16:12:14.408950 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:14.408929 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15a17353-4f84-47ca-a17e-4636c0e6c801-istiod-ca-cert\") pod \"e2e-inf-b-959cca-openshift-default-586f57c5c6-7v96k\" (UID: \"15a17353-4f84-47ca-a17e-4636c0e6c801\") " pod="openshift-ingress/e2e-inf-b-959cca-openshift-default-586f57c5c6-7v96k" Jul 26 16:12:14.408993 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:14.408935 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15a17353-4f84-47ca-a17e-4636c0e6c801-istio-data\") pod \"e2e-inf-b-959cca-openshift-default-586f57c5c6-7v96k\" (UID: \"15a17353-4f84-47ca-a17e-4636c0e6c801\") " pod="openshift-ingress/e2e-inf-b-959cca-openshift-default-586f57c5c6-7v96k" Jul 26 16:12:14.409032 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:14.408996 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15a17353-4f84-47ca-a17e-4636c0e6c801-workload-certs\") pod \"e2e-inf-b-959cca-openshift-default-586f57c5c6-7v96k\" (UID: \"15a17353-4f84-47ca-a17e-4636c0e6c801\") " pod="openshift-ingress/e2e-inf-b-959cca-openshift-default-586f57c5c6-7v96k" Jul 26 16:12:14.410406 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:14.410369 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15a17353-4f84-47ca-a17e-4636c0e6c801-istio-envoy\") pod \"e2e-inf-b-959cca-openshift-default-586f57c5c6-7v96k\" (UID: \"15a17353-4f84-47ca-a17e-4636c0e6c801\") " pod="openshift-ingress/e2e-inf-b-959cca-openshift-default-586f57c5c6-7v96k" Jul 26 16:12:14.410784 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:14.410763 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15a17353-4f84-47ca-a17e-4636c0e6c801-istio-podinfo\") pod \"e2e-inf-b-959cca-openshift-default-586f57c5c6-7v96k\" (UID: \"15a17353-4f84-47ca-a17e-4636c0e6c801\") " pod="openshift-ingress/e2e-inf-b-959cca-openshift-default-586f57c5c6-7v96k" Jul 26 16:12:14.415374 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:14.415351 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15a17353-4f84-47ca-a17e-4636c0e6c801-istio-token\") pod \"e2e-inf-b-959cca-openshift-default-586f57c5c6-7v96k\" (UID: \"15a17353-4f84-47ca-a17e-4636c0e6c801\") " pod="openshift-ingress/e2e-inf-b-959cca-openshift-default-586f57c5c6-7v96k" Jul 26 16:12:14.415549 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:14.415531 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2l65p\" (UniqueName: \"kubernetes.io/projected/15a17353-4f84-47ca-a17e-4636c0e6c801-kube-api-access-2l65p\") pod \"e2e-inf-b-959cca-openshift-default-586f57c5c6-7v96k\" (UID: \"15a17353-4f84-47ca-a17e-4636c0e6c801\") " pod="openshift-ingress/e2e-inf-b-959cca-openshift-default-586f57c5c6-7v96k" Jul 26 16:12:14.495673 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:14.495574 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-959cca-openshift-default-586f57c5c6-7v96k" Jul 26 16:12:14.637181 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:14.637146 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-959cca-openshift-default-586f57c5c6-7v96k"] Jul 26 16:12:14.637761 ip-10-0-138-163 kubenswrapper[2577]: W0726 16:12:14.637729 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15a17353_4f84_47ca_a17e_4636c0e6c801.slice/crio-a1e91fc87e4e0ae7663fc3dcd6cf4e3d06aca58fe39bea311d435aab4bf60cb5 WatchSource:0}: Error finding container a1e91fc87e4e0ae7663fc3dcd6cf4e3d06aca58fe39bea311d435aab4bf60cb5: Status 404 returned error can't find the container with id a1e91fc87e4e0ae7663fc3dcd6cf4e3d06aca58fe39bea311d435aab4bf60cb5 Jul 26 16:12:14.639882 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:14.639851 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:12:14.639983 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:14.639912 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:12:14.639983 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:14.639943 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:12:15.196105 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:15.196072 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-959cca-openshift-default-586f57c5c6-7v96k" event={"ID":"15a17353-4f84-47ca-a17e-4636c0e6c801","Type":"ContainerStarted","Data":"46d86280942781a4cdb951c1645050c607a0923dcdfe3007b9d9d6fd69c84ca2"} Jul 26 16:12:15.196294 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:15.196125 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-959cca-openshift-default-586f57c5c6-7v96k" event={"ID":"15a17353-4f84-47ca-a17e-4636c0e6c801","Type":"ContainerStarted","Data":"a1e91fc87e4e0ae7663fc3dcd6cf4e3d06aca58fe39bea311d435aab4bf60cb5"} Jul 26 16:12:15.217720 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:15.217667 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-959cca-openshift-default-586f57c5c6-7v96k" podStartSLOduration=1.217651046 podStartE2EDuration="1.217651046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 16:12:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 16:12:15.215738391 +0000 UTC m=+2057.955091700" watchObservedRunningTime="2026-07-26 16:12:15.217651046 +0000 UTC m=+2057.957004416" Jul 26 16:12:15.496371 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:15.496284 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-959cca-openshift-default-586f57c5c6-7v96k" Jul 26 16:12:15.501548 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:15.501522 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-959cca-openshift-default-586f57c5c6-7v96k" Jul 26 16:12:16.201765 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:16.201731 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-959cca-openshift-default-586f57c5c6-7v96k" Jul 26 16:12:16.202659 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:16.202639 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-959cca-openshift-default-586f57c5c6-7v96k" Jul 26 16:12:17.134368 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:17.134338 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:12:17.261526 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:17.261492 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:12:17.404123 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:17.404020 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:12:17.509559 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:17.509528 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:12:17.877766 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:17.877729 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:12:17.979121 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:17.979085 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:12:19.190279 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:19.190242 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:12:21.590359 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:21.590327 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:12:24.319057 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:24.318995 2577 patch_prober.go:28] interesting pod/e2e-inf-a-7c7176-openshift-default-568d6865b8-m9qkz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": read tcp 10.132.0.2:58770->10.132.0.64:15021: read: connection reset by peer" start-of-body= Jul 26 16:12:24.319446 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:24.319054 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-7c7176-openshift-default-568d6865b8-m9qkz" podUID="e5cb52e0-d049-45ad-ac1a-64f58ffeec11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": read tcp 10.132.0.2:58770->10.132.0.64:15021: read: connection reset by peer" Jul 26 16:12:25.260729 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:25.260692 2577 generic.go:356] "Generic (PLEG): container finished" podID="e5cb52e0-d049-45ad-ac1a-64f58ffeec11" containerID="af279cdbd19bbd53ed3f17671e05619fd8e27f11f40e62940cc430f0674df166" exitCode=0 Jul 26 16:12:25.260934 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:25.260757 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7c7176-openshift-default-568d6865b8-m9qkz" event={"ID":"e5cb52e0-d049-45ad-ac1a-64f58ffeec11","Type":"ContainerDied","Data":"af279cdbd19bbd53ed3f17671e05619fd8e27f11f40e62940cc430f0674df166"} Jul 26 16:12:25.261257 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:25.261226 2577 scope.go:117] "RemoveContainer" containerID="af279cdbd19bbd53ed3f17671e05619fd8e27f11f40e62940cc430f0674df166" Jul 26 16:12:25.262500 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:25.262468 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:12:25.262633 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:25.262542 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:12:25.262633 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:25.262575 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:12:26.268976 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:26.268946 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7c7176-openshift-default-568d6865b8-m9qkz" event={"ID":"e5cb52e0-d049-45ad-ac1a-64f58ffeec11","Type":"ContainerStarted","Data":"d8ef5e352a8dd86a5314f423bb88a74b796bb76743b2c9b15a50973c20f2f77c"} Jul 26 16:12:26.814815 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:26.814773 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-7c7176-openshift-default-568d6865b8-m9qkz" Jul 26 16:12:26.815019 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:26.814994 2577 patch_prober.go:28] interesting pod/e2e-inf-a-7c7176-openshift-default-568d6865b8-m9qkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Jul 26 16:12:26.815079 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:26.815056 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7c7176-openshift-default-568d6865b8-m9qkz" podUID="e5cb52e0-d049-45ad-ac1a-64f58ffeec11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Jul 26 16:12:27.814500 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:27.814461 2577 patch_prober.go:28] interesting pod/e2e-inf-a-7c7176-openshift-default-568d6865b8-m9qkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Jul 26 16:12:27.814897 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:27.814516 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7c7176-openshift-default-568d6865b8-m9qkz" podUID="e5cb52e0-d049-45ad-ac1a-64f58ffeec11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Jul 26 16:12:28.814332 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:28.814295 2577 patch_prober.go:28] interesting pod/e2e-inf-a-7c7176-openshift-default-568d6865b8-m9qkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Jul 26 16:12:28.814531 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:28.814350 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7c7176-openshift-default-568d6865b8-m9qkz" podUID="e5cb52e0-d049-45ad-ac1a-64f58ffeec11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Jul 26 16:12:29.814386 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:29.814347 2577 patch_prober.go:28] interesting pod/e2e-inf-a-7c7176-openshift-default-568d6865b8-m9qkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Jul 26 16:12:29.814567 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:29.814423 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7c7176-openshift-default-568d6865b8-m9qkz" podUID="e5cb52e0-d049-45ad-ac1a-64f58ffeec11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Jul 26 16:12:30.818266 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:30.818230 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-7c7176-openshift-default-568d6865b8-m9qkz" Jul 26 16:12:30.818685 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:30.818649 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-7c7176-openshift-default-568d6865b8-m9qkz" Jul 26 16:12:30.819445 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:30.819421 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-7c7176-openshift-default-568d6865b8-m9qkz" Jul 26 16:12:31.650877 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:31.650835 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:12:32.243568 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:32.243527 2577 patch_prober.go:28] interesting pod/e2e-inf-b-959cca-openshift-default-586f57c5c6-7v96k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 26 16:12:32.243957 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:32.243592 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-959cca-openshift-default-586f57c5c6-7v96k" podUID="15a17353-4f84-47ca-a17e-4636c0e6c801" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 26 16:12:33.314999 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:33.314965 2577 generic.go:356] "Generic (PLEG): container finished" podID="15a17353-4f84-47ca-a17e-4636c0e6c801" containerID="46d86280942781a4cdb951c1645050c607a0923dcdfe3007b9d9d6fd69c84ca2" exitCode=0 Jul 26 16:12:33.315460 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:33.315018 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-959cca-openshift-default-586f57c5c6-7v96k" event={"ID":"15a17353-4f84-47ca-a17e-4636c0e6c801","Type":"ContainerDied","Data":"46d86280942781a4cdb951c1645050c607a0923dcdfe3007b9d9d6fd69c84ca2"} Jul 26 16:12:33.315460 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:33.315427 2577 scope.go:117] "RemoveContainer" containerID="46d86280942781a4cdb951c1645050c607a0923dcdfe3007b9d9d6fd69c84ca2" Jul 26 16:12:33.316447 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:33.316420 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:12:33.316517 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:33.316484 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:12:33.316554 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:33.316517 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:12:34.322808 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:34.322776 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-959cca-openshift-default-586f57c5c6-7v96k" event={"ID":"15a17353-4f84-47ca-a17e-4636c0e6c801","Type":"ContainerStarted","Data":"f156f8d2a3d4369884f80edf77f33dbaefcf520e03247c23fe3d1f11ae2d0e2a"} Jul 26 16:12:34.496346 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:34.496306 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-959cca-openshift-default-586f57c5c6-7v96k" Jul 26 16:12:34.496563 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:34.496538 2577 patch_prober.go:28] interesting pod/e2e-inf-b-959cca-openshift-default-586f57c5c6-7v96k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Jul 26 16:12:34.496608 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:34.496587 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-959cca-openshift-default-586f57c5c6-7v96k" podUID="15a17353-4f84-47ca-a17e-4636c0e6c801" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Jul 26 16:12:35.496250 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:35.496213 2577 patch_prober.go:28] interesting pod/e2e-inf-b-959cca-openshift-default-586f57c5c6-7v96k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Jul 26 16:12:35.496786 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:35.496275 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-959cca-openshift-default-586f57c5c6-7v96k" podUID="15a17353-4f84-47ca-a17e-4636c0e6c801" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Jul 26 16:12:36.496355 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:36.496312 2577 patch_prober.go:28] interesting pod/e2e-inf-b-959cca-openshift-default-586f57c5c6-7v96k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Jul 26 16:12:36.496791 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:36.496374 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-959cca-openshift-default-586f57c5c6-7v96k" podUID="15a17353-4f84-47ca-a17e-4636c0e6c801" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Jul 26 16:12:37.496850 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:37.496802 2577 patch_prober.go:28] interesting pod/e2e-inf-b-959cca-openshift-default-586f57c5c6-7v96k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Jul 26 16:12:37.497300 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:37.496861 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-959cca-openshift-default-586f57c5c6-7v96k" podUID="15a17353-4f84-47ca-a17e-4636c0e6c801" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Jul 26 16:12:38.500583 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:38.500552 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-959cca-openshift-default-586f57c5c6-7v96k" Jul 26 16:12:38.501042 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:38.500820 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-959cca-openshift-default-586f57c5c6-7v96k" Jul 26 16:12:38.501628 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:38.501610 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-959cca-openshift-default-586f57c5c6-7v96k" Jul 26 16:12:57.919447 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:57.919417 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-8wjqp_b6b8080d-0cee-4d0c-a05c-3e2f1b672a54/console-operator/1.log" Jul 26 16:12:57.920871 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:57.920849 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-8wjqp_b6b8080d-0cee-4d0c-a05c-3e2f1b672a54/console-operator/1.log" Jul 26 16:12:57.923488 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:57.923465 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ztlp6_cbe2cd41-6ebc-4722-a813-57d83fa1d242/ovn-acl-logging/0.log" Jul 26 16:12:57.925018 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:12:57.924999 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ztlp6_cbe2cd41-6ebc-4722-a813-57d83fa1d242/ovn-acl-logging/0.log" Jul 26 16:13:10.500372 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:13:10.500338 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:13:33.853672 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:13:33.853633 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:13:51.580563 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:13:51.580483 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:13:52.526331 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:13:52.526292 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:14:03.050473 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:03.050434 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:14:03.149789 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:03.149754 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:14:03.206442 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:03.206382 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:14:07.699017 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:07.698965 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:14:08.081339 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:08.081303 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:14:13.715708 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:13.715669 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:14:13.817029 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:13.816983 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:14:13.953425 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:13.953378 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-7c7176-openshift-default-568d6865b8-m9qkz"] Jul 26 16:14:13.953686 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:13.953637 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-7c7176-openshift-default-568d6865b8-m9qkz" podUID="e5cb52e0-d049-45ad-ac1a-64f58ffeec11" containerName="istio-proxy" containerID="cri-o://d8ef5e352a8dd86a5314f423bb88a74b796bb76743b2c9b15a50973c20f2f77c" gracePeriod=30 Jul 26 16:14:13.977380 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:13.977297 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:14:14.193153 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:14.193108 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:14:15.821237 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:15.821194 2577 patch_prober.go:28] interesting pod/e2e-inf-a-7c7176-openshift-default-568d6865b8-m9qkz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 26 16:14:15.821654 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:15.821266 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-7c7176-openshift-default-568d6865b8-m9qkz" podUID="e5cb52e0-d049-45ad-ac1a-64f58ffeec11" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 26 16:14:19.226014 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:19.225947 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7c7176-openshift-default-568d6865b8-m9qkz" Jul 26 16:14:19.293551 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:19.293516 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5cb52e0-d049-45ad-ac1a-64f58ffeec11-istio-token\") pod \"e5cb52e0-d049-45ad-ac1a-64f58ffeec11\" (UID: \"e5cb52e0-d049-45ad-ac1a-64f58ffeec11\") " Jul 26 16:14:19.293733 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:19.293573 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8v9jq\" (UniqueName: \"kubernetes.io/projected/e5cb52e0-d049-45ad-ac1a-64f58ffeec11-kube-api-access-8v9jq\") pod \"e5cb52e0-d049-45ad-ac1a-64f58ffeec11\" (UID: \"e5cb52e0-d049-45ad-ac1a-64f58ffeec11\") " Jul 26 16:14:19.293733 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:19.293607 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5cb52e0-d049-45ad-ac1a-64f58ffeec11-istio-podinfo\") pod \"e5cb52e0-d049-45ad-ac1a-64f58ffeec11\" (UID: \"e5cb52e0-d049-45ad-ac1a-64f58ffeec11\") " Jul 26 16:14:19.293733 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:19.293628 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5cb52e0-d049-45ad-ac1a-64f58ffeec11-workload-socket\") pod \"e5cb52e0-d049-45ad-ac1a-64f58ffeec11\" (UID: \"e5cb52e0-d049-45ad-ac1a-64f58ffeec11\") " Jul 26 16:14:19.293733 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:19.293664 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5cb52e0-d049-45ad-ac1a-64f58ffeec11-istio-data\") pod \"e5cb52e0-d049-45ad-ac1a-64f58ffeec11\" (UID: \"e5cb52e0-d049-45ad-ac1a-64f58ffeec11\") " Jul 26 16:14:19.293733 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:19.293720 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5cb52e0-d049-45ad-ac1a-64f58ffeec11-workload-certs\") pod \"e5cb52e0-d049-45ad-ac1a-64f58ffeec11\" (UID: \"e5cb52e0-d049-45ad-ac1a-64f58ffeec11\") " Jul 26 16:14:19.294015 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:19.293768 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5cb52e0-d049-45ad-ac1a-64f58ffeec11-istio-envoy\") pod \"e5cb52e0-d049-45ad-ac1a-64f58ffeec11\" (UID: \"e5cb52e0-d049-45ad-ac1a-64f58ffeec11\") " Jul 26 16:14:19.294015 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:19.293796 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5cb52e0-d049-45ad-ac1a-64f58ffeec11-istiod-ca-cert\") pod \"e5cb52e0-d049-45ad-ac1a-64f58ffeec11\" (UID: \"e5cb52e0-d049-45ad-ac1a-64f58ffeec11\") " Jul 26 16:14:19.294015 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:19.293869 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5cb52e0-d049-45ad-ac1a-64f58ffeec11-credential-socket\") pod \"e5cb52e0-d049-45ad-ac1a-64f58ffeec11\" (UID: \"e5cb52e0-d049-45ad-ac1a-64f58ffeec11\") " Jul 26 16:14:19.294015 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:19.294005 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5cb52e0-d049-45ad-ac1a-64f58ffeec11-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e5cb52e0-d049-45ad-ac1a-64f58ffeec11" (UID: "e5cb52e0-d049-45ad-ac1a-64f58ffeec11"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:14:19.294232 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:19.294096 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5cb52e0-d049-45ad-ac1a-64f58ffeec11-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e5cb52e0-d049-45ad-ac1a-64f58ffeec11" (UID: "e5cb52e0-d049-45ad-ac1a-64f58ffeec11"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:14:19.294232 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:19.294186 2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5cb52e0-d049-45ad-ac1a-64f58ffeec11-workload-certs\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 16:14:19.294232 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:19.294204 2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5cb52e0-d049-45ad-ac1a-64f58ffeec11-workload-socket\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 16:14:19.294438 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:19.294391 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5cb52e0-d049-45ad-ac1a-64f58ffeec11-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e5cb52e0-d049-45ad-ac1a-64f58ffeec11" (UID: "e5cb52e0-d049-45ad-ac1a-64f58ffeec11"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:14:19.294438 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:19.294431 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5cb52e0-d049-45ad-ac1a-64f58ffeec11-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e5cb52e0-d049-45ad-ac1a-64f58ffeec11" (UID: "e5cb52e0-d049-45ad-ac1a-64f58ffeec11"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 16:14:19.294694 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:19.294671 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5cb52e0-d049-45ad-ac1a-64f58ffeec11-istio-data" (OuterVolumeSpecName: "istio-data") pod "e5cb52e0-d049-45ad-ac1a-64f58ffeec11" (UID: "e5cb52e0-d049-45ad-ac1a-64f58ffeec11"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:14:19.296191 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:19.296168 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5cb52e0-d049-45ad-ac1a-64f58ffeec11-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e5cb52e0-d049-45ad-ac1a-64f58ffeec11" (UID: "e5cb52e0-d049-45ad-ac1a-64f58ffeec11"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:14:19.296321 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:19.296268 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5cb52e0-d049-45ad-ac1a-64f58ffeec11-istio-token" (OuterVolumeSpecName: "istio-token") pod "e5cb52e0-d049-45ad-ac1a-64f58ffeec11" (UID: "e5cb52e0-d049-45ad-ac1a-64f58ffeec11"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 16:14:19.296321 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:19.296274 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e5cb52e0-d049-45ad-ac1a-64f58ffeec11-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e5cb52e0-d049-45ad-ac1a-64f58ffeec11" (UID: "e5cb52e0-d049-45ad-ac1a-64f58ffeec11"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 26 16:14:19.296493 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:19.296472 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5cb52e0-d049-45ad-ac1a-64f58ffeec11-kube-api-access-8v9jq" (OuterVolumeSpecName: "kube-api-access-8v9jq") pod "e5cb52e0-d049-45ad-ac1a-64f58ffeec11" (UID: "e5cb52e0-d049-45ad-ac1a-64f58ffeec11"). InnerVolumeSpecName "kube-api-access-8v9jq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 16:14:19.395700 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:19.395659 2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5cb52e0-d049-45ad-ac1a-64f58ffeec11-credential-socket\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 16:14:19.395700 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:19.395691 2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5cb52e0-d049-45ad-ac1a-64f58ffeec11-istio-token\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 16:14:19.395700 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:19.395705 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8v9jq\" (UniqueName: \"kubernetes.io/projected/e5cb52e0-d049-45ad-ac1a-64f58ffeec11-kube-api-access-8v9jq\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 16:14:19.395953 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:19.395718 2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5cb52e0-d049-45ad-ac1a-64f58ffeec11-istio-podinfo\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 16:14:19.395953 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:19.395731 2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5cb52e0-d049-45ad-ac1a-64f58ffeec11-istio-data\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 16:14:19.395953 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:19.395742 2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5cb52e0-d049-45ad-ac1a-64f58ffeec11-istio-envoy\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 16:14:19.395953 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:19.395752 2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5cb52e0-d049-45ad-ac1a-64f58ffeec11-istiod-ca-cert\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 16:14:19.919235 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:19.919205 2577 generic.go:356] "Generic (PLEG): container finished" podID="e5cb52e0-d049-45ad-ac1a-64f58ffeec11" containerID="d8ef5e352a8dd86a5314f423bb88a74b796bb76743b2c9b15a50973c20f2f77c" exitCode=0 Jul 26 16:14:19.919437 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:19.919258 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7c7176-openshift-default-568d6865b8-m9qkz" Jul 26 16:14:19.919437 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:19.919298 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7c7176-openshift-default-568d6865b8-m9qkz" event={"ID":"e5cb52e0-d049-45ad-ac1a-64f58ffeec11","Type":"ContainerDied","Data":"d8ef5e352a8dd86a5314f423bb88a74b796bb76743b2c9b15a50973c20f2f77c"} Jul 26 16:14:19.919437 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:19.919352 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7c7176-openshift-default-568d6865b8-m9qkz" event={"ID":"e5cb52e0-d049-45ad-ac1a-64f58ffeec11","Type":"ContainerDied","Data":"47f0678e6d015daee5fe6d01236446bfc4a30ca4d1c2e72b56356766610eb5d7"} Jul 26 16:14:19.919437 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:19.919374 2577 scope.go:117] "RemoveContainer" containerID="d8ef5e352a8dd86a5314f423bb88a74b796bb76743b2c9b15a50973c20f2f77c" Jul 26 16:14:19.929225 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:19.929208 2577 scope.go:117] "RemoveContainer" containerID="af279cdbd19bbd53ed3f17671e05619fd8e27f11f40e62940cc430f0674df166" Jul 26 16:14:19.937818 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:19.937796 2577 scope.go:117] "RemoveContainer" containerID="d8ef5e352a8dd86a5314f423bb88a74b796bb76743b2c9b15a50973c20f2f77c" Jul 26 16:14:19.938133 ip-10-0-138-163 kubenswrapper[2577]: E0726 16:14:19.938112 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8ef5e352a8dd86a5314f423bb88a74b796bb76743b2c9b15a50973c20f2f77c\": container with ID starting with d8ef5e352a8dd86a5314f423bb88a74b796bb76743b2c9b15a50973c20f2f77c not found: ID does not exist" containerID="d8ef5e352a8dd86a5314f423bb88a74b796bb76743b2c9b15a50973c20f2f77c" Jul 26 16:14:19.938199 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:19.938146 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8ef5e352a8dd86a5314f423bb88a74b796bb76743b2c9b15a50973c20f2f77c"} err="failed to get container status \"d8ef5e352a8dd86a5314f423bb88a74b796bb76743b2c9b15a50973c20f2f77c\": rpc error: code = NotFound desc = could not find container \"d8ef5e352a8dd86a5314f423bb88a74b796bb76743b2c9b15a50973c20f2f77c\": container with ID starting with d8ef5e352a8dd86a5314f423bb88a74b796bb76743b2c9b15a50973c20f2f77c not found: ID does not exist" Jul 26 16:14:19.938199 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:19.938176 2577 scope.go:117] "RemoveContainer" containerID="af279cdbd19bbd53ed3f17671e05619fd8e27f11f40e62940cc430f0674df166" Jul 26 16:14:19.938296 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:19.938156 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-7c7176-openshift-default-568d6865b8-m9qkz"] Jul 26 16:14:19.938492 ip-10-0-138-163 kubenswrapper[2577]: E0726 16:14:19.938474 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af279cdbd19bbd53ed3f17671e05619fd8e27f11f40e62940cc430f0674df166\": container with ID starting with af279cdbd19bbd53ed3f17671e05619fd8e27f11f40e62940cc430f0674df166 not found: ID does not exist" containerID="af279cdbd19bbd53ed3f17671e05619fd8e27f11f40e62940cc430f0674df166" Jul 26 16:14:19.938552 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:19.938499 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af279cdbd19bbd53ed3f17671e05619fd8e27f11f40e62940cc430f0674df166"} err="failed to get container status \"af279cdbd19bbd53ed3f17671e05619fd8e27f11f40e62940cc430f0674df166\": rpc error: code = NotFound desc = could not find container \"af279cdbd19bbd53ed3f17671e05619fd8e27f11f40e62940cc430f0674df166\": container with ID starting with af279cdbd19bbd53ed3f17671e05619fd8e27f11f40e62940cc430f0674df166 not found: ID does not exist" Jul 26 16:14:19.941906 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:19.941886 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-7c7176-openshift-default-568d6865b8-m9qkz"] Jul 26 16:14:21.787767 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:21.787732 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e5cb52e0-d049-45ad-ac1a-64f58ffeec11" path="/var/lib/kubelet/pods/e5cb52e0-d049-45ad-ac1a-64f58ffeec11/volumes" Jul 26 16:14:24.760098 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:24.760057 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:14:24.838971 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:24.838928 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:14:24.902128 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:24.902088 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:14:29.369707 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:29.369667 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:14:29.737525 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:29.737415 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:14:35.441055 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:35.441014 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:14:35.541251 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:35.541218 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:14:35.667223 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:35.667188 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-959cca-openshift-default-586f57c5c6-7v96k"] Jul 26 16:14:35.667495 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:35.667446 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-959cca-openshift-default-586f57c5c6-7v96k" podUID="15a17353-4f84-47ca-a17e-4636c0e6c801" containerName="istio-proxy" containerID="cri-o://f156f8d2a3d4369884f80edf77f33dbaefcf520e03247c23fe3d1f11ae2d0e2a" gracePeriod=30 Jul 26 16:14:35.694306 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:35.694211 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:14:35.913143 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:35.913106 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:14:36.251603 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:36.251567 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:14:36.523974 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:36.523944 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-4b0d96-openshift-default-5d8cc86f8-v8b4l"] Jul 26 16:14:36.524466 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:36.524388 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5cb52e0-d049-45ad-ac1a-64f58ffeec11" containerName="istio-proxy" Jul 26 16:14:36.524466 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:36.524417 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5cb52e0-d049-45ad-ac1a-64f58ffeec11" containerName="istio-proxy" Jul 26 16:14:36.524592 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:36.524501 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5cb52e0-d049-45ad-ac1a-64f58ffeec11" containerName="istio-proxy" Jul 26 16:14:36.524592 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:36.524510 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5cb52e0-d049-45ad-ac1a-64f58ffeec11" containerName="istio-proxy" Jul 26 16:14:36.527449 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:36.527432 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-4b0d96-openshift-default-5d8cc86f8-v8b4l" Jul 26 16:14:36.529848 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:36.529827 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-4b0d96-openshift-default-dockercfg-mp8c8\"" Jul 26 16:14:36.538222 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:36.538200 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-4b0d96-openshift-default-5d8cc86f8-v8b4l"] Jul 26 16:14:36.549858 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:36.549831 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:14:36.653466 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:36.653426 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25d7898d-3961-4da3-a12e-3cce13ff8457-workload-socket\") pod \"e2e-api-a-4b0d96-openshift-default-5d8cc86f8-v8b4l\" (UID: \"25d7898d-3961-4da3-a12e-3cce13ff8457\") " pod="openshift-ingress/e2e-api-a-4b0d96-openshift-default-5d8cc86f8-v8b4l" Jul 26 16:14:36.653620 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:36.653473 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25d7898d-3961-4da3-a12e-3cce13ff8457-workload-certs\") pod \"e2e-api-a-4b0d96-openshift-default-5d8cc86f8-v8b4l\" (UID: \"25d7898d-3961-4da3-a12e-3cce13ff8457\") " pod="openshift-ingress/e2e-api-a-4b0d96-openshift-default-5d8cc86f8-v8b4l" Jul 26 16:14:36.653620 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:36.653570 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5r85m\" (UniqueName: \"kubernetes.io/projected/25d7898d-3961-4da3-a12e-3cce13ff8457-kube-api-access-5r85m\") pod \"e2e-api-a-4b0d96-openshift-default-5d8cc86f8-v8b4l\" (UID: \"25d7898d-3961-4da3-a12e-3cce13ff8457\") " pod="openshift-ingress/e2e-api-a-4b0d96-openshift-default-5d8cc86f8-v8b4l" Jul 26 16:14:36.653719 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:36.653618 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25d7898d-3961-4da3-a12e-3cce13ff8457-istiod-ca-cert\") pod \"e2e-api-a-4b0d96-openshift-default-5d8cc86f8-v8b4l\" (UID: \"25d7898d-3961-4da3-a12e-3cce13ff8457\") " pod="openshift-ingress/e2e-api-a-4b0d96-openshift-default-5d8cc86f8-v8b4l" Jul 26 16:14:36.653719 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:36.653661 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25d7898d-3961-4da3-a12e-3cce13ff8457-istio-podinfo\") pod \"e2e-api-a-4b0d96-openshift-default-5d8cc86f8-v8b4l\" (UID: \"25d7898d-3961-4da3-a12e-3cce13ff8457\") " pod="openshift-ingress/e2e-api-a-4b0d96-openshift-default-5d8cc86f8-v8b4l" Jul 26 16:14:36.653719 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:36.653713 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25d7898d-3961-4da3-a12e-3cce13ff8457-credential-socket\") pod \"e2e-api-a-4b0d96-openshift-default-5d8cc86f8-v8b4l\" (UID: \"25d7898d-3961-4da3-a12e-3cce13ff8457\") " pod="openshift-ingress/e2e-api-a-4b0d96-openshift-default-5d8cc86f8-v8b4l" Jul 26 16:14:36.653886 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:36.653730 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25d7898d-3961-4da3-a12e-3cce13ff8457-istio-data\") pod \"e2e-api-a-4b0d96-openshift-default-5d8cc86f8-v8b4l\" (UID: \"25d7898d-3961-4da3-a12e-3cce13ff8457\") " pod="openshift-ingress/e2e-api-a-4b0d96-openshift-default-5d8cc86f8-v8b4l" Jul 26 16:14:36.653886 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:36.653759 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25d7898d-3961-4da3-a12e-3cce13ff8457-istio-token\") pod \"e2e-api-a-4b0d96-openshift-default-5d8cc86f8-v8b4l\" (UID: \"25d7898d-3961-4da3-a12e-3cce13ff8457\") " pod="openshift-ingress/e2e-api-a-4b0d96-openshift-default-5d8cc86f8-v8b4l" Jul 26 16:14:36.653886 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:36.653793 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25d7898d-3961-4da3-a12e-3cce13ff8457-istio-envoy\") pod \"e2e-api-a-4b0d96-openshift-default-5d8cc86f8-v8b4l\" (UID: \"25d7898d-3961-4da3-a12e-3cce13ff8457\") " pod="openshift-ingress/e2e-api-a-4b0d96-openshift-default-5d8cc86f8-v8b4l" Jul 26 16:14:36.715524 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:36.715488 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:14:36.754713 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:36.754674 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25d7898d-3961-4da3-a12e-3cce13ff8457-credential-socket\") pod \"e2e-api-a-4b0d96-openshift-default-5d8cc86f8-v8b4l\" (UID: \"25d7898d-3961-4da3-a12e-3cce13ff8457\") " pod="openshift-ingress/e2e-api-a-4b0d96-openshift-default-5d8cc86f8-v8b4l" Jul 26 16:14:36.754713 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:36.754713 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25d7898d-3961-4da3-a12e-3cce13ff8457-istio-data\") pod \"e2e-api-a-4b0d96-openshift-default-5d8cc86f8-v8b4l\" (UID: \"25d7898d-3961-4da3-a12e-3cce13ff8457\") " pod="openshift-ingress/e2e-api-a-4b0d96-openshift-default-5d8cc86f8-v8b4l" Jul 26 16:14:36.754948 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:36.754737 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25d7898d-3961-4da3-a12e-3cce13ff8457-istio-token\") pod \"e2e-api-a-4b0d96-openshift-default-5d8cc86f8-v8b4l\" (UID: \"25d7898d-3961-4da3-a12e-3cce13ff8457\") " pod="openshift-ingress/e2e-api-a-4b0d96-openshift-default-5d8cc86f8-v8b4l" Jul 26 16:14:36.754948 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:36.754764 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25d7898d-3961-4da3-a12e-3cce13ff8457-istio-envoy\") pod \"e2e-api-a-4b0d96-openshift-default-5d8cc86f8-v8b4l\" (UID: \"25d7898d-3961-4da3-a12e-3cce13ff8457\") " pod="openshift-ingress/e2e-api-a-4b0d96-openshift-default-5d8cc86f8-v8b4l" Jul 26 16:14:36.754948 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:36.754807 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25d7898d-3961-4da3-a12e-3cce13ff8457-workload-socket\") pod \"e2e-api-a-4b0d96-openshift-default-5d8cc86f8-v8b4l\" (UID: \"25d7898d-3961-4da3-a12e-3cce13ff8457\") " pod="openshift-ingress/e2e-api-a-4b0d96-openshift-default-5d8cc86f8-v8b4l" Jul 26 16:14:36.754948 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:36.754836 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25d7898d-3961-4da3-a12e-3cce13ff8457-workload-certs\") pod \"e2e-api-a-4b0d96-openshift-default-5d8cc86f8-v8b4l\" (UID: \"25d7898d-3961-4da3-a12e-3cce13ff8457\") " pod="openshift-ingress/e2e-api-a-4b0d96-openshift-default-5d8cc86f8-v8b4l" Jul 26 16:14:36.754948 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:36.754882 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5r85m\" (UniqueName: \"kubernetes.io/projected/25d7898d-3961-4da3-a12e-3cce13ff8457-kube-api-access-5r85m\") pod \"e2e-api-a-4b0d96-openshift-default-5d8cc86f8-v8b4l\" (UID: \"25d7898d-3961-4da3-a12e-3cce13ff8457\") " pod="openshift-ingress/e2e-api-a-4b0d96-openshift-default-5d8cc86f8-v8b4l" Jul 26 16:14:36.754948 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:36.754907 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25d7898d-3961-4da3-a12e-3cce13ff8457-istiod-ca-cert\") pod \"e2e-api-a-4b0d96-openshift-default-5d8cc86f8-v8b4l\" (UID: \"25d7898d-3961-4da3-a12e-3cce13ff8457\") " pod="openshift-ingress/e2e-api-a-4b0d96-openshift-default-5d8cc86f8-v8b4l" Jul 26 16:14:36.754948 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:36.754942 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25d7898d-3961-4da3-a12e-3cce13ff8457-istio-podinfo\") pod \"e2e-api-a-4b0d96-openshift-default-5d8cc86f8-v8b4l\" (UID: \"25d7898d-3961-4da3-a12e-3cce13ff8457\") " pod="openshift-ingress/e2e-api-a-4b0d96-openshift-default-5d8cc86f8-v8b4l" Jul 26 16:14:36.755376 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:36.755140 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25d7898d-3961-4da3-a12e-3cce13ff8457-credential-socket\") pod \"e2e-api-a-4b0d96-openshift-default-5d8cc86f8-v8b4l\" (UID: \"25d7898d-3961-4da3-a12e-3cce13ff8457\") " pod="openshift-ingress/e2e-api-a-4b0d96-openshift-default-5d8cc86f8-v8b4l" Jul 26 16:14:36.755376 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:36.755250 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25d7898d-3961-4da3-a12e-3cce13ff8457-workload-certs\") pod \"e2e-api-a-4b0d96-openshift-default-5d8cc86f8-v8b4l\" (UID: \"25d7898d-3961-4da3-a12e-3cce13ff8457\") " pod="openshift-ingress/e2e-api-a-4b0d96-openshift-default-5d8cc86f8-v8b4l" Jul 26 16:14:36.755376 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:36.755318 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25d7898d-3961-4da3-a12e-3cce13ff8457-workload-socket\") pod \"e2e-api-a-4b0d96-openshift-default-5d8cc86f8-v8b4l\" (UID: \"25d7898d-3961-4da3-a12e-3cce13ff8457\") " pod="openshift-ingress/e2e-api-a-4b0d96-openshift-default-5d8cc86f8-v8b4l" Jul 26 16:14:36.755523 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:36.755480 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25d7898d-3961-4da3-a12e-3cce13ff8457-istio-data\") pod \"e2e-api-a-4b0d96-openshift-default-5d8cc86f8-v8b4l\" (UID: \"25d7898d-3961-4da3-a12e-3cce13ff8457\") " pod="openshift-ingress/e2e-api-a-4b0d96-openshift-default-5d8cc86f8-v8b4l" Jul 26 16:14:36.755835 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:36.755812 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25d7898d-3961-4da3-a12e-3cce13ff8457-istiod-ca-cert\") pod \"e2e-api-a-4b0d96-openshift-default-5d8cc86f8-v8b4l\" (UID: \"25d7898d-3961-4da3-a12e-3cce13ff8457\") " pod="openshift-ingress/e2e-api-a-4b0d96-openshift-default-5d8cc86f8-v8b4l" Jul 26 16:14:36.757034 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:36.757017 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25d7898d-3961-4da3-a12e-3cce13ff8457-istio-envoy\") pod \"e2e-api-a-4b0d96-openshift-default-5d8cc86f8-v8b4l\" (UID: \"25d7898d-3961-4da3-a12e-3cce13ff8457\") " pod="openshift-ingress/e2e-api-a-4b0d96-openshift-default-5d8cc86f8-v8b4l" Jul 26 16:14:36.757293 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:36.757277 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25d7898d-3961-4da3-a12e-3cce13ff8457-istio-podinfo\") pod \"e2e-api-a-4b0d96-openshift-default-5d8cc86f8-v8b4l\" (UID: \"25d7898d-3961-4da3-a12e-3cce13ff8457\") " pod="openshift-ingress/e2e-api-a-4b0d96-openshift-default-5d8cc86f8-v8b4l" Jul 26 16:14:36.762548 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:36.762523 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25d7898d-3961-4da3-a12e-3cce13ff8457-istio-token\") pod \"e2e-api-a-4b0d96-openshift-default-5d8cc86f8-v8b4l\" (UID: \"25d7898d-3961-4da3-a12e-3cce13ff8457\") " pod="openshift-ingress/e2e-api-a-4b0d96-openshift-default-5d8cc86f8-v8b4l" Jul 26 16:14:36.762674 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:36.762658 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5r85m\" (UniqueName: \"kubernetes.io/projected/25d7898d-3961-4da3-a12e-3cce13ff8457-kube-api-access-5r85m\") pod \"e2e-api-a-4b0d96-openshift-default-5d8cc86f8-v8b4l\" (UID: \"25d7898d-3961-4da3-a12e-3cce13ff8457\") " pod="openshift-ingress/e2e-api-a-4b0d96-openshift-default-5d8cc86f8-v8b4l" Jul 26 16:14:36.840287 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:36.840184 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-4b0d96-openshift-default-5d8cc86f8-v8b4l" Jul 26 16:14:37.000608 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:37.000584 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-4b0d96-openshift-default-5d8cc86f8-v8b4l"] Jul 26 16:14:37.002630 ip-10-0-138-163 kubenswrapper[2577]: W0726 16:14:37.002604 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25d7898d_3961_4da3_a12e_3cce13ff8457.slice/crio-54c958f67682265cac8174a01254e708cb0cd554cea0cbe8d5e290f667dd7b9e WatchSource:0}: Error finding container 54c958f67682265cac8174a01254e708cb0cd554cea0cbe8d5e290f667dd7b9e: Status 404 returned error can't find the container with id 54c958f67682265cac8174a01254e708cb0cd554cea0cbe8d5e290f667dd7b9e Jul 26 16:14:37.005078 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:37.005043 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:14:37.005179 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:37.005110 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:14:37.005179 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:37.005139 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:14:37.016956 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:37.016933 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4b0d96-openshift-default-5d8cc86f8-v8b4l" event={"ID":"25d7898d-3961-4da3-a12e-3cce13ff8457","Type":"ContainerStarted","Data":"54c958f67682265cac8174a01254e708cb0cd554cea0cbe8d5e290f667dd7b9e"} Jul 26 16:14:38.023640 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:38.023601 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4b0d96-openshift-default-5d8cc86f8-v8b4l" event={"ID":"25d7898d-3961-4da3-a12e-3cce13ff8457","Type":"ContainerStarted","Data":"a1f15cb1bdb96341445752493a0416e2805faa11ddeba498e627768e7c6f9159"} Jul 26 16:14:38.044906 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:38.044837 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-4b0d96-openshift-default-5d8cc86f8-v8b4l" podStartSLOduration=2.044821323 podStartE2EDuration="2.044821323s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 16:14:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 16:14:38.042681908 +0000 UTC m=+2200.782035249" watchObservedRunningTime="2026-07-26 16:14:38.044821323 +0000 UTC m=+2200.784174633" Jul 26 16:14:38.503692 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:38.503592 2577 patch_prober.go:28] interesting pod/e2e-inf-b-959cca-openshift-default-586f57c5c6-7v96k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 26 16:14:38.503692 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:38.503651 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-959cca-openshift-default-586f57c5c6-7v96k" podUID="15a17353-4f84-47ca-a17e-4636c0e6c801" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 26 16:14:38.840952 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:38.840905 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-4b0d96-openshift-default-5d8cc86f8-v8b4l" Jul 26 16:14:38.845755 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:38.845723 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-4b0d96-openshift-default-5d8cc86f8-v8b4l" Jul 26 16:14:39.029821 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:39.029790 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-4b0d96-openshift-default-5d8cc86f8-v8b4l" Jul 26 16:14:39.031072 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:39.031037 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-4b0d96-openshift-default-5d8cc86f8-v8b4l" Jul 26 16:14:40.002515 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:40.002474 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:14:40.172701 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:40.172667 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:14:40.264158 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:40.264051 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:14:40.363016 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:40.362906 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:14:40.772961 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:40.772924 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:14:40.870033 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:40.869996 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:14:40.927250 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:40.927189 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-959cca-openshift-default-586f57c5c6-7v96k" Jul 26 16:14:41.000449 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:41.000417 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15a17353-4f84-47ca-a17e-4636c0e6c801-istio-podinfo\") pod \"15a17353-4f84-47ca-a17e-4636c0e6c801\" (UID: \"15a17353-4f84-47ca-a17e-4636c0e6c801\") " Jul 26 16:14:41.000640 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:41.000463 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2l65p\" (UniqueName: \"kubernetes.io/projected/15a17353-4f84-47ca-a17e-4636c0e6c801-kube-api-access-2l65p\") pod \"15a17353-4f84-47ca-a17e-4636c0e6c801\" (UID: \"15a17353-4f84-47ca-a17e-4636c0e6c801\") " Jul 26 16:14:41.000640 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:41.000488 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15a17353-4f84-47ca-a17e-4636c0e6c801-credential-socket\") pod \"15a17353-4f84-47ca-a17e-4636c0e6c801\" (UID: \"15a17353-4f84-47ca-a17e-4636c0e6c801\") " Jul 26 16:14:41.000640 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:41.000545 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15a17353-4f84-47ca-a17e-4636c0e6c801-istio-token\") pod \"15a17353-4f84-47ca-a17e-4636c0e6c801\" (UID: \"15a17353-4f84-47ca-a17e-4636c0e6c801\") " Jul 26 16:14:41.000640 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:41.000599 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15a17353-4f84-47ca-a17e-4636c0e6c801-workload-socket\") pod \"15a17353-4f84-47ca-a17e-4636c0e6c801\" (UID: \"15a17353-4f84-47ca-a17e-4636c0e6c801\") " Jul 26 16:14:41.000640 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:41.000633 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15a17353-4f84-47ca-a17e-4636c0e6c801-workload-certs\") pod \"15a17353-4f84-47ca-a17e-4636c0e6c801\" (UID: \"15a17353-4f84-47ca-a17e-4636c0e6c801\") " Jul 26 16:14:41.000909 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:41.000669 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15a17353-4f84-47ca-a17e-4636c0e6c801-istiod-ca-cert\") pod \"15a17353-4f84-47ca-a17e-4636c0e6c801\" (UID: \"15a17353-4f84-47ca-a17e-4636c0e6c801\") " Jul 26 16:14:41.000909 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:41.000705 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15a17353-4f84-47ca-a17e-4636c0e6c801-istio-data\") pod \"15a17353-4f84-47ca-a17e-4636c0e6c801\" (UID: \"15a17353-4f84-47ca-a17e-4636c0e6c801\") " Jul 26 16:14:41.000909 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:41.000746 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15a17353-4f84-47ca-a17e-4636c0e6c801-istio-envoy\") pod \"15a17353-4f84-47ca-a17e-4636c0e6c801\" (UID: \"15a17353-4f84-47ca-a17e-4636c0e6c801\") " Jul 26 16:14:41.000909 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:41.000867 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15a17353-4f84-47ca-a17e-4636c0e6c801-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "15a17353-4f84-47ca-a17e-4636c0e6c801" (UID: "15a17353-4f84-47ca-a17e-4636c0e6c801"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:14:41.001102 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:41.000906 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15a17353-4f84-47ca-a17e-4636c0e6c801-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "15a17353-4f84-47ca-a17e-4636c0e6c801" (UID: "15a17353-4f84-47ca-a17e-4636c0e6c801"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:14:41.001102 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:41.000948 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15a17353-4f84-47ca-a17e-4636c0e6c801-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "15a17353-4f84-47ca-a17e-4636c0e6c801" (UID: "15a17353-4f84-47ca-a17e-4636c0e6c801"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:14:41.001102 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:41.001020 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15a17353-4f84-47ca-a17e-4636c0e6c801-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "15a17353-4f84-47ca-a17e-4636c0e6c801" (UID: "15a17353-4f84-47ca-a17e-4636c0e6c801"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 16:14:41.001289 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:41.001110 2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15a17353-4f84-47ca-a17e-4636c0e6c801-istiod-ca-cert\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 16:14:41.001289 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:41.001130 2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15a17353-4f84-47ca-a17e-4636c0e6c801-credential-socket\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 16:14:41.001289 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:41.001146 2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15a17353-4f84-47ca-a17e-4636c0e6c801-workload-socket\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 16:14:41.001289 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:41.001160 2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15a17353-4f84-47ca-a17e-4636c0e6c801-workload-certs\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 16:14:41.001661 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:41.001632 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15a17353-4f84-47ca-a17e-4636c0e6c801-istio-data" (OuterVolumeSpecName: "istio-data") pod "15a17353-4f84-47ca-a17e-4636c0e6c801" (UID: "15a17353-4f84-47ca-a17e-4636c0e6c801"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:14:41.002907 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:41.002880 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15a17353-4f84-47ca-a17e-4636c0e6c801-kube-api-access-2l65p" (OuterVolumeSpecName: "kube-api-access-2l65p") pod "15a17353-4f84-47ca-a17e-4636c0e6c801" (UID: "15a17353-4f84-47ca-a17e-4636c0e6c801"). InnerVolumeSpecName "kube-api-access-2l65p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 16:14:41.003038 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:41.002915 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15a17353-4f84-47ca-a17e-4636c0e6c801-istio-token" (OuterVolumeSpecName: "istio-token") pod "15a17353-4f84-47ca-a17e-4636c0e6c801" (UID: "15a17353-4f84-47ca-a17e-4636c0e6c801"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 16:14:41.003038 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:41.002973 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15a17353-4f84-47ca-a17e-4636c0e6c801-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "15a17353-4f84-47ca-a17e-4636c0e6c801" (UID: "15a17353-4f84-47ca-a17e-4636c0e6c801"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:14:41.003124 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:41.003042 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/15a17353-4f84-47ca-a17e-4636c0e6c801-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "15a17353-4f84-47ca-a17e-4636c0e6c801" (UID: "15a17353-4f84-47ca-a17e-4636c0e6c801"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 26 16:14:41.043992 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:41.043953 2577 generic.go:356] "Generic (PLEG): container finished" podID="15a17353-4f84-47ca-a17e-4636c0e6c801" containerID="f156f8d2a3d4369884f80edf77f33dbaefcf520e03247c23fe3d1f11ae2d0e2a" exitCode=0 Jul 26 16:14:41.044184 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:41.044005 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-959cca-openshift-default-586f57c5c6-7v96k" Jul 26 16:14:41.044184 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:41.044041 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-959cca-openshift-default-586f57c5c6-7v96k" event={"ID":"15a17353-4f84-47ca-a17e-4636c0e6c801","Type":"ContainerDied","Data":"f156f8d2a3d4369884f80edf77f33dbaefcf520e03247c23fe3d1f11ae2d0e2a"} Jul 26 16:14:41.044184 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:41.044094 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-959cca-openshift-default-586f57c5c6-7v96k" event={"ID":"15a17353-4f84-47ca-a17e-4636c0e6c801","Type":"ContainerDied","Data":"a1e91fc87e4e0ae7663fc3dcd6cf4e3d06aca58fe39bea311d435aab4bf60cb5"} Jul 26 16:14:41.044184 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:41.044122 2577 scope.go:117] "RemoveContainer" containerID="f156f8d2a3d4369884f80edf77f33dbaefcf520e03247c23fe3d1f11ae2d0e2a" Jul 26 16:14:41.053970 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:41.053949 2577 scope.go:117] "RemoveContainer" containerID="46d86280942781a4cdb951c1645050c607a0923dcdfe3007b9d9d6fd69c84ca2" Jul 26 16:14:41.062391 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:41.062371 2577 scope.go:117] "RemoveContainer" containerID="f156f8d2a3d4369884f80edf77f33dbaefcf520e03247c23fe3d1f11ae2d0e2a" Jul 26 16:14:41.062682 ip-10-0-138-163 kubenswrapper[2577]: E0726 16:14:41.062662 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f156f8d2a3d4369884f80edf77f33dbaefcf520e03247c23fe3d1f11ae2d0e2a\": container with ID starting with f156f8d2a3d4369884f80edf77f33dbaefcf520e03247c23fe3d1f11ae2d0e2a not found: ID does not exist" containerID="f156f8d2a3d4369884f80edf77f33dbaefcf520e03247c23fe3d1f11ae2d0e2a" Jul 26 16:14:41.062763 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:41.062695 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f156f8d2a3d4369884f80edf77f33dbaefcf520e03247c23fe3d1f11ae2d0e2a"} err="failed to get container status \"f156f8d2a3d4369884f80edf77f33dbaefcf520e03247c23fe3d1f11ae2d0e2a\": rpc error: code = NotFound desc = could not find container \"f156f8d2a3d4369884f80edf77f33dbaefcf520e03247c23fe3d1f11ae2d0e2a\": container with ID starting with f156f8d2a3d4369884f80edf77f33dbaefcf520e03247c23fe3d1f11ae2d0e2a not found: ID does not exist" Jul 26 16:14:41.062763 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:41.062718 2577 scope.go:117] "RemoveContainer" containerID="46d86280942781a4cdb951c1645050c607a0923dcdfe3007b9d9d6fd69c84ca2" Jul 26 16:14:41.062947 ip-10-0-138-163 kubenswrapper[2577]: E0726 16:14:41.062929 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46d86280942781a4cdb951c1645050c607a0923dcdfe3007b9d9d6fd69c84ca2\": container with ID starting with 46d86280942781a4cdb951c1645050c607a0923dcdfe3007b9d9d6fd69c84ca2 not found: ID does not exist" containerID="46d86280942781a4cdb951c1645050c607a0923dcdfe3007b9d9d6fd69c84ca2" Jul 26 16:14:41.062986 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:41.062953 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46d86280942781a4cdb951c1645050c607a0923dcdfe3007b9d9d6fd69c84ca2"} err="failed to get container status \"46d86280942781a4cdb951c1645050c607a0923dcdfe3007b9d9d6fd69c84ca2\": rpc error: code = NotFound desc = could not find container \"46d86280942781a4cdb951c1645050c607a0923dcdfe3007b9d9d6fd69c84ca2\": container with ID starting with 46d86280942781a4cdb951c1645050c607a0923dcdfe3007b9d9d6fd69c84ca2 not found: ID does not exist" Jul 26 16:14:41.066723 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:41.066702 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-959cca-openshift-default-586f57c5c6-7v96k"] Jul 26 16:14:41.071445 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:41.071420 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-959cca-openshift-default-586f57c5c6-7v96k"] Jul 26 16:14:41.102577 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:41.102545 2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15a17353-4f84-47ca-a17e-4636c0e6c801-istio-data\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 16:14:41.102577 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:41.102568 2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15a17353-4f84-47ca-a17e-4636c0e6c801-istio-envoy\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 16:14:41.102577 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:41.102577 2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15a17353-4f84-47ca-a17e-4636c0e6c801-istio-podinfo\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 16:14:41.102795 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:41.102587 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2l65p\" (UniqueName: \"kubernetes.io/projected/15a17353-4f84-47ca-a17e-4636c0e6c801-kube-api-access-2l65p\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 16:14:41.102795 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:41.102597 2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15a17353-4f84-47ca-a17e-4636c0e6c801-istio-token\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 16:14:41.787295 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:41.787260 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="15a17353-4f84-47ca-a17e-4636c0e6c801" path="/var/lib/kubelet/pods/15a17353-4f84-47ca-a17e-4636c0e6c801/volumes" Jul 26 16:14:44.403607 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:44.403566 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:14:44.858222 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:44.858184 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:14:45.129635 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:45.129542 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-7cc89d-openshift-default-6d7c4cf8bb-ckxhl"] Jul 26 16:14:45.130033 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:45.130014 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5cb52e0-d049-45ad-ac1a-64f58ffeec11" containerName="istio-proxy" Jul 26 16:14:45.130127 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:45.130035 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5cb52e0-d049-45ad-ac1a-64f58ffeec11" containerName="istio-proxy" Jul 26 16:14:45.130127 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:45.130064 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15a17353-4f84-47ca-a17e-4636c0e6c801" containerName="istio-proxy" Jul 26 16:14:45.130127 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:45.130072 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="15a17353-4f84-47ca-a17e-4636c0e6c801" containerName="istio-proxy" Jul 26 16:14:45.130127 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:45.130082 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15a17353-4f84-47ca-a17e-4636c0e6c801" containerName="istio-proxy" Jul 26 16:14:45.130127 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:45.130091 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="15a17353-4f84-47ca-a17e-4636c0e6c801" containerName="istio-proxy" Jul 26 16:14:45.130365 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:45.130183 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="15a17353-4f84-47ca-a17e-4636c0e6c801" containerName="istio-proxy" Jul 26 16:14:45.130365 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:45.130196 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="15a17353-4f84-47ca-a17e-4636c0e6c801" containerName="istio-proxy" Jul 26 16:14:45.134822 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:45.134800 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-7cc89d-openshift-default-6d7c4cf8bb-ckxhl" Jul 26 16:14:45.137494 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:45.137468 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-7cc89d-openshift-default-dockercfg-bpz2k\"" Jul 26 16:14:45.145862 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:45.145841 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-7cc89d-openshift-default-6d7c4cf8bb-ckxhl"] Jul 26 16:14:45.158183 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:45.158140 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:14:45.241595 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:45.241563 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/083ae778-ec20-42ba-95b7-74f1f4b788b8-istio-podinfo\") pod \"e2e-api-b-7cc89d-openshift-default-6d7c4cf8bb-ckxhl\" (UID: \"083ae778-ec20-42ba-95b7-74f1f4b788b8\") " pod="openshift-ingress/e2e-api-b-7cc89d-openshift-default-6d7c4cf8bb-ckxhl" Jul 26 16:14:45.241772 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:45.241699 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/083ae778-ec20-42ba-95b7-74f1f4b788b8-workload-certs\") pod \"e2e-api-b-7cc89d-openshift-default-6d7c4cf8bb-ckxhl\" (UID: \"083ae778-ec20-42ba-95b7-74f1f4b788b8\") " pod="openshift-ingress/e2e-api-b-7cc89d-openshift-default-6d7c4cf8bb-ckxhl" Jul 26 16:14:45.241821 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:45.241778 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/083ae778-ec20-42ba-95b7-74f1f4b788b8-istio-token\") pod \"e2e-api-b-7cc89d-openshift-default-6d7c4cf8bb-ckxhl\" (UID: \"083ae778-ec20-42ba-95b7-74f1f4b788b8\") " pod="openshift-ingress/e2e-api-b-7cc89d-openshift-default-6d7c4cf8bb-ckxhl" Jul 26 16:14:45.241821 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:45.241812 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/083ae778-ec20-42ba-95b7-74f1f4b788b8-istiod-ca-cert\") pod \"e2e-api-b-7cc89d-openshift-default-6d7c4cf8bb-ckxhl\" (UID: \"083ae778-ec20-42ba-95b7-74f1f4b788b8\") " pod="openshift-ingress/e2e-api-b-7cc89d-openshift-default-6d7c4cf8bb-ckxhl" Jul 26 16:14:45.241911 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:45.241839 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/083ae778-ec20-42ba-95b7-74f1f4b788b8-istio-data\") pod \"e2e-api-b-7cc89d-openshift-default-6d7c4cf8bb-ckxhl\" (UID: \"083ae778-ec20-42ba-95b7-74f1f4b788b8\") " pod="openshift-ingress/e2e-api-b-7cc89d-openshift-default-6d7c4cf8bb-ckxhl" Jul 26 16:14:45.241911 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:45.241894 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/083ae778-ec20-42ba-95b7-74f1f4b788b8-credential-socket\") pod \"e2e-api-b-7cc89d-openshift-default-6d7c4cf8bb-ckxhl\" (UID: \"083ae778-ec20-42ba-95b7-74f1f4b788b8\") " pod="openshift-ingress/e2e-api-b-7cc89d-openshift-default-6d7c4cf8bb-ckxhl" Jul 26 16:14:45.241985 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:45.241937 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzq7g\" (UniqueName: \"kubernetes.io/projected/083ae778-ec20-42ba-95b7-74f1f4b788b8-kube-api-access-hzq7g\") pod \"e2e-api-b-7cc89d-openshift-default-6d7c4cf8bb-ckxhl\" (UID: \"083ae778-ec20-42ba-95b7-74f1f4b788b8\") " pod="openshift-ingress/e2e-api-b-7cc89d-openshift-default-6d7c4cf8bb-ckxhl" Jul 26 16:14:45.241985 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:45.241962 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/083ae778-ec20-42ba-95b7-74f1f4b788b8-workload-socket\") pod \"e2e-api-b-7cc89d-openshift-default-6d7c4cf8bb-ckxhl\" (UID: \"083ae778-ec20-42ba-95b7-74f1f4b788b8\") " pod="openshift-ingress/e2e-api-b-7cc89d-openshift-default-6d7c4cf8bb-ckxhl" Jul 26 16:14:45.242084 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:45.242044 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/083ae778-ec20-42ba-95b7-74f1f4b788b8-istio-envoy\") pod \"e2e-api-b-7cc89d-openshift-default-6d7c4cf8bb-ckxhl\" (UID: \"083ae778-ec20-42ba-95b7-74f1f4b788b8\") " pod="openshift-ingress/e2e-api-b-7cc89d-openshift-default-6d7c4cf8bb-ckxhl" Jul 26 16:14:45.323632 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:45.323600 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:14:45.342727 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:45.342698 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzq7g\" (UniqueName: \"kubernetes.io/projected/083ae778-ec20-42ba-95b7-74f1f4b788b8-kube-api-access-hzq7g\") pod \"e2e-api-b-7cc89d-openshift-default-6d7c4cf8bb-ckxhl\" (UID: \"083ae778-ec20-42ba-95b7-74f1f4b788b8\") " pod="openshift-ingress/e2e-api-b-7cc89d-openshift-default-6d7c4cf8bb-ckxhl" Jul 26 16:14:45.342893 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:45.342730 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/083ae778-ec20-42ba-95b7-74f1f4b788b8-workload-socket\") pod \"e2e-api-b-7cc89d-openshift-default-6d7c4cf8bb-ckxhl\" (UID: \"083ae778-ec20-42ba-95b7-74f1f4b788b8\") " pod="openshift-ingress/e2e-api-b-7cc89d-openshift-default-6d7c4cf8bb-ckxhl" Jul 26 16:14:45.342893 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:45.342757 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/083ae778-ec20-42ba-95b7-74f1f4b788b8-istio-envoy\") pod \"e2e-api-b-7cc89d-openshift-default-6d7c4cf8bb-ckxhl\" (UID: \"083ae778-ec20-42ba-95b7-74f1f4b788b8\") " pod="openshift-ingress/e2e-api-b-7cc89d-openshift-default-6d7c4cf8bb-ckxhl" Jul 26 16:14:45.343007 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:45.342902 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/083ae778-ec20-42ba-95b7-74f1f4b788b8-istio-podinfo\") pod \"e2e-api-b-7cc89d-openshift-default-6d7c4cf8bb-ckxhl\" (UID: \"083ae778-ec20-42ba-95b7-74f1f4b788b8\") " pod="openshift-ingress/e2e-api-b-7cc89d-openshift-default-6d7c4cf8bb-ckxhl" Jul 26 16:14:45.343007 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:45.342994 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/083ae778-ec20-42ba-95b7-74f1f4b788b8-workload-certs\") pod \"e2e-api-b-7cc89d-openshift-default-6d7c4cf8bb-ckxhl\" (UID: \"083ae778-ec20-42ba-95b7-74f1f4b788b8\") " pod="openshift-ingress/e2e-api-b-7cc89d-openshift-default-6d7c4cf8bb-ckxhl" Jul 26 16:14:45.343121 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:45.343054 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/083ae778-ec20-42ba-95b7-74f1f4b788b8-istio-token\") pod \"e2e-api-b-7cc89d-openshift-default-6d7c4cf8bb-ckxhl\" (UID: \"083ae778-ec20-42ba-95b7-74f1f4b788b8\") " pod="openshift-ingress/e2e-api-b-7cc89d-openshift-default-6d7c4cf8bb-ckxhl" Jul 26 16:14:45.343121 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:45.343084 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/083ae778-ec20-42ba-95b7-74f1f4b788b8-istiod-ca-cert\") pod \"e2e-api-b-7cc89d-openshift-default-6d7c4cf8bb-ckxhl\" (UID: \"083ae778-ec20-42ba-95b7-74f1f4b788b8\") " pod="openshift-ingress/e2e-api-b-7cc89d-openshift-default-6d7c4cf8bb-ckxhl" Jul 26 16:14:45.343121 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:45.343116 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/083ae778-ec20-42ba-95b7-74f1f4b788b8-istio-data\") pod \"e2e-api-b-7cc89d-openshift-default-6d7c4cf8bb-ckxhl\" (UID: \"083ae778-ec20-42ba-95b7-74f1f4b788b8\") " pod="openshift-ingress/e2e-api-b-7cc89d-openshift-default-6d7c4cf8bb-ckxhl" Jul 26 16:14:45.343264 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:45.343197 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/083ae778-ec20-42ba-95b7-74f1f4b788b8-credential-socket\") pod \"e2e-api-b-7cc89d-openshift-default-6d7c4cf8bb-ckxhl\" (UID: \"083ae778-ec20-42ba-95b7-74f1f4b788b8\") " pod="openshift-ingress/e2e-api-b-7cc89d-openshift-default-6d7c4cf8bb-ckxhl" Jul 26 16:14:45.343378 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:45.343352 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/083ae778-ec20-42ba-95b7-74f1f4b788b8-workload-certs\") pod \"e2e-api-b-7cc89d-openshift-default-6d7c4cf8bb-ckxhl\" (UID: \"083ae778-ec20-42ba-95b7-74f1f4b788b8\") " pod="openshift-ingress/e2e-api-b-7cc89d-openshift-default-6d7c4cf8bb-ckxhl" Jul 26 16:14:45.343578 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:45.343552 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/083ae778-ec20-42ba-95b7-74f1f4b788b8-workload-socket\") pod \"e2e-api-b-7cc89d-openshift-default-6d7c4cf8bb-ckxhl\" (UID: \"083ae778-ec20-42ba-95b7-74f1f4b788b8\") " pod="openshift-ingress/e2e-api-b-7cc89d-openshift-default-6d7c4cf8bb-ckxhl" Jul 26 16:14:45.343656 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:45.343577 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/083ae778-ec20-42ba-95b7-74f1f4b788b8-istio-data\") pod \"e2e-api-b-7cc89d-openshift-default-6d7c4cf8bb-ckxhl\" (UID: \"083ae778-ec20-42ba-95b7-74f1f4b788b8\") " pod="openshift-ingress/e2e-api-b-7cc89d-openshift-default-6d7c4cf8bb-ckxhl" Jul 26 16:14:45.343656 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:45.343620 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/083ae778-ec20-42ba-95b7-74f1f4b788b8-credential-socket\") pod \"e2e-api-b-7cc89d-openshift-default-6d7c4cf8bb-ckxhl\" (UID: \"083ae778-ec20-42ba-95b7-74f1f4b788b8\") " pod="openshift-ingress/e2e-api-b-7cc89d-openshift-default-6d7c4cf8bb-ckxhl" Jul 26 16:14:45.343830 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:45.343809 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/083ae778-ec20-42ba-95b7-74f1f4b788b8-istiod-ca-cert\") pod \"e2e-api-b-7cc89d-openshift-default-6d7c4cf8bb-ckxhl\" (UID: \"083ae778-ec20-42ba-95b7-74f1f4b788b8\") " pod="openshift-ingress/e2e-api-b-7cc89d-openshift-default-6d7c4cf8bb-ckxhl" Jul 26 16:14:45.345115 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:45.345096 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/083ae778-ec20-42ba-95b7-74f1f4b788b8-istio-envoy\") pod \"e2e-api-b-7cc89d-openshift-default-6d7c4cf8bb-ckxhl\" (UID: \"083ae778-ec20-42ba-95b7-74f1f4b788b8\") " pod="openshift-ingress/e2e-api-b-7cc89d-openshift-default-6d7c4cf8bb-ckxhl" Jul 26 16:14:45.345390 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:45.345371 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/083ae778-ec20-42ba-95b7-74f1f4b788b8-istio-podinfo\") pod \"e2e-api-b-7cc89d-openshift-default-6d7c4cf8bb-ckxhl\" (UID: \"083ae778-ec20-42ba-95b7-74f1f4b788b8\") " pod="openshift-ingress/e2e-api-b-7cc89d-openshift-default-6d7c4cf8bb-ckxhl" Jul 26 16:14:45.350678 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:45.350658 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/083ae778-ec20-42ba-95b7-74f1f4b788b8-istio-token\") pod \"e2e-api-b-7cc89d-openshift-default-6d7c4cf8bb-ckxhl\" (UID: \"083ae778-ec20-42ba-95b7-74f1f4b788b8\") " pod="openshift-ingress/e2e-api-b-7cc89d-openshift-default-6d7c4cf8bb-ckxhl" Jul 26 16:14:45.351294 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:45.351274 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzq7g\" (UniqueName: \"kubernetes.io/projected/083ae778-ec20-42ba-95b7-74f1f4b788b8-kube-api-access-hzq7g\") pod \"e2e-api-b-7cc89d-openshift-default-6d7c4cf8bb-ckxhl\" (UID: \"083ae778-ec20-42ba-95b7-74f1f4b788b8\") " pod="openshift-ingress/e2e-api-b-7cc89d-openshift-default-6d7c4cf8bb-ckxhl" Jul 26 16:14:45.446795 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:45.446707 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-7cc89d-openshift-default-6d7c4cf8bb-ckxhl" Jul 26 16:14:45.592378 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:45.592346 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-7cc89d-openshift-default-6d7c4cf8bb-ckxhl"] Jul 26 16:14:45.594034 ip-10-0-138-163 kubenswrapper[2577]: W0726 16:14:45.594000 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod083ae778_ec20_42ba_95b7_74f1f4b788b8.slice/crio-6f9985bde3ccc15543138e74f95af0a73bacd49b652619a8d07fe6dc2593849b WatchSource:0}: Error finding container 6f9985bde3ccc15543138e74f95af0a73bacd49b652619a8d07fe6dc2593849b: Status 404 returned error can't find the container with id 6f9985bde3ccc15543138e74f95af0a73bacd49b652619a8d07fe6dc2593849b Jul 26 16:14:45.596230 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:45.596193 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:14:45.596324 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:45.596266 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:14:45.596324 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:45.596311 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:14:46.075222 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:46.075184 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-7cc89d-openshift-default-6d7c4cf8bb-ckxhl" event={"ID":"083ae778-ec20-42ba-95b7-74f1f4b788b8","Type":"ContainerStarted","Data":"c167731f27c3c1de5cd60975abbeed35344eeece4afd91f4e77981e6ee777c23"} Jul 26 16:14:46.075425 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:46.075241 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-7cc89d-openshift-default-6d7c4cf8bb-ckxhl" event={"ID":"083ae778-ec20-42ba-95b7-74f1f4b788b8","Type":"ContainerStarted","Data":"6f9985bde3ccc15543138e74f95af0a73bacd49b652619a8d07fe6dc2593849b"} Jul 26 16:14:46.121998 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:46.121945 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-7cc89d-openshift-default-6d7c4cf8bb-ckxhl" podStartSLOduration=1.121928952 podStartE2EDuration="1.121928952s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 16:14:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 16:14:46.120644347 +0000 UTC m=+2208.859997656" watchObservedRunningTime="2026-07-26 16:14:46.121928952 +0000 UTC m=+2208.861282334" Jul 26 16:14:46.447510 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:46.447390 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-7cc89d-openshift-default-6d7c4cf8bb-ckxhl" Jul 26 16:14:46.455972 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:46.455933 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-7cc89d-openshift-default-6d7c4cf8bb-ckxhl" Jul 26 16:14:47.081536 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:47.081507 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-7cc89d-openshift-default-6d7c4cf8bb-ckxhl" Jul 26 16:14:47.082470 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:47.082452 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-7cc89d-openshift-default-6d7c4cf8bb-ckxhl" Jul 26 16:14:47.987317 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:47.987285 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:14:48.074496 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:48.074462 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:14:48.148896 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:48.148862 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:14:48.350603 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:48.350565 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:14:48.685164 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:48.685078 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:14:48.785339 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:48.785308 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:14:52.474603 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:52.474571 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:14:54.031759 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:54.031709 2577 patch_prober.go:28] interesting pod/e2e-api-a-4b0d96-openshift-default-5d8cc86f8-v8b4l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Jul 26 16:14:54.032305 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:54.031808 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-4b0d96-openshift-default-5d8cc86f8-v8b4l" podUID="25d7898d-3961-4da3-a12e-3cce13ff8457" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Jul 26 16:14:54.123092 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:54.123057 2577 generic.go:356] "Generic (PLEG): container finished" podID="25d7898d-3961-4da3-a12e-3cce13ff8457" containerID="a1f15cb1bdb96341445752493a0416e2805faa11ddeba498e627768e7c6f9159" exitCode=0 Jul 26 16:14:54.123232 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:54.123131 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4b0d96-openshift-default-5d8cc86f8-v8b4l" event={"ID":"25d7898d-3961-4da3-a12e-3cce13ff8457","Type":"ContainerDied","Data":"a1f15cb1bdb96341445752493a0416e2805faa11ddeba498e627768e7c6f9159"} Jul 26 16:14:54.123598 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:54.123582 2577 scope.go:117] "RemoveContainer" containerID="a1f15cb1bdb96341445752493a0416e2805faa11ddeba498e627768e7c6f9159" Jul 26 16:14:54.124622 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:54.124593 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:14:54.124729 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:54.124675 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:14:54.124729 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:54.124721 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:14:55.131063 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:55.131022 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4b0d96-openshift-default-5d8cc86f8-v8b4l" event={"ID":"25d7898d-3961-4da3-a12e-3cce13ff8457","Type":"ContainerStarted","Data":"7ddccd840a8e1fb8ae4a99a323349b397836fe85d34b7f86252a147aeba1f976"} Jul 26 16:14:55.841256 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:55.841158 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-4b0d96-openshift-default-5d8cc86f8-v8b4l" Jul 26 16:14:55.841490 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:55.841286 2577 patch_prober.go:28] interesting pod/e2e-api-a-4b0d96-openshift-default-5d8cc86f8-v8b4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Jul 26 16:14:55.841490 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:55.841351 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-4b0d96-openshift-default-5d8cc86f8-v8b4l" podUID="25d7898d-3961-4da3-a12e-3cce13ff8457" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Jul 26 16:14:56.840721 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:56.840673 2577 patch_prober.go:28] interesting pod/e2e-api-a-4b0d96-openshift-default-5d8cc86f8-v8b4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Jul 26 16:14:56.841237 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:56.840746 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-4b0d96-openshift-default-5d8cc86f8-v8b4l" podUID="25d7898d-3961-4da3-a12e-3cce13ff8457" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Jul 26 16:14:57.840723 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:57.840677 2577 patch_prober.go:28] interesting pod/e2e-api-a-4b0d96-openshift-default-5d8cc86f8-v8b4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Jul 26 16:14:57.841235 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:57.840766 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-4b0d96-openshift-default-5d8cc86f8-v8b4l" podUID="25d7898d-3961-4da3-a12e-3cce13ff8457" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Jul 26 16:14:58.841131 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:58.840652 2577 patch_prober.go:28] interesting pod/e2e-api-a-4b0d96-openshift-default-5d8cc86f8-v8b4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Jul 26 16:14:58.841131 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:58.840720 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-4b0d96-openshift-default-5d8cc86f8-v8b4l" podUID="25d7898d-3961-4da3-a12e-3cce13ff8457" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Jul 26 16:14:59.844979 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:59.844947 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-4b0d96-openshift-default-5d8cc86f8-v8b4l" Jul 26 16:14:59.845478 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:59.845338 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-4b0d96-openshift-default-5d8cc86f8-v8b4l" Jul 26 16:14:59.846214 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:14:59.846198 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-4b0d96-openshift-default-5d8cc86f8-v8b4l" Jul 26 16:15:01.178714 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:01.178625 2577 generic.go:356] "Generic (PLEG): container finished" podID="083ae778-ec20-42ba-95b7-74f1f4b788b8" containerID="c167731f27c3c1de5cd60975abbeed35344eeece4afd91f4e77981e6ee777c23" exitCode=0 Jul 26 16:15:01.179123 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:01.178703 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-7cc89d-openshift-default-6d7c4cf8bb-ckxhl" event={"ID":"083ae778-ec20-42ba-95b7-74f1f4b788b8","Type":"ContainerDied","Data":"c167731f27c3c1de5cd60975abbeed35344eeece4afd91f4e77981e6ee777c23"} Jul 26 16:15:01.179269 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:01.179250 2577 scope.go:117] "RemoveContainer" containerID="c167731f27c3c1de5cd60975abbeed35344eeece4afd91f4e77981e6ee777c23" Jul 26 16:15:01.180376 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:01.180345 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:15:01.180465 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:01.180447 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:15:01.180511 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:01.180498 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:15:02.081772 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:02.081734 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-7cc89d-openshift-default-6d7c4cf8bb-ckxhl" Jul 26 16:15:02.187258 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:02.187222 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-7cc89d-openshift-default-6d7c4cf8bb-ckxhl" event={"ID":"083ae778-ec20-42ba-95b7-74f1f4b788b8","Type":"ContainerStarted","Data":"220e3b7d608f10fcae9087c553daec13c9cdf13419da491cfd97cb1ac52b75e6"} Jul 26 16:15:02.449000 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:02.448877 2577 patch_prober.go:28] interesting pod/e2e-api-b-7cc89d-openshift-default-6d7c4cf8bb-ckxhl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Jul 26 16:15:02.449313 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:02.449282 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-7cc89d-openshift-default-6d7c4cf8bb-ckxhl" podUID="083ae778-ec20-42ba-95b7-74f1f4b788b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Jul 26 16:15:02.449824 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:02.449802 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-7cc89d-openshift-default-6d7c4cf8bb-ckxhl" Jul 26 16:15:03.447456 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:03.447389 2577 patch_prober.go:28] interesting pod/e2e-api-b-7cc89d-openshift-default-6d7c4cf8bb-ckxhl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Jul 26 16:15:03.447964 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:03.447485 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-7cc89d-openshift-default-6d7c4cf8bb-ckxhl" podUID="083ae778-ec20-42ba-95b7-74f1f4b788b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Jul 26 16:15:04.447750 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:04.447706 2577 patch_prober.go:28] interesting pod/e2e-api-b-7cc89d-openshift-default-6d7c4cf8bb-ckxhl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Jul 26 16:15:04.448372 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:04.448339 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-7cc89d-openshift-default-6d7c4cf8bb-ckxhl" podUID="083ae778-ec20-42ba-95b7-74f1f4b788b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Jul 26 16:15:05.447297 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:05.447248 2577 patch_prober.go:28] interesting pod/e2e-api-b-7cc89d-openshift-default-6d7c4cf8bb-ckxhl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Jul 26 16:15:05.447582 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:05.447322 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-7cc89d-openshift-default-6d7c4cf8bb-ckxhl" podUID="083ae778-ec20-42ba-95b7-74f1f4b788b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Jul 26 16:15:06.505437 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:06.505378 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-7cc89d-openshift-default-6d7c4cf8bb-ckxhl" Jul 26 16:15:06.506051 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:06.505940 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-7cc89d-openshift-default-6d7c4cf8bb-ckxhl" Jul 26 16:15:06.506897 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:06.506875 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-7cc89d-openshift-default-6d7c4cf8bb-ckxhl" Jul 26 16:15:15.843190 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:15.843151 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:15:15.940575 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:15.940539 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:15:16.042296 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:16.042256 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:15:20.469325 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:20.469210 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:15:22.067489 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:22.067454 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:15:26.585700 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:26.585664 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:15:26.710267 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:26.710233 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:15:26.790887 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:26.790850 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-4b0d96-openshift-default-5d8cc86f8-v8b4l"] Jul 26 16:15:26.791146 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:26.791117 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-4b0d96-openshift-default-5d8cc86f8-v8b4l" podUID="25d7898d-3961-4da3-a12e-3cce13ff8457" containerName="istio-proxy" containerID="cri-o://7ddccd840a8e1fb8ae4a99a323349b397836fe85d34b7f86252a147aeba1f976" gracePeriod=30 Jul 26 16:15:26.814782 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:26.814751 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:15:27.034040 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:27.034004 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:15:29.848151 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:29.848116 2577 patch_prober.go:28] interesting pod/e2e-api-a-4b0d96-openshift-default-5d8cc86f8-v8b4l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 26 16:15:29.848563 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:29.848168 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-4b0d96-openshift-default-5d8cc86f8-v8b4l" podUID="25d7898d-3961-4da3-a12e-3cce13ff8457" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 26 16:15:32.049736 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:32.049702 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-4b0d96-openshift-default-5d8cc86f8-v8b4l" Jul 26 16:15:32.196647 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:32.196552 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25d7898d-3961-4da3-a12e-3cce13ff8457-istio-data\") pod \"25d7898d-3961-4da3-a12e-3cce13ff8457\" (UID: \"25d7898d-3961-4da3-a12e-3cce13ff8457\") " Jul 26 16:15:32.196647 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:32.196638 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25d7898d-3961-4da3-a12e-3cce13ff8457-credential-socket\") pod \"25d7898d-3961-4da3-a12e-3cce13ff8457\" (UID: \"25d7898d-3961-4da3-a12e-3cce13ff8457\") " Jul 26 16:15:32.196884 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:32.196655 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25d7898d-3961-4da3-a12e-3cce13ff8457-istiod-ca-cert\") pod \"25d7898d-3961-4da3-a12e-3cce13ff8457\" (UID: \"25d7898d-3961-4da3-a12e-3cce13ff8457\") " Jul 26 16:15:32.196884 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:32.196687 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25d7898d-3961-4da3-a12e-3cce13ff8457-workload-socket\") pod \"25d7898d-3961-4da3-a12e-3cce13ff8457\" (UID: \"25d7898d-3961-4da3-a12e-3cce13ff8457\") " Jul 26 16:15:32.196884 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:32.196704 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5r85m\" (UniqueName: \"kubernetes.io/projected/25d7898d-3961-4da3-a12e-3cce13ff8457-kube-api-access-5r85m\") pod \"25d7898d-3961-4da3-a12e-3cce13ff8457\" (UID: \"25d7898d-3961-4da3-a12e-3cce13ff8457\") " Jul 26 16:15:32.196884 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:32.196723 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25d7898d-3961-4da3-a12e-3cce13ff8457-istio-podinfo\") pod \"25d7898d-3961-4da3-a12e-3cce13ff8457\" (UID: \"25d7898d-3961-4da3-a12e-3cce13ff8457\") " Jul 26 16:15:32.196884 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:32.196756 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25d7898d-3961-4da3-a12e-3cce13ff8457-workload-certs\") pod \"25d7898d-3961-4da3-a12e-3cce13ff8457\" (UID: \"25d7898d-3961-4da3-a12e-3cce13ff8457\") " Jul 26 16:15:32.196884 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:32.196779 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25d7898d-3961-4da3-a12e-3cce13ff8457-istio-token\") pod \"25d7898d-3961-4da3-a12e-3cce13ff8457\" (UID: \"25d7898d-3961-4da3-a12e-3cce13ff8457\") " Jul 26 16:15:32.196884 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:32.196838 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25d7898d-3961-4da3-a12e-3cce13ff8457-istio-envoy\") pod \"25d7898d-3961-4da3-a12e-3cce13ff8457\" (UID: \"25d7898d-3961-4da3-a12e-3cce13ff8457\") " Jul 26 16:15:32.197259 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:32.197133 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25d7898d-3961-4da3-a12e-3cce13ff8457-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "25d7898d-3961-4da3-a12e-3cce13ff8457" (UID: "25d7898d-3961-4da3-a12e-3cce13ff8457"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:15:32.197259 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:32.197214 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25d7898d-3961-4da3-a12e-3cce13ff8457-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "25d7898d-3961-4da3-a12e-3cce13ff8457" (UID: "25d7898d-3961-4da3-a12e-3cce13ff8457"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 16:15:32.197259 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:32.197212 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25d7898d-3961-4da3-a12e-3cce13ff8457-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "25d7898d-3961-4da3-a12e-3cce13ff8457" (UID: "25d7898d-3961-4da3-a12e-3cce13ff8457"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:15:32.197550 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:32.197523 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25d7898d-3961-4da3-a12e-3cce13ff8457-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "25d7898d-3961-4da3-a12e-3cce13ff8457" (UID: "25d7898d-3961-4da3-a12e-3cce13ff8457"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:15:32.197628 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:32.197528 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25d7898d-3961-4da3-a12e-3cce13ff8457-istio-data" (OuterVolumeSpecName: "istio-data") pod "25d7898d-3961-4da3-a12e-3cce13ff8457" (UID: "25d7898d-3961-4da3-a12e-3cce13ff8457"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:15:32.199220 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:32.199186 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25d7898d-3961-4da3-a12e-3cce13ff8457-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "25d7898d-3961-4da3-a12e-3cce13ff8457" (UID: "25d7898d-3961-4da3-a12e-3cce13ff8457"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:15:32.199220 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:32.199199 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25d7898d-3961-4da3-a12e-3cce13ff8457-kube-api-access-5r85m" (OuterVolumeSpecName: "kube-api-access-5r85m") pod "25d7898d-3961-4da3-a12e-3cce13ff8457" (UID: "25d7898d-3961-4da3-a12e-3cce13ff8457"). InnerVolumeSpecName "kube-api-access-5r85m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 16:15:32.199501 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:32.199480 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/25d7898d-3961-4da3-a12e-3cce13ff8457-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "25d7898d-3961-4da3-a12e-3cce13ff8457" (UID: "25d7898d-3961-4da3-a12e-3cce13ff8457"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 26 16:15:32.199558 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:32.199532 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25d7898d-3961-4da3-a12e-3cce13ff8457-istio-token" (OuterVolumeSpecName: "istio-token") pod "25d7898d-3961-4da3-a12e-3cce13ff8457" (UID: "25d7898d-3961-4da3-a12e-3cce13ff8457"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 16:15:32.298546 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:32.298509 2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25d7898d-3961-4da3-a12e-3cce13ff8457-istio-data\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 16:15:32.298546 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:32.298536 2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25d7898d-3961-4da3-a12e-3cce13ff8457-credential-socket\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 16:15:32.298546 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:32.298547 2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25d7898d-3961-4da3-a12e-3cce13ff8457-istiod-ca-cert\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 16:15:32.298546 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:32.298555 2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25d7898d-3961-4da3-a12e-3cce13ff8457-workload-socket\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 16:15:32.298810 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:32.298563 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5r85m\" (UniqueName: \"kubernetes.io/projected/25d7898d-3961-4da3-a12e-3cce13ff8457-kube-api-access-5r85m\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 16:15:32.298810 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:32.298572 2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25d7898d-3961-4da3-a12e-3cce13ff8457-istio-podinfo\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 16:15:32.298810 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:32.298580 2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25d7898d-3961-4da3-a12e-3cce13ff8457-workload-certs\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 16:15:32.298810 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:32.298588 2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25d7898d-3961-4da3-a12e-3cce13ff8457-istio-token\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 16:15:32.298810 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:32.298596 2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25d7898d-3961-4da3-a12e-3cce13ff8457-istio-envoy\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 16:15:32.360165 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:32.360130 2577 generic.go:356] "Generic (PLEG): container finished" podID="25d7898d-3961-4da3-a12e-3cce13ff8457" containerID="7ddccd840a8e1fb8ae4a99a323349b397836fe85d34b7f86252a147aeba1f976" exitCode=0 Jul 26 16:15:32.360337 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:32.360177 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-4b0d96-openshift-default-5d8cc86f8-v8b4l" Jul 26 16:15:32.360337 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:32.360217 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4b0d96-openshift-default-5d8cc86f8-v8b4l" event={"ID":"25d7898d-3961-4da3-a12e-3cce13ff8457","Type":"ContainerDied","Data":"7ddccd840a8e1fb8ae4a99a323349b397836fe85d34b7f86252a147aeba1f976"} Jul 26 16:15:32.360337 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:32.360273 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4b0d96-openshift-default-5d8cc86f8-v8b4l" event={"ID":"25d7898d-3961-4da3-a12e-3cce13ff8457","Type":"ContainerDied","Data":"54c958f67682265cac8174a01254e708cb0cd554cea0cbe8d5e290f667dd7b9e"} Jul 26 16:15:32.360337 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:32.360299 2577 scope.go:117] "RemoveContainer" containerID="7ddccd840a8e1fb8ae4a99a323349b397836fe85d34b7f86252a147aeba1f976" Jul 26 16:15:32.370781 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:32.370765 2577 scope.go:117] "RemoveContainer" containerID="a1f15cb1bdb96341445752493a0416e2805faa11ddeba498e627768e7c6f9159" Jul 26 16:15:32.379101 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:32.379084 2577 scope.go:117] "RemoveContainer" containerID="7ddccd840a8e1fb8ae4a99a323349b397836fe85d34b7f86252a147aeba1f976" Jul 26 16:15:32.379388 ip-10-0-138-163 kubenswrapper[2577]: E0726 16:15:32.379357 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ddccd840a8e1fb8ae4a99a323349b397836fe85d34b7f86252a147aeba1f976\": container with ID starting with 7ddccd840a8e1fb8ae4a99a323349b397836fe85d34b7f86252a147aeba1f976 not found: ID does not exist" containerID="7ddccd840a8e1fb8ae4a99a323349b397836fe85d34b7f86252a147aeba1f976" Jul 26 16:15:32.379523 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:32.379416 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ddccd840a8e1fb8ae4a99a323349b397836fe85d34b7f86252a147aeba1f976"} err="failed to get container status \"7ddccd840a8e1fb8ae4a99a323349b397836fe85d34b7f86252a147aeba1f976\": rpc error: code = NotFound desc = could not find container \"7ddccd840a8e1fb8ae4a99a323349b397836fe85d34b7f86252a147aeba1f976\": container with ID starting with 7ddccd840a8e1fb8ae4a99a323349b397836fe85d34b7f86252a147aeba1f976 not found: ID does not exist" Jul 26 16:15:32.379523 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:32.379436 2577 scope.go:117] "RemoveContainer" containerID="a1f15cb1bdb96341445752493a0416e2805faa11ddeba498e627768e7c6f9159" Jul 26 16:15:32.379738 ip-10-0-138-163 kubenswrapper[2577]: E0726 16:15:32.379705 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1f15cb1bdb96341445752493a0416e2805faa11ddeba498e627768e7c6f9159\": container with ID starting with a1f15cb1bdb96341445752493a0416e2805faa11ddeba498e627768e7c6f9159 not found: ID does not exist" containerID="a1f15cb1bdb96341445752493a0416e2805faa11ddeba498e627768e7c6f9159" Jul 26 16:15:32.379821 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:32.379737 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1f15cb1bdb96341445752493a0416e2805faa11ddeba498e627768e7c6f9159"} err="failed to get container status \"a1f15cb1bdb96341445752493a0416e2805faa11ddeba498e627768e7c6f9159\": rpc error: code = NotFound desc = could not find container \"a1f15cb1bdb96341445752493a0416e2805faa11ddeba498e627768e7c6f9159\": container with ID starting with a1f15cb1bdb96341445752493a0416e2805faa11ddeba498e627768e7c6f9159 not found: ID does not exist" Jul 26 16:15:32.383450 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:32.383429 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-4b0d96-openshift-default-5d8cc86f8-v8b4l"] Jul 26 16:15:32.385966 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:32.385946 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-4b0d96-openshift-default-5d8cc86f8-v8b4l"] Jul 26 16:15:33.787609 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:33.787573 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="25d7898d-3961-4da3-a12e-3cce13ff8457" path="/var/lib/kubelet/pods/25d7898d-3961-4da3-a12e-3cce13ff8457/volumes" Jul 26 16:15:42.881877 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:42.881813 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:15:42.952673 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:42.952627 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:15:43.051509 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:43.051474 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:15:47.226900 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:47.226866 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:15:47.592337 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:47.592302 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:15:53.165468 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:53.165421 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:15:53.227314 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:53.227262 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:15:53.398007 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:53.397969 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-7cc89d-openshift-default-6d7c4cf8bb-ckxhl"] Jul 26 16:15:53.398284 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:53.398259 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-7cc89d-openshift-default-6d7c4cf8bb-ckxhl" podUID="083ae778-ec20-42ba-95b7-74f1f4b788b8" containerName="istio-proxy" containerID="cri-o://220e3b7d608f10fcae9087c553daec13c9cdf13419da491cfd97cb1ac52b75e6" gracePeriod=30 Jul 26 16:15:53.421901 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:53.421819 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:15:53.646010 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:53.645979 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:15:53.977651 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:53.977620 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:15:54.250973 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:54.250895 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-aadfb2-openshift-default-76db5f76c6-kc7m7"] Jul 26 16:15:54.251315 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:54.251274 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25d7898d-3961-4da3-a12e-3cce13ff8457" containerName="istio-proxy" Jul 26 16:15:54.251315 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:54.251283 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="25d7898d-3961-4da3-a12e-3cce13ff8457" containerName="istio-proxy" Jul 26 16:15:54.251390 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:54.251357 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="25d7898d-3961-4da3-a12e-3cce13ff8457" containerName="istio-proxy" Jul 26 16:15:54.254719 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:54.254701 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-aadfb2-openshift-default-76db5f76c6-kc7m7" Jul 26 16:15:54.257004 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:54.256985 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-aadfb2-openshift-default-dockercfg-zpf9x\"" Jul 26 16:15:54.269221 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:54.269189 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-aadfb2-openshift-default-76db5f76c6-kc7m7"] Jul 26 16:15:54.272199 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:54.272161 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:15:54.296190 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:54.296163 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b250a529-3b42-40ed-8e16-5eaa7d780d9f-istio-envoy\") pod \"e2e-shared-a-aadfb2-openshift-default-76db5f76c6-kc7m7\" (UID: \"b250a529-3b42-40ed-8e16-5eaa7d780d9f\") " pod="openshift-ingress/e2e-shared-a-aadfb2-openshift-default-76db5f76c6-kc7m7" Jul 26 16:15:54.296190 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:54.296193 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b250a529-3b42-40ed-8e16-5eaa7d780d9f-istio-token\") pod \"e2e-shared-a-aadfb2-openshift-default-76db5f76c6-kc7m7\" (UID: \"b250a529-3b42-40ed-8e16-5eaa7d780d9f\") " pod="openshift-ingress/e2e-shared-a-aadfb2-openshift-default-76db5f76c6-kc7m7" Jul 26 16:15:54.296369 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:54.296255 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b250a529-3b42-40ed-8e16-5eaa7d780d9f-credential-socket\") pod \"e2e-shared-a-aadfb2-openshift-default-76db5f76c6-kc7m7\" (UID: \"b250a529-3b42-40ed-8e16-5eaa7d780d9f\") " pod="openshift-ingress/e2e-shared-a-aadfb2-openshift-default-76db5f76c6-kc7m7" Jul 26 16:15:54.296369 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:54.296279 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b250a529-3b42-40ed-8e16-5eaa7d780d9f-workload-certs\") pod \"e2e-shared-a-aadfb2-openshift-default-76db5f76c6-kc7m7\" (UID: \"b250a529-3b42-40ed-8e16-5eaa7d780d9f\") " pod="openshift-ingress/e2e-shared-a-aadfb2-openshift-default-76db5f76c6-kc7m7" Jul 26 16:15:54.296369 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:54.296295 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b250a529-3b42-40ed-8e16-5eaa7d780d9f-istio-podinfo\") pod \"e2e-shared-a-aadfb2-openshift-default-76db5f76c6-kc7m7\" (UID: \"b250a529-3b42-40ed-8e16-5eaa7d780d9f\") " pod="openshift-ingress/e2e-shared-a-aadfb2-openshift-default-76db5f76c6-kc7m7" Jul 26 16:15:54.296369 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:54.296359 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b250a529-3b42-40ed-8e16-5eaa7d780d9f-workload-socket\") pod \"e2e-shared-a-aadfb2-openshift-default-76db5f76c6-kc7m7\" (UID: \"b250a529-3b42-40ed-8e16-5eaa7d780d9f\") " pod="openshift-ingress/e2e-shared-a-aadfb2-openshift-default-76db5f76c6-kc7m7" Jul 26 16:15:54.296564 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:54.296381 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rwx7\" (UniqueName: \"kubernetes.io/projected/b250a529-3b42-40ed-8e16-5eaa7d780d9f-kube-api-access-7rwx7\") pod \"e2e-shared-a-aadfb2-openshift-default-76db5f76c6-kc7m7\" (UID: \"b250a529-3b42-40ed-8e16-5eaa7d780d9f\") " pod="openshift-ingress/e2e-shared-a-aadfb2-openshift-default-76db5f76c6-kc7m7" Jul 26 16:15:54.296564 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:54.296493 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b250a529-3b42-40ed-8e16-5eaa7d780d9f-istiod-ca-cert\") pod \"e2e-shared-a-aadfb2-openshift-default-76db5f76c6-kc7m7\" (UID: \"b250a529-3b42-40ed-8e16-5eaa7d780d9f\") " pod="openshift-ingress/e2e-shared-a-aadfb2-openshift-default-76db5f76c6-kc7m7" Jul 26 16:15:54.296564 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:54.296524 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b250a529-3b42-40ed-8e16-5eaa7d780d9f-istio-data\") pod \"e2e-shared-a-aadfb2-openshift-default-76db5f76c6-kc7m7\" (UID: \"b250a529-3b42-40ed-8e16-5eaa7d780d9f\") " pod="openshift-ingress/e2e-shared-a-aadfb2-openshift-default-76db5f76c6-kc7m7" Jul 26 16:15:54.397667 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:54.397634 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b250a529-3b42-40ed-8e16-5eaa7d780d9f-credential-socket\") pod \"e2e-shared-a-aadfb2-openshift-default-76db5f76c6-kc7m7\" (UID: \"b250a529-3b42-40ed-8e16-5eaa7d780d9f\") " pod="openshift-ingress/e2e-shared-a-aadfb2-openshift-default-76db5f76c6-kc7m7" Jul 26 16:15:54.397667 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:54.397671 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b250a529-3b42-40ed-8e16-5eaa7d780d9f-workload-certs\") pod \"e2e-shared-a-aadfb2-openshift-default-76db5f76c6-kc7m7\" (UID: \"b250a529-3b42-40ed-8e16-5eaa7d780d9f\") " pod="openshift-ingress/e2e-shared-a-aadfb2-openshift-default-76db5f76c6-kc7m7" Jul 26 16:15:54.397927 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:54.397689 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b250a529-3b42-40ed-8e16-5eaa7d780d9f-istio-podinfo\") pod \"e2e-shared-a-aadfb2-openshift-default-76db5f76c6-kc7m7\" (UID: \"b250a529-3b42-40ed-8e16-5eaa7d780d9f\") " pod="openshift-ingress/e2e-shared-a-aadfb2-openshift-default-76db5f76c6-kc7m7" Jul 26 16:15:54.397927 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:54.397711 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b250a529-3b42-40ed-8e16-5eaa7d780d9f-workload-socket\") pod \"e2e-shared-a-aadfb2-openshift-default-76db5f76c6-kc7m7\" (UID: \"b250a529-3b42-40ed-8e16-5eaa7d780d9f\") " pod="openshift-ingress/e2e-shared-a-aadfb2-openshift-default-76db5f76c6-kc7m7" Jul 26 16:15:54.397927 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:54.397733 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rwx7\" (UniqueName: \"kubernetes.io/projected/b250a529-3b42-40ed-8e16-5eaa7d780d9f-kube-api-access-7rwx7\") pod \"e2e-shared-a-aadfb2-openshift-default-76db5f76c6-kc7m7\" (UID: \"b250a529-3b42-40ed-8e16-5eaa7d780d9f\") " pod="openshift-ingress/e2e-shared-a-aadfb2-openshift-default-76db5f76c6-kc7m7" Jul 26 16:15:54.397927 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:54.397794 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b250a529-3b42-40ed-8e16-5eaa7d780d9f-istiod-ca-cert\") pod \"e2e-shared-a-aadfb2-openshift-default-76db5f76c6-kc7m7\" (UID: \"b250a529-3b42-40ed-8e16-5eaa7d780d9f\") " pod="openshift-ingress/e2e-shared-a-aadfb2-openshift-default-76db5f76c6-kc7m7" Jul 26 16:15:54.397927 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:54.397830 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b250a529-3b42-40ed-8e16-5eaa7d780d9f-istio-data\") pod \"e2e-shared-a-aadfb2-openshift-default-76db5f76c6-kc7m7\" (UID: \"b250a529-3b42-40ed-8e16-5eaa7d780d9f\") " pod="openshift-ingress/e2e-shared-a-aadfb2-openshift-default-76db5f76c6-kc7m7" Jul 26 16:15:54.397927 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:54.397884 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b250a529-3b42-40ed-8e16-5eaa7d780d9f-istio-envoy\") pod \"e2e-shared-a-aadfb2-openshift-default-76db5f76c6-kc7m7\" (UID: \"b250a529-3b42-40ed-8e16-5eaa7d780d9f\") " pod="openshift-ingress/e2e-shared-a-aadfb2-openshift-default-76db5f76c6-kc7m7" Jul 26 16:15:54.397927 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:54.397915 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b250a529-3b42-40ed-8e16-5eaa7d780d9f-istio-token\") pod \"e2e-shared-a-aadfb2-openshift-default-76db5f76c6-kc7m7\" (UID: \"b250a529-3b42-40ed-8e16-5eaa7d780d9f\") " pod="openshift-ingress/e2e-shared-a-aadfb2-openshift-default-76db5f76c6-kc7m7" Jul 26 16:15:54.398288 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:54.398167 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b250a529-3b42-40ed-8e16-5eaa7d780d9f-workload-certs\") pod \"e2e-shared-a-aadfb2-openshift-default-76db5f76c6-kc7m7\" (UID: \"b250a529-3b42-40ed-8e16-5eaa7d780d9f\") " pod="openshift-ingress/e2e-shared-a-aadfb2-openshift-default-76db5f76c6-kc7m7" Jul 26 16:15:54.398288 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:54.398235 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b250a529-3b42-40ed-8e16-5eaa7d780d9f-credential-socket\") pod \"e2e-shared-a-aadfb2-openshift-default-76db5f76c6-kc7m7\" (UID: \"b250a529-3b42-40ed-8e16-5eaa7d780d9f\") " pod="openshift-ingress/e2e-shared-a-aadfb2-openshift-default-76db5f76c6-kc7m7" Jul 26 16:15:54.398453 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:54.398430 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b250a529-3b42-40ed-8e16-5eaa7d780d9f-istio-data\") pod \"e2e-shared-a-aadfb2-openshift-default-76db5f76c6-kc7m7\" (UID: \"b250a529-3b42-40ed-8e16-5eaa7d780d9f\") " pod="openshift-ingress/e2e-shared-a-aadfb2-openshift-default-76db5f76c6-kc7m7" Jul 26 16:15:54.398566 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:54.398474 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b250a529-3b42-40ed-8e16-5eaa7d780d9f-workload-socket\") pod \"e2e-shared-a-aadfb2-openshift-default-76db5f76c6-kc7m7\" (UID: \"b250a529-3b42-40ed-8e16-5eaa7d780d9f\") " pod="openshift-ingress/e2e-shared-a-aadfb2-openshift-default-76db5f76c6-kc7m7" Jul 26 16:15:54.398688 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:54.398662 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b250a529-3b42-40ed-8e16-5eaa7d780d9f-istiod-ca-cert\") pod \"e2e-shared-a-aadfb2-openshift-default-76db5f76c6-kc7m7\" (UID: \"b250a529-3b42-40ed-8e16-5eaa7d780d9f\") " pod="openshift-ingress/e2e-shared-a-aadfb2-openshift-default-76db5f76c6-kc7m7" Jul 26 16:15:54.400231 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:54.400212 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b250a529-3b42-40ed-8e16-5eaa7d780d9f-istio-envoy\") pod \"e2e-shared-a-aadfb2-openshift-default-76db5f76c6-kc7m7\" (UID: \"b250a529-3b42-40ed-8e16-5eaa7d780d9f\") " pod="openshift-ingress/e2e-shared-a-aadfb2-openshift-default-76db5f76c6-kc7m7" Jul 26 16:15:54.400378 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:54.400328 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b250a529-3b42-40ed-8e16-5eaa7d780d9f-istio-podinfo\") pod \"e2e-shared-a-aadfb2-openshift-default-76db5f76c6-kc7m7\" (UID: \"b250a529-3b42-40ed-8e16-5eaa7d780d9f\") " pod="openshift-ingress/e2e-shared-a-aadfb2-openshift-default-76db5f76c6-kc7m7" Jul 26 16:15:54.405312 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:54.405288 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b250a529-3b42-40ed-8e16-5eaa7d780d9f-istio-token\") pod \"e2e-shared-a-aadfb2-openshift-default-76db5f76c6-kc7m7\" (UID: \"b250a529-3b42-40ed-8e16-5eaa7d780d9f\") " pod="openshift-ingress/e2e-shared-a-aadfb2-openshift-default-76db5f76c6-kc7m7" Jul 26 16:15:54.405560 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:54.405539 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rwx7\" (UniqueName: \"kubernetes.io/projected/b250a529-3b42-40ed-8e16-5eaa7d780d9f-kube-api-access-7rwx7\") pod \"e2e-shared-a-aadfb2-openshift-default-76db5f76c6-kc7m7\" (UID: \"b250a529-3b42-40ed-8e16-5eaa7d780d9f\") " pod="openshift-ingress/e2e-shared-a-aadfb2-openshift-default-76db5f76c6-kc7m7" Jul 26 16:15:54.510116 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:54.510027 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:15:54.565112 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:54.565080 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-aadfb2-openshift-default-76db5f76c6-kc7m7" Jul 26 16:15:54.708012 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:54.707982 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-aadfb2-openshift-default-76db5f76c6-kc7m7"] Jul 26 16:15:54.710008 ip-10-0-138-163 kubenswrapper[2577]: W0726 16:15:54.709983 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb250a529_3b42_40ed_8e16_5eaa7d780d9f.slice/crio-681fa5468f7cc7a4d1ce40ea10e962dec30128c206299a9c6ef4a63b9bdc2b47 WatchSource:0}: Error finding container 681fa5468f7cc7a4d1ce40ea10e962dec30128c206299a9c6ef4a63b9bdc2b47: Status 404 returned error can't find the container with id 681fa5468f7cc7a4d1ce40ea10e962dec30128c206299a9c6ef4a63b9bdc2b47 Jul 26 16:15:54.712115 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:54.712081 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:15:54.712204 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:54.712146 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:15:54.712204 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:54.712173 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:15:55.490097 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:55.490062 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-aadfb2-openshift-default-76db5f76c6-kc7m7" event={"ID":"b250a529-3b42-40ed-8e16-5eaa7d780d9f","Type":"ContainerStarted","Data":"74ede80574287b9f2db9f48f4ed3ca61875d5964f37e2e116abf70e03466df72"} Jul 26 16:15:55.490498 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:55.490115 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-aadfb2-openshift-default-76db5f76c6-kc7m7" event={"ID":"b250a529-3b42-40ed-8e16-5eaa7d780d9f","Type":"ContainerStarted","Data":"681fa5468f7cc7a4d1ce40ea10e962dec30128c206299a9c6ef4a63b9bdc2b47"} Jul 26 16:15:55.517499 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:55.517449 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-aadfb2-openshift-default-76db5f76c6-kc7m7" podStartSLOduration=1.51743436 podStartE2EDuration="1.51743436s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 16:15:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 16:15:55.515260193 +0000 UTC m=+2278.254613539" watchObservedRunningTime="2026-07-26 16:15:55.51743436 +0000 UTC m=+2278.256787670" Jul 26 16:15:55.565588 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:55.565548 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-aadfb2-openshift-default-76db5f76c6-kc7m7" Jul 26 16:15:55.570248 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:55.570225 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-aadfb2-openshift-default-76db5f76c6-kc7m7" Jul 26 16:15:56.495896 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:56.495867 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-aadfb2-openshift-default-76db5f76c6-kc7m7" Jul 26 16:15:56.496827 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:56.496810 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-aadfb2-openshift-default-76db5f76c6-kc7m7" Jul 26 16:15:57.399117 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:57.399084 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:15:57.505004 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:57.504972 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:15:57.672851 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:57.672757 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:15:57.758721 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:57.758660 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:15:58.238640 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:58.238604 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:15:58.332411 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:58.332318 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:15:58.656448 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:58.656426 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-7cc89d-openshift-default-6d7c4cf8bb-ckxhl" Jul 26 16:15:58.738665 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:58.738627 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/083ae778-ec20-42ba-95b7-74f1f4b788b8-workload-socket\") pod \"083ae778-ec20-42ba-95b7-74f1f4b788b8\" (UID: \"083ae778-ec20-42ba-95b7-74f1f4b788b8\") " Jul 26 16:15:58.738866 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:58.738670 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hzq7g\" (UniqueName: \"kubernetes.io/projected/083ae778-ec20-42ba-95b7-74f1f4b788b8-kube-api-access-hzq7g\") pod \"083ae778-ec20-42ba-95b7-74f1f4b788b8\" (UID: \"083ae778-ec20-42ba-95b7-74f1f4b788b8\") " Jul 26 16:15:58.738866 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:58.738696 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/083ae778-ec20-42ba-95b7-74f1f4b788b8-istio-envoy\") pod \"083ae778-ec20-42ba-95b7-74f1f4b788b8\" (UID: \"083ae778-ec20-42ba-95b7-74f1f4b788b8\") " Jul 26 16:15:58.738866 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:58.738736 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/083ae778-ec20-42ba-95b7-74f1f4b788b8-istiod-ca-cert\") pod \"083ae778-ec20-42ba-95b7-74f1f4b788b8\" (UID: \"083ae778-ec20-42ba-95b7-74f1f4b788b8\") " Jul 26 16:15:58.738866 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:58.738759 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/083ae778-ec20-42ba-95b7-74f1f4b788b8-istio-data\") pod \"083ae778-ec20-42ba-95b7-74f1f4b788b8\" (UID: \"083ae778-ec20-42ba-95b7-74f1f4b788b8\") " Jul 26 16:15:58.738866 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:58.738810 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/083ae778-ec20-42ba-95b7-74f1f4b788b8-workload-certs\") pod \"083ae778-ec20-42ba-95b7-74f1f4b788b8\" (UID: \"083ae778-ec20-42ba-95b7-74f1f4b788b8\") " Jul 26 16:15:58.739126 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:58.738866 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/083ae778-ec20-42ba-95b7-74f1f4b788b8-credential-socket\") pod \"083ae778-ec20-42ba-95b7-74f1f4b788b8\" (UID: \"083ae778-ec20-42ba-95b7-74f1f4b788b8\") " Jul 26 16:15:58.739126 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:58.738901 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/083ae778-ec20-42ba-95b7-74f1f4b788b8-istio-podinfo\") pod \"083ae778-ec20-42ba-95b7-74f1f4b788b8\" (UID: \"083ae778-ec20-42ba-95b7-74f1f4b788b8\") " Jul 26 16:15:58.739126 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:58.739004 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/083ae778-ec20-42ba-95b7-74f1f4b788b8-istio-token\") pod \"083ae778-ec20-42ba-95b7-74f1f4b788b8\" (UID: \"083ae778-ec20-42ba-95b7-74f1f4b788b8\") " Jul 26 16:15:58.739126 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:58.739080 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/083ae778-ec20-42ba-95b7-74f1f4b788b8-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "083ae778-ec20-42ba-95b7-74f1f4b788b8" (UID: "083ae778-ec20-42ba-95b7-74f1f4b788b8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:15:58.739331 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:58.739203 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/083ae778-ec20-42ba-95b7-74f1f4b788b8-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "083ae778-ec20-42ba-95b7-74f1f4b788b8" (UID: "083ae778-ec20-42ba-95b7-74f1f4b788b8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:15:58.739538 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:58.739461 2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/083ae778-ec20-42ba-95b7-74f1f4b788b8-credential-socket\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 16:15:58.739538 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:58.739488 2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/083ae778-ec20-42ba-95b7-74f1f4b788b8-workload-socket\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 16:15:58.739730 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:58.739556 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/083ae778-ec20-42ba-95b7-74f1f4b788b8-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "083ae778-ec20-42ba-95b7-74f1f4b788b8" (UID: "083ae778-ec20-42ba-95b7-74f1f4b788b8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 16:15:58.739827 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:58.739803 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/083ae778-ec20-42ba-95b7-74f1f4b788b8-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "083ae778-ec20-42ba-95b7-74f1f4b788b8" (UID: "083ae778-ec20-42ba-95b7-74f1f4b788b8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:15:58.740232 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:58.740202 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/083ae778-ec20-42ba-95b7-74f1f4b788b8-istio-data" (OuterVolumeSpecName: "istio-data") pod "083ae778-ec20-42ba-95b7-74f1f4b788b8" (UID: "083ae778-ec20-42ba-95b7-74f1f4b788b8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:15:58.741345 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:58.741324 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/083ae778-ec20-42ba-95b7-74f1f4b788b8-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "083ae778-ec20-42ba-95b7-74f1f4b788b8" (UID: "083ae778-ec20-42ba-95b7-74f1f4b788b8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 26 16:15:58.741475 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:58.741447 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/083ae778-ec20-42ba-95b7-74f1f4b788b8-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "083ae778-ec20-42ba-95b7-74f1f4b788b8" (UID: "083ae778-ec20-42ba-95b7-74f1f4b788b8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:15:58.741596 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:58.741581 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/083ae778-ec20-42ba-95b7-74f1f4b788b8-kube-api-access-hzq7g" (OuterVolumeSpecName: "kube-api-access-hzq7g") pod "083ae778-ec20-42ba-95b7-74f1f4b788b8" (UID: "083ae778-ec20-42ba-95b7-74f1f4b788b8"). InnerVolumeSpecName "kube-api-access-hzq7g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 16:15:58.741747 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:58.741732 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/083ae778-ec20-42ba-95b7-74f1f4b788b8-istio-token" (OuterVolumeSpecName: "istio-token") pod "083ae778-ec20-42ba-95b7-74f1f4b788b8" (UID: "083ae778-ec20-42ba-95b7-74f1f4b788b8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 16:15:58.840472 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:58.840442 2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/083ae778-ec20-42ba-95b7-74f1f4b788b8-istio-token\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 16:15:58.840472 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:58.840467 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzq7g\" (UniqueName: \"kubernetes.io/projected/083ae778-ec20-42ba-95b7-74f1f4b788b8-kube-api-access-hzq7g\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 16:15:58.840472 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:58.840476 2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/083ae778-ec20-42ba-95b7-74f1f4b788b8-istio-envoy\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 16:15:58.840702 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:58.840485 2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/083ae778-ec20-42ba-95b7-74f1f4b788b8-istiod-ca-cert\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 16:15:58.840702 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:58.840493 2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/083ae778-ec20-42ba-95b7-74f1f4b788b8-istio-data\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 16:15:58.840702 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:58.840500 2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/083ae778-ec20-42ba-95b7-74f1f4b788b8-workload-certs\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 16:15:58.840702 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:58.840507 2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/083ae778-ec20-42ba-95b7-74f1f4b788b8-istio-podinfo\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 16:15:59.512856 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:59.512817 2577 generic.go:356] "Generic (PLEG): container finished" podID="083ae778-ec20-42ba-95b7-74f1f4b788b8" containerID="220e3b7d608f10fcae9087c553daec13c9cdf13419da491cfd97cb1ac52b75e6" exitCode=0 Jul 26 16:15:59.513037 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:59.512876 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-7cc89d-openshift-default-6d7c4cf8bb-ckxhl" Jul 26 16:15:59.513037 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:59.512911 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-7cc89d-openshift-default-6d7c4cf8bb-ckxhl" event={"ID":"083ae778-ec20-42ba-95b7-74f1f4b788b8","Type":"ContainerDied","Data":"220e3b7d608f10fcae9087c553daec13c9cdf13419da491cfd97cb1ac52b75e6"} Jul 26 16:15:59.513037 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:59.512976 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-7cc89d-openshift-default-6d7c4cf8bb-ckxhl" event={"ID":"083ae778-ec20-42ba-95b7-74f1f4b788b8","Type":"ContainerDied","Data":"6f9985bde3ccc15543138e74f95af0a73bacd49b652619a8d07fe6dc2593849b"} Jul 26 16:15:59.513037 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:59.513005 2577 scope.go:117] "RemoveContainer" containerID="220e3b7d608f10fcae9087c553daec13c9cdf13419da491cfd97cb1ac52b75e6" Jul 26 16:15:59.523590 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:59.523571 2577 scope.go:117] "RemoveContainer" containerID="c167731f27c3c1de5cd60975abbeed35344eeece4afd91f4e77981e6ee777c23" Jul 26 16:15:59.531924 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:59.531904 2577 scope.go:117] "RemoveContainer" containerID="220e3b7d608f10fcae9087c553daec13c9cdf13419da491cfd97cb1ac52b75e6" Jul 26 16:15:59.532175 ip-10-0-138-163 kubenswrapper[2577]: E0726 16:15:59.532153 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"220e3b7d608f10fcae9087c553daec13c9cdf13419da491cfd97cb1ac52b75e6\": container with ID starting with 220e3b7d608f10fcae9087c553daec13c9cdf13419da491cfd97cb1ac52b75e6 not found: ID does not exist" containerID="220e3b7d608f10fcae9087c553daec13c9cdf13419da491cfd97cb1ac52b75e6" Jul 26 16:15:59.532258 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:59.532187 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"220e3b7d608f10fcae9087c553daec13c9cdf13419da491cfd97cb1ac52b75e6"} err="failed to get container status \"220e3b7d608f10fcae9087c553daec13c9cdf13419da491cfd97cb1ac52b75e6\": rpc error: code = NotFound desc = could not find container \"220e3b7d608f10fcae9087c553daec13c9cdf13419da491cfd97cb1ac52b75e6\": container with ID starting with 220e3b7d608f10fcae9087c553daec13c9cdf13419da491cfd97cb1ac52b75e6 not found: ID does not exist" Jul 26 16:15:59.532258 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:59.532213 2577 scope.go:117] "RemoveContainer" containerID="c167731f27c3c1de5cd60975abbeed35344eeece4afd91f4e77981e6ee777c23" Jul 26 16:15:59.532457 ip-10-0-138-163 kubenswrapper[2577]: E0726 16:15:59.532430 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c167731f27c3c1de5cd60975abbeed35344eeece4afd91f4e77981e6ee777c23\": container with ID starting with c167731f27c3c1de5cd60975abbeed35344eeece4afd91f4e77981e6ee777c23 not found: ID does not exist" containerID="c167731f27c3c1de5cd60975abbeed35344eeece4afd91f4e77981e6ee777c23" Jul 26 16:15:59.532507 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:59.532467 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c167731f27c3c1de5cd60975abbeed35344eeece4afd91f4e77981e6ee777c23"} err="failed to get container status \"c167731f27c3c1de5cd60975abbeed35344eeece4afd91f4e77981e6ee777c23\": rpc error: code = NotFound desc = could not find container \"c167731f27c3c1de5cd60975abbeed35344eeece4afd91f4e77981e6ee777c23\": container with ID starting with c167731f27c3c1de5cd60975abbeed35344eeece4afd91f4e77981e6ee777c23 not found: ID does not exist" Jul 26 16:15:59.539028 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:59.538327 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-7cc89d-openshift-default-6d7c4cf8bb-ckxhl"] Jul 26 16:15:59.544929 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:59.544905 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-7cc89d-openshift-default-6d7c4cf8bb-ckxhl"] Jul 26 16:15:59.787319 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:15:59.787287 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="083ae778-ec20-42ba-95b7-74f1f4b788b8" path="/var/lib/kubelet/pods/083ae778-ec20-42ba-95b7-74f1f4b788b8/volumes" Jul 26 16:16:01.891915 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:16:01.891879 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:16:02.206723 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:16:02.206644 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:16:02.456304 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:16:02.456264 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-e0208c-openshift-default-745b9f8964-jphn4"] Jul 26 16:16:02.456664 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:16:02.456651 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="083ae778-ec20-42ba-95b7-74f1f4b788b8" containerName="istio-proxy" Jul 26 16:16:02.456757 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:16:02.456665 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="083ae778-ec20-42ba-95b7-74f1f4b788b8" containerName="istio-proxy" Jul 26 16:16:02.456757 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:16:02.456681 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="083ae778-ec20-42ba-95b7-74f1f4b788b8" containerName="istio-proxy" Jul 26 16:16:02.456757 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:16:02.456690 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="083ae778-ec20-42ba-95b7-74f1f4b788b8" containerName="istio-proxy" Jul 26 16:16:02.456757 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:16:02.456711 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25d7898d-3961-4da3-a12e-3cce13ff8457" containerName="istio-proxy" Jul 26 16:16:02.456757 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:16:02.456718 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="25d7898d-3961-4da3-a12e-3cce13ff8457" containerName="istio-proxy" Jul 26 16:16:02.456918 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:16:02.456787 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="083ae778-ec20-42ba-95b7-74f1f4b788b8" containerName="istio-proxy" Jul 26 16:16:02.456918 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:16:02.456796 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="083ae778-ec20-42ba-95b7-74f1f4b788b8" containerName="istio-proxy" Jul 26 16:16:02.456918 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:16:02.456804 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="25d7898d-3961-4da3-a12e-3cce13ff8457" containerName="istio-proxy" Jul 26 16:16:02.461519 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:16:02.461499 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-e0208c-openshift-default-745b9f8964-jphn4" Jul 26 16:16:02.463993 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:16:02.463975 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-e0208c-openshift-default-dockercfg-46trv\"" Jul 26 16:16:02.471427 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:16:02.471388 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-e0208c-openshift-default-745b9f8964-jphn4"] Jul 26 16:16:02.484625 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:16:02.484601 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:16:02.571674 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:16:02.571634 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0002d75b-65fd-4dc9-9dbf-72122f738317-istiod-ca-cert\") pod \"e2e-shared-b-e0208c-openshift-default-745b9f8964-jphn4\" (UID: \"0002d75b-65fd-4dc9-9dbf-72122f738317\") " pod="openshift-ingress/e2e-shared-b-e0208c-openshift-default-745b9f8964-jphn4" Jul 26 16:16:02.571901 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:16:02.571722 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0002d75b-65fd-4dc9-9dbf-72122f738317-istio-data\") pod \"e2e-shared-b-e0208c-openshift-default-745b9f8964-jphn4\" (UID: \"0002d75b-65fd-4dc9-9dbf-72122f738317\") " pod="openshift-ingress/e2e-shared-b-e0208c-openshift-default-745b9f8964-jphn4" Jul 26 16:16:02.571901 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:16:02.571746 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0002d75b-65fd-4dc9-9dbf-72122f738317-istio-envoy\") pod \"e2e-shared-b-e0208c-openshift-default-745b9f8964-jphn4\" (UID: \"0002d75b-65fd-4dc9-9dbf-72122f738317\") " pod="openshift-ingress/e2e-shared-b-e0208c-openshift-default-745b9f8964-jphn4" Jul 26 16:16:02.571901 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:16:02.571786 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0002d75b-65fd-4dc9-9dbf-72122f738317-workload-certs\") pod \"e2e-shared-b-e0208c-openshift-default-745b9f8964-jphn4\" (UID: \"0002d75b-65fd-4dc9-9dbf-72122f738317\") " pod="openshift-ingress/e2e-shared-b-e0208c-openshift-default-745b9f8964-jphn4" Jul 26 16:16:02.571901 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:16:02.571809 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0002d75b-65fd-4dc9-9dbf-72122f738317-credential-socket\") pod \"e2e-shared-b-e0208c-openshift-default-745b9f8964-jphn4\" (UID: \"0002d75b-65fd-4dc9-9dbf-72122f738317\") " pod="openshift-ingress/e2e-shared-b-e0208c-openshift-default-745b9f8964-jphn4" Jul 26 16:16:02.571901 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:16:02.571834 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0002d75b-65fd-4dc9-9dbf-72122f738317-istio-podinfo\") pod \"e2e-shared-b-e0208c-openshift-default-745b9f8964-jphn4\" (UID: \"0002d75b-65fd-4dc9-9dbf-72122f738317\") " pod="openshift-ingress/e2e-shared-b-e0208c-openshift-default-745b9f8964-jphn4" Jul 26 16:16:02.571901 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:16:02.571863 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0002d75b-65fd-4dc9-9dbf-72122f738317-workload-socket\") pod \"e2e-shared-b-e0208c-openshift-default-745b9f8964-jphn4\" (UID: \"0002d75b-65fd-4dc9-9dbf-72122f738317\") " pod="openshift-ingress/e2e-shared-b-e0208c-openshift-default-745b9f8964-jphn4" Jul 26 16:16:02.571901 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:16:02.571891 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0002d75b-65fd-4dc9-9dbf-72122f738317-istio-token\") pod \"e2e-shared-b-e0208c-openshift-default-745b9f8964-jphn4\" (UID: \"0002d75b-65fd-4dc9-9dbf-72122f738317\") " pod="openshift-ingress/e2e-shared-b-e0208c-openshift-default-745b9f8964-jphn4" Jul 26 16:16:02.572257 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:16:02.571951 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbkph\" (UniqueName: \"kubernetes.io/projected/0002d75b-65fd-4dc9-9dbf-72122f738317-kube-api-access-dbkph\") pod \"e2e-shared-b-e0208c-openshift-default-745b9f8964-jphn4\" (UID: \"0002d75b-65fd-4dc9-9dbf-72122f738317\") " pod="openshift-ingress/e2e-shared-b-e0208c-openshift-default-745b9f8964-jphn4" Jul 26 16:16:02.672983 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:16:02.672948 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0002d75b-65fd-4dc9-9dbf-72122f738317-istiod-ca-cert\") pod \"e2e-shared-b-e0208c-openshift-default-745b9f8964-jphn4\" (UID: \"0002d75b-65fd-4dc9-9dbf-72122f738317\") " pod="openshift-ingress/e2e-shared-b-e0208c-openshift-default-745b9f8964-jphn4" Jul 26 16:16:02.673187 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:16:02.673015 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0002d75b-65fd-4dc9-9dbf-72122f738317-istio-data\") pod \"e2e-shared-b-e0208c-openshift-default-745b9f8964-jphn4\" (UID: \"0002d75b-65fd-4dc9-9dbf-72122f738317\") " pod="openshift-ingress/e2e-shared-b-e0208c-openshift-default-745b9f8964-jphn4" Jul 26 16:16:02.673187 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:16:02.673043 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0002d75b-65fd-4dc9-9dbf-72122f738317-istio-envoy\") pod \"e2e-shared-b-e0208c-openshift-default-745b9f8964-jphn4\" (UID: \"0002d75b-65fd-4dc9-9dbf-72122f738317\") " pod="openshift-ingress/e2e-shared-b-e0208c-openshift-default-745b9f8964-jphn4" Jul 26 16:16:02.673187 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:16:02.673074 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0002d75b-65fd-4dc9-9dbf-72122f738317-workload-certs\") pod \"e2e-shared-b-e0208c-openshift-default-745b9f8964-jphn4\" (UID: \"0002d75b-65fd-4dc9-9dbf-72122f738317\") " pod="openshift-ingress/e2e-shared-b-e0208c-openshift-default-745b9f8964-jphn4" Jul 26 16:16:02.673359 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:16:02.673234 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0002d75b-65fd-4dc9-9dbf-72122f738317-credential-socket\") pod \"e2e-shared-b-e0208c-openshift-default-745b9f8964-jphn4\" (UID: \"0002d75b-65fd-4dc9-9dbf-72122f738317\") " pod="openshift-ingress/e2e-shared-b-e0208c-openshift-default-745b9f8964-jphn4" Jul 26 16:16:02.673570 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:16:02.673537 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0002d75b-65fd-4dc9-9dbf-72122f738317-istio-podinfo\") pod \"e2e-shared-b-e0208c-openshift-default-745b9f8964-jphn4\" (UID: \"0002d75b-65fd-4dc9-9dbf-72122f738317\") " pod="openshift-ingress/e2e-shared-b-e0208c-openshift-default-745b9f8964-jphn4" Jul 26 16:16:02.673801 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:16:02.673772 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0002d75b-65fd-4dc9-9dbf-72122f738317-workload-socket\") pod \"e2e-shared-b-e0208c-openshift-default-745b9f8964-jphn4\" (UID: \"0002d75b-65fd-4dc9-9dbf-72122f738317\") " pod="openshift-ingress/e2e-shared-b-e0208c-openshift-default-745b9f8964-jphn4" Jul 26 16:16:02.673971 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:16:02.673953 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0002d75b-65fd-4dc9-9dbf-72122f738317-istio-token\") pod \"e2e-shared-b-e0208c-openshift-default-745b9f8964-jphn4\" (UID: \"0002d75b-65fd-4dc9-9dbf-72122f738317\") " pod="openshift-ingress/e2e-shared-b-e0208c-openshift-default-745b9f8964-jphn4" Jul 26 16:16:02.674207 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:16:02.674187 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbkph\" (UniqueName: \"kubernetes.io/projected/0002d75b-65fd-4dc9-9dbf-72122f738317-kube-api-access-dbkph\") pod \"e2e-shared-b-e0208c-openshift-default-745b9f8964-jphn4\" (UID: \"0002d75b-65fd-4dc9-9dbf-72122f738317\") " pod="openshift-ingress/e2e-shared-b-e0208c-openshift-default-745b9f8964-jphn4" Jul 26 16:16:02.674387 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:16:02.673866 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0002d75b-65fd-4dc9-9dbf-72122f738317-credential-socket\") pod \"e2e-shared-b-e0208c-openshift-default-745b9f8964-jphn4\" (UID: \"0002d75b-65fd-4dc9-9dbf-72122f738317\") " pod="openshift-ingress/e2e-shared-b-e0208c-openshift-default-745b9f8964-jphn4" Jul 26 16:16:02.674528 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:16:02.673609 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0002d75b-65fd-4dc9-9dbf-72122f738317-workload-certs\") pod \"e2e-shared-b-e0208c-openshift-default-745b9f8964-jphn4\" (UID: \"0002d75b-65fd-4dc9-9dbf-72122f738317\") " pod="openshift-ingress/e2e-shared-b-e0208c-openshift-default-745b9f8964-jphn4" Jul 26 16:16:02.674528 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:16:02.674058 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0002d75b-65fd-4dc9-9dbf-72122f738317-workload-socket\") pod \"e2e-shared-b-e0208c-openshift-default-745b9f8964-jphn4\" (UID: \"0002d75b-65fd-4dc9-9dbf-72122f738317\") " pod="openshift-ingress/e2e-shared-b-e0208c-openshift-default-745b9f8964-jphn4" Jul 26 16:16:02.674528 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:16:02.674113 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0002d75b-65fd-4dc9-9dbf-72122f738317-istiod-ca-cert\") pod \"e2e-shared-b-e0208c-openshift-default-745b9f8964-jphn4\" (UID: \"0002d75b-65fd-4dc9-9dbf-72122f738317\") " pod="openshift-ingress/e2e-shared-b-e0208c-openshift-default-745b9f8964-jphn4" Jul 26 16:16:02.674528 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:16:02.673572 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0002d75b-65fd-4dc9-9dbf-72122f738317-istio-data\") pod \"e2e-shared-b-e0208c-openshift-default-745b9f8964-jphn4\" (UID: \"0002d75b-65fd-4dc9-9dbf-72122f738317\") " pod="openshift-ingress/e2e-shared-b-e0208c-openshift-default-745b9f8964-jphn4" Jul 26 16:16:02.676018 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:16:02.675990 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0002d75b-65fd-4dc9-9dbf-72122f738317-istio-envoy\") pod \"e2e-shared-b-e0208c-openshift-default-745b9f8964-jphn4\" (UID: \"0002d75b-65fd-4dc9-9dbf-72122f738317\") " pod="openshift-ingress/e2e-shared-b-e0208c-openshift-default-745b9f8964-jphn4" Jul 26 16:16:02.676168 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:16:02.676149 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0002d75b-65fd-4dc9-9dbf-72122f738317-istio-podinfo\") pod \"e2e-shared-b-e0208c-openshift-default-745b9f8964-jphn4\" (UID: \"0002d75b-65fd-4dc9-9dbf-72122f738317\") " pod="openshift-ingress/e2e-shared-b-e0208c-openshift-default-745b9f8964-jphn4" Jul 26 16:16:02.681911 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:16:02.681886 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0002d75b-65fd-4dc9-9dbf-72122f738317-istio-token\") pod \"e2e-shared-b-e0208c-openshift-default-745b9f8964-jphn4\" (UID: \"0002d75b-65fd-4dc9-9dbf-72122f738317\") " pod="openshift-ingress/e2e-shared-b-e0208c-openshift-default-745b9f8964-jphn4" Jul 26 16:16:02.686240 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:16:02.686219 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbkph\" (UniqueName: \"kubernetes.io/projected/0002d75b-65fd-4dc9-9dbf-72122f738317-kube-api-access-dbkph\") pod \"e2e-shared-b-e0208c-openshift-default-745b9f8964-jphn4\" (UID: \"0002d75b-65fd-4dc9-9dbf-72122f738317\") " pod="openshift-ingress/e2e-shared-b-e0208c-openshift-default-745b9f8964-jphn4" Jul 26 16:16:02.773198 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:16:02.773165 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-e0208c-openshift-default-745b9f8964-jphn4" Jul 26 16:16:02.855078 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:16:02.855048 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:16:02.920285 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:16:02.920240 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-e0208c-openshift-default-745b9f8964-jphn4"] Jul 26 16:16:02.921204 ip-10-0-138-163 kubenswrapper[2577]: W0726 16:16:02.921180 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0002d75b_65fd_4dc9_9dbf_72122f738317.slice/crio-9a536be2c9511db1ade2bab9f1aeaad73577e60308d8f2aae08bab00495ddbb5 WatchSource:0}: Error finding container 9a536be2c9511db1ade2bab9f1aeaad73577e60308d8f2aae08bab00495ddbb5: Status 404 returned error can't find the container with id 9a536be2c9511db1ade2bab9f1aeaad73577e60308d8f2aae08bab00495ddbb5 Jul 26 16:16:02.923365 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:16:02.923338 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:16:02.923453 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:16:02.923427 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:16:02.923494 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:16:02.923461 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:16:03.541463 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:16:03.541421 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-e0208c-openshift-default-745b9f8964-jphn4" event={"ID":"0002d75b-65fd-4dc9-9dbf-72122f738317","Type":"ContainerStarted","Data":"a72aaf2ca70da1c251f564beb024d52577f15c40f1c161ae09b23064c0972b46"} Jul 26 16:16:03.541686 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:16:03.541493 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-e0208c-openshift-default-745b9f8964-jphn4" event={"ID":"0002d75b-65fd-4dc9-9dbf-72122f738317","Type":"ContainerStarted","Data":"9a536be2c9511db1ade2bab9f1aeaad73577e60308d8f2aae08bab00495ddbb5"} Jul 26 16:16:03.578309 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:16:03.578256 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-e0208c-openshift-default-745b9f8964-jphn4" podStartSLOduration=1.578239926 podStartE2EDuration="1.578239926s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 16:16:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 16:16:03.575698798 +0000 UTC m=+2286.315052138" watchObservedRunningTime="2026-07-26 16:16:03.578239926 +0000 UTC m=+2286.317593235" Jul 26 16:16:03.773543 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:16:03.773498 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-e0208c-openshift-default-745b9f8964-jphn4" Jul 26 16:16:03.778955 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:16:03.778929 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-e0208c-openshift-default-745b9f8964-jphn4" Jul 26 16:16:04.551084 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:16:04.551044 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-e0208c-openshift-default-745b9f8964-jphn4" Jul 26 16:16:04.557218 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:16:04.557183 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-e0208c-openshift-default-745b9f8964-jphn4" Jul 26 16:16:05.453981 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:16:05.453933 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:16:05.564419 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:16:05.563598 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:16:05.654420 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:16:05.653603 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:16:05.753670 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:16:05.753571 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:16:06.143418 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:16:06.142855 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:16:06.236961 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:16:06.236920 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:16:06.816183 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:16:06.816149 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:16:09.947035 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:16:09.947005 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:16:10.595570 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:16:10.595537 2577 generic.go:356] "Generic (PLEG): container finished" podID="b250a529-3b42-40ed-8e16-5eaa7d780d9f" containerID="74ede80574287b9f2db9f48f4ed3ca61875d5964f37e2e116abf70e03466df72" exitCode=0 Jul 26 16:16:10.595766 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:16:10.595592 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-aadfb2-openshift-default-76db5f76c6-kc7m7" event={"ID":"b250a529-3b42-40ed-8e16-5eaa7d780d9f","Type":"ContainerDied","Data":"74ede80574287b9f2db9f48f4ed3ca61875d5964f37e2e116abf70e03466df72"} Jul 26 16:16:10.596032 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:16:10.596017 2577 scope.go:117] "RemoveContainer" containerID="74ede80574287b9f2db9f48f4ed3ca61875d5964f37e2e116abf70e03466df72" Jul 26 16:16:10.597074 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:16:10.597046 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:16:10.597140 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:16:10.597116 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:16:10.597176 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:16:10.597147 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:16:11.495943 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:16:11.495907 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-aadfb2-openshift-default-76db5f76c6-kc7m7" Jul 26 16:16:11.603252 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:16:11.603221 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-aadfb2-openshift-default-76db5f76c6-kc7m7" event={"ID":"b250a529-3b42-40ed-8e16-5eaa7d780d9f","Type":"ContainerStarted","Data":"da495a676ce303990520016cb83cc5af146fc564feca88d6db7f0f27c01c3b12"} Jul 26 16:16:12.567582 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:16:12.567528 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-aadfb2-openshift-default-76db5f76c6-kc7m7" Jul 26 16:16:12.568097 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:16:12.567693 2577 patch_prober.go:28] interesting pod/e2e-shared-a-aadfb2-openshift-default-76db5f76c6-kc7m7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Jul 26 16:16:12.568097 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:16:12.567758 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-aadfb2-openshift-default-76db5f76c6-kc7m7" podUID="b250a529-3b42-40ed-8e16-5eaa7d780d9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Jul 26 16:16:13.566024 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:16:13.565970 2577 patch_prober.go:28] interesting pod/e2e-shared-a-aadfb2-openshift-default-76db5f76c6-kc7m7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Jul 26 16:16:13.566365 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:16:13.566050 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-aadfb2-openshift-default-76db5f76c6-kc7m7" podUID="b250a529-3b42-40ed-8e16-5eaa7d780d9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Jul 26 16:16:14.566248 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:16:14.566205 2577 patch_prober.go:28] interesting pod/e2e-shared-a-aadfb2-openshift-default-76db5f76c6-kc7m7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Jul 26 16:16:14.566910 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:16:14.566873 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-aadfb2-openshift-default-76db5f76c6-kc7m7" podUID="b250a529-3b42-40ed-8e16-5eaa7d780d9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Jul 26 16:16:15.566035 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:16:15.566002 2577 patch_prober.go:28] interesting pod/e2e-shared-a-aadfb2-openshift-default-76db5f76c6-kc7m7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Jul 26 16:16:15.566219 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:16:15.566061 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-aadfb2-openshift-default-76db5f76c6-kc7m7" podUID="b250a529-3b42-40ed-8e16-5eaa7d780d9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Jul 26 16:16:16.569793 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:16:16.569760 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-aadfb2-openshift-default-76db5f76c6-kc7m7" Jul 26 16:16:16.570179 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:16:16.570026 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-aadfb2-openshift-default-76db5f76c6-kc7m7" Jul 26 16:16:16.570795 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:16:16.570775 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-aadfb2-openshift-default-76db5f76c6-kc7m7" Jul 26 16:16:18.646283 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:16:18.646248 2577 generic.go:356] "Generic (PLEG): container finished" podID="0002d75b-65fd-4dc9-9dbf-72122f738317" containerID="a72aaf2ca70da1c251f564beb024d52577f15c40f1c161ae09b23064c0972b46" exitCode=0 Jul 26 16:16:18.646283 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:16:18.646287 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-e0208c-openshift-default-745b9f8964-jphn4" event={"ID":"0002d75b-65fd-4dc9-9dbf-72122f738317","Type":"ContainerDied","Data":"a72aaf2ca70da1c251f564beb024d52577f15c40f1c161ae09b23064c0972b46"} Jul 26 16:16:18.646771 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:16:18.646679 2577 scope.go:117] "RemoveContainer" containerID="a72aaf2ca70da1c251f564beb024d52577f15c40f1c161ae09b23064c0972b46" Jul 26 16:16:18.647657 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:16:18.647628 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:16:18.647767 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:16:18.647711 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:16:18.647767 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:16:18.647760 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:16:19.551721 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:16:19.551659 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-e0208c-openshift-default-745b9f8964-jphn4" Jul 26 16:16:19.659113 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:16:19.659075 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-e0208c-openshift-default-745b9f8964-jphn4" event={"ID":"0002d75b-65fd-4dc9-9dbf-72122f738317","Type":"ContainerStarted","Data":"6b8986f9838ccae6f74a4dc2e4c4b32a01a01b0173588903d1cdeff7d4b45e4f"} Jul 26 16:16:19.774147 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:16:19.774113 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-e0208c-openshift-default-745b9f8964-jphn4" Jul 26 16:16:19.775226 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:16:19.775201 2577 patch_prober.go:28] interesting pod/e2e-shared-b-e0208c-openshift-default-745b9f8964-jphn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Jul 26 16:16:19.775325 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:16:19.775256 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-e0208c-openshift-default-745b9f8964-jphn4" podUID="0002d75b-65fd-4dc9-9dbf-72122f738317" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Jul 26 16:16:20.774247 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:16:20.774208 2577 patch_prober.go:28] interesting pod/e2e-shared-b-e0208c-openshift-default-745b9f8964-jphn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Jul 26 16:16:20.774771 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:16:20.774284 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-e0208c-openshift-default-745b9f8964-jphn4" podUID="0002d75b-65fd-4dc9-9dbf-72122f738317" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Jul 26 16:16:21.773850 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:16:21.773817 2577 patch_prober.go:28] interesting pod/e2e-shared-b-e0208c-openshift-default-745b9f8964-jphn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Jul 26 16:16:21.774038 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:16:21.773872 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-e0208c-openshift-default-745b9f8964-jphn4" podUID="0002d75b-65fd-4dc9-9dbf-72122f738317" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Jul 26 16:16:22.774412 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:16:22.774361 2577 patch_prober.go:28] interesting pod/e2e-shared-b-e0208c-openshift-default-745b9f8964-jphn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Jul 26 16:16:22.774825 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:16:22.774444 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-e0208c-openshift-default-745b9f8964-jphn4" podUID="0002d75b-65fd-4dc9-9dbf-72122f738317" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Jul 26 16:16:23.530465 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:16:23.530419 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:16:23.777323 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:16:23.777288 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-e0208c-openshift-default-745b9f8964-jphn4" Jul 26 16:16:23.777761 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:16:23.777654 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-e0208c-openshift-default-745b9f8964-jphn4" Jul 26 16:16:23.778443 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:16:23.778424 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-e0208c-openshift-default-745b9f8964-jphn4" Jul 26 16:16:40.398611 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:16:40.398579 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:16:42.331772 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:16:42.331736 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:17:25.130020 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:17:25.129988 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:17:57.960109 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:17:57.960077 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-8wjqp_b6b8080d-0cee-4d0c-a05c-3e2f1b672a54/console-operator/1.log" Jul 26 16:17:57.962546 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:17:57.962524 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-8wjqp_b6b8080d-0cee-4d0c-a05c-3e2f1b672a54/console-operator/1.log" Jul 26 16:17:57.964651 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:17:57.964629 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ztlp6_cbe2cd41-6ebc-4722-a813-57d83fa1d242/ovn-acl-logging/0.log" Jul 26 16:17:57.966884 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:17:57.966864 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ztlp6_cbe2cd41-6ebc-4722-a813-57d83fa1d242/ovn-acl-logging/0.log" Jul 26 16:19:41.934261 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:19:41.934217 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:20:43.754254 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:20:43.754219 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:21:15.104123 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:21:15.104085 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:21:15.904995 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:21:15.904960 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:21:27.300159 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:21:27.300116 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:21:27.882744 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:21:27.882695 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:21:50.972091 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:21:50.972052 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:22:50.395893 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:22:50.395792 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:22:51.286224 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:22:51.286187 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:22:56.937521 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:22:56.937483 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:22:57.999111 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:22:57.999081 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-8wjqp_b6b8080d-0cee-4d0c-a05c-3e2f1b672a54/console-operator/1.log" Jul 26 16:22:58.001753 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:22:58.001728 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-8wjqp_b6b8080d-0cee-4d0c-a05c-3e2f1b672a54/console-operator/1.log" Jul 26 16:22:58.003240 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:22:58.003220 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ztlp6_cbe2cd41-6ebc-4722-a813-57d83fa1d242/ovn-acl-logging/0.log" Jul 26 16:22:58.005602 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:22:58.005583 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ztlp6_cbe2cd41-6ebc-4722-a813-57d83fa1d242/ovn-acl-logging/0.log" Jul 26 16:23:13.722892 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:23:13.722853 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:23:31.424152 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:23:31.424116 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:24:37.345013 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:37.344974 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:24:38.254866 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:38.254830 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:24:39.124894 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:39.124857 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7633-140497990065200"] Jul 26 16:24:39.127197 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:39.127178 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7633-140497990065200" Jul 26 16:24:39.129754 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:39.129732 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-p7n2s\"" Jul 26 16:24:39.135389 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:39.135058 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7633-140497990065200"] Jul 26 16:24:39.174759 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:39.174725 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62pvc\" (UniqueName: \"kubernetes.io/projected/1d466f3a-be25-4b18-b393-a6871ab892c7-kube-api-access-62pvc\") pod \"test-curl-7633-140497990065200\" (UID: \"1d466f3a-be25-4b18-b393-a6871ab892c7\") " pod="opendatahub/test-curl-7633-140497990065200" Jul 26 16:24:39.276236 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:39.276199 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62pvc\" (UniqueName: \"kubernetes.io/projected/1d466f3a-be25-4b18-b393-a6871ab892c7-kube-api-access-62pvc\") pod \"test-curl-7633-140497990065200\" (UID: \"1d466f3a-be25-4b18-b393-a6871ab892c7\") " pod="opendatahub/test-curl-7633-140497990065200" Jul 26 16:24:39.284856 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:39.284828 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-62pvc\" (UniqueName: \"kubernetes.io/projected/1d466f3a-be25-4b18-b393-a6871ab892c7-kube-api-access-62pvc\") pod \"test-curl-7633-140497990065200\" (UID: \"1d466f3a-be25-4b18-b393-a6871ab892c7\") " pod="opendatahub/test-curl-7633-140497990065200" Jul 26 16:24:39.438086 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:39.437992 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7633-140497990065200" Jul 26 16:24:39.572435 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:39.572407 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7633-140497990065200"] Jul 26 16:24:39.575170 ip-10-0-138-163 kubenswrapper[2577]: W0726 16:24:39.575136 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1d466f3a_be25_4b18_b393_a6871ab892c7.slice/crio-b351f80e2e921ef55bc43506c1129225d8646e5b42582ed3e2e3472c2a811b7d WatchSource:0}: Error finding container b351f80e2e921ef55bc43506c1129225d8646e5b42582ed3e2e3472c2a811b7d: Status 404 returned error can't find the container with id b351f80e2e921ef55bc43506c1129225d8646e5b42582ed3e2e3472c2a811b7d Jul 26 16:24:39.576502 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:39.576482 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 26 16:24:40.497365 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:40.497273 2577 generic.go:356] "Generic (PLEG): container finished" podID="1d466f3a-be25-4b18-b393-a6871ab892c7" containerID="7706f3afbe42b03c4be6c9db0a3f02737dcbed399a3970bbd9a0e98e402668d2" exitCode=0 Jul 26 16:24:40.497365 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:40.497324 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7633-140497990065200" event={"ID":"1d466f3a-be25-4b18-b393-a6871ab892c7","Type":"ContainerDied","Data":"7706f3afbe42b03c4be6c9db0a3f02737dcbed399a3970bbd9a0e98e402668d2"} Jul 26 16:24:40.497365 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:40.497363 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7633-140497990065200" event={"ID":"1d466f3a-be25-4b18-b393-a6871ab892c7","Type":"ContainerStarted","Data":"b351f80e2e921ef55bc43506c1129225d8646e5b42582ed3e2e3472c2a811b7d"} Jul 26 16:24:41.641755 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:41.641730 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7633-140497990065200" Jul 26 16:24:41.660365 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:41.660336 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7633-140497990065200_1d466f3a-be25-4b18-b393-a6871ab892c7/test-curl-7633-140497990065200/0.log" Jul 26 16:24:41.683123 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:41.683086 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7633-140497990065200"] Jul 26 16:24:41.685324 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:41.685301 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7633-140497990065200"] Jul 26 16:24:41.700126 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:41.700093 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-62pvc\" (UniqueName: \"kubernetes.io/projected/1d466f3a-be25-4b18-b393-a6871ab892c7-kube-api-access-62pvc\") pod \"1d466f3a-be25-4b18-b393-a6871ab892c7\" (UID: \"1d466f3a-be25-4b18-b393-a6871ab892c7\") " Jul 26 16:24:41.702408 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:41.702373 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d466f3a-be25-4b18-b393-a6871ab892c7-kube-api-access-62pvc" (OuterVolumeSpecName: "kube-api-access-62pvc") pod "1d466f3a-be25-4b18-b393-a6871ab892c7" (UID: "1d466f3a-be25-4b18-b393-a6871ab892c7"). InnerVolumeSpecName "kube-api-access-62pvc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 16:24:41.787492 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:41.787447 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1d466f3a-be25-4b18-b393-a6871ab892c7" path="/var/lib/kubelet/pods/1d466f3a-be25-4b18-b393-a6871ab892c7/volumes" Jul 26 16:24:41.801088 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:41.801063 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62pvc\" (UniqueName: \"kubernetes.io/projected/1d466f3a-be25-4b18-b393-a6871ab892c7-kube-api-access-62pvc\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 16:24:41.805531 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:41.805504 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7633-140497990062640"] Jul 26 16:24:41.805915 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:41.805902 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d466f3a-be25-4b18-b393-a6871ab892c7" containerName="test-curl-7633-140497990065200" Jul 26 16:24:41.805970 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:41.805916 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d466f3a-be25-4b18-b393-a6871ab892c7" containerName="test-curl-7633-140497990065200" Jul 26 16:24:41.806008 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:41.805996 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d466f3a-be25-4b18-b393-a6871ab892c7" containerName="test-curl-7633-140497990065200" Jul 26 16:24:41.808122 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:41.808107 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7633-140497990062640" Jul 26 16:24:41.814459 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:41.814434 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7633-140497990062640"] Jul 26 16:24:41.901841 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:41.901807 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4pvp\" (UniqueName: \"kubernetes.io/projected/f043a7ff-90b7-4d82-b4e9-c6ba8c064d93-kube-api-access-g4pvp\") pod \"test-curl-7633-140497990062640\" (UID: \"f043a7ff-90b7-4d82-b4e9-c6ba8c064d93\") " pod="opendatahub/test-curl-7633-140497990062640" Jul 26 16:24:42.002767 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:42.002729 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4pvp\" (UniqueName: \"kubernetes.io/projected/f043a7ff-90b7-4d82-b4e9-c6ba8c064d93-kube-api-access-g4pvp\") pod \"test-curl-7633-140497990062640\" (UID: \"f043a7ff-90b7-4d82-b4e9-c6ba8c064d93\") " pod="opendatahub/test-curl-7633-140497990062640" Jul 26 16:24:42.010845 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:42.010819 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4pvp\" (UniqueName: \"kubernetes.io/projected/f043a7ff-90b7-4d82-b4e9-c6ba8c064d93-kube-api-access-g4pvp\") pod \"test-curl-7633-140497990062640\" (UID: \"f043a7ff-90b7-4d82-b4e9-c6ba8c064d93\") " pod="opendatahub/test-curl-7633-140497990062640" Jul 26 16:24:42.118933 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:42.118826 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7633-140497990062640" Jul 26 16:24:42.258042 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:42.258014 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7633-140497990062640"] Jul 26 16:24:42.259974 ip-10-0-138-163 kubenswrapper[2577]: W0726 16:24:42.259947 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf043a7ff_90b7_4d82_b4e9_c6ba8c064d93.slice/crio-7112d849abc42774825c1f9eb296e5d67e3bcec8ed6091ad0ade71676673f1ce WatchSource:0}: Error finding container 7112d849abc42774825c1f9eb296e5d67e3bcec8ed6091ad0ade71676673f1ce: Status 404 returned error can't find the container with id 7112d849abc42774825c1f9eb296e5d67e3bcec8ed6091ad0ade71676673f1ce Jul 26 16:24:42.510096 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:42.510010 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7633-140497990062640" event={"ID":"f043a7ff-90b7-4d82-b4e9-c6ba8c064d93","Type":"ContainerStarted","Data":"7112d849abc42774825c1f9eb296e5d67e3bcec8ed6091ad0ade71676673f1ce"} Jul 26 16:24:42.511257 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:42.511234 2577 scope.go:117] "RemoveContainer" containerID="7706f3afbe42b03c4be6c9db0a3f02737dcbed399a3970bbd9a0e98e402668d2" Jul 26 16:24:42.511411 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:42.511235 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7633-140497990065200" Jul 26 16:24:43.517952 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:43.517920 2577 generic.go:356] "Generic (PLEG): container finished" podID="f043a7ff-90b7-4d82-b4e9-c6ba8c064d93" containerID="523883b630290d86b70e95e10f6582dca21ec7e81b73ba09985b4a7cd01a42b5" exitCode=0 Jul 26 16:24:43.518522 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:43.518007 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7633-140497990062640" event={"ID":"f043a7ff-90b7-4d82-b4e9-c6ba8c064d93","Type":"ContainerDied","Data":"523883b630290d86b70e95e10f6582dca21ec7e81b73ba09985b4a7cd01a42b5"} Jul 26 16:24:44.662088 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:44.662055 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7633-140497990062640" Jul 26 16:24:44.683571 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:44.683544 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7633-140497990062640_f043a7ff-90b7-4d82-b4e9-c6ba8c064d93/test-curl-7633-140497990062640/0.log" Jul 26 16:24:44.707505 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:44.707425 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7633-140497990062640"] Jul 26 16:24:44.709824 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:44.709772 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7633-140497990062640"] Jul 26 16:24:44.728283 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:44.728262 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g4pvp\" (UniqueName: \"kubernetes.io/projected/f043a7ff-90b7-4d82-b4e9-c6ba8c064d93-kube-api-access-g4pvp\") pod \"f043a7ff-90b7-4d82-b4e9-c6ba8c064d93\" (UID: \"f043a7ff-90b7-4d82-b4e9-c6ba8c064d93\") " Jul 26 16:24:44.730362 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:44.730338 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f043a7ff-90b7-4d82-b4e9-c6ba8c064d93-kube-api-access-g4pvp" (OuterVolumeSpecName: "kube-api-access-g4pvp") pod "f043a7ff-90b7-4d82-b4e9-c6ba8c064d93" (UID: "f043a7ff-90b7-4d82-b4e9-c6ba8c064d93"). InnerVolumeSpecName "kube-api-access-g4pvp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 16:24:44.829301 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:44.829207 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4pvp\" (UniqueName: \"kubernetes.io/projected/f043a7ff-90b7-4d82-b4e9-c6ba8c064d93-kube-api-access-g4pvp\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 16:24:44.863090 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:44.863053 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7633-140497990784896"] Jul 26 16:24:44.863471 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:44.863457 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f043a7ff-90b7-4d82-b4e9-c6ba8c064d93" containerName="test-curl-7633-140497990062640" Jul 26 16:24:44.863522 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:44.863472 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f043a7ff-90b7-4d82-b4e9-c6ba8c064d93" containerName="test-curl-7633-140497990062640" Jul 26 16:24:44.863558 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:44.863547 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="f043a7ff-90b7-4d82-b4e9-c6ba8c064d93" containerName="test-curl-7633-140497990062640" Jul 26 16:24:44.865644 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:44.865628 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7633-140497990784896" Jul 26 16:24:44.869989 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:44.869964 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7633-140497990784896"] Jul 26 16:24:44.930390 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:44.930355 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7kt2\" (UniqueName: \"kubernetes.io/projected/02b16781-25a0-4115-9a8c-86928441c3dc-kube-api-access-l7kt2\") pod \"test-curl-7633-140497990784896\" (UID: \"02b16781-25a0-4115-9a8c-86928441c3dc\") " pod="opendatahub/test-curl-7633-140497990784896" Jul 26 16:24:45.031152 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:45.031120 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7kt2\" (UniqueName: \"kubernetes.io/projected/02b16781-25a0-4115-9a8c-86928441c3dc-kube-api-access-l7kt2\") pod \"test-curl-7633-140497990784896\" (UID: \"02b16781-25a0-4115-9a8c-86928441c3dc\") " pod="opendatahub/test-curl-7633-140497990784896" Jul 26 16:24:45.038720 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:45.038694 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7kt2\" (UniqueName: \"kubernetes.io/projected/02b16781-25a0-4115-9a8c-86928441c3dc-kube-api-access-l7kt2\") pod \"test-curl-7633-140497990784896\" (UID: \"02b16781-25a0-4115-9a8c-86928441c3dc\") " pod="opendatahub/test-curl-7633-140497990784896" Jul 26 16:24:45.176730 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:45.176637 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7633-140497990784896" Jul 26 16:24:45.315208 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:45.315182 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7633-140497990784896"] Jul 26 16:24:45.317836 ip-10-0-138-163 kubenswrapper[2577]: W0726 16:24:45.317809 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod02b16781_25a0_4115_9a8c_86928441c3dc.slice/crio-5530aa14b33f23e4b82a757c59395918b94e769039e25d526c960eb0a8212ddb WatchSource:0}: Error finding container 5530aa14b33f23e4b82a757c59395918b94e769039e25d526c960eb0a8212ddb: Status 404 returned error can't find the container with id 5530aa14b33f23e4b82a757c59395918b94e769039e25d526c960eb0a8212ddb Jul 26 16:24:45.531748 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:45.531706 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7633-140497990784896" event={"ID":"02b16781-25a0-4115-9a8c-86928441c3dc","Type":"ContainerStarted","Data":"5530aa14b33f23e4b82a757c59395918b94e769039e25d526c960eb0a8212ddb"} Jul 26 16:24:45.533083 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:45.533060 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7633-140497990062640" Jul 26 16:24:45.533083 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:45.533076 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7112d849abc42774825c1f9eb296e5d67e3bcec8ed6091ad0ade71676673f1ce" Jul 26 16:24:45.544054 ip-10-0-138-163 kubenswrapper[2577]: E0726 16:24:45.544027 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7633-140497990062640\" is forbidden: User \"system:node:ip-10-0-138-163.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-163.ec2.internal' and this object" podUID="f043a7ff-90b7-4d82-b4e9-c6ba8c064d93" pod="opendatahub/test-curl-7633-140497990062640" Jul 26 16:24:45.787728 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:45.787649 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f043a7ff-90b7-4d82-b4e9-c6ba8c064d93" path="/var/lib/kubelet/pods/f043a7ff-90b7-4d82-b4e9-c6ba8c064d93/volumes" Jul 26 16:24:46.540567 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:46.540526 2577 generic.go:356] "Generic (PLEG): container finished" podID="02b16781-25a0-4115-9a8c-86928441c3dc" containerID="587c6902d309718fea386c3522116b16ce20fa8172b1ff7a474a14855a61352a" exitCode=0 Jul 26 16:24:46.540567 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:46.540568 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7633-140497990784896" event={"ID":"02b16781-25a0-4115-9a8c-86928441c3dc","Type":"ContainerDied","Data":"587c6902d309718fea386c3522116b16ce20fa8172b1ff7a474a14855a61352a"} Jul 26 16:24:47.706596 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:47.706570 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7633-140497990784896" Jul 26 16:24:47.725595 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:47.725551 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7633-140497990784896_02b16781-25a0-4115-9a8c-86928441c3dc/test-curl-7633-140497990784896/0.log" Jul 26 16:24:47.751598 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:47.751563 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7633-140497990784896"] Jul 26 16:24:47.762421 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:47.761799 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7633-140497990784896"] Jul 26 16:24:47.762421 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:47.762293 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l7kt2\" (UniqueName: \"kubernetes.io/projected/02b16781-25a0-4115-9a8c-86928441c3dc-kube-api-access-l7kt2\") pod \"02b16781-25a0-4115-9a8c-86928441c3dc\" (UID: \"02b16781-25a0-4115-9a8c-86928441c3dc\") " Jul 26 16:24:47.765048 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:47.765019 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02b16781-25a0-4115-9a8c-86928441c3dc-kube-api-access-l7kt2" (OuterVolumeSpecName: "kube-api-access-l7kt2") pod "02b16781-25a0-4115-9a8c-86928441c3dc" (UID: "02b16781-25a0-4115-9a8c-86928441c3dc"). InnerVolumeSpecName "kube-api-access-l7kt2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 16:24:47.787469 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:47.787437 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="02b16781-25a0-4115-9a8c-86928441c3dc" path="/var/lib/kubelet/pods/02b16781-25a0-4115-9a8c-86928441c3dc/volumes" Jul 26 16:24:47.863727 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:47.863639 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7kt2\" (UniqueName: \"kubernetes.io/projected/02b16781-25a0-4115-9a8c-86928441c3dc-kube-api-access-l7kt2\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 16:24:47.871932 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:47.871904 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7633-140497990784896"] Jul 26 16:24:47.872296 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:47.872283 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02b16781-25a0-4115-9a8c-86928441c3dc" containerName="test-curl-7633-140497990784896" Jul 26 16:24:47.872339 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:47.872297 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="02b16781-25a0-4115-9a8c-86928441c3dc" containerName="test-curl-7633-140497990784896" Jul 26 16:24:47.872374 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:47.872367 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="02b16781-25a0-4115-9a8c-86928441c3dc" containerName="test-curl-7633-140497990784896" Jul 26 16:24:47.874675 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:47.874660 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7633-140497990784896" Jul 26 16:24:47.882210 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:47.882183 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7633-140497990784896"] Jul 26 16:24:47.964753 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:47.964713 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vntgl\" (UniqueName: \"kubernetes.io/projected/9700fed0-b211-4a75-b399-9669cb8183b7-kube-api-access-vntgl\") pod \"test-curl-7633-140497990784896\" (UID: \"9700fed0-b211-4a75-b399-9669cb8183b7\") " pod="opendatahub/test-curl-7633-140497990784896" Jul 26 16:24:48.065911 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:48.065875 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vntgl\" (UniqueName: \"kubernetes.io/projected/9700fed0-b211-4a75-b399-9669cb8183b7-kube-api-access-vntgl\") pod \"test-curl-7633-140497990784896\" (UID: \"9700fed0-b211-4a75-b399-9669cb8183b7\") " pod="opendatahub/test-curl-7633-140497990784896" Jul 26 16:24:48.074149 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:48.074122 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vntgl\" (UniqueName: \"kubernetes.io/projected/9700fed0-b211-4a75-b399-9669cb8183b7-kube-api-access-vntgl\") pod \"test-curl-7633-140497990784896\" (UID: \"9700fed0-b211-4a75-b399-9669cb8183b7\") " pod="opendatahub/test-curl-7633-140497990784896" Jul 26 16:24:48.185339 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:48.185249 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7633-140497990784896" Jul 26 16:24:48.337710 ip-10-0-138-163 kubenswrapper[2577]: W0726 16:24:48.337679 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9700fed0_b211_4a75_b399_9669cb8183b7.slice/crio-471852595fb7f8e62cf9c9c6fcb2366452709e9c91c26c82d04b5db418388c7d WatchSource:0}: Error finding container 471852595fb7f8e62cf9c9c6fcb2366452709e9c91c26c82d04b5db418388c7d: Status 404 returned error can't find the container with id 471852595fb7f8e62cf9c9c6fcb2366452709e9c91c26c82d04b5db418388c7d Jul 26 16:24:48.337858 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:48.337795 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7633-140497990784896"] Jul 26 16:24:48.552648 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:48.552613 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7633-140497990784896" event={"ID":"9700fed0-b211-4a75-b399-9669cb8183b7","Type":"ContainerStarted","Data":"471852595fb7f8e62cf9c9c6fcb2366452709e9c91c26c82d04b5db418388c7d"} Jul 26 16:24:48.554229 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:48.554204 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7633-140497990784896" Jul 26 16:24:48.554229 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:48.554218 2577 scope.go:117] "RemoveContainer" containerID="587c6902d309718fea386c3522116b16ce20fa8172b1ff7a474a14855a61352a" Jul 26 16:24:49.562093 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:49.562059 2577 generic.go:356] "Generic (PLEG): container finished" podID="9700fed0-b211-4a75-b399-9669cb8183b7" containerID="0880b8f2a6c90acc0a4231a15473370fad21b6f92f270bb9eb5b55d114c0eaed" exitCode=0 Jul 26 16:24:49.562607 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:49.562155 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7633-140497990784896" event={"ID":"9700fed0-b211-4a75-b399-9669cb8183b7","Type":"ContainerDied","Data":"0880b8f2a6c90acc0a4231a15473370fad21b6f92f270bb9eb5b55d114c0eaed"} Jul 26 16:24:50.706210 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:50.706184 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7633-140497990784896" Jul 26 16:24:50.728048 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:50.728019 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7633-140497990784896_9700fed0-b211-4a75-b399-9669cb8183b7/test-curl-7633-140497990784896/0.log" Jul 26 16:24:50.751295 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:50.751255 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7633-140497990784896"] Jul 26 16:24:50.755787 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:50.755754 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7633-140497990784896"] Jul 26 16:24:50.793750 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:50.793724 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vntgl\" (UniqueName: \"kubernetes.io/projected/9700fed0-b211-4a75-b399-9669cb8183b7-kube-api-access-vntgl\") pod \"9700fed0-b211-4a75-b399-9669cb8183b7\" (UID: \"9700fed0-b211-4a75-b399-9669cb8183b7\") " Jul 26 16:24:50.795804 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:50.795783 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9700fed0-b211-4a75-b399-9669cb8183b7-kube-api-access-vntgl" (OuterVolumeSpecName: "kube-api-access-vntgl") pod "9700fed0-b211-4a75-b399-9669cb8183b7" (UID: "9700fed0-b211-4a75-b399-9669cb8183b7"). InnerVolumeSpecName "kube-api-access-vntgl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 16:24:50.878544 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:50.878460 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7633-140497989957216"] Jul 26 16:24:50.878867 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:50.878853 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9700fed0-b211-4a75-b399-9669cb8183b7" containerName="test-curl-7633-140497990784896" Jul 26 16:24:50.878921 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:50.878869 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9700fed0-b211-4a75-b399-9669cb8183b7" containerName="test-curl-7633-140497990784896" Jul 26 16:24:50.878958 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:50.878932 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="9700fed0-b211-4a75-b399-9669cb8183b7" containerName="test-curl-7633-140497990784896" Jul 26 16:24:50.880750 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:50.880735 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7633-140497989957216" Jul 26 16:24:50.889303 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:50.889277 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7633-140497989957216"] Jul 26 16:24:50.895159 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:50.895084 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vntgl\" (UniqueName: \"kubernetes.io/projected/9700fed0-b211-4a75-b399-9669cb8183b7-kube-api-access-vntgl\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 16:24:50.996179 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:50.996146 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x688c\" (UniqueName: \"kubernetes.io/projected/8a9b13fb-e353-4e1e-9674-da8132570c20-kube-api-access-x688c\") pod \"test-curl-7633-140497989957216\" (UID: \"8a9b13fb-e353-4e1e-9674-da8132570c20\") " pod="opendatahub/test-curl-7633-140497989957216" Jul 26 16:24:51.097449 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:51.097380 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x688c\" (UniqueName: \"kubernetes.io/projected/8a9b13fb-e353-4e1e-9674-da8132570c20-kube-api-access-x688c\") pod \"test-curl-7633-140497989957216\" (UID: \"8a9b13fb-e353-4e1e-9674-da8132570c20\") " pod="opendatahub/test-curl-7633-140497989957216" Jul 26 16:24:51.105750 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:51.105723 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x688c\" (UniqueName: \"kubernetes.io/projected/8a9b13fb-e353-4e1e-9674-da8132570c20-kube-api-access-x688c\") pod \"test-curl-7633-140497989957216\" (UID: \"8a9b13fb-e353-4e1e-9674-da8132570c20\") " pod="opendatahub/test-curl-7633-140497989957216" Jul 26 16:24:51.191787 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:51.191697 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7633-140497989957216" Jul 26 16:24:51.331934 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:51.331771 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7633-140497989957216"] Jul 26 16:24:51.336276 ip-10-0-138-163 kubenswrapper[2577]: W0726 16:24:51.336241 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8a9b13fb_e353_4e1e_9674_da8132570c20.slice/crio-b6e7d87d8e3e826331a15bbc8b8f25dbeff105c0cac4af403bcfb6d799d2f435 WatchSource:0}: Error finding container b6e7d87d8e3e826331a15bbc8b8f25dbeff105c0cac4af403bcfb6d799d2f435: Status 404 returned error can't find the container with id b6e7d87d8e3e826331a15bbc8b8f25dbeff105c0cac4af403bcfb6d799d2f435 Jul 26 16:24:51.575626 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:51.575588 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7633-140497990784896" Jul 26 16:24:51.575806 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:51.575601 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="471852595fb7f8e62cf9c9c6fcb2366452709e9c91c26c82d04b5db418388c7d" Jul 26 16:24:51.576904 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:51.576880 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7633-140497989957216" event={"ID":"8a9b13fb-e353-4e1e-9674-da8132570c20","Type":"ContainerStarted","Data":"b6e7d87d8e3e826331a15bbc8b8f25dbeff105c0cac4af403bcfb6d799d2f435"} Jul 26 16:24:51.586136 ip-10-0-138-163 kubenswrapper[2577]: E0726 16:24:51.586109 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7633-140497990784896\" is forbidden: User \"system:node:ip-10-0-138-163.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-163.ec2.internal' and this object" podUID="9700fed0-b211-4a75-b399-9669cb8183b7" pod="opendatahub/test-curl-7633-140497990784896" Jul 26 16:24:51.788263 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:51.788224 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9700fed0-b211-4a75-b399-9669cb8183b7" path="/var/lib/kubelet/pods/9700fed0-b211-4a75-b399-9669cb8183b7/volumes" Jul 26 16:24:52.583749 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:52.583717 2577 generic.go:356] "Generic (PLEG): container finished" podID="8a9b13fb-e353-4e1e-9674-da8132570c20" containerID="a791f04c47b213b38454daa21edc97f314592460f7ba386b6499e4e3e84f71ba" exitCode=0 Jul 26 16:24:52.583940 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:52.583796 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7633-140497989957216" event={"ID":"8a9b13fb-e353-4e1e-9674-da8132570c20","Type":"ContainerDied","Data":"a791f04c47b213b38454daa21edc97f314592460f7ba386b6499e4e3e84f71ba"} Jul 26 16:24:53.744706 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:53.744682 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7633-140497989957216" Jul 26 16:24:53.765664 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:53.765626 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7633-140497989957216_8a9b13fb-e353-4e1e-9674-da8132570c20/test-curl-7633-140497989957216/0.log" Jul 26 16:24:53.791121 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:53.791095 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7633-140497989957216"] Jul 26 16:24:53.795116 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:53.795082 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7633-140497989957216"] Jul 26 16:24:53.829713 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:53.829677 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x688c\" (UniqueName: \"kubernetes.io/projected/8a9b13fb-e353-4e1e-9674-da8132570c20-kube-api-access-x688c\") pod \"8a9b13fb-e353-4e1e-9674-da8132570c20\" (UID: \"8a9b13fb-e353-4e1e-9674-da8132570c20\") " Jul 26 16:24:53.832258 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:53.832220 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a9b13fb-e353-4e1e-9674-da8132570c20-kube-api-access-x688c" (OuterVolumeSpecName: "kube-api-access-x688c") pod "8a9b13fb-e353-4e1e-9674-da8132570c20" (UID: "8a9b13fb-e353-4e1e-9674-da8132570c20"). InnerVolumeSpecName "kube-api-access-x688c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 16:24:53.930799 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:53.930763 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x688c\" (UniqueName: \"kubernetes.io/projected/8a9b13fb-e353-4e1e-9674-da8132570c20-kube-api-access-x688c\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 16:24:54.595881 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:54.595856 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7633-140497989957216" Jul 26 16:24:54.596030 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:54.595855 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b6e7d87d8e3e826331a15bbc8b8f25dbeff105c0cac4af403bcfb6d799d2f435" Jul 26 16:24:54.605987 ip-10-0-138-163 kubenswrapper[2577]: E0726 16:24:54.605953 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7633-140497989957216\" is forbidden: User \"system:node:ip-10-0-138-163.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-163.ec2.internal' and this object" podUID="8a9b13fb-e353-4e1e-9674-da8132570c20" pod="opendatahub/test-curl-7633-140497989957216" Jul 26 16:24:55.787747 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:55.787709 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a9b13fb-e353-4e1e-9674-da8132570c20" path="/var/lib/kubelet/pods/8a9b13fb-e353-4e1e-9674-da8132570c20/volumes" Jul 26 16:24:57.428433 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:57.428373 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:24:57.711602 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:57.711521 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-10e721-openshift-default-cbb9784bb-94qgh"] Jul 26 16:24:57.711922 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:57.711909 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a9b13fb-e353-4e1e-9674-da8132570c20" containerName="test-curl-7633-140497989957216" Jul 26 16:24:57.711969 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:57.711924 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a9b13fb-e353-4e1e-9674-da8132570c20" containerName="test-curl-7633-140497989957216" Jul 26 16:24:57.712003 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:57.711990 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a9b13fb-e353-4e1e-9674-da8132570c20" containerName="test-curl-7633-140497989957216" Jul 26 16:24:57.714139 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:57.714123 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-10e721-openshift-default-cbb9784bb-94qgh" Jul 26 16:24:57.720462 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:57.720412 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-10e721-openshift-default-dockercfg-p6p9f\"" Jul 26 16:24:57.727742 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:57.727716 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-10e721-openshift-default-cbb9784bb-94qgh"] Jul 26 16:24:57.741031 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:57.740989 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:24:57.868819 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:57.868781 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc949b5c-2665-4ba8-b28d-06a48f5af3d1-workload-socket\") pod \"e2e-ipp-a-10e721-openshift-default-cbb9784bb-94qgh\" (UID: \"bc949b5c-2665-4ba8-b28d-06a48f5af3d1\") " pod="openshift-ingress/e2e-ipp-a-10e721-openshift-default-cbb9784bb-94qgh" Jul 26 16:24:57.868819 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:57.868821 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc949b5c-2665-4ba8-b28d-06a48f5af3d1-istiod-ca-cert\") pod \"e2e-ipp-a-10e721-openshift-default-cbb9784bb-94qgh\" (UID: \"bc949b5c-2665-4ba8-b28d-06a48f5af3d1\") " pod="openshift-ingress/e2e-ipp-a-10e721-openshift-default-cbb9784bb-94qgh" Jul 26 16:24:57.869035 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:57.868860 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc949b5c-2665-4ba8-b28d-06a48f5af3d1-istio-data\") pod \"e2e-ipp-a-10e721-openshift-default-cbb9784bb-94qgh\" (UID: \"bc949b5c-2665-4ba8-b28d-06a48f5af3d1\") " pod="openshift-ingress/e2e-ipp-a-10e721-openshift-default-cbb9784bb-94qgh" Jul 26 16:24:57.869035 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:57.868876 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc949b5c-2665-4ba8-b28d-06a48f5af3d1-istio-podinfo\") pod \"e2e-ipp-a-10e721-openshift-default-cbb9784bb-94qgh\" (UID: \"bc949b5c-2665-4ba8-b28d-06a48f5af3d1\") " pod="openshift-ingress/e2e-ipp-a-10e721-openshift-default-cbb9784bb-94qgh" Jul 26 16:24:57.869035 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:57.868898 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc949b5c-2665-4ba8-b28d-06a48f5af3d1-istio-envoy\") pod \"e2e-ipp-a-10e721-openshift-default-cbb9784bb-94qgh\" (UID: \"bc949b5c-2665-4ba8-b28d-06a48f5af3d1\") " pod="openshift-ingress/e2e-ipp-a-10e721-openshift-default-cbb9784bb-94qgh" Jul 26 16:24:57.869035 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:57.868918 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc949b5c-2665-4ba8-b28d-06a48f5af3d1-credential-socket\") pod \"e2e-ipp-a-10e721-openshift-default-cbb9784bb-94qgh\" (UID: \"bc949b5c-2665-4ba8-b28d-06a48f5af3d1\") " pod="openshift-ingress/e2e-ipp-a-10e721-openshift-default-cbb9784bb-94qgh" Jul 26 16:24:57.869035 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:57.869000 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc949b5c-2665-4ba8-b28d-06a48f5af3d1-workload-certs\") pod \"e2e-ipp-a-10e721-openshift-default-cbb9784bb-94qgh\" (UID: \"bc949b5c-2665-4ba8-b28d-06a48f5af3d1\") " pod="openshift-ingress/e2e-ipp-a-10e721-openshift-default-cbb9784bb-94qgh" Jul 26 16:24:57.869190 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:57.869054 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc949b5c-2665-4ba8-b28d-06a48f5af3d1-istio-token\") pod \"e2e-ipp-a-10e721-openshift-default-cbb9784bb-94qgh\" (UID: \"bc949b5c-2665-4ba8-b28d-06a48f5af3d1\") " pod="openshift-ingress/e2e-ipp-a-10e721-openshift-default-cbb9784bb-94qgh" Jul 26 16:24:57.869190 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:57.869105 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4c625\" (UniqueName: \"kubernetes.io/projected/bc949b5c-2665-4ba8-b28d-06a48f5af3d1-kube-api-access-4c625\") pod \"e2e-ipp-a-10e721-openshift-default-cbb9784bb-94qgh\" (UID: \"bc949b5c-2665-4ba8-b28d-06a48f5af3d1\") " pod="openshift-ingress/e2e-ipp-a-10e721-openshift-default-cbb9784bb-94qgh" Jul 26 16:24:57.970197 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:57.970106 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc949b5c-2665-4ba8-b28d-06a48f5af3d1-istio-token\") pod \"e2e-ipp-a-10e721-openshift-default-cbb9784bb-94qgh\" (UID: \"bc949b5c-2665-4ba8-b28d-06a48f5af3d1\") " pod="openshift-ingress/e2e-ipp-a-10e721-openshift-default-cbb9784bb-94qgh" Jul 26 16:24:57.970197 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:57.970159 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4c625\" (UniqueName: \"kubernetes.io/projected/bc949b5c-2665-4ba8-b28d-06a48f5af3d1-kube-api-access-4c625\") pod \"e2e-ipp-a-10e721-openshift-default-cbb9784bb-94qgh\" (UID: \"bc949b5c-2665-4ba8-b28d-06a48f5af3d1\") " pod="openshift-ingress/e2e-ipp-a-10e721-openshift-default-cbb9784bb-94qgh" Jul 26 16:24:57.970197 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:57.970180 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc949b5c-2665-4ba8-b28d-06a48f5af3d1-workload-socket\") pod \"e2e-ipp-a-10e721-openshift-default-cbb9784bb-94qgh\" (UID: \"bc949b5c-2665-4ba8-b28d-06a48f5af3d1\") " pod="openshift-ingress/e2e-ipp-a-10e721-openshift-default-cbb9784bb-94qgh" Jul 26 16:24:57.970497 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:57.970204 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc949b5c-2665-4ba8-b28d-06a48f5af3d1-istiod-ca-cert\") pod \"e2e-ipp-a-10e721-openshift-default-cbb9784bb-94qgh\" (UID: \"bc949b5c-2665-4ba8-b28d-06a48f5af3d1\") " pod="openshift-ingress/e2e-ipp-a-10e721-openshift-default-cbb9784bb-94qgh" Jul 26 16:24:57.970497 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:57.970243 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc949b5c-2665-4ba8-b28d-06a48f5af3d1-istio-data\") pod \"e2e-ipp-a-10e721-openshift-default-cbb9784bb-94qgh\" (UID: \"bc949b5c-2665-4ba8-b28d-06a48f5af3d1\") " pod="openshift-ingress/e2e-ipp-a-10e721-openshift-default-cbb9784bb-94qgh" Jul 26 16:24:57.970497 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:57.970266 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc949b5c-2665-4ba8-b28d-06a48f5af3d1-istio-podinfo\") pod \"e2e-ipp-a-10e721-openshift-default-cbb9784bb-94qgh\" (UID: \"bc949b5c-2665-4ba8-b28d-06a48f5af3d1\") " pod="openshift-ingress/e2e-ipp-a-10e721-openshift-default-cbb9784bb-94qgh" Jul 26 16:24:57.970497 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:57.970296 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc949b5c-2665-4ba8-b28d-06a48f5af3d1-istio-envoy\") pod \"e2e-ipp-a-10e721-openshift-default-cbb9784bb-94qgh\" (UID: \"bc949b5c-2665-4ba8-b28d-06a48f5af3d1\") " pod="openshift-ingress/e2e-ipp-a-10e721-openshift-default-cbb9784bb-94qgh" Jul 26 16:24:57.970497 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:57.970326 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc949b5c-2665-4ba8-b28d-06a48f5af3d1-credential-socket\") pod \"e2e-ipp-a-10e721-openshift-default-cbb9784bb-94qgh\" (UID: \"bc949b5c-2665-4ba8-b28d-06a48f5af3d1\") " pod="openshift-ingress/e2e-ipp-a-10e721-openshift-default-cbb9784bb-94qgh" Jul 26 16:24:57.970497 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:57.970461 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc949b5c-2665-4ba8-b28d-06a48f5af3d1-workload-certs\") pod \"e2e-ipp-a-10e721-openshift-default-cbb9784bb-94qgh\" (UID: \"bc949b5c-2665-4ba8-b28d-06a48f5af3d1\") " pod="openshift-ingress/e2e-ipp-a-10e721-openshift-default-cbb9784bb-94qgh" Jul 26 16:24:57.970794 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:57.970769 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc949b5c-2665-4ba8-b28d-06a48f5af3d1-istio-data\") pod \"e2e-ipp-a-10e721-openshift-default-cbb9784bb-94qgh\" (UID: \"bc949b5c-2665-4ba8-b28d-06a48f5af3d1\") " pod="openshift-ingress/e2e-ipp-a-10e721-openshift-default-cbb9784bb-94qgh" Jul 26 16:24:57.970878 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:57.970852 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc949b5c-2665-4ba8-b28d-06a48f5af3d1-workload-socket\") pod \"e2e-ipp-a-10e721-openshift-default-cbb9784bb-94qgh\" (UID: \"bc949b5c-2665-4ba8-b28d-06a48f5af3d1\") " pod="openshift-ingress/e2e-ipp-a-10e721-openshift-default-cbb9784bb-94qgh" Jul 26 16:24:57.970927 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:57.970900 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc949b5c-2665-4ba8-b28d-06a48f5af3d1-credential-socket\") pod \"e2e-ipp-a-10e721-openshift-default-cbb9784bb-94qgh\" (UID: \"bc949b5c-2665-4ba8-b28d-06a48f5af3d1\") " pod="openshift-ingress/e2e-ipp-a-10e721-openshift-default-cbb9784bb-94qgh" Jul 26 16:24:57.970960 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:57.970925 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc949b5c-2665-4ba8-b28d-06a48f5af3d1-workload-certs\") pod \"e2e-ipp-a-10e721-openshift-default-cbb9784bb-94qgh\" (UID: \"bc949b5c-2665-4ba8-b28d-06a48f5af3d1\") " pod="openshift-ingress/e2e-ipp-a-10e721-openshift-default-cbb9784bb-94qgh" Jul 26 16:24:57.971091 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:57.971075 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc949b5c-2665-4ba8-b28d-06a48f5af3d1-istiod-ca-cert\") pod \"e2e-ipp-a-10e721-openshift-default-cbb9784bb-94qgh\" (UID: \"bc949b5c-2665-4ba8-b28d-06a48f5af3d1\") " pod="openshift-ingress/e2e-ipp-a-10e721-openshift-default-cbb9784bb-94qgh" Jul 26 16:24:57.972530 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:57.972511 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc949b5c-2665-4ba8-b28d-06a48f5af3d1-istio-envoy\") pod \"e2e-ipp-a-10e721-openshift-default-cbb9784bb-94qgh\" (UID: \"bc949b5c-2665-4ba8-b28d-06a48f5af3d1\") " pod="openshift-ingress/e2e-ipp-a-10e721-openshift-default-cbb9784bb-94qgh" Jul 26 16:24:57.972761 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:57.972745 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc949b5c-2665-4ba8-b28d-06a48f5af3d1-istio-podinfo\") pod \"e2e-ipp-a-10e721-openshift-default-cbb9784bb-94qgh\" (UID: \"bc949b5c-2665-4ba8-b28d-06a48f5af3d1\") " pod="openshift-ingress/e2e-ipp-a-10e721-openshift-default-cbb9784bb-94qgh" Jul 26 16:24:57.980167 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:57.980136 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc949b5c-2665-4ba8-b28d-06a48f5af3d1-istio-token\") pod \"e2e-ipp-a-10e721-openshift-default-cbb9784bb-94qgh\" (UID: \"bc949b5c-2665-4ba8-b28d-06a48f5af3d1\") " pod="openshift-ingress/e2e-ipp-a-10e721-openshift-default-cbb9784bb-94qgh" Jul 26 16:24:57.980279 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:57.980170 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4c625\" (UniqueName: \"kubernetes.io/projected/bc949b5c-2665-4ba8-b28d-06a48f5af3d1-kube-api-access-4c625\") pod \"e2e-ipp-a-10e721-openshift-default-cbb9784bb-94qgh\" (UID: \"bc949b5c-2665-4ba8-b28d-06a48f5af3d1\") " pod="openshift-ingress/e2e-ipp-a-10e721-openshift-default-cbb9784bb-94qgh" Jul 26 16:24:58.024145 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:58.024097 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-10e721-openshift-default-cbb9784bb-94qgh" Jul 26 16:24:58.168533 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:58.168508 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-10e721-openshift-default-cbb9784bb-94qgh"] Jul 26 16:24:58.171004 ip-10-0-138-163 kubenswrapper[2577]: W0726 16:24:58.170976 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc949b5c_2665_4ba8_b28d_06a48f5af3d1.slice/crio-671b656734a6106fcd2ae5f06145aee6a3545fcdf4d0fa373984b0c9a4c584bb WatchSource:0}: Error finding container 671b656734a6106fcd2ae5f06145aee6a3545fcdf4d0fa373984b0c9a4c584bb: Status 404 returned error can't find the container with id 671b656734a6106fcd2ae5f06145aee6a3545fcdf4d0fa373984b0c9a4c584bb Jul 26 16:24:58.173921 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:58.173874 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:24:58.174126 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:58.173959 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:24:58.174126 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:58.174003 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:24:58.189645 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:58.189618 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:24:58.621058 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:58.621023 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-10e721-openshift-default-cbb9784bb-94qgh" event={"ID":"bc949b5c-2665-4ba8-b28d-06a48f5af3d1","Type":"ContainerStarted","Data":"48edabcde199814fe86a28d0ae2950426e62f1703e9d0477e3304cf1d7dbfe79"} Jul 26 16:24:58.621469 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:58.621073 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-10e721-openshift-default-cbb9784bb-94qgh" event={"ID":"bc949b5c-2665-4ba8-b28d-06a48f5af3d1","Type":"ContainerStarted","Data":"671b656734a6106fcd2ae5f06145aee6a3545fcdf4d0fa373984b0c9a4c584bb"} Jul 26 16:24:58.645791 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:58.645718 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-10e721-openshift-default-cbb9784bb-94qgh" podStartSLOduration=1.645700258 podStartE2EDuration="1.645700258s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 16:24:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 16:24:58.642669627 +0000 UTC m=+2821.382022963" watchObservedRunningTime="2026-07-26 16:24:58.645700258 +0000 UTC m=+2821.385053568" Jul 26 16:24:59.024932 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:59.024901 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-10e721-openshift-default-cbb9784bb-94qgh" Jul 26 16:24:59.031070 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:59.031042 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-10e721-openshift-default-cbb9784bb-94qgh" Jul 26 16:24:59.626789 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:59.626752 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-10e721-openshift-default-cbb9784bb-94qgh" Jul 26 16:24:59.627833 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:24:59.627813 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-10e721-openshift-default-cbb9784bb-94qgh" Jul 26 16:25:00.653566 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:25:00.653529 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:25:00.746256 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:25:00.746220 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:25:00.846306 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:25:00.846266 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:25:01.193715 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:25:01.193683 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:25:01.407278 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:25:01.407242 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:25:01.510094 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:25:01.509994 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:25:05.022737 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:25:05.022694 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:25:14.628498 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:25:14.628385 2577 patch_prober.go:28] interesting pod/e2e-ipp-a-10e721-openshift-default-cbb9784bb-94qgh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Jul 26 16:25:14.628498 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:25:14.628469 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-10e721-openshift-default-cbb9784bb-94qgh" podUID="bc949b5c-2665-4ba8-b28d-06a48f5af3d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Jul 26 16:25:14.717939 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:25:14.717906 2577 generic.go:356] "Generic (PLEG): container finished" podID="bc949b5c-2665-4ba8-b28d-06a48f5af3d1" containerID="48edabcde199814fe86a28d0ae2950426e62f1703e9d0477e3304cf1d7dbfe79" exitCode=0 Jul 26 16:25:14.718113 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:25:14.717972 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-10e721-openshift-default-cbb9784bb-94qgh" event={"ID":"bc949b5c-2665-4ba8-b28d-06a48f5af3d1","Type":"ContainerDied","Data":"48edabcde199814fe86a28d0ae2950426e62f1703e9d0477e3304cf1d7dbfe79"} Jul 26 16:25:14.718365 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:25:14.718352 2577 scope.go:117] "RemoveContainer" containerID="48edabcde199814fe86a28d0ae2950426e62f1703e9d0477e3304cf1d7dbfe79" Jul 26 16:25:14.719423 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:25:14.719374 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:25:14.719541 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:25:14.719477 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:25:14.719541 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:25:14.719526 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:25:15.725905 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:25:15.725871 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-10e721-openshift-default-cbb9784bb-94qgh" event={"ID":"bc949b5c-2665-4ba8-b28d-06a48f5af3d1","Type":"ContainerStarted","Data":"f81ba603e8dc27331f9c9cda2278c9ded7d4fcd323ee720c007638d7234a101e"} Jul 26 16:25:15.954377 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:25:15.954292 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:25:16.024427 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:25:16.024377 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-10e721-openshift-default-cbb9784bb-94qgh" Jul 26 16:25:16.024610 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:25:16.024588 2577 patch_prober.go:28] interesting pod/e2e-ipp-a-10e721-openshift-default-cbb9784bb-94qgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Jul 26 16:25:16.024672 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:25:16.024636 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-10e721-openshift-default-cbb9784bb-94qgh" podUID="bc949b5c-2665-4ba8-b28d-06a48f5af3d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Jul 26 16:25:16.263543 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:25:16.263504 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:25:16.557456 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:25:16.557422 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:25:17.025466 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:25:17.025430 2577 patch_prober.go:28] interesting pod/e2e-ipp-a-10e721-openshift-default-cbb9784bb-94qgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Jul 26 16:25:17.025957 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:25:17.025501 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-10e721-openshift-default-cbb9784bb-94qgh" podUID="bc949b5c-2665-4ba8-b28d-06a48f5af3d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Jul 26 16:25:18.025419 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:25:18.025367 2577 patch_prober.go:28] interesting pod/e2e-ipp-a-10e721-openshift-default-cbb9784bb-94qgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Jul 26 16:25:18.025603 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:25:18.025438 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-10e721-openshift-default-cbb9784bb-94qgh" podUID="bc949b5c-2665-4ba8-b28d-06a48f5af3d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Jul 26 16:25:19.025448 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:25:19.025418 2577 patch_prober.go:28] interesting pod/e2e-ipp-a-10e721-openshift-default-cbb9784bb-94qgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Jul 26 16:25:19.025625 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:25:19.025470 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-10e721-openshift-default-cbb9784bb-94qgh" podUID="bc949b5c-2665-4ba8-b28d-06a48f5af3d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Jul 26 16:25:20.028990 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:25:20.028959 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-10e721-openshift-default-cbb9784bb-94qgh" Jul 26 16:25:20.029371 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:25:20.029308 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-10e721-openshift-default-cbb9784bb-94qgh" Jul 26 16:25:20.029943 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:25:20.029923 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-10e721-openshift-default-cbb9784bb-94qgh" Jul 26 16:25:21.765142 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:25:21.765098 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:25:21.864134 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:25:21.864095 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:25:22.249981 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:25:22.249943 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:25:22.346653 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:25:22.346614 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:25:22.822806 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:25:22.822774 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:25:22.926392 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:25:22.926355 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:25:26.264612 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:25:26.264579 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:25:31.448412 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:25:31.448349 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:26:02.219932 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:26:02.219848 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:26:26.757519 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:26:26.757478 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:26:26.860630 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:26:26.860598 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:26:46.250915 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:26:46.250870 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:26:46.556195 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:26:46.556160 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:26:46.620939 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:26:46.620905 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:26:49.294456 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:26:49.294418 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:26:49.388566 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:26:49.388527 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:26:51.815927 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:26:51.815887 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:26:52.132706 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:26:52.132604 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:26:52.271841 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:26:52.271812 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:26:52.373047 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:26:52.373003 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:26:53.723745 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:26:53.723710 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:27:21.411988 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:27:21.411899 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:27:21.473412 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:27:21.473372 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:27:21.530957 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:27:21.530924 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:27:24.835612 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:27:24.835572 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:27:25.194358 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:27:25.194262 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:27:31.221993 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:27:31.221957 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:27:31.321483 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:27:31.321449 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:27:31.473411 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:27:31.473301 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:27:31.692282 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:27:31.692243 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:27:44.954378 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:27:44.954337 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:27:53.457345 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:27:53.457296 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:27:53.530968 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:27:53.530933 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:27:53.585544 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:27:53.585508 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:27:55.120173 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:27:55.120139 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:27:55.488084 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:27:55.487996 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:27:58.038409 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:27:58.038362 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-8wjqp_b6b8080d-0cee-4d0c-a05c-3e2f1b672a54/console-operator/1.log" Jul 26 16:27:58.043164 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:27:58.043139 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-8wjqp_b6b8080d-0cee-4d0c-a05c-3e2f1b672a54/console-operator/1.log" Jul 26 16:27:58.043327 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:27:58.043202 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ztlp6_cbe2cd41-6ebc-4722-a813-57d83fa1d242/ovn-acl-logging/0.log" Jul 26 16:27:58.047572 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:27:58.047552 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ztlp6_cbe2cd41-6ebc-4722-a813-57d83fa1d242/ovn-acl-logging/0.log" Jul 26 16:28:00.568335 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:28:00.568302 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:28:22.317834 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:28:22.317795 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:28:24.255741 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:28:24.255702 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:28:36.854639 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:28:36.854600 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:28:37.641086 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:28:37.641049 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:28:43.360545 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:28:43.360499 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:28:43.520846 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:28:43.520808 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:28:43.625063 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:28:43.624972 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-10e721-openshift-default-cbb9784bb-94qgh"] Jul 26 16:28:43.625233 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:28:43.625210 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-10e721-openshift-default-cbb9784bb-94qgh" podUID="bc949b5c-2665-4ba8-b28d-06a48f5af3d1" containerName="istio-proxy" containerID="cri-o://f81ba603e8dc27331f9c9cda2278c9ded7d4fcd323ee720c007638d7234a101e" gracePeriod=30 Jul 26 16:28:43.650780 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:28:43.650748 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:28:43.873095 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:28:43.873062 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:28:48.892346 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:28:48.892276 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-10e721-openshift-default-cbb9784bb-94qgh" Jul 26 16:28:48.955892 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:28:48.955858 2577 generic.go:356] "Generic (PLEG): container finished" podID="bc949b5c-2665-4ba8-b28d-06a48f5af3d1" containerID="f81ba603e8dc27331f9c9cda2278c9ded7d4fcd323ee720c007638d7234a101e" exitCode=0 Jul 26 16:28:48.956099 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:28:48.955908 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-10e721-openshift-default-cbb9784bb-94qgh" Jul 26 16:28:48.956099 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:28:48.955940 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-10e721-openshift-default-cbb9784bb-94qgh" event={"ID":"bc949b5c-2665-4ba8-b28d-06a48f5af3d1","Type":"ContainerDied","Data":"f81ba603e8dc27331f9c9cda2278c9ded7d4fcd323ee720c007638d7234a101e"} Jul 26 16:28:48.956099 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:28:48.956004 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-10e721-openshift-default-cbb9784bb-94qgh" event={"ID":"bc949b5c-2665-4ba8-b28d-06a48f5af3d1","Type":"ContainerDied","Data":"671b656734a6106fcd2ae5f06145aee6a3545fcdf4d0fa373984b0c9a4c584bb"} Jul 26 16:28:48.956099 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:28:48.956034 2577 scope.go:117] "RemoveContainer" containerID="f81ba603e8dc27331f9c9cda2278c9ded7d4fcd323ee720c007638d7234a101e" Jul 26 16:28:48.959125 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:28:48.959100 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc949b5c-2665-4ba8-b28d-06a48f5af3d1-workload-certs\") pod \"bc949b5c-2665-4ba8-b28d-06a48f5af3d1\" (UID: \"bc949b5c-2665-4ba8-b28d-06a48f5af3d1\") " Jul 26 16:28:48.959273 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:28:48.959144 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc949b5c-2665-4ba8-b28d-06a48f5af3d1-istio-token\") pod \"bc949b5c-2665-4ba8-b28d-06a48f5af3d1\" (UID: \"bc949b5c-2665-4ba8-b28d-06a48f5af3d1\") " Jul 26 16:28:48.959273 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:28:48.959176 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc949b5c-2665-4ba8-b28d-06a48f5af3d1-istio-envoy\") pod \"bc949b5c-2665-4ba8-b28d-06a48f5af3d1\" (UID: \"bc949b5c-2665-4ba8-b28d-06a48f5af3d1\") " Jul 26 16:28:48.959273 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:28:48.959240 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc949b5c-2665-4ba8-b28d-06a48f5af3d1-istio-data\") pod \"bc949b5c-2665-4ba8-b28d-06a48f5af3d1\" (UID: \"bc949b5c-2665-4ba8-b28d-06a48f5af3d1\") " Jul 26 16:28:48.959476 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:28:48.959284 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc949b5c-2665-4ba8-b28d-06a48f5af3d1-istio-podinfo\") pod \"bc949b5c-2665-4ba8-b28d-06a48f5af3d1\" (UID: \"bc949b5c-2665-4ba8-b28d-06a48f5af3d1\") " Jul 26 16:28:48.959476 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:28:48.959314 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc949b5c-2665-4ba8-b28d-06a48f5af3d1-credential-socket\") pod \"bc949b5c-2665-4ba8-b28d-06a48f5af3d1\" (UID: \"bc949b5c-2665-4ba8-b28d-06a48f5af3d1\") " Jul 26 16:28:48.959476 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:28:48.959340 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4c625\" (UniqueName: \"kubernetes.io/projected/bc949b5c-2665-4ba8-b28d-06a48f5af3d1-kube-api-access-4c625\") pod \"bc949b5c-2665-4ba8-b28d-06a48f5af3d1\" (UID: \"bc949b5c-2665-4ba8-b28d-06a48f5af3d1\") " Jul 26 16:28:48.959476 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:28:48.959384 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc949b5c-2665-4ba8-b28d-06a48f5af3d1-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "bc949b5c-2665-4ba8-b28d-06a48f5af3d1" (UID: "bc949b5c-2665-4ba8-b28d-06a48f5af3d1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:28:48.959476 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:28:48.959419 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc949b5c-2665-4ba8-b28d-06a48f5af3d1-istiod-ca-cert\") pod \"bc949b5c-2665-4ba8-b28d-06a48f5af3d1\" (UID: \"bc949b5c-2665-4ba8-b28d-06a48f5af3d1\") " Jul 26 16:28:48.959476 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:28:48.959459 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc949b5c-2665-4ba8-b28d-06a48f5af3d1-workload-socket\") pod \"bc949b5c-2665-4ba8-b28d-06a48f5af3d1\" (UID: \"bc949b5c-2665-4ba8-b28d-06a48f5af3d1\") " Jul 26 16:28:48.959795 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:28:48.959619 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc949b5c-2665-4ba8-b28d-06a48f5af3d1-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "bc949b5c-2665-4ba8-b28d-06a48f5af3d1" (UID: "bc949b5c-2665-4ba8-b28d-06a48f5af3d1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:28:48.959855 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:28:48.959797 2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc949b5c-2665-4ba8-b28d-06a48f5af3d1-credential-socket\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 16:28:48.959855 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:28:48.959825 2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc949b5c-2665-4ba8-b28d-06a48f5af3d1-workload-certs\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 16:28:48.959855 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:28:48.959825 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc949b5c-2665-4ba8-b28d-06a48f5af3d1-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "bc949b5c-2665-4ba8-b28d-06a48f5af3d1" (UID: "bc949b5c-2665-4ba8-b28d-06a48f5af3d1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 16:28:48.960112 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:28:48.960058 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc949b5c-2665-4ba8-b28d-06a48f5af3d1-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "bc949b5c-2665-4ba8-b28d-06a48f5af3d1" (UID: "bc949b5c-2665-4ba8-b28d-06a48f5af3d1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:28:48.960775 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:28:48.960746 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc949b5c-2665-4ba8-b28d-06a48f5af3d1-istio-data" (OuterVolumeSpecName: "istio-data") pod "bc949b5c-2665-4ba8-b28d-06a48f5af3d1" (UID: "bc949b5c-2665-4ba8-b28d-06a48f5af3d1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:28:48.961886 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:28:48.961859 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc949b5c-2665-4ba8-b28d-06a48f5af3d1-istio-token" (OuterVolumeSpecName: "istio-token") pod "bc949b5c-2665-4ba8-b28d-06a48f5af3d1" (UID: "bc949b5c-2665-4ba8-b28d-06a48f5af3d1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 16:28:48.961986 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:28:48.961954 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc949b5c-2665-4ba8-b28d-06a48f5af3d1-kube-api-access-4c625" (OuterVolumeSpecName: "kube-api-access-4c625") pod "bc949b5c-2665-4ba8-b28d-06a48f5af3d1" (UID: "bc949b5c-2665-4ba8-b28d-06a48f5af3d1"). InnerVolumeSpecName "kube-api-access-4c625". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 16:28:48.962118 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:28:48.962096 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc949b5c-2665-4ba8-b28d-06a48f5af3d1-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "bc949b5c-2665-4ba8-b28d-06a48f5af3d1" (UID: "bc949b5c-2665-4ba8-b28d-06a48f5af3d1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:28:48.962118 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:28:48.962093 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bc949b5c-2665-4ba8-b28d-06a48f5af3d1-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "bc949b5c-2665-4ba8-b28d-06a48f5af3d1" (UID: "bc949b5c-2665-4ba8-b28d-06a48f5af3d1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 26 16:28:48.968097 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:28:48.968082 2577 scope.go:117] "RemoveContainer" containerID="48edabcde199814fe86a28d0ae2950426e62f1703e9d0477e3304cf1d7dbfe79" Jul 26 16:28:48.984152 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:28:48.984127 2577 scope.go:117] "RemoveContainer" containerID="f81ba603e8dc27331f9c9cda2278c9ded7d4fcd323ee720c007638d7234a101e" Jul 26 16:28:48.984498 ip-10-0-138-163 kubenswrapper[2577]: E0726 16:28:48.984479 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f81ba603e8dc27331f9c9cda2278c9ded7d4fcd323ee720c007638d7234a101e\": container with ID starting with f81ba603e8dc27331f9c9cda2278c9ded7d4fcd323ee720c007638d7234a101e not found: ID does not exist" containerID="f81ba603e8dc27331f9c9cda2278c9ded7d4fcd323ee720c007638d7234a101e" Jul 26 16:28:48.984560 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:28:48.984509 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f81ba603e8dc27331f9c9cda2278c9ded7d4fcd323ee720c007638d7234a101e"} err="failed to get container status \"f81ba603e8dc27331f9c9cda2278c9ded7d4fcd323ee720c007638d7234a101e\": rpc error: code = NotFound desc = could not find container \"f81ba603e8dc27331f9c9cda2278c9ded7d4fcd323ee720c007638d7234a101e\": container with ID starting with f81ba603e8dc27331f9c9cda2278c9ded7d4fcd323ee720c007638d7234a101e not found: ID does not exist" Jul 26 16:28:48.984560 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:28:48.984528 2577 scope.go:117] "RemoveContainer" containerID="48edabcde199814fe86a28d0ae2950426e62f1703e9d0477e3304cf1d7dbfe79" Jul 26 16:28:48.984779 ip-10-0-138-163 kubenswrapper[2577]: E0726 16:28:48.984755 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48edabcde199814fe86a28d0ae2950426e62f1703e9d0477e3304cf1d7dbfe79\": container with ID starting with 48edabcde199814fe86a28d0ae2950426e62f1703e9d0477e3304cf1d7dbfe79 not found: ID does not exist" containerID="48edabcde199814fe86a28d0ae2950426e62f1703e9d0477e3304cf1d7dbfe79" Jul 26 16:28:48.984822 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:28:48.984786 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48edabcde199814fe86a28d0ae2950426e62f1703e9d0477e3304cf1d7dbfe79"} err="failed to get container status \"48edabcde199814fe86a28d0ae2950426e62f1703e9d0477e3304cf1d7dbfe79\": rpc error: code = NotFound desc = could not find container \"48edabcde199814fe86a28d0ae2950426e62f1703e9d0477e3304cf1d7dbfe79\": container with ID starting with 48edabcde199814fe86a28d0ae2950426e62f1703e9d0477e3304cf1d7dbfe79 not found: ID does not exist" Jul 26 16:28:49.060515 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:28:49.060484 2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc949b5c-2665-4ba8-b28d-06a48f5af3d1-istiod-ca-cert\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 16:28:49.060515 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:28:49.060510 2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc949b5c-2665-4ba8-b28d-06a48f5af3d1-workload-socket\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 16:28:49.060515 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:28:49.060519 2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc949b5c-2665-4ba8-b28d-06a48f5af3d1-istio-token\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 16:28:49.060755 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:28:49.060528 2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc949b5c-2665-4ba8-b28d-06a48f5af3d1-istio-envoy\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 16:28:49.060755 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:28:49.060537 2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc949b5c-2665-4ba8-b28d-06a48f5af3d1-istio-data\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 16:28:49.060755 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:28:49.060549 2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc949b5c-2665-4ba8-b28d-06a48f5af3d1-istio-podinfo\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 16:28:49.060755 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:28:49.060560 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4c625\" (UniqueName: \"kubernetes.io/projected/bc949b5c-2665-4ba8-b28d-06a48f5af3d1-kube-api-access-4c625\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 16:28:49.278990 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:28:49.278959 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-10e721-openshift-default-cbb9784bb-94qgh"] Jul 26 16:28:49.285112 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:28:49.285086 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-10e721-openshift-default-cbb9784bb-94qgh"] Jul 26 16:28:49.788448 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:28:49.788384 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bc949b5c-2665-4ba8-b28d-06a48f5af3d1" path="/var/lib/kubelet/pods/bc949b5c-2665-4ba8-b28d-06a48f5af3d1/volumes" Jul 26 16:28:55.622040 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:28:55.621978 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:28:55.721479 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:28:55.721449 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:28:55.823650 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:28:55.823614 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:28:59.096307 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:28:59.096262 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:28:59.478819 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:28:59.478711 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:29:05.651030 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:29:05.650989 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:29:05.751872 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:29:05.751825 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:29:05.912534 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:29:05.912504 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:29:06.146108 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:29:06.146064 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:29:16.384558 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:29:16.384516 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:29:16.482871 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:29:16.482842 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:29:16.583576 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:29:16.583539 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:29:21.386583 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:29:21.386537 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:29:21.774200 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:29:21.774170 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:29:27.989857 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:29:27.989823 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:29:28.091563 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:29:28.091522 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:29:28.256841 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:29:28.256757 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-aadfb2-openshift-default-76db5f76c6-kc7m7"] Jul 26 16:29:28.256984 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:29:28.256957 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-aadfb2-openshift-default-76db5f76c6-kc7m7" podUID="b250a529-3b42-40ed-8e16-5eaa7d780d9f" containerName="istio-proxy" containerID="cri-o://da495a676ce303990520016cb83cc5af146fc564feca88d6db7f0f27c01c3b12" gracePeriod=30 Jul 26 16:29:28.285510 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:29:28.285468 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:29:28.487843 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:29:28.487806 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:29:31.572931 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:29:31.572892 2577 patch_prober.go:28] interesting pod/e2e-shared-a-aadfb2-openshift-default-76db5f76c6-kc7m7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 26 16:29:31.573315 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:29:31.572951 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-aadfb2-openshift-default-76db5f76c6-kc7m7" podUID="b250a529-3b42-40ed-8e16-5eaa7d780d9f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 26 16:29:33.518525 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:29:33.518445 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-aadfb2-openshift-default-76db5f76c6-kc7m7" Jul 26 16:29:33.583204 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:29:33.583170 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b250a529-3b42-40ed-8e16-5eaa7d780d9f-credential-socket\") pod \"b250a529-3b42-40ed-8e16-5eaa7d780d9f\" (UID: \"b250a529-3b42-40ed-8e16-5eaa7d780d9f\") " Jul 26 16:29:33.583382 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:29:33.583285 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7rwx7\" (UniqueName: \"kubernetes.io/projected/b250a529-3b42-40ed-8e16-5eaa7d780d9f-kube-api-access-7rwx7\") pod \"b250a529-3b42-40ed-8e16-5eaa7d780d9f\" (UID: \"b250a529-3b42-40ed-8e16-5eaa7d780d9f\") " Jul 26 16:29:33.583382 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:29:33.583304 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b250a529-3b42-40ed-8e16-5eaa7d780d9f-workload-socket\") pod \"b250a529-3b42-40ed-8e16-5eaa7d780d9f\" (UID: \"b250a529-3b42-40ed-8e16-5eaa7d780d9f\") " Jul 26 16:29:33.583382 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:29:33.583321 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b250a529-3b42-40ed-8e16-5eaa7d780d9f-istio-data\") pod \"b250a529-3b42-40ed-8e16-5eaa7d780d9f\" (UID: \"b250a529-3b42-40ed-8e16-5eaa7d780d9f\") " Jul 26 16:29:33.583382 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:29:33.583340 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b250a529-3b42-40ed-8e16-5eaa7d780d9f-istio-envoy\") pod \"b250a529-3b42-40ed-8e16-5eaa7d780d9f\" (UID: \"b250a529-3b42-40ed-8e16-5eaa7d780d9f\") " Jul 26 16:29:33.583382 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:29:33.583356 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b250a529-3b42-40ed-8e16-5eaa7d780d9f-istio-token\") pod \"b250a529-3b42-40ed-8e16-5eaa7d780d9f\" (UID: \"b250a529-3b42-40ed-8e16-5eaa7d780d9f\") " Jul 26 16:29:33.583382 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:29:33.583372 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b250a529-3b42-40ed-8e16-5eaa7d780d9f-istio-podinfo\") pod \"b250a529-3b42-40ed-8e16-5eaa7d780d9f\" (UID: \"b250a529-3b42-40ed-8e16-5eaa7d780d9f\") " Jul 26 16:29:33.583728 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:29:33.583389 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b250a529-3b42-40ed-8e16-5eaa7d780d9f-istiod-ca-cert\") pod \"b250a529-3b42-40ed-8e16-5eaa7d780d9f\" (UID: \"b250a529-3b42-40ed-8e16-5eaa7d780d9f\") " Jul 26 16:29:33.583728 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:29:33.583472 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b250a529-3b42-40ed-8e16-5eaa7d780d9f-workload-certs\") pod \"b250a529-3b42-40ed-8e16-5eaa7d780d9f\" (UID: \"b250a529-3b42-40ed-8e16-5eaa7d780d9f\") " Jul 26 16:29:33.583728 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:29:33.583502 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b250a529-3b42-40ed-8e16-5eaa7d780d9f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b250a529-3b42-40ed-8e16-5eaa7d780d9f" (UID: "b250a529-3b42-40ed-8e16-5eaa7d780d9f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:29:33.583728 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:29:33.583664 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b250a529-3b42-40ed-8e16-5eaa7d780d9f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b250a529-3b42-40ed-8e16-5eaa7d780d9f" (UID: "b250a529-3b42-40ed-8e16-5eaa7d780d9f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:29:33.583914 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:29:33.583823 2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b250a529-3b42-40ed-8e16-5eaa7d780d9f-workload-socket\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 16:29:33.583914 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:29:33.583843 2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b250a529-3b42-40ed-8e16-5eaa7d780d9f-credential-socket\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 16:29:33.583914 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:29:33.583871 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b250a529-3b42-40ed-8e16-5eaa7d780d9f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b250a529-3b42-40ed-8e16-5eaa7d780d9f" (UID: "b250a529-3b42-40ed-8e16-5eaa7d780d9f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:29:33.584081 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:29:33.584059 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b250a529-3b42-40ed-8e16-5eaa7d780d9f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b250a529-3b42-40ed-8e16-5eaa7d780d9f" (UID: "b250a529-3b42-40ed-8e16-5eaa7d780d9f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 16:29:33.584308 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:29:33.584283 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b250a529-3b42-40ed-8e16-5eaa7d780d9f-istio-data" (OuterVolumeSpecName: "istio-data") pod "b250a529-3b42-40ed-8e16-5eaa7d780d9f" (UID: "b250a529-3b42-40ed-8e16-5eaa7d780d9f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:29:33.585766 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:29:33.585741 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b250a529-3b42-40ed-8e16-5eaa7d780d9f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b250a529-3b42-40ed-8e16-5eaa7d780d9f" (UID: "b250a529-3b42-40ed-8e16-5eaa7d780d9f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 26 16:29:33.585862 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:29:33.585783 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b250a529-3b42-40ed-8e16-5eaa7d780d9f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b250a529-3b42-40ed-8e16-5eaa7d780d9f" (UID: "b250a529-3b42-40ed-8e16-5eaa7d780d9f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:29:33.586027 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:29:33.585999 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b250a529-3b42-40ed-8e16-5eaa7d780d9f-istio-token" (OuterVolumeSpecName: "istio-token") pod "b250a529-3b42-40ed-8e16-5eaa7d780d9f" (UID: "b250a529-3b42-40ed-8e16-5eaa7d780d9f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 16:29:33.586027 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:29:33.586009 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b250a529-3b42-40ed-8e16-5eaa7d780d9f-kube-api-access-7rwx7" (OuterVolumeSpecName: "kube-api-access-7rwx7") pod "b250a529-3b42-40ed-8e16-5eaa7d780d9f" (UID: "b250a529-3b42-40ed-8e16-5eaa7d780d9f"). InnerVolumeSpecName "kube-api-access-7rwx7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 16:29:33.685189 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:29:33.685157 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rwx7\" (UniqueName: \"kubernetes.io/projected/b250a529-3b42-40ed-8e16-5eaa7d780d9f-kube-api-access-7rwx7\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 16:29:33.685189 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:29:33.685186 2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b250a529-3b42-40ed-8e16-5eaa7d780d9f-istio-data\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 16:29:33.685189 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:29:33.685195 2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b250a529-3b42-40ed-8e16-5eaa7d780d9f-istio-envoy\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 16:29:33.685457 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:29:33.685202 2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b250a529-3b42-40ed-8e16-5eaa7d780d9f-istio-token\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 16:29:33.685457 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:29:33.685213 2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b250a529-3b42-40ed-8e16-5eaa7d780d9f-istio-podinfo\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 16:29:33.685457 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:29:33.685221 2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b250a529-3b42-40ed-8e16-5eaa7d780d9f-istiod-ca-cert\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 16:29:33.685457 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:29:33.685228 2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b250a529-3b42-40ed-8e16-5eaa7d780d9f-workload-certs\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 16:29:34.217590 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:29:34.217493 2577 generic.go:356] "Generic (PLEG): container finished" podID="b250a529-3b42-40ed-8e16-5eaa7d780d9f" containerID="da495a676ce303990520016cb83cc5af146fc564feca88d6db7f0f27c01c3b12" exitCode=0 Jul 26 16:29:34.217590 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:29:34.217540 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-aadfb2-openshift-default-76db5f76c6-kc7m7" Jul 26 16:29:34.217590 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:29:34.217574 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-aadfb2-openshift-default-76db5f76c6-kc7m7" event={"ID":"b250a529-3b42-40ed-8e16-5eaa7d780d9f","Type":"ContainerDied","Data":"da495a676ce303990520016cb83cc5af146fc564feca88d6db7f0f27c01c3b12"} Jul 26 16:29:34.217854 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:29:34.217629 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-aadfb2-openshift-default-76db5f76c6-kc7m7" event={"ID":"b250a529-3b42-40ed-8e16-5eaa7d780d9f","Type":"ContainerDied","Data":"681fa5468f7cc7a4d1ce40ea10e962dec30128c206299a9c6ef4a63b9bdc2b47"} Jul 26 16:29:34.217854 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:29:34.217650 2577 scope.go:117] "RemoveContainer" containerID="da495a676ce303990520016cb83cc5af146fc564feca88d6db7f0f27c01c3b12" Jul 26 16:29:34.226662 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:29:34.226635 2577 scope.go:117] "RemoveContainer" containerID="74ede80574287b9f2db9f48f4ed3ca61875d5964f37e2e116abf70e03466df72" Jul 26 16:29:34.234806 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:29:34.234781 2577 scope.go:117] "RemoveContainer" containerID="da495a676ce303990520016cb83cc5af146fc564feca88d6db7f0f27c01c3b12" Jul 26 16:29:34.235040 ip-10-0-138-163 kubenswrapper[2577]: E0726 16:29:34.235021 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da495a676ce303990520016cb83cc5af146fc564feca88d6db7f0f27c01c3b12\": container with ID starting with da495a676ce303990520016cb83cc5af146fc564feca88d6db7f0f27c01c3b12 not found: ID does not exist" containerID="da495a676ce303990520016cb83cc5af146fc564feca88d6db7f0f27c01c3b12" Jul 26 16:29:34.235093 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:29:34.235050 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da495a676ce303990520016cb83cc5af146fc564feca88d6db7f0f27c01c3b12"} err="failed to get container status \"da495a676ce303990520016cb83cc5af146fc564feca88d6db7f0f27c01c3b12\": rpc error: code = NotFound desc = could not find container \"da495a676ce303990520016cb83cc5af146fc564feca88d6db7f0f27c01c3b12\": container with ID starting with da495a676ce303990520016cb83cc5af146fc564feca88d6db7f0f27c01c3b12 not found: ID does not exist" Jul 26 16:29:34.235093 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:29:34.235068 2577 scope.go:117] "RemoveContainer" containerID="74ede80574287b9f2db9f48f4ed3ca61875d5964f37e2e116abf70e03466df72" Jul 26 16:29:34.235300 ip-10-0-138-163 kubenswrapper[2577]: E0726 16:29:34.235279 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74ede80574287b9f2db9f48f4ed3ca61875d5964f37e2e116abf70e03466df72\": container with ID starting with 74ede80574287b9f2db9f48f4ed3ca61875d5964f37e2e116abf70e03466df72 not found: ID does not exist" containerID="74ede80574287b9f2db9f48f4ed3ca61875d5964f37e2e116abf70e03466df72" Jul 26 16:29:34.235344 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:29:34.235307 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74ede80574287b9f2db9f48f4ed3ca61875d5964f37e2e116abf70e03466df72"} err="failed to get container status \"74ede80574287b9f2db9f48f4ed3ca61875d5964f37e2e116abf70e03466df72\": rpc error: code = NotFound desc = could not find container \"74ede80574287b9f2db9f48f4ed3ca61875d5964f37e2e116abf70e03466df72\": container with ID starting with 74ede80574287b9f2db9f48f4ed3ca61875d5964f37e2e116abf70e03466df72 not found: ID does not exist" Jul 26 16:29:34.253793 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:29:34.253763 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-aadfb2-openshift-default-76db5f76c6-kc7m7"] Jul 26 16:29:34.281666 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:29:34.281641 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-aadfb2-openshift-default-76db5f76c6-kc7m7"] Jul 26 16:29:35.788246 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:29:35.788210 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b250a529-3b42-40ed-8e16-5eaa7d780d9f" path="/var/lib/kubelet/pods/b250a529-3b42-40ed-8e16-5eaa7d780d9f/volumes" Jul 26 16:29:39.583538 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:29:39.583501 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:29:39.680246 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:29:39.680212 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:29:39.750317 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:29:39.750283 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:29:43.666115 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:29:43.666077 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:29:44.039985 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:29:44.039947 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:29:50.086032 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:29:50.085997 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:29:50.160807 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:29:50.160771 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:29:50.345667 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:29:50.345579 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-e0208c-openshift-default-745b9f8964-jphn4"] Jul 26 16:29:50.345839 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:29:50.345781 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-e0208c-openshift-default-745b9f8964-jphn4" podUID="0002d75b-65fd-4dc9-9dbf-72122f738317" containerName="istio-proxy" containerID="cri-o://6b8986f9838ccae6f74a4dc2e4c4b32a01a01b0173588903d1cdeff7d4b45e4f" gracePeriod=30 Jul 26 16:29:50.366983 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:29:50.366952 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:29:50.579380 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:29:50.579344 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:29:50.979192 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:29:50.979153 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-997bd858f-b6n8s_9b08503b-3207-43e6-8fa2-1fc702a76fc5/authorino/0.log" Jul 26 16:29:53.780227 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:29:53.780190 2577 patch_prober.go:28] interesting pod/e2e-shared-b-e0208c-openshift-default-745b9f8964-jphn4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 26 16:29:53.780694 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:29:53.780252 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-e0208c-openshift-default-745b9f8964-jphn4" podUID="0002d75b-65fd-4dc9-9dbf-72122f738317" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 26 16:29:55.604327 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:29:55.604300 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-e0208c-openshift-default-745b9f8964-jphn4" Jul 26 16:29:55.684986 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:29:55.684901 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0002d75b-65fd-4dc9-9dbf-72122f738317-workload-socket\") pod \"0002d75b-65fd-4dc9-9dbf-72122f738317\" (UID: \"0002d75b-65fd-4dc9-9dbf-72122f738317\") " Jul 26 16:29:55.684986 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:29:55.684978 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0002d75b-65fd-4dc9-9dbf-72122f738317-istio-podinfo\") pod \"0002d75b-65fd-4dc9-9dbf-72122f738317\" (UID: \"0002d75b-65fd-4dc9-9dbf-72122f738317\") " Jul 26 16:29:55.685179 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:29:55.685010 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0002d75b-65fd-4dc9-9dbf-72122f738317-istiod-ca-cert\") pod \"0002d75b-65fd-4dc9-9dbf-72122f738317\" (UID: \"0002d75b-65fd-4dc9-9dbf-72122f738317\") " Jul 26 16:29:55.685179 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:29:55.685044 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0002d75b-65fd-4dc9-9dbf-72122f738317-workload-certs\") pod \"0002d75b-65fd-4dc9-9dbf-72122f738317\" (UID: \"0002d75b-65fd-4dc9-9dbf-72122f738317\") " Jul 26 16:29:55.685179 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:29:55.685074 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0002d75b-65fd-4dc9-9dbf-72122f738317-istio-token\") pod \"0002d75b-65fd-4dc9-9dbf-72122f738317\" (UID: \"0002d75b-65fd-4dc9-9dbf-72122f738317\") " Jul 26 16:29:55.685179 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:29:55.685162 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dbkph\" (UniqueName: \"kubernetes.io/projected/0002d75b-65fd-4dc9-9dbf-72122f738317-kube-api-access-dbkph\") pod \"0002d75b-65fd-4dc9-9dbf-72122f738317\" (UID: \"0002d75b-65fd-4dc9-9dbf-72122f738317\") " Jul 26 16:29:55.685423 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:29:55.685201 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0002d75b-65fd-4dc9-9dbf-72122f738317-istio-data\") pod \"0002d75b-65fd-4dc9-9dbf-72122f738317\" (UID: \"0002d75b-65fd-4dc9-9dbf-72122f738317\") " Jul 26 16:29:55.685423 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:29:55.685202 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0002d75b-65fd-4dc9-9dbf-72122f738317-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0002d75b-65fd-4dc9-9dbf-72122f738317" (UID: "0002d75b-65fd-4dc9-9dbf-72122f738317"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:29:55.685423 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:29:55.685228 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0002d75b-65fd-4dc9-9dbf-72122f738317-istio-envoy\") pod \"0002d75b-65fd-4dc9-9dbf-72122f738317\" (UID: \"0002d75b-65fd-4dc9-9dbf-72122f738317\") " Jul 26 16:29:55.685423 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:29:55.685326 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0002d75b-65fd-4dc9-9dbf-72122f738317-credential-socket\") pod \"0002d75b-65fd-4dc9-9dbf-72122f738317\" (UID: \"0002d75b-65fd-4dc9-9dbf-72122f738317\") " Jul 26 16:29:55.685423 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:29:55.685356 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0002d75b-65fd-4dc9-9dbf-72122f738317-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0002d75b-65fd-4dc9-9dbf-72122f738317" (UID: "0002d75b-65fd-4dc9-9dbf-72122f738317"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:29:55.685670 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:29:55.685425 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0002d75b-65fd-4dc9-9dbf-72122f738317-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0002d75b-65fd-4dc9-9dbf-72122f738317" (UID: "0002d75b-65fd-4dc9-9dbf-72122f738317"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 16:29:55.686284 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:29:55.686186 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0002d75b-65fd-4dc9-9dbf-72122f738317-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0002d75b-65fd-4dc9-9dbf-72122f738317" (UID: "0002d75b-65fd-4dc9-9dbf-72122f738317"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:29:55.686758 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:29:55.686702 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0002d75b-65fd-4dc9-9dbf-72122f738317-istio-data" (OuterVolumeSpecName: "istio-data") pod "0002d75b-65fd-4dc9-9dbf-72122f738317" (UID: "0002d75b-65fd-4dc9-9dbf-72122f738317"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:29:55.686885 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:29:55.686798 2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0002d75b-65fd-4dc9-9dbf-72122f738317-workload-socket\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 16:29:55.686885 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:29:55.686834 2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0002d75b-65fd-4dc9-9dbf-72122f738317-istiod-ca-cert\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 16:29:55.686885 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:29:55.686849 2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0002d75b-65fd-4dc9-9dbf-72122f738317-workload-certs\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 16:29:55.688820 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:29:55.687619 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0002d75b-65fd-4dc9-9dbf-72122f738317-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0002d75b-65fd-4dc9-9dbf-72122f738317" (UID: "0002d75b-65fd-4dc9-9dbf-72122f738317"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 26 16:29:55.691475 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:29:55.689272 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0002d75b-65fd-4dc9-9dbf-72122f738317-kube-api-access-dbkph" (OuterVolumeSpecName: "kube-api-access-dbkph") pod "0002d75b-65fd-4dc9-9dbf-72122f738317" (UID: "0002d75b-65fd-4dc9-9dbf-72122f738317"). InnerVolumeSpecName "kube-api-access-dbkph". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 16:29:55.691475 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:29:55.689344 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0002d75b-65fd-4dc9-9dbf-72122f738317-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0002d75b-65fd-4dc9-9dbf-72122f738317" (UID: "0002d75b-65fd-4dc9-9dbf-72122f738317"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:29:55.694957 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:29:55.694936 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0002d75b-65fd-4dc9-9dbf-72122f738317-istio-token" (OuterVolumeSpecName: "istio-token") pod "0002d75b-65fd-4dc9-9dbf-72122f738317" (UID: "0002d75b-65fd-4dc9-9dbf-72122f738317"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 16:29:55.788223 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:29:55.788189 2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0002d75b-65fd-4dc9-9dbf-72122f738317-istio-podinfo\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 16:29:55.788223 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:29:55.788215 2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0002d75b-65fd-4dc9-9dbf-72122f738317-istio-token\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 16:29:55.788223 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:29:55.788229 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbkph\" (UniqueName: \"kubernetes.io/projected/0002d75b-65fd-4dc9-9dbf-72122f738317-kube-api-access-dbkph\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 16:29:55.788473 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:29:55.788240 2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0002d75b-65fd-4dc9-9dbf-72122f738317-istio-data\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 16:29:55.788473 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:29:55.788249 2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0002d75b-65fd-4dc9-9dbf-72122f738317-istio-envoy\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 16:29:55.788473 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:29:55.788257 2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0002d75b-65fd-4dc9-9dbf-72122f738317-credential-socket\") on node \"ip-10-0-138-163.ec2.internal\" DevicePath \"\"" Jul 26 16:29:56.345057 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:29:56.345016 2577 generic.go:356] "Generic (PLEG): container finished" podID="0002d75b-65fd-4dc9-9dbf-72122f738317" containerID="6b8986f9838ccae6f74a4dc2e4c4b32a01a01b0173588903d1cdeff7d4b45e4f" exitCode=0 Jul 26 16:29:56.345251 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:29:56.345090 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-e0208c-openshift-default-745b9f8964-jphn4" Jul 26 16:29:56.345251 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:29:56.345095 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-e0208c-openshift-default-745b9f8964-jphn4" event={"ID":"0002d75b-65fd-4dc9-9dbf-72122f738317","Type":"ContainerDied","Data":"6b8986f9838ccae6f74a4dc2e4c4b32a01a01b0173588903d1cdeff7d4b45e4f"} Jul 26 16:29:56.345251 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:29:56.345153 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-e0208c-openshift-default-745b9f8964-jphn4" event={"ID":"0002d75b-65fd-4dc9-9dbf-72122f738317","Type":"ContainerDied","Data":"9a536be2c9511db1ade2bab9f1aeaad73577e60308d8f2aae08bab00495ddbb5"} Jul 26 16:29:56.345251 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:29:56.345174 2577 scope.go:117] "RemoveContainer" containerID="6b8986f9838ccae6f74a4dc2e4c4b32a01a01b0173588903d1cdeff7d4b45e4f" Jul 26 16:29:56.354557 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:29:56.354537 2577 scope.go:117] "RemoveContainer" containerID="a72aaf2ca70da1c251f564beb024d52577f15c40f1c161ae09b23064c0972b46" Jul 26 16:29:56.363441 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:29:56.363418 2577 scope.go:117] "RemoveContainer" containerID="6b8986f9838ccae6f74a4dc2e4c4b32a01a01b0173588903d1cdeff7d4b45e4f" Jul 26 16:29:56.363785 ip-10-0-138-163 kubenswrapper[2577]: E0726 16:29:56.363765 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b8986f9838ccae6f74a4dc2e4c4b32a01a01b0173588903d1cdeff7d4b45e4f\": container with ID starting with 6b8986f9838ccae6f74a4dc2e4c4b32a01a01b0173588903d1cdeff7d4b45e4f not found: ID does not exist" containerID="6b8986f9838ccae6f74a4dc2e4c4b32a01a01b0173588903d1cdeff7d4b45e4f" Jul 26 16:29:56.363868 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:29:56.363795 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b8986f9838ccae6f74a4dc2e4c4b32a01a01b0173588903d1cdeff7d4b45e4f"} err="failed to get container status \"6b8986f9838ccae6f74a4dc2e4c4b32a01a01b0173588903d1cdeff7d4b45e4f\": rpc error: code = NotFound desc = could not find container \"6b8986f9838ccae6f74a4dc2e4c4b32a01a01b0173588903d1cdeff7d4b45e4f\": container with ID starting with 6b8986f9838ccae6f74a4dc2e4c4b32a01a01b0173588903d1cdeff7d4b45e4f not found: ID does not exist" Jul 26 16:29:56.363868 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:29:56.363814 2577 scope.go:117] "RemoveContainer" containerID="a72aaf2ca70da1c251f564beb024d52577f15c40f1c161ae09b23064c0972b46" Jul 26 16:29:56.364099 ip-10-0-138-163 kubenswrapper[2577]: E0726 16:29:56.364081 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a72aaf2ca70da1c251f564beb024d52577f15c40f1c161ae09b23064c0972b46\": container with ID starting with a72aaf2ca70da1c251f564beb024d52577f15c40f1c161ae09b23064c0972b46 not found: ID does not exist" containerID="a72aaf2ca70da1c251f564beb024d52577f15c40f1c161ae09b23064c0972b46" Jul 26 16:29:56.364141 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:29:56.364107 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a72aaf2ca70da1c251f564beb024d52577f15c40f1c161ae09b23064c0972b46"} err="failed to get container status \"a72aaf2ca70da1c251f564beb024d52577f15c40f1c161ae09b23064c0972b46\": rpc error: code = NotFound desc = could not find container \"a72aaf2ca70da1c251f564beb024d52577f15c40f1c161ae09b23064c0972b46\": container with ID starting with a72aaf2ca70da1c251f564beb024d52577f15c40f1c161ae09b23064c0972b46 not found: ID does not exist" Jul 26 16:29:56.368222 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:29:56.368195 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-e0208c-openshift-default-745b9f8964-jphn4"] Jul 26 16:29:56.372861 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:29:56.372838 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-e0208c-openshift-default-745b9f8964-jphn4"] Jul 26 16:29:56.571158 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:29:56.571132 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-mblzh_f230bc42-6931-4613-ac67-8e30d1b81094/server/0.log" Jul 26 16:29:56.690609 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:29:56.690535 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-rg7qh_f24e879c-5719-4461-a065-c15b3bfc8513/manager/0.log" Jul 26 16:29:56.795589 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:29:56.795559 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-66d4685587-5rddj_881b4d20-4178-492f-8183-7d1011be3602/copy-manifests/0.log" Jul 26 16:29:56.802676 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:29:56.802651 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-66d4685587-5rddj_881b4d20-4178-492f-8183-7d1011be3602/manager/0.log" Jul 26 16:29:57.518557 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:29:57.518528 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29751360-vx2hl_f7e3b4ec-623e-435c-a990-b9c4f02652b1/cleanup/0.log" Jul 26 16:29:57.787414 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:29:57.787359 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0002d75b-65fd-4dc9-9dbf-72122f738317" path="/var/lib/kubelet/pods/0002d75b-65fd-4dc9-9dbf-72122f738317/volumes" Jul 26 16:29:58.499547 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:29:58.499476 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hdscz_ac43ac70-9ab9-4bdd-ba8a-ea16fe1c6936/util/0.log" Jul 26 16:29:58.506151 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:29:58.506123 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hdscz_ac43ac70-9ab9-4bdd-ba8a-ea16fe1c6936/pull/0.log" Jul 26 16:29:58.512594 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:29:58.512575 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hdscz_ac43ac70-9ab9-4bdd-ba8a-ea16fe1c6936/extract/0.log" Jul 26 16:29:58.617657 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:29:58.617626 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xndcq_1435a9c3-05a1-4b00-9365-4db5b74bb22c/util/0.log" Jul 26 16:29:58.623308 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:29:58.623284 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xndcq_1435a9c3-05a1-4b00-9365-4db5b74bb22c/pull/0.log" Jul 26 16:29:58.629029 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:29:58.629006 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xndcq_1435a9c3-05a1-4b00-9365-4db5b74bb22c/extract/0.log" Jul 26 16:29:58.740865 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:29:58.740836 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73g75nl_cb6ec50d-8695-49c0-8eab-bad100286052/util/0.log" Jul 26 16:29:58.747172 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:29:58.747134 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73g75nl_cb6ec50d-8695-49c0-8eab-bad100286052/pull/0.log" Jul 26 16:29:58.753300 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:29:58.753232 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73g75nl_cb6ec50d-8695-49c0-8eab-bad100286052/extract/0.log" Jul 26 16:29:58.862768 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:29:58.862740 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1674ps_b8633ddc-0c3b-4ae5-b5a1-36716c97108f/util/0.log" Jul 26 16:29:58.869139 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:29:58.869110 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1674ps_b8633ddc-0c3b-4ae5-b5a1-36716c97108f/pull/0.log" Jul 26 16:29:58.875547 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:29:58.875526 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1674ps_b8633ddc-0c3b-4ae5-b5a1-36716c97108f/extract/0.log" Jul 26 16:29:58.997489 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:29:58.997460 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-997bd858f-b6n8s_9b08503b-3207-43e6-8fa2-1fc702a76fc5/authorino/0.log" Jul 26 16:29:59.118788 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:29:59.118759 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-5qlgd_8fa87b24-e0fc-4e83-adc1-d9fcf1395fc6/manager/0.log" Jul 26 16:29:59.229731 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:29:59.229700 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-q7jfk_da528638-84d1-40cf-9208-7929f925421a/manager/0.log" Jul 26 16:29:59.459696 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:29:59.459618 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-s5mxx_e3fb5dbe-c712-4c56-82fc-dbcb11fad428/registry-server/0.log" Jul 26 16:29:59.579476 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:29:59.579445 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-cpvmq_52da11a3-f37a-413e-b8c5-934a0aaf9526/manager/0.log" Jul 26 16:30:00.181245 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:30:00.181216 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-rx9js_b03798f1-6dbf-4cee-bf15-062b468c5206/istio-proxy/0.log" Jul 26 16:30:00.655019 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:30:00.654982 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-596h4_34f385df-2d11-4b82-ad20-404ee0104f33/istio-proxy/1.log" Jul 26 16:30:01.015208 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:30:01.015124 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5b79d74577-h8slk_9af43da1-c25b-4687-820f-6a95afe88786/router/0.log" Jul 26 16:30:01.346186 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:30:01.346158 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-knsvs_92d710d9-8053-4983-aa56-06536dc79041/main/0.log" Jul 26 16:30:01.562297 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:30:01.562270 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-bjb5v_2694dc63-b18b-42d3-9cab-8fef40fdbf33/main/0.log" Jul 26 16:30:07.697018 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:30:07.696979 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:30:07.795253 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:30:07.795214 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:30:07.823367 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:30:07.823335 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-4m86b/must-gather-kmclw"] Jul 26 16:30:07.823735 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:30:07.823723 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc949b5c-2665-4ba8-b28d-06a48f5af3d1" containerName="istio-proxy" Jul 26 16:30:07.823784 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:30:07.823736 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc949b5c-2665-4ba8-b28d-06a48f5af3d1" containerName="istio-proxy" Jul 26 16:30:07.823784 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:30:07.823746 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc949b5c-2665-4ba8-b28d-06a48f5af3d1" containerName="istio-proxy" Jul 26 16:30:07.823784 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:30:07.823753 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc949b5c-2665-4ba8-b28d-06a48f5af3d1" containerName="istio-proxy" Jul 26 16:30:07.823784 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:30:07.823766 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0002d75b-65fd-4dc9-9dbf-72122f738317" containerName="istio-proxy" Jul 26 16:30:07.823784 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:30:07.823771 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="0002d75b-65fd-4dc9-9dbf-72122f738317" containerName="istio-proxy" Jul 26 16:30:07.823784 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:30:07.823779 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b250a529-3b42-40ed-8e16-5eaa7d780d9f" containerName="istio-proxy" Jul 26 16:30:07.823784 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:30:07.823784 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b250a529-3b42-40ed-8e16-5eaa7d780d9f" containerName="istio-proxy" Jul 26 16:30:07.824019 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:30:07.823790 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b250a529-3b42-40ed-8e16-5eaa7d780d9f" containerName="istio-proxy" Jul 26 16:30:07.824019 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:30:07.823795 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b250a529-3b42-40ed-8e16-5eaa7d780d9f" containerName="istio-proxy" Jul 26 16:30:07.824019 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:30:07.823803 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0002d75b-65fd-4dc9-9dbf-72122f738317" containerName="istio-proxy" Jul 26 16:30:07.824019 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:30:07.823808 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="0002d75b-65fd-4dc9-9dbf-72122f738317" containerName="istio-proxy" Jul 26 16:30:07.824019 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:30:07.823873 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="0002d75b-65fd-4dc9-9dbf-72122f738317" containerName="istio-proxy" Jul 26 16:30:07.824019 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:30:07.823880 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="b250a529-3b42-40ed-8e16-5eaa7d780d9f" containerName="istio-proxy" Jul 26 16:30:07.824019 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:30:07.823887 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc949b5c-2665-4ba8-b28d-06a48f5af3d1" containerName="istio-proxy" Jul 26 16:30:07.824019 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:30:07.823893 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc949b5c-2665-4ba8-b28d-06a48f5af3d1" containerName="istio-proxy" Jul 26 16:30:07.824019 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:30:07.824006 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="0002d75b-65fd-4dc9-9dbf-72122f738317" containerName="istio-proxy" Jul 26 16:30:07.824019 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:30:07.824012 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="b250a529-3b42-40ed-8e16-5eaa7d780d9f" containerName="istio-proxy" Jul 26 16:30:07.828948 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:30:07.828932 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4m86b/must-gather-kmclw" Jul 26 16:30:07.831796 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:30:07.831770 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-4m86b\"/\"openshift-service-ca.crt\"" Jul 26 16:30:07.832956 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:30:07.832931 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-4m86b\"/\"kube-root-ca.crt\"" Jul 26 16:30:07.833061 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:30:07.832931 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-4m86b\"/\"default-dockercfg-gbd2s\"" Jul 26 16:30:07.836037 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:30:07.836013 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4m86b/must-gather-kmclw"] Jul 26 16:30:07.898666 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:30:07.898622 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/3e2971dc-687b-42bb-882b-32171d9b5b67-must-gather-output\") pod \"must-gather-kmclw\" (UID: \"3e2971dc-687b-42bb-882b-32171d9b5b67\") " pod="openshift-must-gather-4m86b/must-gather-kmclw" Jul 26 16:30:07.898666 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:30:07.898665 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjtdh\" (UniqueName: \"kubernetes.io/projected/3e2971dc-687b-42bb-882b-32171d9b5b67-kube-api-access-wjtdh\") pod \"must-gather-kmclw\" (UID: \"3e2971dc-687b-42bb-882b-32171d9b5b67\") " pod="openshift-must-gather-4m86b/must-gather-kmclw" Jul 26 16:30:07.999620 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:30:07.999533 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/3e2971dc-687b-42bb-882b-32171d9b5b67-must-gather-output\") pod \"must-gather-kmclw\" (UID: \"3e2971dc-687b-42bb-882b-32171d9b5b67\") " pod="openshift-must-gather-4m86b/must-gather-kmclw" Jul 26 16:30:07.999620 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:30:07.999572 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjtdh\" (UniqueName: \"kubernetes.io/projected/3e2971dc-687b-42bb-882b-32171d9b5b67-kube-api-access-wjtdh\") pod \"must-gather-kmclw\" (UID: \"3e2971dc-687b-42bb-882b-32171d9b5b67\") " pod="openshift-must-gather-4m86b/must-gather-kmclw" Jul 26 16:30:07.999888 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:30:07.999869 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/3e2971dc-687b-42bb-882b-32171d9b5b67-must-gather-output\") pod \"must-gather-kmclw\" (UID: \"3e2971dc-687b-42bb-882b-32171d9b5b67\") " pod="openshift-must-gather-4m86b/must-gather-kmclw" Jul 26 16:30:08.007630 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:30:08.007607 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjtdh\" (UniqueName: \"kubernetes.io/projected/3e2971dc-687b-42bb-882b-32171d9b5b67-kube-api-access-wjtdh\") pod \"must-gather-kmclw\" (UID: \"3e2971dc-687b-42bb-882b-32171d9b5b67\") " pod="openshift-must-gather-4m86b/must-gather-kmclw" Jul 26 16:30:08.139261 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:30:08.139225 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4m86b/must-gather-kmclw" Jul 26 16:30:08.275121 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:30:08.275091 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4m86b/must-gather-kmclw"] Jul 26 16:30:08.276454 ip-10-0-138-163 kubenswrapper[2577]: W0726 16:30:08.276429 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3e2971dc_687b_42bb_882b_32171d9b5b67.slice/crio-4c99c19c440acb4b99fcc61401d905c2be185083b5f86141a1f2bffbbe8e5fc6 WatchSource:0}: Error finding container 4c99c19c440acb4b99fcc61401d905c2be185083b5f86141a1f2bffbbe8e5fc6: Status 404 returned error can't find the container with id 4c99c19c440acb4b99fcc61401d905c2be185083b5f86141a1f2bffbbe8e5fc6 Jul 26 16:30:08.278430 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:30:08.278387 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 26 16:30:08.412109 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:30:08.412067 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4m86b/must-gather-kmclw" event={"ID":"3e2971dc-687b-42bb-882b-32171d9b5b67","Type":"ContainerStarted","Data":"4c99c19c440acb4b99fcc61401d905c2be185083b5f86141a1f2bffbbe8e5fc6"} Jul 26 16:30:09.422438 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:30:09.422328 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4m86b/must-gather-kmclw" event={"ID":"3e2971dc-687b-42bb-882b-32171d9b5b67","Type":"ContainerStarted","Data":"ee6768006cb262c2430376a52efeaee382e1d4ab5c8a8b2fc7e294425bb1ab77"} Jul 26 16:30:09.422438 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:30:09.422376 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4m86b/must-gather-kmclw" event={"ID":"3e2971dc-687b-42bb-882b-32171d9b5b67","Type":"ContainerStarted","Data":"8ac6f40e752a3d7cb8aad598d372fb45f75dc5264f68b4ba5f8db3766ad4678b"} Jul 26 16:30:09.440736 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:30:09.440667 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-4m86b/must-gather-kmclw" podStartSLOduration=1.6891805949999998 podStartE2EDuration="2.440646108s" totalImagesPullingTime="751.465513ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 16:30:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 16:30:09.438608938 +0000 UTC m=+3132.177962267" watchObservedRunningTime="2026-07-26 16:30:09.440646108 +0000 UTC m=+3132.179999419" Jul 26 16:30:10.817082 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:30:10.817043 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-l56b4_f6bd9013-635a-4af8-b0ee-621a280a459a/global-pull-secret-syncer/0.log" Jul 26 16:30:10.883228 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:30:10.883192 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" Jul 26 16:30:11.082167 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:30:11.082083 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-138-163.ec2.internal_0d03d973abb0a97c200617d39e2704ee/haproxy/0.log" Jul 26 16:30:15.245072 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:30:15.245011 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hdscz_ac43ac70-9ab9-4bdd-ba8a-ea16fe1c6936/extract/0.log" Jul 26 16:30:15.269802 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:30:15.269777 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hdscz_ac43ac70-9ab9-4bdd-ba8a-ea16fe1c6936/util/0.log" Jul 26 16:30:15.294455 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:30:15.294425 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hdscz_ac43ac70-9ab9-4bdd-ba8a-ea16fe1c6936/pull/0.log" Jul 26 16:30:15.332636 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:30:15.332602 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xndcq_1435a9c3-05a1-4b00-9365-4db5b74bb22c/extract/0.log" Jul 26 16:30:15.356801 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:30:15.356761 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xndcq_1435a9c3-05a1-4b00-9365-4db5b74bb22c/util/0.log" Jul 26 16:30:15.384342 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:30:15.384299 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xndcq_1435a9c3-05a1-4b00-9365-4db5b74bb22c/pull/0.log" Jul 26 16:30:15.412998 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:30:15.412906 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73g75nl_cb6ec50d-8695-49c0-8eab-bad100286052/extract/0.log" Jul 26 16:30:15.436349 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:30:15.436318 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73g75nl_cb6ec50d-8695-49c0-8eab-bad100286052/util/0.log" Jul 26 16:30:15.460187 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:30:15.460156 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73g75nl_cb6ec50d-8695-49c0-8eab-bad100286052/pull/0.log" Jul 26 16:30:15.505027 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:30:15.505001 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1674ps_b8633ddc-0c3b-4ae5-b5a1-36716c97108f/extract/0.log" Jul 26 16:30:15.535924 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:30:15.535894 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1674ps_b8633ddc-0c3b-4ae5-b5a1-36716c97108f/util/0.log" Jul 26 16:30:15.562674 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:30:15.562625 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1674ps_b8633ddc-0c3b-4ae5-b5a1-36716c97108f/pull/0.log" Jul 26 16:30:16.144361 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:30:16.144322 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-997bd858f-b6n8s_9b08503b-3207-43e6-8fa2-1fc702a76fc5/authorino/0.log" Jul 26 16:30:16.182048 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:30:16.182013 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-5qlgd_8fa87b24-e0fc-4e83-adc1-d9fcf1395fc6/manager/0.log" Jul 26 16:30:16.212709 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:30:16.212671 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-q7jfk_da528638-84d1-40cf-9208-7929f925421a/manager/0.log" Jul 26 16:30:16.302617 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:30:16.302578 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-s5mxx_e3fb5dbe-c712-4c56-82fc-dbcb11fad428/registry-server/0.log" Jul 26 16:30:16.581273 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:30:16.581172 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-cpvmq_52da11a3-f37a-413e-b8c5-934a0aaf9526/manager/0.log" Jul 26 16:30:17.644867 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:30:17.644825 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbpx6_806292e6-a776-4195-a595-263edb35e68b/konnectivity-agent/0.log" limit=1024 Jul 26 16:30:18.627929 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:30:18.627894 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-548b489cb8-2q2tq_60d326a6-77da-4f72-965f-880f5c94f2b7/cluster-monitoring-operator/0.log" Jul 26 16:30:18.986649 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:30:18.986572 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-s5rxf_e8a7bf77-ea86-49c6-9111-73932e0359bb/node-exporter/0.log" Jul 26 16:30:19.013080 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:30:19.013029 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-s5rxf_e8a7bf77-ea86-49c6-9111-73932e0359bb/kube-rbac-proxy/0.log" Jul 26 16:30:19.048499 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:30:19.048474 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-s5rxf_e8a7bf77-ea86-49c6-9111-73932e0359bb/init-textfile/0.log" Jul 26 16:30:19.489728 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:30:19.489692 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-7f8d78f54b-9rkgx_ef70e16b-47fb-49f0-ab9d-d54f5cce38fb/prometheus-operator-admission-webhook/0.log" Jul 26 16:30:20.545644 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:30:20.545543 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-4m86b/perf-node-gather-daemonset-7xklb"] Jul 26 16:30:20.549771 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:30:20.549745 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4m86b/perf-node-gather-daemonset-7xklb" Jul 26 16:30:20.561366 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:30:20.561339 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4m86b/perf-node-gather-daemonset-7xklb"] Jul 26 16:30:20.644545 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:30:20.644503 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/66e6da90-90b0-452f-8476-3db3bfa69ffe-podres\") pod \"perf-node-gather-daemonset-7xklb\" (UID: \"66e6da90-90b0-452f-8476-3db3bfa69ffe\") " pod="openshift-must-gather-4m86b/perf-node-gather-daemonset-7xklb" Jul 26 16:30:20.644741 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:30:20.644625 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/66e6da90-90b0-452f-8476-3db3bfa69ffe-sys\") pod \"perf-node-gather-daemonset-7xklb\" (UID: \"66e6da90-90b0-452f-8476-3db3bfa69ffe\") " pod="openshift-must-gather-4m86b/perf-node-gather-daemonset-7xklb" Jul 26 16:30:20.644741 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:30:20.644666 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/66e6da90-90b0-452f-8476-3db3bfa69ffe-lib-modules\") pod \"perf-node-gather-daemonset-7xklb\" (UID: \"66e6da90-90b0-452f-8476-3db3bfa69ffe\") " pod="openshift-must-gather-4m86b/perf-node-gather-daemonset-7xklb" Jul 26 16:30:20.644741 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:30:20.644700 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/66e6da90-90b0-452f-8476-3db3bfa69ffe-proc\") pod \"perf-node-gather-daemonset-7xklb\" (UID: \"66e6da90-90b0-452f-8476-3db3bfa69ffe\") " pod="openshift-must-gather-4m86b/perf-node-gather-daemonset-7xklb" Jul 26 16:30:20.644741 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:30:20.644728 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jccl9\" (UniqueName: \"kubernetes.io/projected/66e6da90-90b0-452f-8476-3db3bfa69ffe-kube-api-access-jccl9\") pod \"perf-node-gather-daemonset-7xklb\" (UID: \"66e6da90-90b0-452f-8476-3db3bfa69ffe\") " pod="openshift-must-gather-4m86b/perf-node-gather-daemonset-7xklb" Jul 26 16:30:20.745735 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:30:20.745670 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jccl9\" (UniqueName: \"kubernetes.io/projected/66e6da90-90b0-452f-8476-3db3bfa69ffe-kube-api-access-jccl9\") pod \"perf-node-gather-daemonset-7xklb\" (UID: \"66e6da90-90b0-452f-8476-3db3bfa69ffe\") " pod="openshift-must-gather-4m86b/perf-node-gather-daemonset-7xklb" Jul 26 16:30:20.745935 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:30:20.745778 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/66e6da90-90b0-452f-8476-3db3bfa69ffe-podres\") pod \"perf-node-gather-daemonset-7xklb\" (UID: \"66e6da90-90b0-452f-8476-3db3bfa69ffe\") " pod="openshift-must-gather-4m86b/perf-node-gather-daemonset-7xklb" Jul 26 16:30:20.745935 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:30:20.745869 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/66e6da90-90b0-452f-8476-3db3bfa69ffe-sys\") pod \"perf-node-gather-daemonset-7xklb\" (UID: \"66e6da90-90b0-452f-8476-3db3bfa69ffe\") " pod="openshift-must-gather-4m86b/perf-node-gather-daemonset-7xklb" Jul 26 16:30:20.745935 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:30:20.745912 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/66e6da90-90b0-452f-8476-3db3bfa69ffe-lib-modules\") pod \"perf-node-gather-daemonset-7xklb\" (UID: \"66e6da90-90b0-452f-8476-3db3bfa69ffe\") " pod="openshift-must-gather-4m86b/perf-node-gather-daemonset-7xklb" Jul 26 16:30:20.746116 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:30:20.745945 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/66e6da90-90b0-452f-8476-3db3bfa69ffe-proc\") pod \"perf-node-gather-daemonset-7xklb\" (UID: \"66e6da90-90b0-452f-8476-3db3bfa69ffe\") " pod="openshift-must-gather-4m86b/perf-node-gather-daemonset-7xklb" Jul 26 16:30:20.746116 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:30:20.746055 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/66e6da90-90b0-452f-8476-3db3bfa69ffe-proc\") pod \"perf-node-gather-daemonset-7xklb\" (UID: \"66e6da90-90b0-452f-8476-3db3bfa69ffe\") " pod="openshift-must-gather-4m86b/perf-node-gather-daemonset-7xklb" Jul 26 16:30:20.746567 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:30:20.746541 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/66e6da90-90b0-452f-8476-3db3bfa69ffe-podres\") pod \"perf-node-gather-daemonset-7xklb\" (UID: \"66e6da90-90b0-452f-8476-3db3bfa69ffe\") " pod="openshift-must-gather-4m86b/perf-node-gather-daemonset-7xklb" Jul 26 16:30:20.746687 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:30:20.746607 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/66e6da90-90b0-452f-8476-3db3bfa69ffe-sys\") pod \"perf-node-gather-daemonset-7xklb\" (UID: \"66e6da90-90b0-452f-8476-3db3bfa69ffe\") " pod="openshift-must-gather-4m86b/perf-node-gather-daemonset-7xklb" Jul 26 16:30:20.746755 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:30:20.746691 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/66e6da90-90b0-452f-8476-3db3bfa69ffe-lib-modules\") pod \"perf-node-gather-daemonset-7xklb\" (UID: \"66e6da90-90b0-452f-8476-3db3bfa69ffe\") " pod="openshift-must-gather-4m86b/perf-node-gather-daemonset-7xklb" Jul 26 16:30:20.756352 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:30:20.756320 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jccl9\" (UniqueName: \"kubernetes.io/projected/66e6da90-90b0-452f-8476-3db3bfa69ffe-kube-api-access-jccl9\") pod \"perf-node-gather-daemonset-7xklb\" (UID: \"66e6da90-90b0-452f-8476-3db3bfa69ffe\") " pod="openshift-must-gather-4m86b/perf-node-gather-daemonset-7xklb" Jul 26 16:30:20.863360 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:30:20.863262 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4m86b/perf-node-gather-daemonset-7xklb" Jul 26 16:30:20.956794 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:30:20.956765 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-74c9f884b7-5pdf8_3d7ef445-5c4a-4c4e-92f4-2da8ee29b868/networking-console-plugin/0.log" Jul 26 16:30:21.063923 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:30:21.063244 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4m86b/perf-node-gather-daemonset-7xklb"] Jul 26 16:30:21.541440 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:30:21.541380 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4m86b/perf-node-gather-daemonset-7xklb" event={"ID":"66e6da90-90b0-452f-8476-3db3bfa69ffe","Type":"ContainerStarted","Data":"886c7c3f6a0ae37aec36efd8d96aa7dfa8d23fa01767cfd1fd0cbbe4cb0d0ead"} Jul 26 16:30:21.541652 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:30:21.541465 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4m86b/perf-node-gather-daemonset-7xklb" event={"ID":"66e6da90-90b0-452f-8476-3db3bfa69ffe","Type":"ContainerStarted","Data":"9e4e590f8579c56c938b739a299fbbc04ff57a4b0dc186055784bc6790ee7e6d"} Jul 26 16:30:21.541728 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:30:21.541671 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-4m86b/perf-node-gather-daemonset-7xklb" Jul 26 16:30:21.559630 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:30:21.559578 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-4m86b/perf-node-gather-daemonset-7xklb" podStartSLOduration=1.559563071 podStartE2EDuration="1.559563071s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-26 16:30:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 16:30:21.557500258 +0000 UTC m=+3144.296853591" watchObservedRunningTime="2026-07-26 16:30:21.559563071 +0000 UTC m=+3144.298916399" Jul 26 16:30:21.594404 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:30:21.594352 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-8wjqp_b6b8080d-0cee-4d0c-a05c-3e2f1b672a54/console-operator/1.log" Jul 26 16:30:21.606179 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:30:21.606147 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-8wjqp_b6b8080d-0cee-4d0c-a05c-3e2f1b672a54/console-operator/2.log" Jul 26 16:30:22.234452 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:30:22.234416 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-7c97dc4555-rt6ww_3a938ae2-0e14-4c59-8347-52d26294c17f/download-server/0.log" Jul 26 16:30:22.825686 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:30:22.825659 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-549c6d54cf-6s9pz_cb333fda-51e9-4488-acb3-85b027240581/volume-data-source-validator/0.log" Jul 26 16:30:23.896269 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:30:23.896235 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-65h7n_839598d1-d772-47cf-a460-5e88affbac0e/dns/0.log" Jul 26 16:30:23.922091 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:30:23.922061 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-65h7n_839598d1-d772-47cf-a460-5e88affbac0e/kube-rbac-proxy/0.log" Jul 26 16:30:24.105363 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:30:24.105328 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-v4mgw_356a056a-d4d5-48ed-a10f-928b36becc81/dns-node-resolver/0.log" Jul 26 16:30:24.753284 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:30:24.753245 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-mfwqb_098dd2e5-9540-425f-ba4e-71ea0b064be7/node-ca/0.log" Jul 26 16:30:25.778003 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:30:25.777971 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-rx9js_b03798f1-6dbf-4cee-bf15-062b468c5206/istio-proxy/0.log" Jul 26 16:30:26.124533 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:30:26.124384 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-596h4_34f385df-2d11-4b82-ad20-404ee0104f33/istio-proxy/0.log" Jul 26 16:30:26.181966 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:30:26.181929 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-596h4_34f385df-2d11-4b82-ad20-404ee0104f33/istio-proxy/1.log" Jul 26 16:30:26.299251 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:30:26.299218 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5b79d74577-h8slk_9af43da1-c25b-4687-820f-6a95afe88786/router/0.log" Jul 26 16:30:27.192034 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:30:27.192000 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-5xf7l_fceea9e2-8f78-4c66-a5b4-3ed7666a93af/serve-healthcheck-canary/0.log" Jul 26 16:30:27.563219 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:30:27.563194 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-4m86b/perf-node-gather-daemonset-7xklb" Jul 26 16:30:27.732795 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:30:27.732763 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5885fdb759-7qj5q_d48d98c5-9029-4b14-ab01-2234e4ecc1f7/insights-operator/0.log" Jul 26 16:30:27.733898 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:30:27.733872 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5885fdb759-7qj5q_d48d98c5-9029-4b14-ab01-2234e4ecc1f7/insights-operator/1.log" Jul 26 16:30:27.757509 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:30:27.757470 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5q8xh_76cd20bd-3a66-40ad-99c0-fefa173500d6/kube-rbac-proxy/0.log" Jul 26 16:30:27.785110 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:30:27.785083 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5q8xh_76cd20bd-3a66-40ad-99c0-fefa173500d6/exporter/0.log" Jul 26 16:30:27.809080 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:30:27.809051 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5q8xh_76cd20bd-3a66-40ad-99c0-fefa173500d6/extractor/0.log" Jul 26 16:30:31.128819 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:30:31.128792 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-mblzh_f230bc42-6931-4613-ac67-8e30d1b81094/server/0.log" Jul 26 16:30:31.268561 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:30:31.268529 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-rg7qh_f24e879c-5719-4461-a065-c15b3bfc8513/manager/0.log" Jul 26 16:30:31.293927 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:30:31.293891 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-66d4685587-5rddj_881b4d20-4178-492f-8183-7d1011be3602/manager/0.log" Jul 26 16:30:31.315662 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:30:31.315635 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-66d4685587-5rddj_881b4d20-4178-492f-8183-7d1011be3602/copy-manifests/0.log" Jul 26 16:30:31.507822 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:30:31.507743 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-rx9js_b03798f1-6dbf-4cee-bf15-062b468c5206/istio-proxy/0.log" Jul 26 16:30:31.828580 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:30:31.828501 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-596h4_34f385df-2d11-4b82-ad20-404ee0104f33/istio-proxy/0.log" Jul 26 16:30:31.881022 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:30:31.880993 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-596h4_34f385df-2d11-4b82-ad20-404ee0104f33/istio-proxy/1.log" Jul 26 16:30:32.009238 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:30:32.009206 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5b79d74577-h8slk_9af43da1-c25b-4687-820f-6a95afe88786/router/0.log" Jul 26 16:30:32.861233 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:30:32.861146 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-54c98bf85c-znvqm_534eb529-23ca-407a-9c54-bae5531d8360/manager/0.log" Jul 26 16:30:33.365110 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:30:33.365081 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-rx9js_b03798f1-6dbf-4cee-bf15-062b468c5206/istio-proxy/0.log" Jul 26 16:30:33.690979 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:30:33.690890 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-596h4_34f385df-2d11-4b82-ad20-404ee0104f33/istio-proxy/0.log" Jul 26 16:30:33.753903 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:30:33.753875 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-596h4_34f385df-2d11-4b82-ad20-404ee0104f33/istio-proxy/1.log" Jul 26 16:30:33.831241 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:30:33.831199 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5b79d74577-h8slk_9af43da1-c25b-4687-820f-6a95afe88786/router/0.log" Jul 26 16:30:38.077901 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:30:38.077852 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-tzgd6_e6f72321-bfbd-415c-9e47-cb4ef4fa0d46/kube-storage-version-migrator-operator/1.log" Jul 26 16:30:38.079031 ip-10-0-138-163 kubenswrapper[2577]: I0726 16:30:38.079000 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-tzgd6_e6f72321-bfbd-415c-9e47-cb4ef4fa0d46/kube-storage-version-migrator-operator/0.log"