Jul 27 18:26:42.756610 ip-10-0-128-32 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 27 18:26:42.756625 ip-10-0-128-32 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 27 18:26:42.756635 ip-10-0-128-32 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 27 18:26:42.756999 ip-10-0-128-32 systemd[1]: Failed to start Kubernetes Kubelet. Jul 27 18:26:52.481806 ip-10-0-128-32 systemd[1]: kubelet.service: Failed to schedule restart job: Unit openvswitch.service not found. Jul 27 18:26:52.481818 ip-10-0-128-32 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 3e917d5eeded40fdbf2481eb7a5c43d6 -- Jul 27 18:28:43.377113 ip-10-0-128-32 systemd[1]: Starting Kubernetes Kubelet... Jul 27 18:28:43.827206 ip-10-0-128-32 kubenswrapper[2579]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 27 18:28:43.827206 ip-10-0-128-32 kubenswrapper[2579]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 27 18:28:43.827206 ip-10-0-128-32 kubenswrapper[2579]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 27 18:28:43.827206 ip-10-0-128-32 kubenswrapper[2579]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 27 18:28:43.827206 ip-10-0-128-32 kubenswrapper[2579]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 27 18:28:43.828256 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.828106 2579 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 27 18:28:43.832554 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.832532 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 27 18:28:43.832554 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.832550 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 27 18:28:43.832554 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.832555 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 27 18:28:43.832554 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.832559 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 27 18:28:43.832808 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.832565 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 27 18:28:43.832808 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.832569 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 27 18:28:43.832808 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.832573 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 27 18:28:43.832808 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.832577 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 27 18:28:43.832808 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.832582 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 27 18:28:43.832808 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.832586 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 27 18:28:43.832808 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.832589 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 27 18:28:43.832808 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.832593 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 27 18:28:43.832808 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.832597 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 27 18:28:43.832808 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.832601 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 27 18:28:43.832808 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.832606 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 27 18:28:43.832808 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.832609 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 27 18:28:43.832808 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.832613 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 27 18:28:43.832808 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.832617 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 27 18:28:43.832808 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.832621 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 27 18:28:43.832808 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.832625 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 27 18:28:43.832808 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.832631 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 27 18:28:43.832808 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.832637 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 27 18:28:43.832808 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.832642 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 27 18:28:43.833623 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.832646 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 27 18:28:43.833623 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.832651 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 27 18:28:43.833623 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.832656 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 27 18:28:43.833623 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.832665 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 27 18:28:43.833623 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.832670 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 27 18:28:43.833623 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.832674 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 27 18:28:43.833623 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.832679 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 27 18:28:43.833623 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.832684 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 27 18:28:43.833623 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.832689 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 27 18:28:43.833623 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.832694 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 27 18:28:43.833623 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.832698 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 27 18:28:43.833623 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.832706 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 27 18:28:43.833623 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.832713 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 27 18:28:43.833623 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.832718 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 27 18:28:43.833623 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.832723 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 27 18:28:43.833623 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.832728 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 27 18:28:43.833623 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.832732 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 27 18:28:43.833623 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.832737 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 27 18:28:43.833623 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.832742 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 27 18:28:43.834521 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.832746 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 27 18:28:43.834521 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.832750 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 27 18:28:43.834521 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.832755 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 27 18:28:43.834521 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.832760 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 27 18:28:43.834521 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.832764 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 27 18:28:43.834521 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.832768 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 27 18:28:43.834521 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.832772 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 27 18:28:43.834521 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.832776 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 27 18:28:43.834521 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.832781 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 27 18:28:43.834521 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.832786 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 27 18:28:43.834521 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.832790 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 27 18:28:43.834521 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.832794 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 27 18:28:43.834521 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.832799 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 27 18:28:43.834521 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.832805 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 27 18:28:43.834521 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.832811 2579 feature_gate.go:328] unrecognized feature gate: Example Jul 27 18:28:43.834521 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.832816 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 27 18:28:43.834521 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.832839 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 27 18:28:43.834521 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.832845 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 27 18:28:43.834521 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.832850 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 27 18:28:43.834521 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.832856 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 27 18:28:43.835418 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.832860 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 27 18:28:43.835418 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.832866 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 27 18:28:43.835418 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.832872 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 27 18:28:43.835418 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.832877 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 27 18:28:43.835418 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.832882 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 27 18:28:43.835418 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.832886 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 27 18:28:43.835418 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.832891 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 27 18:28:43.835418 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.832896 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 27 18:28:43.835418 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.832901 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 27 18:28:43.835418 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.832905 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 27 18:28:43.835418 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.832909 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 27 18:28:43.835418 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.832914 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 27 18:28:43.835418 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.832918 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 27 18:28:43.835418 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.832923 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 27 18:28:43.835418 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.832928 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 27 18:28:43.835418 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.832933 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 27 18:28:43.835418 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.832938 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 27 18:28:43.835418 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.832942 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 27 18:28:43.835418 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.832946 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 27 18:28:43.835418 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.832950 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 27 18:28:43.836296 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.832955 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 27 18:28:43.836296 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.832959 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 27 18:28:43.836296 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.832964 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 27 18:28:43.836296 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.832968 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 27 18:28:43.836296 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.832972 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 27 18:28:43.836296 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.832977 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 27 18:28:43.836296 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.832981 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 27 18:28:43.836296 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.832985 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 27 18:28:43.836296 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.832989 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 27 18:28:43.836296 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.832994 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 27 18:28:43.836296 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.832998 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 27 18:28:43.836296 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.833003 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 27 18:28:43.836296 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.833007 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 27 18:28:43.836296 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.833012 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 27 18:28:43.836296 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.833016 2579 feature_gate.go:328] unrecognized feature gate: Example2 Jul 27 18:28:43.836296 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.833020 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 27 18:28:43.836296 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.833026 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 27 18:28:43.836296 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.834236 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 27 18:28:43.836296 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.834247 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 27 18:28:43.836296 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.834252 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 27 18:28:43.837181 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.834257 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 27 18:28:43.837181 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.834261 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 27 18:28:43.837181 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.834265 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 27 18:28:43.837181 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.834270 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 27 18:28:43.837181 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.834274 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 27 18:28:43.837181 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.834278 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 27 18:28:43.837181 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.834284 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 27 18:28:43.837181 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.834290 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 27 18:28:43.837181 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.834294 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 27 18:28:43.837181 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.834299 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 27 18:28:43.837181 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.834303 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 27 18:28:43.837181 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.834307 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 27 18:28:43.837181 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.834312 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 27 18:28:43.837181 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.834316 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 27 18:28:43.837181 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.834321 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 27 18:28:43.837181 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.834326 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 27 18:28:43.837181 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.834330 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 27 18:28:43.837181 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.834335 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 27 18:28:43.837181 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.834339 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 27 18:28:43.837181 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.834344 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 27 18:28:43.837792 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.834348 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 27 18:28:43.837792 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.834352 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 27 18:28:43.837792 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.834356 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 27 18:28:43.837792 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.834361 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 27 18:28:43.837792 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.834366 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 27 18:28:43.837792 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.834370 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 27 18:28:43.837792 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.834375 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 27 18:28:43.837792 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.834379 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 27 18:28:43.837792 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.834383 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 27 18:28:43.837792 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.834387 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 27 18:28:43.837792 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.834391 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 27 18:28:43.837792 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.834396 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 27 18:28:43.837792 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.834401 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 27 18:28:43.837792 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.834405 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 27 18:28:43.837792 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.834410 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 27 18:28:43.837792 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.834415 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 27 18:28:43.837792 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.834419 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 27 18:28:43.837792 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.834424 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 27 18:28:43.837792 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.834428 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 27 18:28:43.837792 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.834434 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 27 18:28:43.838363 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.834438 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 27 18:28:43.838363 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.834444 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 27 18:28:43.838363 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.834449 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 27 18:28:43.838363 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.834453 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 27 18:28:43.838363 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.834458 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 27 18:28:43.838363 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.834462 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 27 18:28:43.838363 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.834466 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 27 18:28:43.838363 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.834471 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 27 18:28:43.838363 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.834478 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 27 18:28:43.838363 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.834484 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 27 18:28:43.838363 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.834490 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 27 18:28:43.838363 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.834494 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 27 18:28:43.838363 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.834499 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 27 18:28:43.838363 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.834503 2579 feature_gate.go:328] unrecognized feature gate: Example Jul 27 18:28:43.838363 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.834507 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 27 18:28:43.838363 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.834512 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 27 18:28:43.838363 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.834516 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 27 18:28:43.838363 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.834521 2579 feature_gate.go:328] unrecognized feature gate: Example2 Jul 27 18:28:43.838363 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.834525 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 27 18:28:43.838363 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.834529 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 27 18:28:43.838969 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.834533 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 27 18:28:43.838969 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.834537 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 27 18:28:43.838969 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.834541 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 27 18:28:43.838969 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.834546 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 27 18:28:43.838969 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.834550 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 27 18:28:43.838969 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.834554 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 27 18:28:43.838969 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.834559 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 27 18:28:43.838969 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.834563 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 27 18:28:43.838969 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.834569 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 27 18:28:43.838969 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.834576 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 27 18:28:43.838969 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.834583 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 27 18:28:43.838969 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.834589 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 27 18:28:43.838969 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.834595 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 27 18:28:43.838969 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.834600 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 27 18:28:43.838969 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.834605 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 27 18:28:43.838969 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.834609 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 27 18:28:43.838969 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.834614 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 27 18:28:43.838969 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.834619 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 27 18:28:43.838969 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.834624 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 27 18:28:43.839655 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.834629 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 27 18:28:43.839655 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.834634 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 27 18:28:43.839655 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.834638 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 27 18:28:43.839655 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.834642 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 27 18:28:43.839655 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.834647 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 27 18:28:43.839655 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.834651 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 27 18:28:43.839655 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.834655 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 27 18:28:43.839655 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.834659 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 27 18:28:43.839655 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.834664 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 27 18:28:43.839655 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.834669 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 27 18:28:43.839655 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.834674 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 27 18:28:43.839655 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.834678 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 27 18:28:43.839655 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.834682 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 27 18:28:43.839655 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.834686 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 27 18:28:43.839655 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.834690 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 27 18:28:43.839655 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.834695 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 27 18:28:43.839655 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.834699 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 27 18:28:43.839655 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.835291 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 27 18:28:43.839655 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.835299 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 27 18:28:43.840154 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.835304 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 27 18:28:43.840154 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.835309 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 27 18:28:43.840154 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.835314 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 27 18:28:43.840154 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.835318 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 27 18:28:43.840154 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.835323 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 27 18:28:43.840154 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.835327 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 27 18:28:43.840154 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.835332 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 27 18:28:43.840154 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.835337 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 27 18:28:43.840154 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.835341 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 27 18:28:43.840154 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.835346 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 27 18:28:43.840154 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.835350 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 27 18:28:43.840154 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.835354 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 27 18:28:43.840154 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.835358 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 27 18:28:43.840154 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.835363 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 27 18:28:43.840154 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.835367 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 27 18:28:43.840154 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.835372 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 27 18:28:43.840154 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.835379 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 27 18:28:43.840154 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.835385 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 27 18:28:43.840154 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.835392 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 27 18:28:43.840639 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.835398 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 27 18:28:43.840639 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.835402 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 27 18:28:43.840639 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.835410 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 27 18:28:43.840639 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.835415 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 27 18:28:43.840639 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.835420 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 27 18:28:43.840639 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.835424 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 27 18:28:43.840639 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.835428 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 27 18:28:43.840639 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.835433 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 27 18:28:43.840639 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.835438 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 27 18:28:43.840639 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.835442 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 27 18:28:43.840639 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.835447 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 27 18:28:43.840639 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.835452 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 27 18:28:43.840639 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.835456 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 27 18:28:43.840639 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.835460 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 27 18:28:43.840639 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.835465 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 27 18:28:43.840639 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.835469 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 27 18:28:43.840639 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.835473 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 27 18:28:43.840639 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.835478 2579 feature_gate.go:328] unrecognized feature gate: Example Jul 27 18:28:43.840639 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.835482 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 27 18:28:43.840639 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.835486 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 27 18:28:43.841313 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.835491 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 27 18:28:43.841313 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.835496 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 27 18:28:43.841313 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.835500 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 27 18:28:43.841313 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.835506 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 27 18:28:43.841313 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.835510 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 27 18:28:43.841313 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.835514 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 27 18:28:43.841313 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.835518 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 27 18:28:43.841313 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.835523 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 27 18:28:43.841313 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.835527 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 27 18:28:43.841313 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.835531 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 27 18:28:43.841313 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.835535 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 27 18:28:43.841313 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.835540 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 27 18:28:43.841313 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.835544 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 27 18:28:43.841313 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.835548 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 27 18:28:43.841313 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.835555 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 27 18:28:43.841313 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.835559 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 27 18:28:43.841313 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.835563 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 27 18:28:43.841313 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.835567 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 27 18:28:43.841313 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.835572 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 27 18:28:43.841786 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.835577 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 27 18:28:43.841786 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.835581 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 27 18:28:43.841786 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.835585 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 27 18:28:43.841786 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.835590 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 27 18:28:43.841786 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.835595 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 27 18:28:43.841786 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.835601 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 27 18:28:43.841786 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.835605 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 27 18:28:43.841786 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.835609 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 27 18:28:43.841786 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.835614 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 27 18:28:43.841786 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.835618 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 27 18:28:43.841786 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.835622 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 27 18:28:43.841786 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.835627 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 27 18:28:43.841786 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.835631 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 27 18:28:43.841786 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.835635 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 27 18:28:43.841786 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.835642 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 27 18:28:43.841786 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.835647 2579 feature_gate.go:328] unrecognized feature gate: Example2 Jul 27 18:28:43.841786 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.835651 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 27 18:28:43.841786 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.835655 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 27 18:28:43.841786 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.835659 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 27 18:28:43.841786 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.835663 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 27 18:28:43.842301 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.835668 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 27 18:28:43.842301 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.835672 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 27 18:28:43.842301 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.835676 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 27 18:28:43.842301 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.835681 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 27 18:28:43.842301 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.835685 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 27 18:28:43.842301 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.835689 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 27 18:28:43.842301 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.835693 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 27 18:28:43.842301 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.835698 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 27 18:28:43.842301 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.835703 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 27 18:28:43.842301 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.835707 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 27 18:28:43.842301 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.835711 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 27 18:28:43.842301 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.835716 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 27 18:28:43.842301 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.835721 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 27 18:28:43.842301 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.835725 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 27 18:28:43.842301 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.835729 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 27 18:28:43.842301 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.835734 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 27 18:28:43.842301 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.835738 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 27 18:28:43.842301 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.835742 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 27 18:28:43.842301 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.835746 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 27 18:28:43.842301 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.835865 2579 flags.go:64] FLAG: --address="0.0.0.0" Jul 27 18:28:43.842772 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.835877 2579 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 27 18:28:43.842772 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.835887 2579 flags.go:64] FLAG: --anonymous-auth="true" Jul 27 18:28:43.842772 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.835895 2579 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 27 18:28:43.842772 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.835902 2579 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 27 18:28:43.842772 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.835907 2579 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 27 18:28:43.842772 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.835914 2579 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 27 18:28:43.842772 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.835921 2579 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 27 18:28:43.842772 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.835928 2579 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 27 18:28:43.842772 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.835934 2579 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 27 18:28:43.842772 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.835940 2579 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 27 18:28:43.842772 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.835945 2579 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 27 18:28:43.842772 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.835950 2579 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 27 18:28:43.842772 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.835956 2579 flags.go:64] FLAG: --cgroup-root="" Jul 27 18:28:43.842772 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.835960 2579 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 27 18:28:43.842772 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.835965 2579 flags.go:64] FLAG: --client-ca-file="" Jul 27 18:28:43.842772 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.835970 2579 flags.go:64] FLAG: --cloud-provider="external" Jul 27 18:28:43.842772 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.835975 2579 flags.go:64] FLAG: --cluster-dns="[]" Jul 27 18:28:43.842772 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.835984 2579 flags.go:64] FLAG: --cluster-domain="" Jul 27 18:28:43.842772 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.835989 2579 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 27 18:28:43.842772 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.835997 2579 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 27 18:28:43.842772 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.836003 2579 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 27 18:28:43.842772 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.836009 2579 flags.go:64] FLAG: --container-log-max-files="5" Jul 27 18:28:43.842772 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.836016 2579 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 27 18:28:43.842772 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.836021 2579 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 27 18:28:43.843369 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.836026 2579 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 27 18:28:43.843369 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.836032 2579 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 27 18:28:43.843369 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.836038 2579 flags.go:64] FLAG: --contention-profiling="false" Jul 27 18:28:43.843369 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.836043 2579 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 27 18:28:43.843369 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.836047 2579 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 27 18:28:43.843369 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.836052 2579 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 27 18:28:43.843369 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.836057 2579 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 27 18:28:43.843369 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.836065 2579 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 27 18:28:43.843369 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.836070 2579 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 27 18:28:43.843369 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.836075 2579 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 27 18:28:43.843369 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.836080 2579 flags.go:64] FLAG: --enable-load-reader="false" Jul 27 18:28:43.843369 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.836086 2579 flags.go:64] FLAG: --enable-server="true" Jul 27 18:28:43.843369 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.836090 2579 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 27 18:28:43.843369 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.836098 2579 flags.go:64] FLAG: --event-burst="100" Jul 27 18:28:43.843369 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.836103 2579 flags.go:64] FLAG: --event-qps="50" Jul 27 18:28:43.843369 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.836108 2579 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 27 18:28:43.843369 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.836114 2579 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 27 18:28:43.843369 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.836119 2579 flags.go:64] FLAG: --eviction-hard="" Jul 27 18:28:43.843369 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.836126 2579 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 27 18:28:43.843369 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.836131 2579 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 27 18:28:43.843369 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.836137 2579 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 27 18:28:43.843369 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.836142 2579 flags.go:64] FLAG: --eviction-soft="" Jul 27 18:28:43.843369 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.836147 2579 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 27 18:28:43.843369 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.836152 2579 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 27 18:28:43.843369 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.836157 2579 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 27 18:28:43.844063 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.836162 2579 flags.go:64] FLAG: --experimental-mounter-path="" Jul 27 18:28:43.844063 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.836167 2579 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 27 18:28:43.844063 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.836173 2579 flags.go:64] FLAG: --fail-swap-on="true" Jul 27 18:28:43.844063 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.836178 2579 flags.go:64] FLAG: --feature-gates="" Jul 27 18:28:43.844063 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.836185 2579 flags.go:64] FLAG: --file-check-frequency="20s" Jul 27 18:28:43.844063 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.836190 2579 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 27 18:28:43.844063 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.836195 2579 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 27 18:28:43.844063 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.836200 2579 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 27 18:28:43.844063 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.836206 2579 flags.go:64] FLAG: --healthz-port="10248" Jul 27 18:28:43.844063 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.836211 2579 flags.go:64] FLAG: --help="false" Jul 27 18:28:43.844063 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.836216 2579 flags.go:64] FLAG: --hostname-override="ip-10-0-128-32.ec2.internal" Jul 27 18:28:43.844063 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.836221 2579 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 27 18:28:43.844063 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.836226 2579 flags.go:64] FLAG: --http-check-frequency="20s" Jul 27 18:28:43.844063 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.836231 2579 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 27 18:28:43.844063 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.836237 2579 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 27 18:28:43.844063 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.836243 2579 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 27 18:28:43.844063 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.836248 2579 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 27 18:28:43.844063 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.836252 2579 flags.go:64] FLAG: --image-service-endpoint="" Jul 27 18:28:43.844063 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.836257 2579 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 27 18:28:43.844063 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.836262 2579 flags.go:64] FLAG: --kube-api-burst="100" Jul 27 18:28:43.844063 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.836267 2579 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 27 18:28:43.844063 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.836273 2579 flags.go:64] FLAG: --kube-api-qps="50" Jul 27 18:28:43.844063 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.836280 2579 flags.go:64] FLAG: --kube-reserved="" Jul 27 18:28:43.844063 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.836285 2579 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 27 18:28:43.844653 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.836291 2579 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 27 18:28:43.844653 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.836296 2579 flags.go:64] FLAG: --kubelet-cgroups="" Jul 27 18:28:43.844653 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.836301 2579 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 27 18:28:43.844653 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.836306 2579 flags.go:64] FLAG: --lock-file="" Jul 27 18:28:43.844653 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.836310 2579 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 27 18:28:43.844653 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.836316 2579 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 27 18:28:43.844653 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.836320 2579 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 27 18:28:43.844653 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.836331 2579 flags.go:64] FLAG: --log-json-split-stream="false" Jul 27 18:28:43.844653 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.836336 2579 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 27 18:28:43.844653 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.836341 2579 flags.go:64] FLAG: --log-text-split-stream="false" Jul 27 18:28:43.844653 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.836348 2579 flags.go:64] FLAG: --logging-format="text" Jul 27 18:28:43.844653 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.836353 2579 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 27 18:28:43.844653 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.836359 2579 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 27 18:28:43.844653 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.836364 2579 flags.go:64] FLAG: --manifest-url="" Jul 27 18:28:43.844653 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.836368 2579 flags.go:64] FLAG: --manifest-url-header="" Jul 27 18:28:43.844653 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.836376 2579 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 27 18:28:43.844653 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.836382 2579 flags.go:64] FLAG: --max-open-files="1000000" Jul 27 18:28:43.844653 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.836396 2579 flags.go:64] FLAG: --max-pods="110" Jul 27 18:28:43.844653 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.836401 2579 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 27 18:28:43.844653 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.836406 2579 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 27 18:28:43.844653 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.836410 2579 flags.go:64] FLAG: --memory-manager-policy="None" Jul 27 18:28:43.844653 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.836415 2579 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 27 18:28:43.844653 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.836421 2579 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 27 18:28:43.844653 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.836426 2579 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 27 18:28:43.844653 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.836431 2579 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 27 18:28:43.845292 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.836444 2579 flags.go:64] FLAG: --node-status-max-images="50" Jul 27 18:28:43.845292 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.836449 2579 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 27 18:28:43.845292 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.836453 2579 flags.go:64] FLAG: --oom-score-adj="-999" Jul 27 18:28:43.845292 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.836459 2579 flags.go:64] FLAG: --pod-cidr="" Jul 27 18:28:43.845292 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.836463 2579 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb" Jul 27 18:28:43.845292 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.836472 2579 flags.go:64] FLAG: --pod-manifest-path="" Jul 27 18:28:43.845292 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.836478 2579 flags.go:64] FLAG: --pod-max-pids="-1" Jul 27 18:28:43.845292 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.836484 2579 flags.go:64] FLAG: --pods-per-core="0" Jul 27 18:28:43.845292 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.836490 2579 flags.go:64] FLAG: --port="10250" Jul 27 18:28:43.845292 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.836495 2579 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 27 18:28:43.845292 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.836500 2579 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-063bb647f50d19748" Jul 27 18:28:43.845292 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.836505 2579 flags.go:64] FLAG: --qos-reserved="" Jul 27 18:28:43.845292 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.836511 2579 flags.go:64] FLAG: --read-only-port="10255" Jul 27 18:28:43.845292 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.836516 2579 flags.go:64] FLAG: --register-node="true" Jul 27 18:28:43.845292 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.836521 2579 flags.go:64] FLAG: --register-with-taints="" Jul 27 18:28:43.845292 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.836527 2579 flags.go:64] FLAG: --registry-burst="10" Jul 27 18:28:43.845292 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.836532 2579 flags.go:64] FLAG: --registry-qps="5" Jul 27 18:28:43.845292 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.836538 2579 flags.go:64] FLAG: --reserved-cpus="" Jul 27 18:28:43.845292 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.836543 2579 flags.go:64] FLAG: --reserved-memory="" Jul 27 18:28:43.845292 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.836550 2579 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 27 18:28:43.845292 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.836555 2579 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 27 18:28:43.845292 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.836561 2579 flags.go:64] FLAG: --rotate-certificates="false" Jul 27 18:28:43.845292 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.836565 2579 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 27 18:28:43.845292 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.836570 2579 flags.go:64] FLAG: --runonce="false" Jul 27 18:28:43.845292 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.836575 2579 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 27 18:28:43.845925 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.836580 2579 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 27 18:28:43.845925 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.836585 2579 flags.go:64] FLAG: --seccomp-default="false" Jul 27 18:28:43.845925 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.836590 2579 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 27 18:28:43.845925 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.836595 2579 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 27 18:28:43.845925 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.836600 2579 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 27 18:28:43.845925 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.836605 2579 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 27 18:28:43.845925 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.836610 2579 flags.go:64] FLAG: --storage-driver-password="root" Jul 27 18:28:43.845925 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.836615 2579 flags.go:64] FLAG: --storage-driver-secure="false" Jul 27 18:28:43.845925 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.836620 2579 flags.go:64] FLAG: --storage-driver-table="stats" Jul 27 18:28:43.845925 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.836625 2579 flags.go:64] FLAG: --storage-driver-user="root" Jul 27 18:28:43.845925 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.836630 2579 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 27 18:28:43.845925 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.836635 2579 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 27 18:28:43.845925 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.836640 2579 flags.go:64] FLAG: --system-cgroups="" Jul 27 18:28:43.845925 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.836647 2579 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 27 18:28:43.845925 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.836656 2579 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 27 18:28:43.845925 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.836661 2579 flags.go:64] FLAG: --tls-cert-file="" Jul 27 18:28:43.845925 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.836666 2579 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 27 18:28:43.845925 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.836674 2579 flags.go:64] FLAG: --tls-min-version="" Jul 27 18:28:43.845925 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.836679 2579 flags.go:64] FLAG: --tls-private-key-file="" Jul 27 18:28:43.845925 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.836684 2579 flags.go:64] FLAG: --topology-manager-policy="none" Jul 27 18:28:43.845925 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.836689 2579 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 27 18:28:43.845925 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.836694 2579 flags.go:64] FLAG: --topology-manager-scope="container" Jul 27 18:28:43.845925 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.836699 2579 flags.go:64] FLAG: --v="2" Jul 27 18:28:43.845925 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.836706 2579 flags.go:64] FLAG: --version="false" Jul 27 18:28:43.845925 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.836716 2579 flags.go:64] FLAG: --vmodule="" Jul 27 18:28:43.846542 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.836722 2579 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 27 18:28:43.846542 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.836728 2579 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 27 18:28:43.846542 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.836932 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 27 18:28:43.846542 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.836940 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 27 18:28:43.846542 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.836945 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 27 18:28:43.846542 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.836950 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 27 18:28:43.846542 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.836954 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 27 18:28:43.846542 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.836959 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 27 18:28:43.846542 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.836963 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 27 18:28:43.846542 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.836968 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 27 18:28:43.846542 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.836972 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 27 18:28:43.846542 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.836976 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 27 18:28:43.846542 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.836980 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 27 18:28:43.846542 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.836985 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 27 18:28:43.846542 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.836991 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 27 18:28:43.846542 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.836998 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 27 18:28:43.846542 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.837002 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 27 18:28:43.846542 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.837007 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 27 18:28:43.846542 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.837011 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 27 18:28:43.846542 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.837015 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 27 18:28:43.847065 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.837024 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 27 18:28:43.847065 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.837031 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 27 18:28:43.847065 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.837036 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 27 18:28:43.847065 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.837040 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 27 18:28:43.847065 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.837046 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 27 18:28:43.847065 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.837051 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 27 18:28:43.847065 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.837055 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 27 18:28:43.847065 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.837059 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 27 18:28:43.847065 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.837065 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 27 18:28:43.847065 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.837070 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 27 18:28:43.847065 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.837075 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 27 18:28:43.847065 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.837081 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 27 18:28:43.847065 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.837086 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 27 18:28:43.847065 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.837090 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 27 18:28:43.847065 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.837094 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 27 18:28:43.847065 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.837098 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 27 18:28:43.847065 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.837103 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 27 18:28:43.847065 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.837107 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 27 18:28:43.847065 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.837112 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 27 18:28:43.847565 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.837116 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 27 18:28:43.847565 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.837121 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 27 18:28:43.847565 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.837125 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 27 18:28:43.847565 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.837129 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 27 18:28:43.847565 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.837134 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 27 18:28:43.847565 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.837138 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 27 18:28:43.847565 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.837142 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 27 18:28:43.847565 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.837146 2579 feature_gate.go:328] unrecognized feature gate: Example2 Jul 27 18:28:43.847565 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.837151 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 27 18:28:43.847565 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.837155 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 27 18:28:43.847565 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.837160 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 27 18:28:43.847565 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.837164 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 27 18:28:43.847565 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.837168 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 27 18:28:43.847565 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.837174 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 27 18:28:43.847565 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.837178 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 27 18:28:43.847565 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.837182 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 27 18:28:43.847565 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.837186 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 27 18:28:43.847565 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.837190 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 27 18:28:43.847565 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.837195 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 27 18:28:43.847565 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.837200 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 27 18:28:43.848071 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.837204 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 27 18:28:43.848071 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.837208 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 27 18:28:43.848071 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.837212 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 27 18:28:43.848071 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.837217 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 27 18:28:43.848071 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.837223 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 27 18:28:43.848071 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.837227 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 27 18:28:43.848071 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.837231 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 27 18:28:43.848071 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.837236 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 27 18:28:43.848071 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.837240 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 27 18:28:43.848071 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.837245 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 27 18:28:43.848071 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.837249 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 27 18:28:43.848071 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.837253 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 27 18:28:43.848071 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.837258 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 27 18:28:43.848071 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.837262 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 27 18:28:43.848071 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.837266 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 27 18:28:43.848071 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.837271 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 27 18:28:43.848071 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.837276 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 27 18:28:43.848071 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.837280 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 27 18:28:43.848071 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.837285 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 27 18:28:43.848071 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.837289 2579 feature_gate.go:328] unrecognized feature gate: Example Jul 27 18:28:43.848557 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.837294 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 27 18:28:43.848557 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.837298 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 27 18:28:43.848557 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.837302 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 27 18:28:43.848557 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.837307 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 27 18:28:43.848557 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.837311 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 27 18:28:43.848557 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.837317 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 27 18:28:43.848557 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.837321 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 27 18:28:43.848557 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.837325 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 27 18:28:43.848557 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.837330 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 27 18:28:43.848557 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.837334 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 27 18:28:43.848557 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.837338 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 27 18:28:43.848557 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.837343 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 27 18:28:43.848557 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.837348 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 27 18:28:43.848557 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.837354 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 27 18:28:43.848557 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.837360 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 27 18:28:43.848557 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.837365 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 27 18:28:43.848557 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.837371 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 27 18:28:43.848557 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.837375 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 27 18:28:43.848557 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.837379 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 27 18:28:43.849042 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.837384 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 27 18:28:43.849042 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.837388 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 27 18:28:43.849042 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.837393 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 27 18:28:43.849042 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.838520 2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 27 18:28:43.849042 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.845447 2579 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 27 18:28:43.849042 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.845462 2579 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 27 18:28:43.849042 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.845482 2579 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 27 18:28:43.849042 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.845491 2579 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 27 18:28:43.849042 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.845547 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 27 18:28:43.849042 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.845553 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 27 18:28:43.849042 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.845556 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 27 18:28:43.849042 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.845559 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 27 18:28:43.849042 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.845562 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 27 18:28:43.849042 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.845565 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 27 18:28:43.849042 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.845567 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 27 18:28:43.849444 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.845570 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 27 18:28:43.849444 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.845573 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 27 18:28:43.849444 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.845575 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 27 18:28:43.849444 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.845578 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 27 18:28:43.849444 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.845581 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 27 18:28:43.849444 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.845583 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 27 18:28:43.849444 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.845585 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 27 18:28:43.849444 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.845588 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 27 18:28:43.849444 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.845590 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 27 18:28:43.849444 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.845593 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 27 18:28:43.849444 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.845595 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 27 18:28:43.849444 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.845598 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 27 18:28:43.849444 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.845602 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 27 18:28:43.849444 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.845605 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 27 18:28:43.849444 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.845608 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 27 18:28:43.849444 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.845611 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 27 18:28:43.849444 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.845614 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 27 18:28:43.849444 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.845616 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 27 18:28:43.849444 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.845619 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 27 18:28:43.849992 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.845621 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 27 18:28:43.849992 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.845624 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 27 18:28:43.849992 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.845627 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 27 18:28:43.849992 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.845630 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 27 18:28:43.849992 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.845633 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 27 18:28:43.849992 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.845635 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 27 18:28:43.849992 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.845638 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 27 18:28:43.849992 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.845640 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 27 18:28:43.849992 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.845643 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 27 18:28:43.849992 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.845645 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 27 18:28:43.849992 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.845648 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 27 18:28:43.849992 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.845650 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 27 18:28:43.849992 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.845652 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 27 18:28:43.849992 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.845655 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 27 18:28:43.849992 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.845657 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 27 18:28:43.849992 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.845660 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 27 18:28:43.849992 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.845662 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 27 18:28:43.849992 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.845665 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 27 18:28:43.849992 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.845668 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 27 18:28:43.849992 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.845671 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 27 18:28:43.850520 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.845675 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 27 18:28:43.850520 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.845677 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 27 18:28:43.850520 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.845680 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 27 18:28:43.850520 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.845683 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 27 18:28:43.850520 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.845685 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 27 18:28:43.850520 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.845688 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 27 18:28:43.850520 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.845691 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 27 18:28:43.850520 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.845693 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 27 18:28:43.850520 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.845696 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 27 18:28:43.850520 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.845698 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 27 18:28:43.850520 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.845701 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 27 18:28:43.850520 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.845703 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 27 18:28:43.850520 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.845706 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 27 18:28:43.850520 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.845708 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 27 18:28:43.850520 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.845711 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 27 18:28:43.850520 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.845714 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 27 18:28:43.850520 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.845717 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 27 18:28:43.850520 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.845719 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 27 18:28:43.850520 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.845722 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 27 18:28:43.850520 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.845724 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 27 18:28:43.851041 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.845727 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 27 18:28:43.851041 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.845730 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 27 18:28:43.851041 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.845732 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 27 18:28:43.851041 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.845734 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 27 18:28:43.851041 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.845737 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 27 18:28:43.851041 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.845739 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 27 18:28:43.851041 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.845742 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 27 18:28:43.851041 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.845744 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 27 18:28:43.851041 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.845747 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 27 18:28:43.851041 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.845749 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 27 18:28:43.851041 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.845752 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 27 18:28:43.851041 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.845754 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 27 18:28:43.851041 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.845757 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 27 18:28:43.851041 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.845759 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 27 18:28:43.851041 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.845761 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 27 18:28:43.851041 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.845764 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 27 18:28:43.851041 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.845766 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 27 18:28:43.851041 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.845769 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 27 18:28:43.851041 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.845772 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 27 18:28:43.851041 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.845774 2579 feature_gate.go:328] unrecognized feature gate: Example2 Jul 27 18:28:43.851544 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.845776 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 27 18:28:43.851544 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.845779 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 27 18:28:43.851544 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.845782 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 27 18:28:43.851544 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.845784 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 27 18:28:43.851544 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.845786 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 27 18:28:43.851544 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.845789 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 27 18:28:43.851544 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.845792 2579 feature_gate.go:328] unrecognized feature gate: Example Jul 27 18:28:43.851544 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.845795 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 27 18:28:43.851544 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.845798 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 27 18:28:43.851544 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.845802 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 27 18:28:43.851544 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.845805 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 27 18:28:43.851544 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.845807 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 27 18:28:43.851544 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.845810 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 27 18:28:43.851544 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.845814 2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 27 18:28:43.851913 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.845961 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 27 18:28:43.851913 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.845966 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 27 18:28:43.851913 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.845969 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 27 18:28:43.851913 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.845972 2579 feature_gate.go:328] unrecognized feature gate: Example Jul 27 18:28:43.851913 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.845975 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 27 18:28:43.851913 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.845977 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 27 18:28:43.851913 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.845980 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 27 18:28:43.851913 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.845983 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 27 18:28:43.851913 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.845985 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 27 18:28:43.851913 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.845988 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 27 18:28:43.851913 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.845990 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 27 18:28:43.851913 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.845993 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 27 18:28:43.851913 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.845995 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 27 18:28:43.851913 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.845998 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 27 18:28:43.851913 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.846001 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 27 18:28:43.851913 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.846003 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 27 18:28:43.851913 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.846005 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 27 18:28:43.851913 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.846007 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 27 18:28:43.851913 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.846011 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 27 18:28:43.851913 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.846013 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 27 18:28:43.852398 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.846016 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 27 18:28:43.852398 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.846018 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 27 18:28:43.852398 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.846021 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 27 18:28:43.852398 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.846023 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 27 18:28:43.852398 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.846026 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 27 18:28:43.852398 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.846029 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 27 18:28:43.852398 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.846032 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 27 18:28:43.852398 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.846034 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 27 18:28:43.852398 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.846037 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 27 18:28:43.852398 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.846040 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 27 18:28:43.852398 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.846042 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 27 18:28:43.852398 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.846045 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 27 18:28:43.852398 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.846047 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 27 18:28:43.852398 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.846049 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 27 18:28:43.852398 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.846052 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 27 18:28:43.852398 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.846055 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 27 18:28:43.852398 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.846057 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 27 18:28:43.852398 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.846059 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 27 18:28:43.852398 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.846062 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 27 18:28:43.852398 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.846064 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 27 18:28:43.852877 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.846067 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 27 18:28:43.852877 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.846069 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 27 18:28:43.852877 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.846071 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 27 18:28:43.852877 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.846074 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 27 18:28:43.852877 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.846076 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 27 18:28:43.852877 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.846078 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 27 18:28:43.852877 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.846081 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 27 18:28:43.852877 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.846084 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 27 18:28:43.852877 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.846086 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 27 18:28:43.852877 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.846089 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 27 18:28:43.852877 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.846091 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 27 18:28:43.852877 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.846093 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 27 18:28:43.852877 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.846096 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 27 18:28:43.852877 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.846098 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 27 18:28:43.852877 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.846101 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 27 18:28:43.852877 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.846103 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 27 18:28:43.852877 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.846106 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 27 18:28:43.852877 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.846108 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 27 18:28:43.852877 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.846112 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 27 18:28:43.852877 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.846116 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 27 18:28:43.853360 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.846118 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 27 18:28:43.853360 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.846121 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 27 18:28:43.853360 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.846125 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 27 18:28:43.853360 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.846129 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 27 18:28:43.853360 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.846132 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 27 18:28:43.853360 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.846135 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 27 18:28:43.853360 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.846138 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 27 18:28:43.853360 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.846140 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 27 18:28:43.853360 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.846143 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 27 18:28:43.853360 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.846146 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 27 18:28:43.853360 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.846149 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 27 18:28:43.853360 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.846151 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 27 18:28:43.853360 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.846154 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 27 18:28:43.853360 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.846156 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 27 18:28:43.853360 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.846158 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 27 18:28:43.853360 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.846161 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 27 18:28:43.853360 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.846163 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 27 18:28:43.853360 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.846166 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 27 18:28:43.853360 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.846168 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 27 18:28:43.853811 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.846170 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 27 18:28:43.853811 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.846173 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 27 18:28:43.853811 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.846175 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 27 18:28:43.853811 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.846178 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 27 18:28:43.853811 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.846180 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 27 18:28:43.853811 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.846183 2579 feature_gate.go:328] unrecognized feature gate: Example2 Jul 27 18:28:43.853811 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.846185 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 27 18:28:43.853811 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.846187 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 27 18:28:43.853811 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.846190 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 27 18:28:43.853811 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.846192 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 27 18:28:43.853811 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.846194 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 27 18:28:43.853811 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.846197 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 27 18:28:43.853811 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.846199 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 27 18:28:43.853811 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.846202 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 27 18:28:43.853811 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.846204 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 27 18:28:43.853811 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.846207 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 27 18:28:43.853811 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.846209 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 27 18:28:43.853811 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.846212 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 27 18:28:43.853811 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.846214 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 27 18:28:43.854389 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:43.846218 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 27 18:28:43.854389 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.846223 2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 27 18:28:43.854389 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.846323 2579 server.go:962] "Client rotation is on, will bootstrap in background" Jul 27 18:28:43.854389 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.849171 2579 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 27 18:28:43.854389 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.850605 2579 server.go:1019] "Starting client certificate rotation" Jul 27 18:28:43.854389 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.850705 2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 27 18:28:43.854389 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.851688 2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 27 18:28:43.854389 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.854302 2579 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 27 18:28:43.858770 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.858740 2579 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 27 18:28:43.869693 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.869673 2579 log.go:25] "Validated CRI v1 runtime API" Jul 27 18:28:43.871917 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.871898 2579 log.go:25] "Validated CRI v1 image API" Jul 27 18:28:43.874322 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.874303 2579 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 27 18:28:43.877031 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.877013 2579 fs.go:135] Filesystem UUIDs: map[0f2af1aa-586e-47ce-af6e-9677eff83cb3:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 b8eb89ed-9f4f-47ba-83b7-c0ac2619afe3:/dev/nvme0n1p3] Jul 27 18:28:43.877100 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.877031 2579 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 27 18:28:43.882588 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.882479 2579 manager.go:216] Machine: {Timestamp:2026-07-27 18:28:43.881418361 +0000 UTC m=+0.376511298 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3126258 MemoryCapacity:32812175360 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2e1c53e972e39b0ee424ee812421d7 SystemUUID:ec2e1c53-e972-e39b-0ee4-24ee812421d7 BootID:3e917d5e-eded-40fd-bf24-81eb7a5c43d6 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:22:2a:ae:5d:1b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:22:2a:ae:5d:1b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:86:24:8f:f1:96:84 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812175360 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 27 18:28:43.882588 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.882578 2579 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 27 18:28:43.882687 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.882661 2579 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 27 18:28:43.883880 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.883854 2579 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 27 18:28:43.884011 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.883883 2579 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-32.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 27 18:28:43.884774 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.884764 2579 topology_manager.go:138] "Creating topology manager with none policy" Jul 27 18:28:43.884808 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.884777 2579 container_manager_linux.go:309] "Creating device plugin manager" Jul 27 18:28:43.884808 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.884784 2579 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 27 18:28:43.885405 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.885395 2579 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 27 18:28:43.885898 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.885883 2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 27 18:28:43.886247 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.886235 2579 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 27 18:28:43.887922 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.887911 2579 state_mem.go:36] "Initialized new in-memory state store" Jul 27 18:28:43.888063 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.888053 2579 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 27 18:28:43.888106 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.888085 2579 kubelet.go:486] "Attempting to sync node with API server" Jul 27 18:28:43.888106 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.888100 2579 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 27 18:28:43.888173 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.888121 2579 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 27 18:28:43.888173 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.888134 2579 kubelet.go:398] "Adding apiserver pod source" Jul 27 18:28:43.888173 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.888145 2579 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 27 18:28:43.888548 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.888536 2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 27 18:28:43.888584 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.888555 2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 27 18:28:43.890788 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.890769 2579 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 27 18:28:43.893223 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.893195 2579 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 27 18:28:43.893316 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.893240 2579 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 27 18:28:43.894807 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.894791 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 27 18:28:43.894862 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.894812 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 27 18:28:43.894862 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.894819 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 27 18:28:43.894916 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.894859 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 27 18:28:43.894916 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.894890 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 27 18:28:43.894916 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.894901 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 27 18:28:43.895014 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.894920 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 27 18:28:43.895014 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.894929 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 27 18:28:43.895014 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.894936 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 27 18:28:43.895014 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.894943 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 27 18:28:43.895014 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.894951 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 27 18:28:43.895014 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.894957 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 27 18:28:43.896187 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.895893 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 27 18:28:43.896230 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.896197 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 27 18:28:43.899387 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.899371 2579 server.go:1268] "Started kubelet" Jul 27 18:28:43.899473 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.899441 2579 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 27 18:28:43.899887 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.899518 2579 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 27 18:28:43.899887 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.899589 2579 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 27 18:28:43.900351 ip-10-0-128-32 systemd[1]: Started Kubernetes Kubelet. Jul 27 18:28:43.900886 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.900798 2579 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-128-32.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 27 18:28:43.900886 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:28:43.900802 2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-128-32.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 27 18:28:43.900886 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:28:43.900853 2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 27 18:28:43.901334 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.901314 2579 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 27 18:28:43.901644 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.901632 2579 server.go:310] "Adding debug handlers to kubelet server" Jul 27 18:28:43.904899 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:28:43.904877 2579 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 27 18:28:43.909003 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:28:43.907975 2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-32.ec2.internal.18c638f43a3ec244 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-32.ec2.internal,UID:ip-10-0-128-32.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-128-32.ec2.internal,},FirstTimestamp:2026-07-27 18:28:43.899347524 +0000 UTC m=+0.394440461,LastTimestamp:2026-07-27 18:28:43.899347524 +0000 UTC m=+0.394440461,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-32.ec2.internal,}" Jul 27 18:28:43.909908 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.909881 2579 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 27 18:28:43.909908 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.909885 2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 27 18:28:43.910126 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:28:43.910107 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-32.ec2.internal\" not found" Jul 27 18:28:43.910758 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.910740 2579 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 27 18:28:43.910758 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.910743 2579 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 27 18:28:43.910917 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.910768 2579 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 27 18:28:43.910917 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.910871 2579 reconstruct.go:97] "Volume reconstruction finished" Jul 27 18:28:43.910917 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.910881 2579 reconciler.go:29] "Reconciler: start to sync state" Jul 27 18:28:43.911045 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.910962 2579 factory.go:55] Registering systemd factory Jul 27 18:28:43.911045 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.910978 2579 factory.go:223] Registration of the systemd container factory successfully Jul 27 18:28:43.911192 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.911176 2579 factory.go:153] Registering CRI-O factory Jul 27 18:28:43.911192 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.911190 2579 factory.go:223] Registration of the crio container factory successfully Jul 27 18:28:43.911336 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.911245 2579 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 27 18:28:43.911336 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.911263 2579 factory.go:103] Registering Raw factory Jul 27 18:28:43.911336 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.911279 2579 manager.go:1196] Started watching for new ooms in manager Jul 27 18:28:43.911843 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.911812 2579 manager.go:349] Starting recovery of all containers Jul 27 18:28:43.914061 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:28:43.914038 2579 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-128-32.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 27 18:28:43.914569 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:28:43.914546 2579 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 27 18:28:43.924044 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.923885 2579 manager.go:354] Recovery completed Jul 27 18:28:43.925385 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:28:43.925366 2579 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory Jul 27 18:28:43.928453 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.928438 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 27 18:28:43.930461 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.930435 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-32.ec2.internal" event="NodeHasSufficientMemory" Jul 27 18:28:43.930543 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.930480 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-32.ec2.internal" event="NodeHasNoDiskPressure" Jul 27 18:28:43.930543 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.930495 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-32.ec2.internal" event="NodeHasSufficientPID" Jul 27 18:28:43.931032 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.931017 2579 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 27 18:28:43.931086 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.931032 2579 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 27 18:28:43.931086 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.931048 2579 state_mem.go:36] "Initialized new in-memory state store" Jul 27 18:28:43.931150 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.931023 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 27 18:28:43.931762 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.931746 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-32.ec2.internal" event="NodeHasSufficientMemory" Jul 27 18:28:43.931871 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.931773 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-32.ec2.internal" event="NodeHasNoDiskPressure" Jul 27 18:28:43.931871 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.931783 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-32.ec2.internal" event="NodeHasSufficientPID" Jul 27 18:28:43.932762 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.932749 2579 policy_none.go:49] "None policy: Start" Jul 27 18:28:43.932845 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.932770 2579 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 27 18:28:43.932845 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.932785 2579 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 27 18:28:43.933028 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:28:43.932959 2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-32.ec2.internal.18c638f43c1986fe default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-32.ec2.internal,UID:ip-10-0-128-32.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-128-32.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-128-32.ec2.internal,},FirstTimestamp:2026-07-27 18:28:43.93046195 +0000 UTC m=+0.425554890,LastTimestamp:2026-07-27 18:28:43.93046195 +0000 UTC m=+0.425554890,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-32.ec2.internal,}" Jul 27 18:28:43.933889 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.933874 2579 policy_none.go:47] "Start" Jul 27 18:28:43.941586 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:28:43.941519 2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-32.ec2.internal.18c638f43c19e92d default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-32.ec2.internal,UID:ip-10-0-128-32.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-128-32.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-128-32.ec2.internal,},FirstTimestamp:2026-07-27 18:28:43.930487085 +0000 UTC m=+0.425580025,LastTimestamp:2026-07-27 18:28:43.930487085 +0000 UTC m=+0.425580025,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-32.ec2.internal,}" Jul 27 18:28:43.944129 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:28:43.944052 2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-32.ec2.internal.18c638f43c1a1dda default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-32.ec2.internal,UID:ip-10-0-128-32.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-128-32.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-128-32.ec2.internal,},FirstTimestamp:2026-07-27 18:28:43.93050057 +0000 UTC m=+0.425593509,LastTimestamp:2026-07-27 18:28:43.93050057 +0000 UTC m=+0.425593509,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-32.ec2.internal,}" Jul 27 18:28:43.944235 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.944117 2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-lxl6s" Jul 27 18:28:43.951049 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:28:43.950952 2579 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-128-32.ec2.internal.18c638f43c1986fe\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-32.ec2.internal.18c638f43c1986fe default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-32.ec2.internal,UID:ip-10-0-128-32.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-128-32.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-128-32.ec2.internal,},FirstTimestamp:2026-07-27 18:28:43.93046195 +0000 UTC m=+0.425554890,LastTimestamp:2026-07-27 18:28:43.931759758 +0000 UTC m=+0.426852700,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-32.ec2.internal,}" Jul 27 18:28:43.953707 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:28:43.953630 2579 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-128-32.ec2.internal.18c638f43c19e92d\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-32.ec2.internal.18c638f43c19e92d default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-32.ec2.internal,UID:ip-10-0-128-32.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-128-32.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-128-32.ec2.internal,},FirstTimestamp:2026-07-27 18:28:43.930487085 +0000 UTC m=+0.425580025,LastTimestamp:2026-07-27 18:28:43.931777633 +0000 UTC m=+0.426870570,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-32.ec2.internal,}" Jul 27 18:28:43.953972 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.953951 2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-lxl6s" Jul 27 18:28:43.971409 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.971394 2579 manager.go:337] "Starting Device Plugin manager" Jul 27 18:28:43.971493 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:28:43.971430 2579 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 27 18:28:43.971493 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.971446 2579 server.go:87] "Starting device plugin registration server" Jul 27 18:28:43.971706 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.971694 2579 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 27 18:28:43.971755 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.971706 2579 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 27 18:28:43.971883 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.971803 2579 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 27 18:28:43.972286 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.972216 2579 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 27 18:28:43.972286 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:43.972235 2579 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 27 18:28:43.972473 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:28:43.972454 2579 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 27 18:28:43.972517 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:28:43.972500 2579 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-32.ec2.internal\" not found" Jul 27 18:28:44.051557 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:44.051523 2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 27 18:28:44.052857 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:44.052839 2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 27 18:28:44.052948 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:44.052883 2579 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 27 18:28:44.052948 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:44.052908 2579 kubelet.go:2448] "Starting kubelet main sync loop" Jul 27 18:28:44.053039 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:28:44.052959 2579 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 27 18:28:44.055286 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:44.055264 2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 27 18:28:44.072157 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:44.072135 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 27 18:28:44.072992 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:44.072974 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-32.ec2.internal" event="NodeHasSufficientMemory" Jul 27 18:28:44.073072 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:44.073002 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-32.ec2.internal" event="NodeHasNoDiskPressure" Jul 27 18:28:44.073072 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:44.073011 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-32.ec2.internal" event="NodeHasSufficientPID" Jul 27 18:28:44.073072 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:44.073033 2579 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-128-32.ec2.internal" Jul 27 18:28:44.081347 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:44.081300 2579 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-128-32.ec2.internal" Jul 27 18:28:44.081347 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:28:44.081321 2579 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-128-32.ec2.internal\": node \"ip-10-0-128-32.ec2.internal\" not found" Jul 27 18:28:44.109524 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:28:44.109503 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-32.ec2.internal\" not found" Jul 27 18:28:44.153295 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:44.153266 2579 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-32.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-32.ec2.internal"] Jul 27 18:28:44.153405 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:44.153336 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 27 18:28:44.154241 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:44.154225 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-32.ec2.internal" event="NodeHasSufficientMemory" Jul 27 18:28:44.154328 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:44.154258 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-32.ec2.internal" event="NodeHasNoDiskPressure" Jul 27 18:28:44.154328 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:44.154271 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-32.ec2.internal" event="NodeHasSufficientPID" Jul 27 18:28:44.155680 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:44.155664 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 27 18:28:44.155789 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:44.155774 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-32.ec2.internal" Jul 27 18:28:44.155856 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:44.155807 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 27 18:28:44.156346 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:44.156332 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-32.ec2.internal" event="NodeHasSufficientMemory" Jul 27 18:28:44.156415 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:44.156364 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-32.ec2.internal" event="NodeHasNoDiskPressure" Jul 27 18:28:44.156415 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:44.156374 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-32.ec2.internal" event="NodeHasSufficientPID" Jul 27 18:28:44.156415 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:44.156403 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-32.ec2.internal" event="NodeHasSufficientMemory" Jul 27 18:28:44.156524 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:44.156432 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-32.ec2.internal" event="NodeHasNoDiskPressure" Jul 27 18:28:44.156524 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:44.156442 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-32.ec2.internal" event="NodeHasSufficientPID" Jul 27 18:28:44.157487 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:44.157473 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-32.ec2.internal" Jul 27 18:28:44.157544 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:44.157500 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 27 18:28:44.158176 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:44.158159 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-32.ec2.internal" event="NodeHasSufficientMemory" Jul 27 18:28:44.158265 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:44.158190 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-32.ec2.internal" event="NodeHasNoDiskPressure" Jul 27 18:28:44.158265 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:44.158200 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-32.ec2.internal" event="NodeHasSufficientPID" Jul 27 18:28:44.181439 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:28:44.181422 2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-32.ec2.internal\" not found" node="ip-10-0-128-32.ec2.internal" Jul 27 18:28:44.185790 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:28:44.185774 2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-32.ec2.internal\" not found" node="ip-10-0-128-32.ec2.internal" Jul 27 18:28:44.209992 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:28:44.209971 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-32.ec2.internal\" not found" Jul 27 18:28:44.212223 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:44.212208 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8bf56391e54c962db9801fd2ce6f6f03-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-32.ec2.internal\" (UID: \"8bf56391e54c962db9801fd2ce6f6f03\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-32.ec2.internal" Jul 27 18:28:44.212287 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:44.212230 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8aa0e86aff493755e5519df8f9d22c9f-config\") pod \"kube-apiserver-proxy-ip-10-0-128-32.ec2.internal\" (UID: \"8aa0e86aff493755e5519df8f9d22c9f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-32.ec2.internal" Jul 27 18:28:44.212287 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:44.212245 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8bf56391e54c962db9801fd2ce6f6f03-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-32.ec2.internal\" (UID: \"8bf56391e54c962db9801fd2ce6f6f03\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-32.ec2.internal" Jul 27 18:28:44.310461 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:28:44.310430 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-32.ec2.internal\" not found" Jul 27 18:28:44.312704 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:44.312684 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8bf56391e54c962db9801fd2ce6f6f03-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-32.ec2.internal\" (UID: \"8bf56391e54c962db9801fd2ce6f6f03\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-32.ec2.internal" Jul 27 18:28:44.312769 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:44.312721 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8bf56391e54c962db9801fd2ce6f6f03-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-32.ec2.internal\" (UID: \"8bf56391e54c962db9801fd2ce6f6f03\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-32.ec2.internal" Jul 27 18:28:44.312769 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:44.312742 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8aa0e86aff493755e5519df8f9d22c9f-config\") pod \"kube-apiserver-proxy-ip-10-0-128-32.ec2.internal\" (UID: \"8aa0e86aff493755e5519df8f9d22c9f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-32.ec2.internal" Jul 27 18:28:44.312888 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:44.312779 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8bf56391e54c962db9801fd2ce6f6f03-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-32.ec2.internal\" (UID: \"8bf56391e54c962db9801fd2ce6f6f03\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-32.ec2.internal" Jul 27 18:28:44.312888 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:44.312821 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8bf56391e54c962db9801fd2ce6f6f03-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-32.ec2.internal\" (UID: \"8bf56391e54c962db9801fd2ce6f6f03\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-32.ec2.internal" Jul 27 18:28:44.312888 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:44.312847 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8aa0e86aff493755e5519df8f9d22c9f-config\") pod \"kube-apiserver-proxy-ip-10-0-128-32.ec2.internal\" (UID: \"8aa0e86aff493755e5519df8f9d22c9f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-32.ec2.internal" Jul 27 18:28:44.411134 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:28:44.411102 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-32.ec2.internal\" not found" Jul 27 18:28:44.484497 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:44.484467 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-32.ec2.internal" Jul 27 18:28:44.489635 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:44.489612 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-32.ec2.internal" Jul 27 18:28:44.512056 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:28:44.512033 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-32.ec2.internal\" not found" Jul 27 18:28:44.612625 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:28:44.612589 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-32.ec2.internal\" not found" Jul 27 18:28:46.146526 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:28:46.146501 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-32.ec2.internal\" not found" Jul 27 18:28:46.162990 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.162969 2579 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 27 18:28:46.244513 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.244486 2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-32.ec2.internal" Jul 27 18:28:46.263725 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.263700 2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 27 18:28:46.265959 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.265944 2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-32.ec2.internal" Jul 27 18:28:46.272552 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.272538 2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 27 18:28:46.283683 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.283665 2579 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 27 18:28:46.283816 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.283798 2579 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 27 18:28:46.283874 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.283830 2579 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 27 18:28:46.311544 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.311520 2579 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 27 18:28:46.322017 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.321989 2579 apiserver.go:52] "Watching apiserver" Jul 27 18:28:46.331368 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.331346 2579 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 27 18:28:46.331702 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.331677 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-ttprc","openshift-image-registry/node-ca-fdsdr","openshift-multus/multus-additional-cni-plugins-6jzx9","openshift-multus/multus-pgzw4","openshift-network-operator/iptables-alerter-8phm4","openshift-ovn-kubernetes/ovnkube-node-vnbl8","kube-system/global-pull-secret-syncer-pr8j9","kube-system/kube-apiserver-proxy-ip-10-0-128-32.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h9d4j","openshift-cluster-node-tuning-operator/tuned-sxj58","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-32.ec2.internal","openshift-multus/network-metrics-daemon-6htnl","openshift-network-diagnostics/network-check-target-tgxn8","kube-system/konnectivity-agent-67t26"] Jul 27 18:28:46.333170 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.333152 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pr8j9" Jul 27 18:28:46.333270 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:28:46.333251 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pr8j9" podUID="bd3d598d-7458-4407-ba85-49402348da87" Jul 27 18:28:46.334305 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.334289 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgxn8" Jul 27 18:28:46.334374 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:28:46.334355 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tgxn8" podUID="8ce054d9-d9c1-44ef-9e32-5a4c502e0ced" Jul 27 18:28:46.335353 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.335341 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-6jzx9" Jul 27 18:28:46.336369 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.336350 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-pgzw4" Jul 27 18:28:46.337357 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.337341 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-8phm4" Jul 27 18:28:46.337929 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.337909 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 27 18:28:46.338137 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.338121 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 27 18:28:46.338325 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.338310 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 27 18:28:46.338396 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.338339 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 27 18:28:46.338476 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.338463 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 27 18:28:46.338686 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.338671 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-vnbl8" Jul 27 18:28:46.339627 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.339608 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 27 18:28:46.339724 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.339632 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-mcct4\"" Jul 27 18:28:46.339724 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.339641 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-vmnbn\"" Jul 27 18:28:46.339837 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.339745 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h9d4j" Jul 27 18:28:46.339950 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.339932 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 27 18:28:46.340164 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.340150 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 27 18:28:46.340297 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.340182 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 27 18:28:46.340297 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.340182 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-vsnt9\"" Jul 27 18:28:46.341764 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.341286 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-ttprc" Jul 27 18:28:46.343355 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.343336 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-fdsdr" Jul 27 18:28:46.343476 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.343445 2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 27 18:28:46.344120 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.344098 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 27 18:28:46.344120 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.344115 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 27 18:28:46.344300 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.344139 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 27 18:28:46.344350 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.344304 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-sxj58" Jul 27 18:28:46.344629 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.344602 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 27 18:28:46.344629 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.344619 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 27 18:28:46.344750 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.344657 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 27 18:28:46.345014 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.344995 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 27 18:28:46.345117 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.345036 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-7gqrv\"" Jul 27 18:28:46.345177 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.345063 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 27 18:28:46.345276 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.345258 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-cqfq8\"" Jul 27 18:28:46.345397 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.345365 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 27 18:28:46.345471 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.345427 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-cc5kb\"" Jul 27 18:28:46.345471 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.345457 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 27 18:28:46.345471 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.345466 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 27 18:28:46.345676 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.345661 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-67t26" Jul 27 18:28:46.346393 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.346376 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-msf2r\"" Jul 27 18:28:46.346393 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.346385 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 27 18:28:46.346393 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.346394 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 27 18:28:46.346798 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.346784 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6htnl" Jul 27 18:28:46.346845 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:28:46.346829 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6htnl" podUID="270a2080-a3dd-4227-9abc-c093f70df909" Jul 27 18:28:46.347943 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.347923 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 27 18:28:46.348119 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.348101 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 27 18:28:46.348192 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.348111 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-stdww\"" Jul 27 18:28:46.348192 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.348112 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 27 18:28:46.348704 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.348676 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 27 18:28:46.349156 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.349134 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4f79aa08-0ec5-4fd9-a5a2-06c0d9a755ef-run-openvswitch\") pod \"ovnkube-node-vnbl8\" (UID: \"4f79aa08-0ec5-4fd9-a5a2-06c0d9a755ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-vnbl8" Jul 27 18:28:46.349292 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.349173 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bd3d598d-7458-4407-ba85-49402348da87-original-pull-secret\") pod \"global-pull-secret-syncer-pr8j9\" (UID: \"bd3d598d-7458-4407-ba85-49402348da87\") " pod="kube-system/global-pull-secret-syncer-pr8j9" Jul 27 18:28:46.349292 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.349192 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5ba4cb7b-80f6-470b-8ab2-7e22c50af412-system-cni-dir\") pod \"multus-additional-cni-plugins-6jzx9\" (UID: \"5ba4cb7b-80f6-470b-8ab2-7e22c50af412\") " pod="openshift-multus/multus-additional-cni-plugins-6jzx9" Jul 27 18:28:46.349292 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.349210 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7d279661-f5f8-49ca-86c1-3c6f863340c8-system-cni-dir\") pod \"multus-pgzw4\" (UID: \"7d279661-f5f8-49ca-86c1-3c6f863340c8\") " pod="openshift-multus/multus-pgzw4" Jul 27 18:28:46.349292 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.349242 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7d279661-f5f8-49ca-86c1-3c6f863340c8-cnibin\") pod \"multus-pgzw4\" (UID: \"7d279661-f5f8-49ca-86c1-3c6f863340c8\") " pod="openshift-multus/multus-pgzw4" Jul 27 18:28:46.349292 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.349266 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7d279661-f5f8-49ca-86c1-3c6f863340c8-host-run-netns\") pod \"multus-pgzw4\" (UID: \"7d279661-f5f8-49ca-86c1-3c6f863340c8\") " pod="openshift-multus/multus-pgzw4" Jul 27 18:28:46.349550 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.349307 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4f79aa08-0ec5-4fd9-a5a2-06c0d9a755ef-host-run-ovn-kubernetes\") pod \"ovnkube-node-vnbl8\" (UID: \"4f79aa08-0ec5-4fd9-a5a2-06c0d9a755ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-vnbl8" Jul 27 18:28:46.349550 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.349323 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-dxz2n\"" Jul 27 18:28:46.349550 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.349342 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4f79aa08-0ec5-4fd9-a5a2-06c0d9a755ef-ovnkube-script-lib\") pod \"ovnkube-node-vnbl8\" (UID: \"4f79aa08-0ec5-4fd9-a5a2-06c0d9a755ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-vnbl8" Jul 27 18:28:46.349550 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.349351 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 27 18:28:46.349550 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.349368 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8bbbb432-a482-444d-99d3-f2615e9b99bb-serviceca\") pod \"node-ca-fdsdr\" (UID: \"8bbbb432-a482-444d-99d3-f2615e9b99bb\") " pod="openshift-image-registry/node-ca-fdsdr" Jul 27 18:28:46.349550 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.349393 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3706a9c3-0323-4665-99b3-8bd8889dcd85-run\") pod \"tuned-sxj58\" (UID: \"3706a9c3-0323-4665-99b3-8bd8889dcd85\") " pod="openshift-cluster-node-tuning-operator/tuned-sxj58" Jul 27 18:28:46.349550 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.349416 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bd3d598d-7458-4407-ba85-49402348da87-kubelet-config\") pod \"global-pull-secret-syncer-pr8j9\" (UID: \"bd3d598d-7458-4407-ba85-49402348da87\") " pod="kube-system/global-pull-secret-syncer-pr8j9" Jul 27 18:28:46.349550 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.349441 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/83c21902-488a-44d1-b494-7000443a9202-iptables-alerter-script\") pod \"iptables-alerter-8phm4\" (UID: \"83c21902-488a-44d1-b494-7000443a9202\") " pod="openshift-network-operator/iptables-alerter-8phm4" Jul 27 18:28:46.349550 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.349465 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnk5m\" (UniqueName: \"kubernetes.io/projected/52bcf393-1ae7-4102-b762-569badf3538c-kube-api-access-pnk5m\") pod \"node-resolver-ttprc\" (UID: \"52bcf393-1ae7-4102-b762-569badf3538c\") " pod="openshift-dns/node-resolver-ttprc" Jul 27 18:28:46.349550 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.349488 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4f79aa08-0ec5-4fd9-a5a2-06c0d9a755ef-log-socket\") pod \"ovnkube-node-vnbl8\" (UID: \"4f79aa08-0ec5-4fd9-a5a2-06c0d9a755ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-vnbl8" Jul 27 18:28:46.349550 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.349511 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4f79aa08-0ec5-4fd9-a5a2-06c0d9a755ef-host-cni-bin\") pod \"ovnkube-node-vnbl8\" (UID: \"4f79aa08-0ec5-4fd9-a5a2-06c0d9a755ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-vnbl8" Jul 27 18:28:46.349550 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.349533 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4f79aa08-0ec5-4fd9-a5a2-06c0d9a755ef-ovnkube-config\") pod \"ovnkube-node-vnbl8\" (UID: \"4f79aa08-0ec5-4fd9-a5a2-06c0d9a755ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-vnbl8" Jul 27 18:28:46.349939 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.349568 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5ba4cb7b-80f6-470b-8ab2-7e22c50af412-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6jzx9\" (UID: \"5ba4cb7b-80f6-470b-8ab2-7e22c50af412\") " pod="openshift-multus/multus-additional-cni-plugins-6jzx9" Jul 27 18:28:46.349939 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.349603 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3706a9c3-0323-4665-99b3-8bd8889dcd85-etc-modprobe-d\") pod \"tuned-sxj58\" (UID: \"3706a9c3-0323-4665-99b3-8bd8889dcd85\") " pod="openshift-cluster-node-tuning-operator/tuned-sxj58" Jul 27 18:28:46.349939 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.349627 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9c37bc5b-9ef7-4e2b-b555-52de0d9861b8-agent-certs\") pod \"konnectivity-agent-67t26\" (UID: \"9c37bc5b-9ef7-4e2b-b555-52de0d9861b8\") " pod="kube-system/konnectivity-agent-67t26" Jul 27 18:28:46.349939 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.349644 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9c37bc5b-9ef7-4e2b-b555-52de0d9861b8-konnectivity-ca\") pod \"konnectivity-agent-67t26\" (UID: \"9c37bc5b-9ef7-4e2b-b555-52de0d9861b8\") " pod="kube-system/konnectivity-agent-67t26" Jul 27 18:28:46.349939 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.349663 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bd3d598d-7458-4407-ba85-49402348da87-dbus\") pod \"global-pull-secret-syncer-pr8j9\" (UID: \"bd3d598d-7458-4407-ba85-49402348da87\") " pod="kube-system/global-pull-secret-syncer-pr8j9" Jul 27 18:28:46.349939 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.349685 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/af18a2fc-71f3-4f54-9907-89e31bd65bc9-socket-dir\") pod \"aws-ebs-csi-driver-node-h9d4j\" (UID: \"af18a2fc-71f3-4f54-9907-89e31bd65bc9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h9d4j" Jul 27 18:28:46.349939 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.349718 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/af18a2fc-71f3-4f54-9907-89e31bd65bc9-sys-fs\") pod \"aws-ebs-csi-driver-node-h9d4j\" (UID: \"af18a2fc-71f3-4f54-9907-89e31bd65bc9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h9d4j" Jul 27 18:28:46.349939 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.349741 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rr8z7\" (UniqueName: \"kubernetes.io/projected/8ce054d9-d9c1-44ef-9e32-5a4c502e0ced-kube-api-access-rr8z7\") pod \"network-check-target-tgxn8\" (UID: \"8ce054d9-d9c1-44ef-9e32-5a4c502e0ced\") " pod="openshift-network-diagnostics/network-check-target-tgxn8" Jul 27 18:28:46.349939 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.349784 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5ba4cb7b-80f6-470b-8ab2-7e22c50af412-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6jzx9\" (UID: \"5ba4cb7b-80f6-470b-8ab2-7e22c50af412\") " pod="openshift-multus/multus-additional-cni-plugins-6jzx9" Jul 27 18:28:46.349939 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.349815 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/af18a2fc-71f3-4f54-9907-89e31bd65bc9-registration-dir\") pod \"aws-ebs-csi-driver-node-h9d4j\" (UID: \"af18a2fc-71f3-4f54-9907-89e31bd65bc9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h9d4j" Jul 27 18:28:46.349939 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.349830 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3706a9c3-0323-4665-99b3-8bd8889dcd85-tmp\") pod \"tuned-sxj58\" (UID: \"3706a9c3-0323-4665-99b3-8bd8889dcd85\") " pod="openshift-cluster-node-tuning-operator/tuned-sxj58" Jul 27 18:28:46.349939 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.349850 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7d279661-f5f8-49ca-86c1-3c6f863340c8-multus-daemon-config\") pod \"multus-pgzw4\" (UID: \"7d279661-f5f8-49ca-86c1-3c6f863340c8\") " pod="openshift-multus/multus-pgzw4" Jul 27 18:28:46.349939 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.349884 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/52bcf393-1ae7-4102-b762-569badf3538c-hosts-file\") pod \"node-resolver-ttprc\" (UID: \"52bcf393-1ae7-4102-b762-569badf3538c\") " pod="openshift-dns/node-resolver-ttprc" Jul 27 18:28:46.349939 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.349911 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4f79aa08-0ec5-4fd9-a5a2-06c0d9a755ef-host-kubelet\") pod \"ovnkube-node-vnbl8\" (UID: \"4f79aa08-0ec5-4fd9-a5a2-06c0d9a755ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-vnbl8" Jul 27 18:28:46.349939 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.349940 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4f79aa08-0ec5-4fd9-a5a2-06c0d9a755ef-etc-openvswitch\") pod \"ovnkube-node-vnbl8\" (UID: \"4f79aa08-0ec5-4fd9-a5a2-06c0d9a755ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-vnbl8" Jul 27 18:28:46.350631 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.349967 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4f79aa08-0ec5-4fd9-a5a2-06c0d9a755ef-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vnbl8\" (UID: \"4f79aa08-0ec5-4fd9-a5a2-06c0d9a755ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-vnbl8" Jul 27 18:28:46.350631 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.349992 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3706a9c3-0323-4665-99b3-8bd8889dcd85-host\") pod \"tuned-sxj58\" (UID: \"3706a9c3-0323-4665-99b3-8bd8889dcd85\") " pod="openshift-cluster-node-tuning-operator/tuned-sxj58" Jul 27 18:28:46.350631 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.350017 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7d279661-f5f8-49ca-86c1-3c6f863340c8-os-release\") pod \"multus-pgzw4\" (UID: \"7d279661-f5f8-49ca-86c1-3c6f863340c8\") " pod="openshift-multus/multus-pgzw4" Jul 27 18:28:46.350631 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.350042 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/af18a2fc-71f3-4f54-9907-89e31bd65bc9-etc-selinux\") pod \"aws-ebs-csi-driver-node-h9d4j\" (UID: \"af18a2fc-71f3-4f54-9907-89e31bd65bc9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h9d4j" Jul 27 18:28:46.350631 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.350068 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4f79aa08-0ec5-4fd9-a5a2-06c0d9a755ef-node-log\") pod \"ovnkube-node-vnbl8\" (UID: \"4f79aa08-0ec5-4fd9-a5a2-06c0d9a755ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-vnbl8" Jul 27 18:28:46.350631 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.350109 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5ba4cb7b-80f6-470b-8ab2-7e22c50af412-cni-binary-copy\") pod \"multus-additional-cni-plugins-6jzx9\" (UID: \"5ba4cb7b-80f6-470b-8ab2-7e22c50af412\") " pod="openshift-multus/multus-additional-cni-plugins-6jzx9" Jul 27 18:28:46.350631 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.350134 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5ba4cb7b-80f6-470b-8ab2-7e22c50af412-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6jzx9\" (UID: \"5ba4cb7b-80f6-470b-8ab2-7e22c50af412\") " pod="openshift-multus/multus-additional-cni-plugins-6jzx9" Jul 27 18:28:46.350631 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.350149 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7d279661-f5f8-49ca-86c1-3c6f863340c8-cni-binary-copy\") pod \"multus-pgzw4\" (UID: \"7d279661-f5f8-49ca-86c1-3c6f863340c8\") " pod="openshift-multus/multus-pgzw4" Jul 27 18:28:46.350631 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.350165 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7d279661-f5f8-49ca-86c1-3c6f863340c8-hostroot\") pod \"multus-pgzw4\" (UID: \"7d279661-f5f8-49ca-86c1-3c6f863340c8\") " pod="openshift-multus/multus-pgzw4" Jul 27 18:28:46.350631 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.350195 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/af18a2fc-71f3-4f54-9907-89e31bd65bc9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-h9d4j\" (UID: \"af18a2fc-71f3-4f54-9907-89e31bd65bc9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h9d4j" Jul 27 18:28:46.350631 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.350220 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4f79aa08-0ec5-4fd9-a5a2-06c0d9a755ef-host-cni-netd\") pod \"ovnkube-node-vnbl8\" (UID: \"4f79aa08-0ec5-4fd9-a5a2-06c0d9a755ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-vnbl8" Jul 27 18:28:46.350631 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.350258 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52bcf393-1ae7-4102-b762-569badf3538c-tmp-dir\") pod \"node-resolver-ttprc\" (UID: \"52bcf393-1ae7-4102-b762-569badf3538c\") " pod="openshift-dns/node-resolver-ttprc" Jul 27 18:28:46.350631 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.350280 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4f79aa08-0ec5-4fd9-a5a2-06c0d9a755ef-env-overrides\") pod \"ovnkube-node-vnbl8\" (UID: \"4f79aa08-0ec5-4fd9-a5a2-06c0d9a755ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-vnbl8" Jul 27 18:28:46.350631 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.350303 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3706a9c3-0323-4665-99b3-8bd8889dcd85-etc-sysctl-d\") pod \"tuned-sxj58\" (UID: \"3706a9c3-0323-4665-99b3-8bd8889dcd85\") " pod="openshift-cluster-node-tuning-operator/tuned-sxj58" Jul 27 18:28:46.350631 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.350331 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3706a9c3-0323-4665-99b3-8bd8889dcd85-etc-systemd\") pod \"tuned-sxj58\" (UID: \"3706a9c3-0323-4665-99b3-8bd8889dcd85\") " pod="openshift-cluster-node-tuning-operator/tuned-sxj58" Jul 27 18:28:46.350631 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.350360 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqb8v\" (UniqueName: \"kubernetes.io/projected/5ba4cb7b-80f6-470b-8ab2-7e22c50af412-kube-api-access-gqb8v\") pod \"multus-additional-cni-plugins-6jzx9\" (UID: \"5ba4cb7b-80f6-470b-8ab2-7e22c50af412\") " pod="openshift-multus/multus-additional-cni-plugins-6jzx9" Jul 27 18:28:46.351565 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.350384 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4f79aa08-0ec5-4fd9-a5a2-06c0d9a755ef-host-run-netns\") pod \"ovnkube-node-vnbl8\" (UID: \"4f79aa08-0ec5-4fd9-a5a2-06c0d9a755ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-vnbl8" Jul 27 18:28:46.351565 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.350400 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4f79aa08-0ec5-4fd9-a5a2-06c0d9a755ef-ovn-node-metrics-cert\") pod \"ovnkube-node-vnbl8\" (UID: \"4f79aa08-0ec5-4fd9-a5a2-06c0d9a755ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-vnbl8" Jul 27 18:28:46.351565 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.350445 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3706a9c3-0323-4665-99b3-8bd8889dcd85-etc-tuned\") pod \"tuned-sxj58\" (UID: \"3706a9c3-0323-4665-99b3-8bd8889dcd85\") " pod="openshift-cluster-node-tuning-operator/tuned-sxj58" Jul 27 18:28:46.351565 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.350508 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5ba4cb7b-80f6-470b-8ab2-7e22c50af412-os-release\") pod \"multus-additional-cni-plugins-6jzx9\" (UID: \"5ba4cb7b-80f6-470b-8ab2-7e22c50af412\") " pod="openshift-multus/multus-additional-cni-plugins-6jzx9" Jul 27 18:28:46.351565 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.350535 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7d279661-f5f8-49ca-86c1-3c6f863340c8-host-var-lib-cni-bin\") pod \"multus-pgzw4\" (UID: \"7d279661-f5f8-49ca-86c1-3c6f863340c8\") " pod="openshift-multus/multus-pgzw4" Jul 27 18:28:46.351565 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.350590 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5ba4cb7b-80f6-470b-8ab2-7e22c50af412-cnibin\") pod \"multus-additional-cni-plugins-6jzx9\" (UID: \"5ba4cb7b-80f6-470b-8ab2-7e22c50af412\") " pod="openshift-multus/multus-additional-cni-plugins-6jzx9" Jul 27 18:28:46.351565 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.350620 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7d279661-f5f8-49ca-86c1-3c6f863340c8-multus-cni-dir\") pod \"multus-pgzw4\" (UID: \"7d279661-f5f8-49ca-86c1-3c6f863340c8\") " pod="openshift-multus/multus-pgzw4" Jul 27 18:28:46.351565 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.350647 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7d279661-f5f8-49ca-86c1-3c6f863340c8-host-var-lib-kubelet\") pod \"multus-pgzw4\" (UID: \"7d279661-f5f8-49ca-86c1-3c6f863340c8\") " pod="openshift-multus/multus-pgzw4" Jul 27 18:28:46.351565 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.350670 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7d279661-f5f8-49ca-86c1-3c6f863340c8-host-run-multus-certs\") pod \"multus-pgzw4\" (UID: \"7d279661-f5f8-49ca-86c1-3c6f863340c8\") " pod="openshift-multus/multus-pgzw4" Jul 27 18:28:46.351565 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.350692 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/af18a2fc-71f3-4f54-9907-89e31bd65bc9-device-dir\") pod \"aws-ebs-csi-driver-node-h9d4j\" (UID: \"af18a2fc-71f3-4f54-9907-89e31bd65bc9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h9d4j" Jul 27 18:28:46.351565 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.350716 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4f79aa08-0ec5-4fd9-a5a2-06c0d9a755ef-run-systemd\") pod \"ovnkube-node-vnbl8\" (UID: \"4f79aa08-0ec5-4fd9-a5a2-06c0d9a755ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-vnbl8" Jul 27 18:28:46.351565 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.350741 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxzwx\" (UniqueName: \"kubernetes.io/projected/8bbbb432-a482-444d-99d3-f2615e9b99bb-kube-api-access-nxzwx\") pod \"node-ca-fdsdr\" (UID: \"8bbbb432-a482-444d-99d3-f2615e9b99bb\") " pod="openshift-image-registry/node-ca-fdsdr" Jul 27 18:28:46.351565 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.350764 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3706a9c3-0323-4665-99b3-8bd8889dcd85-etc-sysctl-conf\") pod \"tuned-sxj58\" (UID: \"3706a9c3-0323-4665-99b3-8bd8889dcd85\") " pod="openshift-cluster-node-tuning-operator/tuned-sxj58" Jul 27 18:28:46.351565 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.350786 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3706a9c3-0323-4665-99b3-8bd8889dcd85-lib-modules\") pod \"tuned-sxj58\" (UID: \"3706a9c3-0323-4665-99b3-8bd8889dcd85\") " pod="openshift-cluster-node-tuning-operator/tuned-sxj58" Jul 27 18:28:46.351565 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.350808 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7d279661-f5f8-49ca-86c1-3c6f863340c8-host-var-lib-cni-multus\") pod \"multus-pgzw4\" (UID: \"7d279661-f5f8-49ca-86c1-3c6f863340c8\") " pod="openshift-multus/multus-pgzw4" Jul 27 18:28:46.351565 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.350831 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgq78\" (UniqueName: \"kubernetes.io/projected/7d279661-f5f8-49ca-86c1-3c6f863340c8-kube-api-access-vgq78\") pod \"multus-pgzw4\" (UID: \"7d279661-f5f8-49ca-86c1-3c6f863340c8\") " pod="openshift-multus/multus-pgzw4" Jul 27 18:28:46.352020 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.350845 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6tzv\" (UniqueName: \"kubernetes.io/projected/af18a2fc-71f3-4f54-9907-89e31bd65bc9-kube-api-access-c6tzv\") pod \"aws-ebs-csi-driver-node-h9d4j\" (UID: \"af18a2fc-71f3-4f54-9907-89e31bd65bc9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h9d4j" Jul 27 18:28:46.352020 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.350869 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4f79aa08-0ec5-4fd9-a5a2-06c0d9a755ef-var-lib-openvswitch\") pod \"ovnkube-node-vnbl8\" (UID: \"4f79aa08-0ec5-4fd9-a5a2-06c0d9a755ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-vnbl8" Jul 27 18:28:46.352020 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.350892 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4f79aa08-0ec5-4fd9-a5a2-06c0d9a755ef-run-ovn\") pod \"ovnkube-node-vnbl8\" (UID: \"4f79aa08-0ec5-4fd9-a5a2-06c0d9a755ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-vnbl8" Jul 27 18:28:46.352020 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.350914 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7d279661-f5f8-49ca-86c1-3c6f863340c8-etc-kubernetes\") pod \"multus-pgzw4\" (UID: \"7d279661-f5f8-49ca-86c1-3c6f863340c8\") " pod="openshift-multus/multus-pgzw4" Jul 27 18:28:46.352020 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.350937 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fv4z\" (UniqueName: \"kubernetes.io/projected/83c21902-488a-44d1-b494-7000443a9202-kube-api-access-5fv4z\") pod \"iptables-alerter-8phm4\" (UID: \"83c21902-488a-44d1-b494-7000443a9202\") " pod="openshift-network-operator/iptables-alerter-8phm4" Jul 27 18:28:46.352020 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.350953 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjwxt\" (UniqueName: \"kubernetes.io/projected/4f79aa08-0ec5-4fd9-a5a2-06c0d9a755ef-kube-api-access-zjwxt\") pod \"ovnkube-node-vnbl8\" (UID: \"4f79aa08-0ec5-4fd9-a5a2-06c0d9a755ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-vnbl8" Jul 27 18:28:46.352020 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.350967 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8bbbb432-a482-444d-99d3-f2615e9b99bb-host\") pod \"node-ca-fdsdr\" (UID: \"8bbbb432-a482-444d-99d3-f2615e9b99bb\") " pod="openshift-image-registry/node-ca-fdsdr" Jul 27 18:28:46.352020 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.350985 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3706a9c3-0323-4665-99b3-8bd8889dcd85-sys\") pod \"tuned-sxj58\" (UID: \"3706a9c3-0323-4665-99b3-8bd8889dcd85\") " pod="openshift-cluster-node-tuning-operator/tuned-sxj58" Jul 27 18:28:46.352020 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.351034 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3706a9c3-0323-4665-99b3-8bd8889dcd85-var-lib-kubelet\") pod \"tuned-sxj58\" (UID: \"3706a9c3-0323-4665-99b3-8bd8889dcd85\") " pod="openshift-cluster-node-tuning-operator/tuned-sxj58" Jul 27 18:28:46.352020 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.351059 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsp6g\" (UniqueName: \"kubernetes.io/projected/3706a9c3-0323-4665-99b3-8bd8889dcd85-kube-api-access-gsp6g\") pod \"tuned-sxj58\" (UID: \"3706a9c3-0323-4665-99b3-8bd8889dcd85\") " pod="openshift-cluster-node-tuning-operator/tuned-sxj58" Jul 27 18:28:46.352020 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.351074 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7d279661-f5f8-49ca-86c1-3c6f863340c8-multus-socket-dir-parent\") pod \"multus-pgzw4\" (UID: \"7d279661-f5f8-49ca-86c1-3c6f863340c8\") " pod="openshift-multus/multus-pgzw4" Jul 27 18:28:46.352020 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.351089 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7d279661-f5f8-49ca-86c1-3c6f863340c8-host-run-k8s-cni-cncf-io\") pod \"multus-pgzw4\" (UID: \"7d279661-f5f8-49ca-86c1-3c6f863340c8\") " pod="openshift-multus/multus-pgzw4" Jul 27 18:28:46.352020 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.351110 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3706a9c3-0323-4665-99b3-8bd8889dcd85-etc-sysconfig\") pod \"tuned-sxj58\" (UID: \"3706a9c3-0323-4665-99b3-8bd8889dcd85\") " pod="openshift-cluster-node-tuning-operator/tuned-sxj58" Jul 27 18:28:46.352020 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.351132 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3706a9c3-0323-4665-99b3-8bd8889dcd85-etc-kubernetes\") pod \"tuned-sxj58\" (UID: \"3706a9c3-0323-4665-99b3-8bd8889dcd85\") " pod="openshift-cluster-node-tuning-operator/tuned-sxj58" Jul 27 18:28:46.352020 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.351151 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7d279661-f5f8-49ca-86c1-3c6f863340c8-multus-conf-dir\") pod \"multus-pgzw4\" (UID: \"7d279661-f5f8-49ca-86c1-3c6f863340c8\") " pod="openshift-multus/multus-pgzw4" Jul 27 18:28:46.352020 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.351165 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/83c21902-488a-44d1-b494-7000443a9202-host-slash\") pod \"iptables-alerter-8phm4\" (UID: \"83c21902-488a-44d1-b494-7000443a9202\") " pod="openshift-network-operator/iptables-alerter-8phm4" Jul 27 18:28:46.352500 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.351179 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4f79aa08-0ec5-4fd9-a5a2-06c0d9a755ef-systemd-units\") pod \"ovnkube-node-vnbl8\" (UID: \"4f79aa08-0ec5-4fd9-a5a2-06c0d9a755ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-vnbl8" Jul 27 18:28:46.352500 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.351199 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4f79aa08-0ec5-4fd9-a5a2-06c0d9a755ef-host-slash\") pod \"ovnkube-node-vnbl8\" (UID: \"4f79aa08-0ec5-4fd9-a5a2-06c0d9a755ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-vnbl8" Jul 27 18:28:46.362612 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.362591 2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 27 18:28:46.377025 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.377009 2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-rkr94" Jul 27 18:28:46.383031 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.382979 2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-rkr94" Jul 27 18:28:46.391722 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.391672 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-26 18:23:45 +0000 UTC" deadline="2027-12-21 18:30:10.116158215 +0000 UTC" Jul 27 18:28:46.391722 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.391719 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12288h1m23.724441496s" Jul 27 18:28:46.444971 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.444926 2579 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 27 18:28:46.451512 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.451489 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4f79aa08-0ec5-4fd9-a5a2-06c0d9a755ef-host-kubelet\") pod \"ovnkube-node-vnbl8\" (UID: \"4f79aa08-0ec5-4fd9-a5a2-06c0d9a755ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-vnbl8" Jul 27 18:28:46.451642 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.451524 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4f79aa08-0ec5-4fd9-a5a2-06c0d9a755ef-etc-openvswitch\") pod \"ovnkube-node-vnbl8\" (UID: \"4f79aa08-0ec5-4fd9-a5a2-06c0d9a755ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-vnbl8" Jul 27 18:28:46.451642 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.451553 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4f79aa08-0ec5-4fd9-a5a2-06c0d9a755ef-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vnbl8\" (UID: \"4f79aa08-0ec5-4fd9-a5a2-06c0d9a755ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-vnbl8" Jul 27 18:28:46.451642 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.451579 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3706a9c3-0323-4665-99b3-8bd8889dcd85-host\") pod \"tuned-sxj58\" (UID: \"3706a9c3-0323-4665-99b3-8bd8889dcd85\") " pod="openshift-cluster-node-tuning-operator/tuned-sxj58" Jul 27 18:28:46.451642 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.451597 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4f79aa08-0ec5-4fd9-a5a2-06c0d9a755ef-host-kubelet\") pod \"ovnkube-node-vnbl8\" (UID: \"4f79aa08-0ec5-4fd9-a5a2-06c0d9a755ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-vnbl8" Jul 27 18:28:46.451642 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.451604 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7d279661-f5f8-49ca-86c1-3c6f863340c8-os-release\") pod \"multus-pgzw4\" (UID: \"7d279661-f5f8-49ca-86c1-3c6f863340c8\") " pod="openshift-multus/multus-pgzw4" Jul 27 18:28:46.451642 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.451610 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4f79aa08-0ec5-4fd9-a5a2-06c0d9a755ef-etc-openvswitch\") pod \"ovnkube-node-vnbl8\" (UID: \"4f79aa08-0ec5-4fd9-a5a2-06c0d9a755ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-vnbl8" Jul 27 18:28:46.451642 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.451630 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/af18a2fc-71f3-4f54-9907-89e31bd65bc9-etc-selinux\") pod \"aws-ebs-csi-driver-node-h9d4j\" (UID: \"af18a2fc-71f3-4f54-9907-89e31bd65bc9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h9d4j" Jul 27 18:28:46.451975 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.451650 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3706a9c3-0323-4665-99b3-8bd8889dcd85-host\") pod \"tuned-sxj58\" (UID: \"3706a9c3-0323-4665-99b3-8bd8889dcd85\") " pod="openshift-cluster-node-tuning-operator/tuned-sxj58" Jul 27 18:28:46.451975 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.451667 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4f79aa08-0ec5-4fd9-a5a2-06c0d9a755ef-node-log\") pod \"ovnkube-node-vnbl8\" (UID: \"4f79aa08-0ec5-4fd9-a5a2-06c0d9a755ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-vnbl8" Jul 27 18:28:46.451975 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.451673 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7d279661-f5f8-49ca-86c1-3c6f863340c8-os-release\") pod \"multus-pgzw4\" (UID: \"7d279661-f5f8-49ca-86c1-3c6f863340c8\") " pod="openshift-multus/multus-pgzw4" Jul 27 18:28:46.451975 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.451685 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5ba4cb7b-80f6-470b-8ab2-7e22c50af412-cni-binary-copy\") pod \"multus-additional-cni-plugins-6jzx9\" (UID: \"5ba4cb7b-80f6-470b-8ab2-7e22c50af412\") " pod="openshift-multus/multus-additional-cni-plugins-6jzx9" Jul 27 18:28:46.451975 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.451612 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4f79aa08-0ec5-4fd9-a5a2-06c0d9a755ef-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vnbl8\" (UID: \"4f79aa08-0ec5-4fd9-a5a2-06c0d9a755ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-vnbl8" Jul 27 18:28:46.451975 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.451701 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5ba4cb7b-80f6-470b-8ab2-7e22c50af412-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6jzx9\" (UID: \"5ba4cb7b-80f6-470b-8ab2-7e22c50af412\") " pod="openshift-multus/multus-additional-cni-plugins-6jzx9" Jul 27 18:28:46.451975 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.451703 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4f79aa08-0ec5-4fd9-a5a2-06c0d9a755ef-node-log\") pod \"ovnkube-node-vnbl8\" (UID: \"4f79aa08-0ec5-4fd9-a5a2-06c0d9a755ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-vnbl8" Jul 27 18:28:46.451975 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.451718 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7d279661-f5f8-49ca-86c1-3c6f863340c8-cni-binary-copy\") pod \"multus-pgzw4\" (UID: \"7d279661-f5f8-49ca-86c1-3c6f863340c8\") " pod="openshift-multus/multus-pgzw4" Jul 27 18:28:46.451975 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.451725 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/af18a2fc-71f3-4f54-9907-89e31bd65bc9-etc-selinux\") pod \"aws-ebs-csi-driver-node-h9d4j\" (UID: \"af18a2fc-71f3-4f54-9907-89e31bd65bc9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h9d4j" Jul 27 18:28:46.451975 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.451738 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7d279661-f5f8-49ca-86c1-3c6f863340c8-hostroot\") pod \"multus-pgzw4\" (UID: \"7d279661-f5f8-49ca-86c1-3c6f863340c8\") " pod="openshift-multus/multus-pgzw4" Jul 27 18:28:46.451975 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.451752 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/af18a2fc-71f3-4f54-9907-89e31bd65bc9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-h9d4j\" (UID: \"af18a2fc-71f3-4f54-9907-89e31bd65bc9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h9d4j" Jul 27 18:28:46.451975 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.451766 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4f79aa08-0ec5-4fd9-a5a2-06c0d9a755ef-host-cni-netd\") pod \"ovnkube-node-vnbl8\" (UID: \"4f79aa08-0ec5-4fd9-a5a2-06c0d9a755ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-vnbl8" Jul 27 18:28:46.451975 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.451783 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52bcf393-1ae7-4102-b762-569badf3538c-tmp-dir\") pod \"node-resolver-ttprc\" (UID: \"52bcf393-1ae7-4102-b762-569badf3538c\") " pod="openshift-dns/node-resolver-ttprc" Jul 27 18:28:46.451975 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.451805 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4f79aa08-0ec5-4fd9-a5a2-06c0d9a755ef-env-overrides\") pod \"ovnkube-node-vnbl8\" (UID: \"4f79aa08-0ec5-4fd9-a5a2-06c0d9a755ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-vnbl8" Jul 27 18:28:46.451975 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.451829 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3706a9c3-0323-4665-99b3-8bd8889dcd85-etc-sysctl-d\") pod \"tuned-sxj58\" (UID: \"3706a9c3-0323-4665-99b3-8bd8889dcd85\") " pod="openshift-cluster-node-tuning-operator/tuned-sxj58" Jul 27 18:28:46.451975 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.451829 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7d279661-f5f8-49ca-86c1-3c6f863340c8-hostroot\") pod \"multus-pgzw4\" (UID: \"7d279661-f5f8-49ca-86c1-3c6f863340c8\") " pod="openshift-multus/multus-pgzw4" Jul 27 18:28:46.451975 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.451850 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3706a9c3-0323-4665-99b3-8bd8889dcd85-etc-systemd\") pod \"tuned-sxj58\" (UID: \"3706a9c3-0323-4665-99b3-8bd8889dcd85\") " pod="openshift-cluster-node-tuning-operator/tuned-sxj58" Jul 27 18:28:46.452790 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.451872 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqb8v\" (UniqueName: \"kubernetes.io/projected/5ba4cb7b-80f6-470b-8ab2-7e22c50af412-kube-api-access-gqb8v\") pod \"multus-additional-cni-plugins-6jzx9\" (UID: \"5ba4cb7b-80f6-470b-8ab2-7e22c50af412\") " pod="openshift-multus/multus-additional-cni-plugins-6jzx9" Jul 27 18:28:46.452790 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.451888 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/af18a2fc-71f3-4f54-9907-89e31bd65bc9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-h9d4j\" (UID: \"af18a2fc-71f3-4f54-9907-89e31bd65bc9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h9d4j" Jul 27 18:28:46.452790 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.451902 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkrwl\" (UniqueName: \"kubernetes.io/projected/270a2080-a3dd-4227-9abc-c093f70df909-kube-api-access-fkrwl\") pod \"network-metrics-daemon-6htnl\" (UID: \"270a2080-a3dd-4227-9abc-c093f70df909\") " pod="openshift-multus/network-metrics-daemon-6htnl" Jul 27 18:28:46.452790 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.451927 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4f79aa08-0ec5-4fd9-a5a2-06c0d9a755ef-host-cni-netd\") pod \"ovnkube-node-vnbl8\" (UID: \"4f79aa08-0ec5-4fd9-a5a2-06c0d9a755ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-vnbl8" Jul 27 18:28:46.452790 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.451962 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4f79aa08-0ec5-4fd9-a5a2-06c0d9a755ef-host-run-netns\") pod \"ovnkube-node-vnbl8\" (UID: \"4f79aa08-0ec5-4fd9-a5a2-06c0d9a755ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-vnbl8" Jul 27 18:28:46.452790 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.452274 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52bcf393-1ae7-4102-b762-569badf3538c-tmp-dir\") pod \"node-resolver-ttprc\" (UID: \"52bcf393-1ae7-4102-b762-569badf3538c\") " pod="openshift-dns/node-resolver-ttprc" Jul 27 18:28:46.452790 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.452328 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5ba4cb7b-80f6-470b-8ab2-7e22c50af412-cni-binary-copy\") pod \"multus-additional-cni-plugins-6jzx9\" (UID: \"5ba4cb7b-80f6-470b-8ab2-7e22c50af412\") " pod="openshift-multus/multus-additional-cni-plugins-6jzx9" Jul 27 18:28:46.452790 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.452331 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5ba4cb7b-80f6-470b-8ab2-7e22c50af412-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6jzx9\" (UID: \"5ba4cb7b-80f6-470b-8ab2-7e22c50af412\") " pod="openshift-multus/multus-additional-cni-plugins-6jzx9" Jul 27 18:28:46.452790 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.452349 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7d279661-f5f8-49ca-86c1-3c6f863340c8-cni-binary-copy\") pod \"multus-pgzw4\" (UID: \"7d279661-f5f8-49ca-86c1-3c6f863340c8\") " pod="openshift-multus/multus-pgzw4" Jul 27 18:28:46.452790 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.451928 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4f79aa08-0ec5-4fd9-a5a2-06c0d9a755ef-host-run-netns\") pod \"ovnkube-node-vnbl8\" (UID: \"4f79aa08-0ec5-4fd9-a5a2-06c0d9a755ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-vnbl8" Jul 27 18:28:46.452790 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.452413 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4f79aa08-0ec5-4fd9-a5a2-06c0d9a755ef-ovn-node-metrics-cert\") pod \"ovnkube-node-vnbl8\" (UID: \"4f79aa08-0ec5-4fd9-a5a2-06c0d9a755ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-vnbl8" Jul 27 18:28:46.452790 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.452415 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4f79aa08-0ec5-4fd9-a5a2-06c0d9a755ef-env-overrides\") pod \"ovnkube-node-vnbl8\" (UID: \"4f79aa08-0ec5-4fd9-a5a2-06c0d9a755ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-vnbl8" Jul 27 18:28:46.452790 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.452431 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3706a9c3-0323-4665-99b3-8bd8889dcd85-etc-systemd\") pod \"tuned-sxj58\" (UID: \"3706a9c3-0323-4665-99b3-8bd8889dcd85\") " pod="openshift-cluster-node-tuning-operator/tuned-sxj58" Jul 27 18:28:46.452790 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.452450 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3706a9c3-0323-4665-99b3-8bd8889dcd85-etc-tuned\") pod \"tuned-sxj58\" (UID: \"3706a9c3-0323-4665-99b3-8bd8889dcd85\") " pod="openshift-cluster-node-tuning-operator/tuned-sxj58" Jul 27 18:28:46.452790 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.452479 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3706a9c3-0323-4665-99b3-8bd8889dcd85-etc-sysctl-d\") pod \"tuned-sxj58\" (UID: \"3706a9c3-0323-4665-99b3-8bd8889dcd85\") " pod="openshift-cluster-node-tuning-operator/tuned-sxj58" Jul 27 18:28:46.452790 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.452478 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5ba4cb7b-80f6-470b-8ab2-7e22c50af412-os-release\") pod \"multus-additional-cni-plugins-6jzx9\" (UID: \"5ba4cb7b-80f6-470b-8ab2-7e22c50af412\") " pod="openshift-multus/multus-additional-cni-plugins-6jzx9" Jul 27 18:28:46.452790 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.452512 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7d279661-f5f8-49ca-86c1-3c6f863340c8-host-var-lib-cni-bin\") pod \"multus-pgzw4\" (UID: \"7d279661-f5f8-49ca-86c1-3c6f863340c8\") " pod="openshift-multus/multus-pgzw4" Jul 27 18:28:46.453437 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.452539 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5ba4cb7b-80f6-470b-8ab2-7e22c50af412-cnibin\") pod \"multus-additional-cni-plugins-6jzx9\" (UID: \"5ba4cb7b-80f6-470b-8ab2-7e22c50af412\") " pod="openshift-multus/multus-additional-cni-plugins-6jzx9" Jul 27 18:28:46.453437 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.452565 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7d279661-f5f8-49ca-86c1-3c6f863340c8-multus-cni-dir\") pod \"multus-pgzw4\" (UID: \"7d279661-f5f8-49ca-86c1-3c6f863340c8\") " pod="openshift-multus/multus-pgzw4" Jul 27 18:28:46.453437 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.452573 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7d279661-f5f8-49ca-86c1-3c6f863340c8-host-var-lib-cni-bin\") pod \"multus-pgzw4\" (UID: \"7d279661-f5f8-49ca-86c1-3c6f863340c8\") " pod="openshift-multus/multus-pgzw4" Jul 27 18:28:46.453437 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.452589 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7d279661-f5f8-49ca-86c1-3c6f863340c8-host-var-lib-kubelet\") pod \"multus-pgzw4\" (UID: \"7d279661-f5f8-49ca-86c1-3c6f863340c8\") " pod="openshift-multus/multus-pgzw4" Jul 27 18:28:46.453437 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.452600 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5ba4cb7b-80f6-470b-8ab2-7e22c50af412-cnibin\") pod \"multus-additional-cni-plugins-6jzx9\" (UID: \"5ba4cb7b-80f6-470b-8ab2-7e22c50af412\") " pod="openshift-multus/multus-additional-cni-plugins-6jzx9" Jul 27 18:28:46.453437 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.452698 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7d279661-f5f8-49ca-86c1-3c6f863340c8-multus-cni-dir\") pod \"multus-pgzw4\" (UID: \"7d279661-f5f8-49ca-86c1-3c6f863340c8\") " pod="openshift-multus/multus-pgzw4" Jul 27 18:28:46.453437 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.452643 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5ba4cb7b-80f6-470b-8ab2-7e22c50af412-os-release\") pod \"multus-additional-cni-plugins-6jzx9\" (UID: \"5ba4cb7b-80f6-470b-8ab2-7e22c50af412\") " pod="openshift-multus/multus-additional-cni-plugins-6jzx9" Jul 27 18:28:46.453437 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.452664 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7d279661-f5f8-49ca-86c1-3c6f863340c8-host-run-multus-certs\") pod \"multus-pgzw4\" (UID: \"7d279661-f5f8-49ca-86c1-3c6f863340c8\") " pod="openshift-multus/multus-pgzw4" Jul 27 18:28:46.453437 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.452675 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7d279661-f5f8-49ca-86c1-3c6f863340c8-host-var-lib-kubelet\") pod \"multus-pgzw4\" (UID: \"7d279661-f5f8-49ca-86c1-3c6f863340c8\") " pod="openshift-multus/multus-pgzw4" Jul 27 18:28:46.453437 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.452610 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7d279661-f5f8-49ca-86c1-3c6f863340c8-host-run-multus-certs\") pod \"multus-pgzw4\" (UID: \"7d279661-f5f8-49ca-86c1-3c6f863340c8\") " pod="openshift-multus/multus-pgzw4" Jul 27 18:28:46.453437 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.452726 2579 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 27 18:28:46.453437 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.452756 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/af18a2fc-71f3-4f54-9907-89e31bd65bc9-device-dir\") pod \"aws-ebs-csi-driver-node-h9d4j\" (UID: \"af18a2fc-71f3-4f54-9907-89e31bd65bc9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h9d4j" Jul 27 18:28:46.453437 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.452773 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/270a2080-a3dd-4227-9abc-c093f70df909-metrics-certs\") pod \"network-metrics-daemon-6htnl\" (UID: \"270a2080-a3dd-4227-9abc-c093f70df909\") " pod="openshift-multus/network-metrics-daemon-6htnl" Jul 27 18:28:46.453437 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.452807 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/af18a2fc-71f3-4f54-9907-89e31bd65bc9-device-dir\") pod \"aws-ebs-csi-driver-node-h9d4j\" (UID: \"af18a2fc-71f3-4f54-9907-89e31bd65bc9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h9d4j" Jul 27 18:28:46.453437 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.452803 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4f79aa08-0ec5-4fd9-a5a2-06c0d9a755ef-run-systemd\") pod \"ovnkube-node-vnbl8\" (UID: \"4f79aa08-0ec5-4fd9-a5a2-06c0d9a755ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-vnbl8" Jul 27 18:28:46.453437 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.452834 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxzwx\" (UniqueName: \"kubernetes.io/projected/8bbbb432-a482-444d-99d3-f2615e9b99bb-kube-api-access-nxzwx\") pod \"node-ca-fdsdr\" (UID: \"8bbbb432-a482-444d-99d3-f2615e9b99bb\") " pod="openshift-image-registry/node-ca-fdsdr" Jul 27 18:28:46.453437 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.452846 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4f79aa08-0ec5-4fd9-a5a2-06c0d9a755ef-run-systemd\") pod \"ovnkube-node-vnbl8\" (UID: \"4f79aa08-0ec5-4fd9-a5a2-06c0d9a755ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-vnbl8" Jul 27 18:28:46.453437 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.452886 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3706a9c3-0323-4665-99b3-8bd8889dcd85-etc-sysctl-conf\") pod \"tuned-sxj58\" (UID: \"3706a9c3-0323-4665-99b3-8bd8889dcd85\") " pod="openshift-cluster-node-tuning-operator/tuned-sxj58" Jul 27 18:28:46.454185 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.452907 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3706a9c3-0323-4665-99b3-8bd8889dcd85-lib-modules\") pod \"tuned-sxj58\" (UID: \"3706a9c3-0323-4665-99b3-8bd8889dcd85\") " pod="openshift-cluster-node-tuning-operator/tuned-sxj58" Jul 27 18:28:46.454185 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.453003 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3706a9c3-0323-4665-99b3-8bd8889dcd85-lib-modules\") pod \"tuned-sxj58\" (UID: \"3706a9c3-0323-4665-99b3-8bd8889dcd85\") " pod="openshift-cluster-node-tuning-operator/tuned-sxj58" Jul 27 18:28:46.454185 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.453010 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3706a9c3-0323-4665-99b3-8bd8889dcd85-etc-sysctl-conf\") pod \"tuned-sxj58\" (UID: \"3706a9c3-0323-4665-99b3-8bd8889dcd85\") " pod="openshift-cluster-node-tuning-operator/tuned-sxj58" Jul 27 18:28:46.454185 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.453032 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7d279661-f5f8-49ca-86c1-3c6f863340c8-host-var-lib-cni-multus\") pod \"multus-pgzw4\" (UID: \"7d279661-f5f8-49ca-86c1-3c6f863340c8\") " pod="openshift-multus/multus-pgzw4" Jul 27 18:28:46.454185 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.453134 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgq78\" (UniqueName: \"kubernetes.io/projected/7d279661-f5f8-49ca-86c1-3c6f863340c8-kube-api-access-vgq78\") pod \"multus-pgzw4\" (UID: \"7d279661-f5f8-49ca-86c1-3c6f863340c8\") " pod="openshift-multus/multus-pgzw4" Jul 27 18:28:46.454185 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.453160 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6tzv\" (UniqueName: \"kubernetes.io/projected/af18a2fc-71f3-4f54-9907-89e31bd65bc9-kube-api-access-c6tzv\") pod \"aws-ebs-csi-driver-node-h9d4j\" (UID: \"af18a2fc-71f3-4f54-9907-89e31bd65bc9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h9d4j" Jul 27 18:28:46.454185 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.453294 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7d279661-f5f8-49ca-86c1-3c6f863340c8-host-var-lib-cni-multus\") pod \"multus-pgzw4\" (UID: \"7d279661-f5f8-49ca-86c1-3c6f863340c8\") " pod="openshift-multus/multus-pgzw4" Jul 27 18:28:46.454185 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.453302 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4f79aa08-0ec5-4fd9-a5a2-06c0d9a755ef-var-lib-openvswitch\") pod \"ovnkube-node-vnbl8\" (UID: \"4f79aa08-0ec5-4fd9-a5a2-06c0d9a755ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-vnbl8" Jul 27 18:28:46.454185 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.453348 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4f79aa08-0ec5-4fd9-a5a2-06c0d9a755ef-var-lib-openvswitch\") pod \"ovnkube-node-vnbl8\" (UID: \"4f79aa08-0ec5-4fd9-a5a2-06c0d9a755ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-vnbl8" Jul 27 18:28:46.454185 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.453348 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4f79aa08-0ec5-4fd9-a5a2-06c0d9a755ef-run-ovn\") pod \"ovnkube-node-vnbl8\" (UID: \"4f79aa08-0ec5-4fd9-a5a2-06c0d9a755ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-vnbl8" Jul 27 18:28:46.454185 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.453387 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7d279661-f5f8-49ca-86c1-3c6f863340c8-etc-kubernetes\") pod \"multus-pgzw4\" (UID: \"7d279661-f5f8-49ca-86c1-3c6f863340c8\") " pod="openshift-multus/multus-pgzw4" Jul 27 18:28:46.454185 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.453411 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fv4z\" (UniqueName: \"kubernetes.io/projected/83c21902-488a-44d1-b494-7000443a9202-kube-api-access-5fv4z\") pod \"iptables-alerter-8phm4\" (UID: \"83c21902-488a-44d1-b494-7000443a9202\") " pod="openshift-network-operator/iptables-alerter-8phm4" Jul 27 18:28:46.454185 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.453387 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4f79aa08-0ec5-4fd9-a5a2-06c0d9a755ef-run-ovn\") pod \"ovnkube-node-vnbl8\" (UID: \"4f79aa08-0ec5-4fd9-a5a2-06c0d9a755ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-vnbl8" Jul 27 18:28:46.454185 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.453438 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjwxt\" (UniqueName: \"kubernetes.io/projected/4f79aa08-0ec5-4fd9-a5a2-06c0d9a755ef-kube-api-access-zjwxt\") pod \"ovnkube-node-vnbl8\" (UID: \"4f79aa08-0ec5-4fd9-a5a2-06c0d9a755ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-vnbl8" Jul 27 18:28:46.454185 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.453485 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8bbbb432-a482-444d-99d3-f2615e9b99bb-host\") pod \"node-ca-fdsdr\" (UID: \"8bbbb432-a482-444d-99d3-f2615e9b99bb\") " pod="openshift-image-registry/node-ca-fdsdr" Jul 27 18:28:46.454185 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.453471 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7d279661-f5f8-49ca-86c1-3c6f863340c8-etc-kubernetes\") pod \"multus-pgzw4\" (UID: \"7d279661-f5f8-49ca-86c1-3c6f863340c8\") " pod="openshift-multus/multus-pgzw4" Jul 27 18:28:46.454185 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.453548 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3706a9c3-0323-4665-99b3-8bd8889dcd85-sys\") pod \"tuned-sxj58\" (UID: \"3706a9c3-0323-4665-99b3-8bd8889dcd85\") " pod="openshift-cluster-node-tuning-operator/tuned-sxj58" Jul 27 18:28:46.454185 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.453538 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8bbbb432-a482-444d-99d3-f2615e9b99bb-host\") pod \"node-ca-fdsdr\" (UID: \"8bbbb432-a482-444d-99d3-f2615e9b99bb\") " pod="openshift-image-registry/node-ca-fdsdr" Jul 27 18:28:46.455021 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.453511 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3706a9c3-0323-4665-99b3-8bd8889dcd85-sys\") pod \"tuned-sxj58\" (UID: \"3706a9c3-0323-4665-99b3-8bd8889dcd85\") " pod="openshift-cluster-node-tuning-operator/tuned-sxj58" Jul 27 18:28:46.455021 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.453652 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3706a9c3-0323-4665-99b3-8bd8889dcd85-var-lib-kubelet\") pod \"tuned-sxj58\" (UID: \"3706a9c3-0323-4665-99b3-8bd8889dcd85\") " pod="openshift-cluster-node-tuning-operator/tuned-sxj58" Jul 27 18:28:46.455021 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.453668 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsp6g\" (UniqueName: \"kubernetes.io/projected/3706a9c3-0323-4665-99b3-8bd8889dcd85-kube-api-access-gsp6g\") pod \"tuned-sxj58\" (UID: \"3706a9c3-0323-4665-99b3-8bd8889dcd85\") " pod="openshift-cluster-node-tuning-operator/tuned-sxj58" Jul 27 18:28:46.455021 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.453684 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7d279661-f5f8-49ca-86c1-3c6f863340c8-multus-socket-dir-parent\") pod \"multus-pgzw4\" (UID: \"7d279661-f5f8-49ca-86c1-3c6f863340c8\") " pod="openshift-multus/multus-pgzw4" Jul 27 18:28:46.455021 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.453699 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3706a9c3-0323-4665-99b3-8bd8889dcd85-var-lib-kubelet\") pod \"tuned-sxj58\" (UID: \"3706a9c3-0323-4665-99b3-8bd8889dcd85\") " pod="openshift-cluster-node-tuning-operator/tuned-sxj58" Jul 27 18:28:46.455021 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.453703 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7d279661-f5f8-49ca-86c1-3c6f863340c8-host-run-k8s-cni-cncf-io\") pod \"multus-pgzw4\" (UID: \"7d279661-f5f8-49ca-86c1-3c6f863340c8\") " pod="openshift-multus/multus-pgzw4" Jul 27 18:28:46.455021 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.453723 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3706a9c3-0323-4665-99b3-8bd8889dcd85-etc-sysconfig\") pod \"tuned-sxj58\" (UID: \"3706a9c3-0323-4665-99b3-8bd8889dcd85\") " pod="openshift-cluster-node-tuning-operator/tuned-sxj58" Jul 27 18:28:46.455021 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.453743 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3706a9c3-0323-4665-99b3-8bd8889dcd85-etc-kubernetes\") pod \"tuned-sxj58\" (UID: \"3706a9c3-0323-4665-99b3-8bd8889dcd85\") " pod="openshift-cluster-node-tuning-operator/tuned-sxj58" Jul 27 18:28:46.455021 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.453746 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7d279661-f5f8-49ca-86c1-3c6f863340c8-multus-socket-dir-parent\") pod \"multus-pgzw4\" (UID: \"7d279661-f5f8-49ca-86c1-3c6f863340c8\") " pod="openshift-multus/multus-pgzw4" Jul 27 18:28:46.455021 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.453762 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7d279661-f5f8-49ca-86c1-3c6f863340c8-host-run-k8s-cni-cncf-io\") pod \"multus-pgzw4\" (UID: \"7d279661-f5f8-49ca-86c1-3c6f863340c8\") " pod="openshift-multus/multus-pgzw4" Jul 27 18:28:46.455021 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.453776 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7d279661-f5f8-49ca-86c1-3c6f863340c8-multus-conf-dir\") pod \"multus-pgzw4\" (UID: \"7d279661-f5f8-49ca-86c1-3c6f863340c8\") " pod="openshift-multus/multus-pgzw4" Jul 27 18:28:46.455021 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.453790 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3706a9c3-0323-4665-99b3-8bd8889dcd85-etc-kubernetes\") pod \"tuned-sxj58\" (UID: \"3706a9c3-0323-4665-99b3-8bd8889dcd85\") " pod="openshift-cluster-node-tuning-operator/tuned-sxj58" Jul 27 18:28:46.455021 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.453806 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/83c21902-488a-44d1-b494-7000443a9202-host-slash\") pod \"iptables-alerter-8phm4\" (UID: \"83c21902-488a-44d1-b494-7000443a9202\") " pod="openshift-network-operator/iptables-alerter-8phm4" Jul 27 18:28:46.455021 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.453827 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3706a9c3-0323-4665-99b3-8bd8889dcd85-etc-sysconfig\") pod \"tuned-sxj58\" (UID: \"3706a9c3-0323-4665-99b3-8bd8889dcd85\") " pod="openshift-cluster-node-tuning-operator/tuned-sxj58" Jul 27 18:28:46.455021 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.453835 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7d279661-f5f8-49ca-86c1-3c6f863340c8-multus-conf-dir\") pod \"multus-pgzw4\" (UID: \"7d279661-f5f8-49ca-86c1-3c6f863340c8\") " pod="openshift-multus/multus-pgzw4" Jul 27 18:28:46.455021 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.453874 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4f79aa08-0ec5-4fd9-a5a2-06c0d9a755ef-systemd-units\") pod \"ovnkube-node-vnbl8\" (UID: \"4f79aa08-0ec5-4fd9-a5a2-06c0d9a755ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-vnbl8" Jul 27 18:28:46.455021 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.453888 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/83c21902-488a-44d1-b494-7000443a9202-host-slash\") pod \"iptables-alerter-8phm4\" (UID: \"83c21902-488a-44d1-b494-7000443a9202\") " pod="openshift-network-operator/iptables-alerter-8phm4" Jul 27 18:28:46.455021 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.453898 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4f79aa08-0ec5-4fd9-a5a2-06c0d9a755ef-host-slash\") pod \"ovnkube-node-vnbl8\" (UID: \"4f79aa08-0ec5-4fd9-a5a2-06c0d9a755ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-vnbl8" Jul 27 18:28:46.455908 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.453901 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4f79aa08-0ec5-4fd9-a5a2-06c0d9a755ef-systemd-units\") pod \"ovnkube-node-vnbl8\" (UID: \"4f79aa08-0ec5-4fd9-a5a2-06c0d9a755ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-vnbl8" Jul 27 18:28:46.455908 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.453924 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4f79aa08-0ec5-4fd9-a5a2-06c0d9a755ef-run-openvswitch\") pod \"ovnkube-node-vnbl8\" (UID: \"4f79aa08-0ec5-4fd9-a5a2-06c0d9a755ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-vnbl8" Jul 27 18:28:46.455908 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.453940 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4f79aa08-0ec5-4fd9-a5a2-06c0d9a755ef-host-slash\") pod \"ovnkube-node-vnbl8\" (UID: \"4f79aa08-0ec5-4fd9-a5a2-06c0d9a755ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-vnbl8" Jul 27 18:28:46.455908 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.453949 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bd3d598d-7458-4407-ba85-49402348da87-original-pull-secret\") pod \"global-pull-secret-syncer-pr8j9\" (UID: \"bd3d598d-7458-4407-ba85-49402348da87\") " pod="kube-system/global-pull-secret-syncer-pr8j9" Jul 27 18:28:46.455908 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.453975 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4f79aa08-0ec5-4fd9-a5a2-06c0d9a755ef-run-openvswitch\") pod \"ovnkube-node-vnbl8\" (UID: \"4f79aa08-0ec5-4fd9-a5a2-06c0d9a755ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-vnbl8" Jul 27 18:28:46.455908 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.453977 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5ba4cb7b-80f6-470b-8ab2-7e22c50af412-system-cni-dir\") pod \"multus-additional-cni-plugins-6jzx9\" (UID: \"5ba4cb7b-80f6-470b-8ab2-7e22c50af412\") " pod="openshift-multus/multus-additional-cni-plugins-6jzx9" Jul 27 18:28:46.455908 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:28:46.454012 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 27 18:28:46.455908 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.454017 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5ba4cb7b-80f6-470b-8ab2-7e22c50af412-system-cni-dir\") pod \"multus-additional-cni-plugins-6jzx9\" (UID: \"5ba4cb7b-80f6-470b-8ab2-7e22c50af412\") " pod="openshift-multus/multus-additional-cni-plugins-6jzx9" Jul 27 18:28:46.455908 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.454051 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7d279661-f5f8-49ca-86c1-3c6f863340c8-system-cni-dir\") pod \"multus-pgzw4\" (UID: \"7d279661-f5f8-49ca-86c1-3c6f863340c8\") " pod="openshift-multus/multus-pgzw4" Jul 27 18:28:46.455908 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:28:46.454068 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd3d598d-7458-4407-ba85-49402348da87-original-pull-secret podName:bd3d598d-7458-4407-ba85-49402348da87 nodeName:}" failed. No retries permitted until 2026-07-27 18:28:46.954043547 +0000 UTC m=+2.015732555 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bd3d598d-7458-4407-ba85-49402348da87-original-pull-secret") pod "global-pull-secret-syncer-pr8j9" (UID: "bd3d598d-7458-4407-ba85-49402348da87") : object "kube-system"/"original-pull-secret" not registered Jul 27 18:28:46.455908 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.454013 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7d279661-f5f8-49ca-86c1-3c6f863340c8-system-cni-dir\") pod \"multus-pgzw4\" (UID: \"7d279661-f5f8-49ca-86c1-3c6f863340c8\") " pod="openshift-multus/multus-pgzw4" Jul 27 18:28:46.455908 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.454093 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7d279661-f5f8-49ca-86c1-3c6f863340c8-cnibin\") pod \"multus-pgzw4\" (UID: \"7d279661-f5f8-49ca-86c1-3c6f863340c8\") " pod="openshift-multus/multus-pgzw4" Jul 27 18:28:46.455908 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.454106 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7d279661-f5f8-49ca-86c1-3c6f863340c8-host-run-netns\") pod \"multus-pgzw4\" (UID: \"7d279661-f5f8-49ca-86c1-3c6f863340c8\") " pod="openshift-multus/multus-pgzw4" Jul 27 18:28:46.455908 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.454124 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4f79aa08-0ec5-4fd9-a5a2-06c0d9a755ef-host-run-ovn-kubernetes\") pod \"ovnkube-node-vnbl8\" (UID: \"4f79aa08-0ec5-4fd9-a5a2-06c0d9a755ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-vnbl8" Jul 27 18:28:46.455908 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.454150 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4f79aa08-0ec5-4fd9-a5a2-06c0d9a755ef-ovnkube-script-lib\") pod \"ovnkube-node-vnbl8\" (UID: \"4f79aa08-0ec5-4fd9-a5a2-06c0d9a755ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-vnbl8" Jul 27 18:28:46.455908 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.454172 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8bbbb432-a482-444d-99d3-f2615e9b99bb-serviceca\") pod \"node-ca-fdsdr\" (UID: \"8bbbb432-a482-444d-99d3-f2615e9b99bb\") " pod="openshift-image-registry/node-ca-fdsdr" Jul 27 18:28:46.455908 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.454197 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3706a9c3-0323-4665-99b3-8bd8889dcd85-run\") pod \"tuned-sxj58\" (UID: \"3706a9c3-0323-4665-99b3-8bd8889dcd85\") " pod="openshift-cluster-node-tuning-operator/tuned-sxj58" Jul 27 18:28:46.456567 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.454203 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7d279661-f5f8-49ca-86c1-3c6f863340c8-host-run-netns\") pod \"multus-pgzw4\" (UID: \"7d279661-f5f8-49ca-86c1-3c6f863340c8\") " pod="openshift-multus/multus-pgzw4" Jul 27 18:28:46.456567 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.454224 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4f79aa08-0ec5-4fd9-a5a2-06c0d9a755ef-host-run-ovn-kubernetes\") pod \"ovnkube-node-vnbl8\" (UID: \"4f79aa08-0ec5-4fd9-a5a2-06c0d9a755ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-vnbl8" Jul 27 18:28:46.456567 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.454261 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bd3d598d-7458-4407-ba85-49402348da87-kubelet-config\") pod \"global-pull-secret-syncer-pr8j9\" (UID: \"bd3d598d-7458-4407-ba85-49402348da87\") " pod="kube-system/global-pull-secret-syncer-pr8j9" Jul 27 18:28:46.456567 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.454290 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/83c21902-488a-44d1-b494-7000443a9202-iptables-alerter-script\") pod \"iptables-alerter-8phm4\" (UID: \"83c21902-488a-44d1-b494-7000443a9202\") " pod="openshift-network-operator/iptables-alerter-8phm4" Jul 27 18:28:46.456567 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.454309 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3706a9c3-0323-4665-99b3-8bd8889dcd85-run\") pod \"tuned-sxj58\" (UID: \"3706a9c3-0323-4665-99b3-8bd8889dcd85\") " pod="openshift-cluster-node-tuning-operator/tuned-sxj58" Jul 27 18:28:46.456567 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.454312 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnk5m\" (UniqueName: \"kubernetes.io/projected/52bcf393-1ae7-4102-b762-569badf3538c-kube-api-access-pnk5m\") pod \"node-resolver-ttprc\" (UID: \"52bcf393-1ae7-4102-b762-569badf3538c\") " pod="openshift-dns/node-resolver-ttprc" Jul 27 18:28:46.456567 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.454349 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4f79aa08-0ec5-4fd9-a5a2-06c0d9a755ef-log-socket\") pod \"ovnkube-node-vnbl8\" (UID: \"4f79aa08-0ec5-4fd9-a5a2-06c0d9a755ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-vnbl8" Jul 27 18:28:46.456567 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.454372 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bd3d598d-7458-4407-ba85-49402348da87-kubelet-config\") pod \"global-pull-secret-syncer-pr8j9\" (UID: \"bd3d598d-7458-4407-ba85-49402348da87\") " pod="kube-system/global-pull-secret-syncer-pr8j9" Jul 27 18:28:46.456567 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.454420 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4f79aa08-0ec5-4fd9-a5a2-06c0d9a755ef-host-cni-bin\") pod \"ovnkube-node-vnbl8\" (UID: \"4f79aa08-0ec5-4fd9-a5a2-06c0d9a755ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-vnbl8" Jul 27 18:28:46.456567 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.454173 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7d279661-f5f8-49ca-86c1-3c6f863340c8-cnibin\") pod \"multus-pgzw4\" (UID: \"7d279661-f5f8-49ca-86c1-3c6f863340c8\") " pod="openshift-multus/multus-pgzw4" Jul 27 18:28:46.456567 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.454378 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4f79aa08-0ec5-4fd9-a5a2-06c0d9a755ef-host-cni-bin\") pod \"ovnkube-node-vnbl8\" (UID: \"4f79aa08-0ec5-4fd9-a5a2-06c0d9a755ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-vnbl8" Jul 27 18:28:46.456567 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.454591 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4f79aa08-0ec5-4fd9-a5a2-06c0d9a755ef-log-socket\") pod \"ovnkube-node-vnbl8\" (UID: \"4f79aa08-0ec5-4fd9-a5a2-06c0d9a755ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-vnbl8" Jul 27 18:28:46.456567 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.454634 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4f79aa08-0ec5-4fd9-a5a2-06c0d9a755ef-ovnkube-config\") pod \"ovnkube-node-vnbl8\" (UID: \"4f79aa08-0ec5-4fd9-a5a2-06c0d9a755ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-vnbl8" Jul 27 18:28:46.456567 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.454667 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5ba4cb7b-80f6-470b-8ab2-7e22c50af412-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6jzx9\" (UID: \"5ba4cb7b-80f6-470b-8ab2-7e22c50af412\") " pod="openshift-multus/multus-additional-cni-plugins-6jzx9" Jul 27 18:28:46.456567 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.454690 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3706a9c3-0323-4665-99b3-8bd8889dcd85-etc-modprobe-d\") pod \"tuned-sxj58\" (UID: \"3706a9c3-0323-4665-99b3-8bd8889dcd85\") " pod="openshift-cluster-node-tuning-operator/tuned-sxj58" Jul 27 18:28:46.456567 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.454703 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4f79aa08-0ec5-4fd9-a5a2-06c0d9a755ef-ovnkube-script-lib\") pod \"ovnkube-node-vnbl8\" (UID: \"4f79aa08-0ec5-4fd9-a5a2-06c0d9a755ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-vnbl8" Jul 27 18:28:46.456567 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.454735 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9c37bc5b-9ef7-4e2b-b555-52de0d9861b8-agent-certs\") pod \"konnectivity-agent-67t26\" (UID: \"9c37bc5b-9ef7-4e2b-b555-52de0d9861b8\") " pod="kube-system/konnectivity-agent-67t26" Jul 27 18:28:46.456567 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.454757 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9c37bc5b-9ef7-4e2b-b555-52de0d9861b8-konnectivity-ca\") pod \"konnectivity-agent-67t26\" (UID: \"9c37bc5b-9ef7-4e2b-b555-52de0d9861b8\") " pod="kube-system/konnectivity-agent-67t26" Jul 27 18:28:46.457183 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.454774 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8bbbb432-a482-444d-99d3-f2615e9b99bb-serviceca\") pod \"node-ca-fdsdr\" (UID: \"8bbbb432-a482-444d-99d3-f2615e9b99bb\") " pod="openshift-image-registry/node-ca-fdsdr" Jul 27 18:28:46.457183 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.454776 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bd3d598d-7458-4407-ba85-49402348da87-dbus\") pod \"global-pull-secret-syncer-pr8j9\" (UID: \"bd3d598d-7458-4407-ba85-49402348da87\") " pod="kube-system/global-pull-secret-syncer-pr8j9" Jul 27 18:28:46.457183 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.454820 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bd3d598d-7458-4407-ba85-49402348da87-dbus\") pod \"global-pull-secret-syncer-pr8j9\" (UID: \"bd3d598d-7458-4407-ba85-49402348da87\") " pod="kube-system/global-pull-secret-syncer-pr8j9" Jul 27 18:28:46.457183 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.454822 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/af18a2fc-71f3-4f54-9907-89e31bd65bc9-socket-dir\") pod \"aws-ebs-csi-driver-node-h9d4j\" (UID: \"af18a2fc-71f3-4f54-9907-89e31bd65bc9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h9d4j" Jul 27 18:28:46.457183 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.454843 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5ba4cb7b-80f6-470b-8ab2-7e22c50af412-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6jzx9\" (UID: \"5ba4cb7b-80f6-470b-8ab2-7e22c50af412\") " pod="openshift-multus/multus-additional-cni-plugins-6jzx9" Jul 27 18:28:46.457183 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.454861 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/af18a2fc-71f3-4f54-9907-89e31bd65bc9-sys-fs\") pod \"aws-ebs-csi-driver-node-h9d4j\" (UID: \"af18a2fc-71f3-4f54-9907-89e31bd65bc9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h9d4j" Jul 27 18:28:46.457183 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.454885 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rr8z7\" (UniqueName: \"kubernetes.io/projected/8ce054d9-d9c1-44ef-9e32-5a4c502e0ced-kube-api-access-rr8z7\") pod \"network-check-target-tgxn8\" (UID: \"8ce054d9-d9c1-44ef-9e32-5a4c502e0ced\") " pod="openshift-network-diagnostics/network-check-target-tgxn8" Jul 27 18:28:46.457183 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.454911 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/af18a2fc-71f3-4f54-9907-89e31bd65bc9-socket-dir\") pod \"aws-ebs-csi-driver-node-h9d4j\" (UID: \"af18a2fc-71f3-4f54-9907-89e31bd65bc9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h9d4j" Jul 27 18:28:46.457183 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.454911 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5ba4cb7b-80f6-470b-8ab2-7e22c50af412-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6jzx9\" (UID: \"5ba4cb7b-80f6-470b-8ab2-7e22c50af412\") " pod="openshift-multus/multus-additional-cni-plugins-6jzx9" Jul 27 18:28:46.457183 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.454939 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/83c21902-488a-44d1-b494-7000443a9202-iptables-alerter-script\") pod \"iptables-alerter-8phm4\" (UID: \"83c21902-488a-44d1-b494-7000443a9202\") " pod="openshift-network-operator/iptables-alerter-8phm4" Jul 27 18:28:46.457183 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.454952 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/af18a2fc-71f3-4f54-9907-89e31bd65bc9-registration-dir\") pod \"aws-ebs-csi-driver-node-h9d4j\" (UID: \"af18a2fc-71f3-4f54-9907-89e31bd65bc9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h9d4j" Jul 27 18:28:46.457183 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.454976 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3706a9c3-0323-4665-99b3-8bd8889dcd85-tmp\") pod \"tuned-sxj58\" (UID: \"3706a9c3-0323-4665-99b3-8bd8889dcd85\") " pod="openshift-cluster-node-tuning-operator/tuned-sxj58" Jul 27 18:28:46.457183 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.454993 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/af18a2fc-71f3-4f54-9907-89e31bd65bc9-sys-fs\") pod \"aws-ebs-csi-driver-node-h9d4j\" (UID: \"af18a2fc-71f3-4f54-9907-89e31bd65bc9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h9d4j" Jul 27 18:28:46.457183 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.454997 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7d279661-f5f8-49ca-86c1-3c6f863340c8-multus-daemon-config\") pod \"multus-pgzw4\" (UID: \"7d279661-f5f8-49ca-86c1-3c6f863340c8\") " pod="openshift-multus/multus-pgzw4" Jul 27 18:28:46.457183 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.455030 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/52bcf393-1ae7-4102-b762-569badf3538c-hosts-file\") pod \"node-resolver-ttprc\" (UID: \"52bcf393-1ae7-4102-b762-569badf3538c\") " pod="openshift-dns/node-resolver-ttprc" Jul 27 18:28:46.457183 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.455098 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/52bcf393-1ae7-4102-b762-569badf3538c-hosts-file\") pod \"node-resolver-ttprc\" (UID: \"52bcf393-1ae7-4102-b762-569badf3538c\") " pod="openshift-dns/node-resolver-ttprc" Jul 27 18:28:46.457183 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.455352 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4f79aa08-0ec5-4fd9-a5a2-06c0d9a755ef-ovnkube-config\") pod \"ovnkube-node-vnbl8\" (UID: \"4f79aa08-0ec5-4fd9-a5a2-06c0d9a755ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-vnbl8" Jul 27 18:28:46.457830 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.455461 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3706a9c3-0323-4665-99b3-8bd8889dcd85-etc-modprobe-d\") pod \"tuned-sxj58\" (UID: \"3706a9c3-0323-4665-99b3-8bd8889dcd85\") " pod="openshift-cluster-node-tuning-operator/tuned-sxj58" Jul 27 18:28:46.457830 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.455494 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7d279661-f5f8-49ca-86c1-3c6f863340c8-multus-daemon-config\") pod \"multus-pgzw4\" (UID: \"7d279661-f5f8-49ca-86c1-3c6f863340c8\") " pod="openshift-multus/multus-pgzw4" Jul 27 18:28:46.457830 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.455518 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/af18a2fc-71f3-4f54-9907-89e31bd65bc9-registration-dir\") pod \"aws-ebs-csi-driver-node-h9d4j\" (UID: \"af18a2fc-71f3-4f54-9907-89e31bd65bc9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h9d4j" Jul 27 18:28:46.457830 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.455721 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9c37bc5b-9ef7-4e2b-b555-52de0d9861b8-konnectivity-ca\") pod \"konnectivity-agent-67t26\" (UID: \"9c37bc5b-9ef7-4e2b-b555-52de0d9861b8\") " pod="kube-system/konnectivity-agent-67t26" Jul 27 18:28:46.457830 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.455732 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5ba4cb7b-80f6-470b-8ab2-7e22c50af412-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6jzx9\" (UID: \"5ba4cb7b-80f6-470b-8ab2-7e22c50af412\") " pod="openshift-multus/multus-additional-cni-plugins-6jzx9" Jul 27 18:28:46.457830 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.456164 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3706a9c3-0323-4665-99b3-8bd8889dcd85-etc-tuned\") pod \"tuned-sxj58\" (UID: \"3706a9c3-0323-4665-99b3-8bd8889dcd85\") " pod="openshift-cluster-node-tuning-operator/tuned-sxj58" Jul 27 18:28:46.457830 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.456317 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4f79aa08-0ec5-4fd9-a5a2-06c0d9a755ef-ovn-node-metrics-cert\") pod \"ovnkube-node-vnbl8\" (UID: \"4f79aa08-0ec5-4fd9-a5a2-06c0d9a755ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-vnbl8" Jul 27 18:28:46.458169 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.457946 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3706a9c3-0323-4665-99b3-8bd8889dcd85-tmp\") pod \"tuned-sxj58\" (UID: \"3706a9c3-0323-4665-99b3-8bd8889dcd85\") " pod="openshift-cluster-node-tuning-operator/tuned-sxj58" Jul 27 18:28:46.458169 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.457952 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9c37bc5b-9ef7-4e2b-b555-52de0d9861b8-agent-certs\") pod \"konnectivity-agent-67t26\" (UID: \"9c37bc5b-9ef7-4e2b-b555-52de0d9861b8\") " pod="kube-system/konnectivity-agent-67t26" Jul 27 18:28:46.469102 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:28:46.469079 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 27 18:28:46.469220 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:28:46.469112 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 27 18:28:46.469220 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:28:46.469128 2579 projected.go:196] Error preparing data for projected volume kube-api-access-rr8z7 for pod openshift-network-diagnostics/network-check-target-tgxn8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 18:28:46.469354 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:28:46.469257 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8ce054d9-d9c1-44ef-9e32-5a4c502e0ced-kube-api-access-rr8z7 podName:8ce054d9-d9c1-44ef-9e32-5a4c502e0ced nodeName:}" failed. No retries permitted until 2026-07-27 18:28:46.96921588 +0000 UTC m=+2.030904903 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-rr8z7" (UniqueName: "kubernetes.io/projected/8ce054d9-d9c1-44ef-9e32-5a4c502e0ced-kube-api-access-rr8z7") pod "network-check-target-tgxn8" (UID: "8ce054d9-d9c1-44ef-9e32-5a4c502e0ced") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 18:28:46.469475 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.469421 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqb8v\" (UniqueName: \"kubernetes.io/projected/5ba4cb7b-80f6-470b-8ab2-7e22c50af412-kube-api-access-gqb8v\") pod \"multus-additional-cni-plugins-6jzx9\" (UID: \"5ba4cb7b-80f6-470b-8ab2-7e22c50af412\") " pod="openshift-multus/multus-additional-cni-plugins-6jzx9" Jul 27 18:28:46.470457 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.470433 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fv4z\" (UniqueName: \"kubernetes.io/projected/83c21902-488a-44d1-b494-7000443a9202-kube-api-access-5fv4z\") pod \"iptables-alerter-8phm4\" (UID: \"83c21902-488a-44d1-b494-7000443a9202\") " pod="openshift-network-operator/iptables-alerter-8phm4" Jul 27 18:28:46.470631 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.470615 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6tzv\" (UniqueName: \"kubernetes.io/projected/af18a2fc-71f3-4f54-9907-89e31bd65bc9-kube-api-access-c6tzv\") pod \"aws-ebs-csi-driver-node-h9d4j\" (UID: \"af18a2fc-71f3-4f54-9907-89e31bd65bc9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h9d4j" Jul 27 18:28:46.471167 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.471144 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjwxt\" (UniqueName: \"kubernetes.io/projected/4f79aa08-0ec5-4fd9-a5a2-06c0d9a755ef-kube-api-access-zjwxt\") pod \"ovnkube-node-vnbl8\" (UID: \"4f79aa08-0ec5-4fd9-a5a2-06c0d9a755ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-vnbl8" Jul 27 18:28:46.471510 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.471488 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxzwx\" (UniqueName: \"kubernetes.io/projected/8bbbb432-a482-444d-99d3-f2615e9b99bb-kube-api-access-nxzwx\") pod \"node-ca-fdsdr\" (UID: \"8bbbb432-a482-444d-99d3-f2615e9b99bb\") " pod="openshift-image-registry/node-ca-fdsdr" Jul 27 18:28:46.471606 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.471547 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgq78\" (UniqueName: \"kubernetes.io/projected/7d279661-f5f8-49ca-86c1-3c6f863340c8-kube-api-access-vgq78\") pod \"multus-pgzw4\" (UID: \"7d279661-f5f8-49ca-86c1-3c6f863340c8\") " pod="openshift-multus/multus-pgzw4" Jul 27 18:28:46.471721 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.471705 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnk5m\" (UniqueName: \"kubernetes.io/projected/52bcf393-1ae7-4102-b762-569badf3538c-kube-api-access-pnk5m\") pod \"node-resolver-ttprc\" (UID: \"52bcf393-1ae7-4102-b762-569badf3538c\") " pod="openshift-dns/node-resolver-ttprc" Jul 27 18:28:46.473114 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.473093 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsp6g\" (UniqueName: \"kubernetes.io/projected/3706a9c3-0323-4665-99b3-8bd8889dcd85-kube-api-access-gsp6g\") pod \"tuned-sxj58\" (UID: \"3706a9c3-0323-4665-99b3-8bd8889dcd85\") " pod="openshift-cluster-node-tuning-operator/tuned-sxj58" Jul 27 18:28:46.487063 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.487044 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-sxj58" Jul 27 18:28:46.492542 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.492527 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-67t26" Jul 27 18:28:46.511063 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:46.511032 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c37bc5b_9ef7_4e2b_b555_52de0d9861b8.slice/crio-edae0eac3666faf987ec3a5e69e6896c75b67d8f72c230cd28cb791ee6a9cd5a WatchSource:0}: Error finding container edae0eac3666faf987ec3a5e69e6896c75b67d8f72c230cd28cb791ee6a9cd5a: Status 404 returned error can't find the container with id edae0eac3666faf987ec3a5e69e6896c75b67d8f72c230cd28cb791ee6a9cd5a Jul 27 18:28:46.511414 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:46.511392 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bf56391e54c962db9801fd2ce6f6f03.slice/crio-30325422e723ad8ada8e0dffa28960e76a26e3d6a44fe81648e692e9a11c3331 WatchSource:0}: Error finding container 30325422e723ad8ada8e0dffa28960e76a26e3d6a44fe81648e692e9a11c3331: Status 404 returned error can't find the container with id 30325422e723ad8ada8e0dffa28960e76a26e3d6a44fe81648e692e9a11c3331 Jul 27 18:28:46.511650 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:46.511632 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8aa0e86aff493755e5519df8f9d22c9f.slice/crio-7f3d7bd97cd8fc1d925c8ce5610036d41c5b3ff7275a6f3a3a148e52c458c786 WatchSource:0}: Error finding container 7f3d7bd97cd8fc1d925c8ce5610036d41c5b3ff7275a6f3a3a148e52c458c786: Status 404 returned error can't find the container with id 7f3d7bd97cd8fc1d925c8ce5610036d41c5b3ff7275a6f3a3a148e52c458c786 Jul 27 18:28:46.512127 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:46.512096 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3706a9c3_0323_4665_99b3_8bd8889dcd85.slice/crio-d80300ff41cdd587ebe8c114d6d06688873fe3f97cd0d3bf83c6e680ef464ce9 WatchSource:0}: Error finding container d80300ff41cdd587ebe8c114d6d06688873fe3f97cd0d3bf83c6e680ef464ce9: Status 404 returned error can't find the container with id d80300ff41cdd587ebe8c114d6d06688873fe3f97cd0d3bf83c6e680ef464ce9 Jul 27 18:28:46.516041 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.516027 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 27 18:28:46.537192 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.537170 2579 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 27 18:28:46.555489 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.555462 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkrwl\" (UniqueName: \"kubernetes.io/projected/270a2080-a3dd-4227-9abc-c093f70df909-kube-api-access-fkrwl\") pod \"network-metrics-daemon-6htnl\" (UID: \"270a2080-a3dd-4227-9abc-c093f70df909\") " pod="openshift-multus/network-metrics-daemon-6htnl" Jul 27 18:28:46.555598 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.555501 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/270a2080-a3dd-4227-9abc-c093f70df909-metrics-certs\") pod \"network-metrics-daemon-6htnl\" (UID: \"270a2080-a3dd-4227-9abc-c093f70df909\") " pod="openshift-multus/network-metrics-daemon-6htnl" Jul 27 18:28:46.555662 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:28:46.555605 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 18:28:46.555715 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:28:46.555663 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/270a2080-a3dd-4227-9abc-c093f70df909-metrics-certs podName:270a2080-a3dd-4227-9abc-c093f70df909 nodeName:}" failed. No retries permitted until 2026-07-27 18:28:47.055642646 +0000 UTC m=+2.117331672 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/270a2080-a3dd-4227-9abc-c093f70df909-metrics-certs") pod "network-metrics-daemon-6htnl" (UID: "270a2080-a3dd-4227-9abc-c093f70df909") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 18:28:46.578707 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.578679 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkrwl\" (UniqueName: \"kubernetes.io/projected/270a2080-a3dd-4227-9abc-c093f70df909-kube-api-access-fkrwl\") pod \"network-metrics-daemon-6htnl\" (UID: \"270a2080-a3dd-4227-9abc-c093f70df909\") " pod="openshift-multus/network-metrics-daemon-6htnl" Jul 27 18:28:46.667384 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.667315 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-6jzx9" Jul 27 18:28:46.688075 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.688057 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-pgzw4" Jul 27 18:28:46.694549 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:46.694528 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d279661_f5f8_49ca_86c1_3c6f863340c8.slice/crio-cd1834a916ae512cb3b7e09a2707c61c768b7a698dac9e9bb0c380c11040dc51 WatchSource:0}: Error finding container cd1834a916ae512cb3b7e09a2707c61c768b7a698dac9e9bb0c380c11040dc51: Status 404 returned error can't find the container with id cd1834a916ae512cb3b7e09a2707c61c768b7a698dac9e9bb0c380c11040dc51 Jul 27 18:28:46.696619 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.696599 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-8phm4" Jul 27 18:28:46.705450 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:46.705426 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83c21902_488a_44d1_b494_7000443a9202.slice/crio-04b9626ecbf3afdc3ccea91fcf3edb14307c620a452cdeb758c6bee18c1815bd WatchSource:0}: Error finding container 04b9626ecbf3afdc3ccea91fcf3edb14307c620a452cdeb758c6bee18c1815bd: Status 404 returned error can't find the container with id 04b9626ecbf3afdc3ccea91fcf3edb14307c620a452cdeb758c6bee18c1815bd Jul 27 18:28:46.719984 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.719965 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-vnbl8" Jul 27 18:28:46.726692 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.726667 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h9d4j" Jul 27 18:28:46.726906 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:46.726883 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f79aa08_0ec5_4fd9_a5a2_06c0d9a755ef.slice/crio-d3a6a45d415288304829e3a89874f9a57f37d37e5b38727dd3ba11ea55ec581f WatchSource:0}: Error finding container d3a6a45d415288304829e3a89874f9a57f37d37e5b38727dd3ba11ea55ec581f: Status 404 returned error can't find the container with id d3a6a45d415288304829e3a89874f9a57f37d37e5b38727dd3ba11ea55ec581f Jul 27 18:28:46.733511 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:46.733488 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf18a2fc_71f3_4f54_9907_89e31bd65bc9.slice/crio-92de07b95d7c8c455e71863a3fcbbbc1d549eaa97011373ee2c4f5220dbf5505 WatchSource:0}: Error finding container 92de07b95d7c8c455e71863a3fcbbbc1d549eaa97011373ee2c4f5220dbf5505: Status 404 returned error can't find the container with id 92de07b95d7c8c455e71863a3fcbbbc1d549eaa97011373ee2c4f5220dbf5505 Jul 27 18:28:46.741480 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.741461 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-ttprc" Jul 27 18:28:46.748492 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:46.748472 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52bcf393_1ae7_4102_b762_569badf3538c.slice/crio-bec344a67e7141f37984bcc746d3569084750ba496c31c4dd1076816b5aed1fe WatchSource:0}: Error finding container bec344a67e7141f37984bcc746d3569084750ba496c31c4dd1076816b5aed1fe: Status 404 returned error can't find the container with id bec344a67e7141f37984bcc746d3569084750ba496c31c4dd1076816b5aed1fe Jul 27 18:28:46.757314 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.757298 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-fdsdr" Jul 27 18:28:46.765761 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:28:46.765743 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bbbb432_a482_444d_99d3_f2615e9b99bb.slice/crio-ccc6719590b6a7e4a94038c309782a0d6ccfc457208115a0420d27aa32c75bf1 WatchSource:0}: Error finding container ccc6719590b6a7e4a94038c309782a0d6ccfc457208115a0420d27aa32c75bf1: Status 404 returned error can't find the container with id ccc6719590b6a7e4a94038c309782a0d6ccfc457208115a0420d27aa32c75bf1 Jul 27 18:28:46.959484 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:46.959403 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bd3d598d-7458-4407-ba85-49402348da87-original-pull-secret\") pod \"global-pull-secret-syncer-pr8j9\" (UID: \"bd3d598d-7458-4407-ba85-49402348da87\") " pod="kube-system/global-pull-secret-syncer-pr8j9" Jul 27 18:28:46.959613 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:28:46.959514 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 27 18:28:46.959613 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:28:46.959561 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd3d598d-7458-4407-ba85-49402348da87-original-pull-secret podName:bd3d598d-7458-4407-ba85-49402348da87 nodeName:}" failed. No retries permitted until 2026-07-27 18:28:47.959547474 +0000 UTC m=+3.021236483 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bd3d598d-7458-4407-ba85-49402348da87-original-pull-secret") pod "global-pull-secret-syncer-pr8j9" (UID: "bd3d598d-7458-4407-ba85-49402348da87") : object "kube-system"/"original-pull-secret" not registered Jul 27 18:28:47.060620 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:47.060590 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rr8z7\" (UniqueName: \"kubernetes.io/projected/8ce054d9-d9c1-44ef-9e32-5a4c502e0ced-kube-api-access-rr8z7\") pod \"network-check-target-tgxn8\" (UID: \"8ce054d9-d9c1-44ef-9e32-5a4c502e0ced\") " pod="openshift-network-diagnostics/network-check-target-tgxn8" Jul 27 18:28:47.060782 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:47.060629 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/270a2080-a3dd-4227-9abc-c093f70df909-metrics-certs\") pod \"network-metrics-daemon-6htnl\" (UID: \"270a2080-a3dd-4227-9abc-c093f70df909\") " pod="openshift-multus/network-metrics-daemon-6htnl" Jul 27 18:28:47.060782 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:28:47.060712 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 18:28:47.060782 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:28:47.060731 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 27 18:28:47.060782 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:28:47.060751 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 27 18:28:47.060782 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:28:47.060755 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/270a2080-a3dd-4227-9abc-c093f70df909-metrics-certs podName:270a2080-a3dd-4227-9abc-c093f70df909 nodeName:}" failed. No retries permitted until 2026-07-27 18:28:48.06074345 +0000 UTC m=+3.122432459 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/270a2080-a3dd-4227-9abc-c093f70df909-metrics-certs") pod "network-metrics-daemon-6htnl" (UID: "270a2080-a3dd-4227-9abc-c093f70df909") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 18:28:47.060782 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:28:47.060761 2579 projected.go:196] Error preparing data for projected volume kube-api-access-rr8z7 for pod openshift-network-diagnostics/network-check-target-tgxn8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 18:28:47.061030 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:28:47.060796 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8ce054d9-d9c1-44ef-9e32-5a4c502e0ced-kube-api-access-rr8z7 podName:8ce054d9-d9c1-44ef-9e32-5a4c502e0ced nodeName:}" failed. No retries permitted until 2026-07-27 18:28:48.060785097 +0000 UTC m=+3.122474107 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-rr8z7" (UniqueName: "kubernetes.io/projected/8ce054d9-d9c1-44ef-9e32-5a4c502e0ced-kube-api-access-rr8z7") pod "network-check-target-tgxn8" (UID: "8ce054d9-d9c1-44ef-9e32-5a4c502e0ced") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 18:28:47.259435 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:47.259359 2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 27 18:28:47.385163 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:47.385097 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-26 18:23:46 +0000 UTC" deadline="2028-03-16 20:42:59.264694798 +0000 UTC" Jul 27 18:28:47.385163 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:47.385162 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14354h14m11.879536912s" Jul 27 18:28:47.489663 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:47.489629 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6htnl" Jul 27 18:28:47.489842 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:28:47.489768 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6htnl" podUID="270a2080-a3dd-4227-9abc-c093f70df909" Jul 27 18:28:47.489904 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:47.489854 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgxn8" Jul 27 18:28:47.489957 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:28:47.489934 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tgxn8" podUID="8ce054d9-d9c1-44ef-9e32-5a4c502e0ced" Jul 27 18:28:47.506540 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:47.506475 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6jzx9" event={"ID":"5ba4cb7b-80f6-470b-8ab2-7e22c50af412","Type":"ContainerStarted","Data":"f7a121c7aa6b9ed15831781691ecfb1013e28f4daa2b89b1f87bcd035eaf3717"} Jul 27 18:28:47.511615 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:47.511546 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-67t26" event={"ID":"9c37bc5b-9ef7-4e2b-b555-52de0d9861b8","Type":"ContainerStarted","Data":"edae0eac3666faf987ec3a5e69e6896c75b67d8f72c230cd28cb791ee6a9cd5a"} Jul 27 18:28:47.525916 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:47.525885 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-sxj58" event={"ID":"3706a9c3-0323-4665-99b3-8bd8889dcd85","Type":"ContainerStarted","Data":"d80300ff41cdd587ebe8c114d6d06688873fe3f97cd0d3bf83c6e680ef464ce9"} Jul 27 18:28:47.541865 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:47.541721 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vnbl8" event={"ID":"4f79aa08-0ec5-4fd9-a5a2-06c0d9a755ef","Type":"ContainerStarted","Data":"d3a6a45d415288304829e3a89874f9a57f37d37e5b38727dd3ba11ea55ec581f"} Jul 27 18:28:47.548299 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:47.548211 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-32.ec2.internal" event={"ID":"8bf56391e54c962db9801fd2ce6f6f03","Type":"ContainerStarted","Data":"30325422e723ad8ada8e0dffa28960e76a26e3d6a44fe81648e692e9a11c3331"} Jul 27 18:28:47.562494 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:47.562459 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-32.ec2.internal" event={"ID":"8aa0e86aff493755e5519df8f9d22c9f","Type":"ContainerStarted","Data":"7f3d7bd97cd8fc1d925c8ce5610036d41c5b3ff7275a6f3a3a148e52c458c786"} Jul 27 18:28:47.573085 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:47.573055 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-fdsdr" event={"ID":"8bbbb432-a482-444d-99d3-f2615e9b99bb","Type":"ContainerStarted","Data":"ccc6719590b6a7e4a94038c309782a0d6ccfc457208115a0420d27aa32c75bf1"} Jul 27 18:28:47.582320 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:47.580422 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-ttprc" event={"ID":"52bcf393-1ae7-4102-b762-569badf3538c","Type":"ContainerStarted","Data":"bec344a67e7141f37984bcc746d3569084750ba496c31c4dd1076816b5aed1fe"} Jul 27 18:28:47.583169 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:47.583140 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h9d4j" event={"ID":"af18a2fc-71f3-4f54-9907-89e31bd65bc9","Type":"ContainerStarted","Data":"92de07b95d7c8c455e71863a3fcbbbc1d549eaa97011373ee2c4f5220dbf5505"} Jul 27 18:28:47.605922 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:47.605891 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-8phm4" event={"ID":"83c21902-488a-44d1-b494-7000443a9202","Type":"ContainerStarted","Data":"04b9626ecbf3afdc3ccea91fcf3edb14307c620a452cdeb758c6bee18c1815bd"} Jul 27 18:28:47.612265 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:47.612221 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-pgzw4" event={"ID":"7d279661-f5f8-49ca-86c1-3c6f863340c8","Type":"ContainerStarted","Data":"cd1834a916ae512cb3b7e09a2707c61c768b7a698dac9e9bb0c380c11040dc51"} Jul 27 18:28:47.967297 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:47.967185 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bd3d598d-7458-4407-ba85-49402348da87-original-pull-secret\") pod \"global-pull-secret-syncer-pr8j9\" (UID: \"bd3d598d-7458-4407-ba85-49402348da87\") " pod="kube-system/global-pull-secret-syncer-pr8j9" Jul 27 18:28:47.967510 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:28:47.967369 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 27 18:28:47.967510 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:28:47.967444 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd3d598d-7458-4407-ba85-49402348da87-original-pull-secret podName:bd3d598d-7458-4407-ba85-49402348da87 nodeName:}" failed. No retries permitted until 2026-07-27 18:28:49.967423546 +0000 UTC m=+5.029112577 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bd3d598d-7458-4407-ba85-49402348da87-original-pull-secret") pod "global-pull-secret-syncer-pr8j9" (UID: "bd3d598d-7458-4407-ba85-49402348da87") : object "kube-system"/"original-pull-secret" not registered Jul 27 18:28:48.068286 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:48.068247 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rr8z7\" (UniqueName: \"kubernetes.io/projected/8ce054d9-d9c1-44ef-9e32-5a4c502e0ced-kube-api-access-rr8z7\") pod \"network-check-target-tgxn8\" (UID: \"8ce054d9-d9c1-44ef-9e32-5a4c502e0ced\") " pod="openshift-network-diagnostics/network-check-target-tgxn8" Jul 27 18:28:48.068465 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:48.068326 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/270a2080-a3dd-4227-9abc-c093f70df909-metrics-certs\") pod \"network-metrics-daemon-6htnl\" (UID: \"270a2080-a3dd-4227-9abc-c093f70df909\") " pod="openshift-multus/network-metrics-daemon-6htnl" Jul 27 18:28:48.068531 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:28:48.068477 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 18:28:48.068583 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:28:48.068537 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/270a2080-a3dd-4227-9abc-c093f70df909-metrics-certs podName:270a2080-a3dd-4227-9abc-c093f70df909 nodeName:}" failed. No retries permitted until 2026-07-27 18:28:50.068518708 +0000 UTC m=+5.130207722 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/270a2080-a3dd-4227-9abc-c093f70df909-metrics-certs") pod "network-metrics-daemon-6htnl" (UID: "270a2080-a3dd-4227-9abc-c093f70df909") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 18:28:48.068968 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:28:48.068949 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 27 18:28:48.069034 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:28:48.068974 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 27 18:28:48.069034 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:28:48.068987 2579 projected.go:196] Error preparing data for projected volume kube-api-access-rr8z7 for pod openshift-network-diagnostics/network-check-target-tgxn8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 18:28:48.069034 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:28:48.069031 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8ce054d9-d9c1-44ef-9e32-5a4c502e0ced-kube-api-access-rr8z7 podName:8ce054d9-d9c1-44ef-9e32-5a4c502e0ced nodeName:}" failed. No retries permitted until 2026-07-27 18:28:50.06901593 +0000 UTC m=+5.130704947 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-rr8z7" (UniqueName: "kubernetes.io/projected/8ce054d9-d9c1-44ef-9e32-5a4c502e0ced-kube-api-access-rr8z7") pod "network-check-target-tgxn8" (UID: "8ce054d9-d9c1-44ef-9e32-5a4c502e0ced") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 18:28:48.385853 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:48.385787 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-26 18:23:46 +0000 UTC" deadline="2027-12-21 14:37:02.062119484 +0000 UTC" Jul 27 18:28:48.385853 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:48.385824 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12284h8m13.676300155s" Jul 27 18:28:48.486756 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:48.486722 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pr8j9" Jul 27 18:28:48.486932 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:28:48.486911 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pr8j9" podUID="bd3d598d-7458-4407-ba85-49402348da87" Jul 27 18:28:49.249780 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:49.249640 2579 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 27 18:28:49.487360 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:49.487329 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6htnl" Jul 27 18:28:49.487805 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:28:49.487472 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6htnl" podUID="270a2080-a3dd-4227-9abc-c093f70df909" Jul 27 18:28:49.488062 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:49.488040 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgxn8" Jul 27 18:28:49.488172 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:28:49.488150 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tgxn8" podUID="8ce054d9-d9c1-44ef-9e32-5a4c502e0ced" Jul 27 18:28:49.985814 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:49.985767 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bd3d598d-7458-4407-ba85-49402348da87-original-pull-secret\") pod \"global-pull-secret-syncer-pr8j9\" (UID: \"bd3d598d-7458-4407-ba85-49402348da87\") " pod="kube-system/global-pull-secret-syncer-pr8j9" Jul 27 18:28:49.986049 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:28:49.985945 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 27 18:28:49.986049 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:28:49.986012 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd3d598d-7458-4407-ba85-49402348da87-original-pull-secret podName:bd3d598d-7458-4407-ba85-49402348da87 nodeName:}" failed. No retries permitted until 2026-07-27 18:28:53.985992915 +0000 UTC m=+9.047681930 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bd3d598d-7458-4407-ba85-49402348da87-original-pull-secret") pod "global-pull-secret-syncer-pr8j9" (UID: "bd3d598d-7458-4407-ba85-49402348da87") : object "kube-system"/"original-pull-secret" not registered Jul 27 18:28:50.086466 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:50.086429 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/270a2080-a3dd-4227-9abc-c093f70df909-metrics-certs\") pod \"network-metrics-daemon-6htnl\" (UID: \"270a2080-a3dd-4227-9abc-c093f70df909\") " pod="openshift-multus/network-metrics-daemon-6htnl" Jul 27 18:28:50.086665 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:50.086516 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rr8z7\" (UniqueName: \"kubernetes.io/projected/8ce054d9-d9c1-44ef-9e32-5a4c502e0ced-kube-api-access-rr8z7\") pod \"network-check-target-tgxn8\" (UID: \"8ce054d9-d9c1-44ef-9e32-5a4c502e0ced\") " pod="openshift-network-diagnostics/network-check-target-tgxn8" Jul 27 18:28:50.086752 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:28:50.086667 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 27 18:28:50.086815 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:28:50.086759 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 18:28:50.086868 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:28:50.086819 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/270a2080-a3dd-4227-9abc-c093f70df909-metrics-certs podName:270a2080-a3dd-4227-9abc-c093f70df909 nodeName:}" failed. No retries permitted until 2026-07-27 18:28:54.086800518 +0000 UTC m=+9.148489532 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/270a2080-a3dd-4227-9abc-c093f70df909-metrics-certs") pod "network-metrics-daemon-6htnl" (UID: "270a2080-a3dd-4227-9abc-c093f70df909") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 18:28:50.087176 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:28:50.087155 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 27 18:28:50.087329 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:28:50.087183 2579 projected.go:196] Error preparing data for projected volume kube-api-access-rr8z7 for pod openshift-network-diagnostics/network-check-target-tgxn8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 18:28:50.087329 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:28:50.087255 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8ce054d9-d9c1-44ef-9e32-5a4c502e0ced-kube-api-access-rr8z7 podName:8ce054d9-d9c1-44ef-9e32-5a4c502e0ced nodeName:}" failed. No retries permitted until 2026-07-27 18:28:54.087220765 +0000 UTC m=+9.148909773 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-rr8z7" (UniqueName: "kubernetes.io/projected/8ce054d9-d9c1-44ef-9e32-5a4c502e0ced-kube-api-access-rr8z7") pod "network-check-target-tgxn8" (UID: "8ce054d9-d9c1-44ef-9e32-5a4c502e0ced") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 18:28:50.486702 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:50.486667 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pr8j9" Jul 27 18:28:50.486865 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:28:50.486808 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pr8j9" podUID="bd3d598d-7458-4407-ba85-49402348da87" Jul 27 18:28:51.489473 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:51.489034 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6htnl" Jul 27 18:28:51.489473 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:28:51.489167 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6htnl" podUID="270a2080-a3dd-4227-9abc-c093f70df909" Jul 27 18:28:51.489473 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:51.489357 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgxn8" Jul 27 18:28:51.489473 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:28:51.489435 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tgxn8" podUID="8ce054d9-d9c1-44ef-9e32-5a4c502e0ced" Jul 27 18:28:52.487903 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:52.487407 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pr8j9" Jul 27 18:28:52.487903 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:28:52.487562 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pr8j9" podUID="bd3d598d-7458-4407-ba85-49402348da87" Jul 27 18:28:53.489312 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:53.489215 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6htnl" Jul 27 18:28:53.489689 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:28:53.489348 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6htnl" podUID="270a2080-a3dd-4227-9abc-c093f70df909" Jul 27 18:28:53.489689 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:53.489411 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgxn8" Jul 27 18:28:53.489689 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:28:53.489476 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tgxn8" podUID="8ce054d9-d9c1-44ef-9e32-5a4c502e0ced" Jul 27 18:28:54.020601 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:54.020559 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bd3d598d-7458-4407-ba85-49402348da87-original-pull-secret\") pod \"global-pull-secret-syncer-pr8j9\" (UID: \"bd3d598d-7458-4407-ba85-49402348da87\") " pod="kube-system/global-pull-secret-syncer-pr8j9" Jul 27 18:28:54.020783 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:28:54.020762 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 27 18:28:54.020848 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:28:54.020831 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd3d598d-7458-4407-ba85-49402348da87-original-pull-secret podName:bd3d598d-7458-4407-ba85-49402348da87 nodeName:}" failed. No retries permitted until 2026-07-27 18:29:02.020810682 +0000 UTC m=+17.082499701 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bd3d598d-7458-4407-ba85-49402348da87-original-pull-secret") pod "global-pull-secret-syncer-pr8j9" (UID: "bd3d598d-7458-4407-ba85-49402348da87") : object "kube-system"/"original-pull-secret" not registered Jul 27 18:28:54.121568 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:54.121528 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rr8z7\" (UniqueName: \"kubernetes.io/projected/8ce054d9-d9c1-44ef-9e32-5a4c502e0ced-kube-api-access-rr8z7\") pod \"network-check-target-tgxn8\" (UID: \"8ce054d9-d9c1-44ef-9e32-5a4c502e0ced\") " pod="openshift-network-diagnostics/network-check-target-tgxn8" Jul 27 18:28:54.121758 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:54.121586 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/270a2080-a3dd-4227-9abc-c093f70df909-metrics-certs\") pod \"network-metrics-daemon-6htnl\" (UID: \"270a2080-a3dd-4227-9abc-c093f70df909\") " pod="openshift-multus/network-metrics-daemon-6htnl" Jul 27 18:28:54.121758 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:28:54.121707 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 18:28:54.121758 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:28:54.121760 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/270a2080-a3dd-4227-9abc-c093f70df909-metrics-certs podName:270a2080-a3dd-4227-9abc-c093f70df909 nodeName:}" failed. No retries permitted until 2026-07-27 18:29:02.121745472 +0000 UTC m=+17.183434481 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/270a2080-a3dd-4227-9abc-c093f70df909-metrics-certs") pod "network-metrics-daemon-6htnl" (UID: "270a2080-a3dd-4227-9abc-c093f70df909") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 18:28:54.122132 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:28:54.122115 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 27 18:28:54.122132 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:28:54.122134 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 27 18:28:54.122268 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:28:54.122143 2579 projected.go:196] Error preparing data for projected volume kube-api-access-rr8z7 for pod openshift-network-diagnostics/network-check-target-tgxn8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 18:28:54.122268 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:28:54.122194 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8ce054d9-d9c1-44ef-9e32-5a4c502e0ced-kube-api-access-rr8z7 podName:8ce054d9-d9c1-44ef-9e32-5a4c502e0ced nodeName:}" failed. No retries permitted until 2026-07-27 18:29:02.122167393 +0000 UTC m=+17.183856403 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-rr8z7" (UniqueName: "kubernetes.io/projected/8ce054d9-d9c1-44ef-9e32-5a4c502e0ced-kube-api-access-rr8z7") pod "network-check-target-tgxn8" (UID: "8ce054d9-d9c1-44ef-9e32-5a4c502e0ced") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 18:28:54.487376 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:54.486813 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pr8j9" Jul 27 18:28:54.487376 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:28:54.486960 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pr8j9" podUID="bd3d598d-7458-4407-ba85-49402348da87" Jul 27 18:28:55.488038 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:55.487577 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgxn8" Jul 27 18:28:55.488038 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:55.487666 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6htnl" Jul 27 18:28:55.488038 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:28:55.487680 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tgxn8" podUID="8ce054d9-d9c1-44ef-9e32-5a4c502e0ced" Jul 27 18:28:55.488038 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:28:55.487804 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6htnl" podUID="270a2080-a3dd-4227-9abc-c093f70df909" Jul 27 18:28:56.487142 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:56.487093 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pr8j9" Jul 27 18:28:56.487334 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:28:56.487253 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pr8j9" podUID="bd3d598d-7458-4407-ba85-49402348da87" Jul 27 18:28:57.487460 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:57.487426 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6htnl" Jul 27 18:28:57.487898 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:57.487426 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgxn8" Jul 27 18:28:57.487898 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:28:57.487577 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6htnl" podUID="270a2080-a3dd-4227-9abc-c093f70df909" Jul 27 18:28:57.487898 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:28:57.487623 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tgxn8" podUID="8ce054d9-d9c1-44ef-9e32-5a4c502e0ced" Jul 27 18:28:58.487338 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:58.487293 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pr8j9" Jul 27 18:28:58.487510 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:28:58.487431 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pr8j9" podUID="bd3d598d-7458-4407-ba85-49402348da87" Jul 27 18:28:59.487488 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:59.487248 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgxn8" Jul 27 18:28:59.487637 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:28:59.487285 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6htnl" Jul 27 18:28:59.487637 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:28:59.487534 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tgxn8" podUID="8ce054d9-d9c1-44ef-9e32-5a4c502e0ced" Jul 27 18:28:59.487985 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:28:59.487647 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6htnl" podUID="270a2080-a3dd-4227-9abc-c093f70df909" Jul 27 18:29:00.486946 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:00.486899 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pr8j9" Jul 27 18:29:00.487135 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:29:00.487004 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pr8j9" podUID="bd3d598d-7458-4407-ba85-49402348da87" Jul 27 18:29:01.487394 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:01.487349 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6htnl" Jul 27 18:29:01.487394 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:01.487392 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgxn8" Jul 27 18:29:01.487858 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:29:01.487496 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6htnl" podUID="270a2080-a3dd-4227-9abc-c093f70df909" Jul 27 18:29:01.487858 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:29:01.487588 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tgxn8" podUID="8ce054d9-d9c1-44ef-9e32-5a4c502e0ced" Jul 27 18:29:02.076830 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:02.076785 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bd3d598d-7458-4407-ba85-49402348da87-original-pull-secret\") pod \"global-pull-secret-syncer-pr8j9\" (UID: \"bd3d598d-7458-4407-ba85-49402348da87\") " pod="kube-system/global-pull-secret-syncer-pr8j9" Jul 27 18:29:02.077022 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:29:02.076942 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 27 18:29:02.077022 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:29:02.077014 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd3d598d-7458-4407-ba85-49402348da87-original-pull-secret podName:bd3d598d-7458-4407-ba85-49402348da87 nodeName:}" failed. No retries permitted until 2026-07-27 18:29:18.076992714 +0000 UTC m=+33.138681725 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bd3d598d-7458-4407-ba85-49402348da87-original-pull-secret") pod "global-pull-secret-syncer-pr8j9" (UID: "bd3d598d-7458-4407-ba85-49402348da87") : object "kube-system"/"original-pull-secret" not registered Jul 27 18:29:02.177796 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:02.177751 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rr8z7\" (UniqueName: \"kubernetes.io/projected/8ce054d9-d9c1-44ef-9e32-5a4c502e0ced-kube-api-access-rr8z7\") pod \"network-check-target-tgxn8\" (UID: \"8ce054d9-d9c1-44ef-9e32-5a4c502e0ced\") " pod="openshift-network-diagnostics/network-check-target-tgxn8" Jul 27 18:29:02.177967 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:02.177830 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/270a2080-a3dd-4227-9abc-c093f70df909-metrics-certs\") pod \"network-metrics-daemon-6htnl\" (UID: \"270a2080-a3dd-4227-9abc-c093f70df909\") " pod="openshift-multus/network-metrics-daemon-6htnl" Jul 27 18:29:02.177967 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:29:02.177933 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 27 18:29:02.177967 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:29:02.177948 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 18:29:02.177967 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:29:02.177956 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 27 18:29:02.177967 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:29:02.177968 2579 projected.go:196] Error preparing data for projected volume kube-api-access-rr8z7 for pod openshift-network-diagnostics/network-check-target-tgxn8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 18:29:02.178123 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:29:02.178007 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/270a2080-a3dd-4227-9abc-c093f70df909-metrics-certs podName:270a2080-a3dd-4227-9abc-c093f70df909 nodeName:}" failed. No retries permitted until 2026-07-27 18:29:18.177990068 +0000 UTC m=+33.239679077 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/270a2080-a3dd-4227-9abc-c093f70df909-metrics-certs") pod "network-metrics-daemon-6htnl" (UID: "270a2080-a3dd-4227-9abc-c093f70df909") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 18:29:02.178123 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:29:02.178024 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8ce054d9-d9c1-44ef-9e32-5a4c502e0ced-kube-api-access-rr8z7 podName:8ce054d9-d9c1-44ef-9e32-5a4c502e0ced nodeName:}" failed. No retries permitted until 2026-07-27 18:29:18.17801749 +0000 UTC m=+33.239706499 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-rr8z7" (UniqueName: "kubernetes.io/projected/8ce054d9-d9c1-44ef-9e32-5a4c502e0ced-kube-api-access-rr8z7") pod "network-check-target-tgxn8" (UID: "8ce054d9-d9c1-44ef-9e32-5a4c502e0ced") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 18:29:02.486531 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:02.486449 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pr8j9" Jul 27 18:29:02.486680 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:29:02.486578 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pr8j9" podUID="bd3d598d-7458-4407-ba85-49402348da87" Jul 27 18:29:03.486802 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:03.486769 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgxn8" Jul 27 18:29:03.486802 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:03.486801 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6htnl" Jul 27 18:29:03.487410 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:29:03.486895 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tgxn8" podUID="8ce054d9-d9c1-44ef-9e32-5a4c502e0ced" Jul 27 18:29:03.487410 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:29:03.487009 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6htnl" podUID="270a2080-a3dd-4227-9abc-c093f70df909" Jul 27 18:29:04.487142 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:04.487095 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pr8j9" Jul 27 18:29:04.487599 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:29:04.487224 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pr8j9" podUID="bd3d598d-7458-4407-ba85-49402348da87" Jul 27 18:29:05.487472 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:05.487437 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgxn8" Jul 27 18:29:05.487472 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:05.487479 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6htnl" Jul 27 18:29:05.487997 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:29:05.487561 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tgxn8" podUID="8ce054d9-d9c1-44ef-9e32-5a4c502e0ced" Jul 27 18:29:05.487997 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:29:05.487712 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6htnl" podUID="270a2080-a3dd-4227-9abc-c093f70df909" Jul 27 18:29:06.486849 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:06.486811 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pr8j9" Jul 27 18:29:06.487034 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:29:06.486936 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pr8j9" podUID="bd3d598d-7458-4407-ba85-49402348da87" Jul 27 18:29:07.489298 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:07.489095 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6htnl" Jul 27 18:29:07.489725 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:29:07.489493 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6htnl" podUID="270a2080-a3dd-4227-9abc-c093f70df909" Jul 27 18:29:07.489936 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:07.489893 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgxn8" Jul 27 18:29:07.490012 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:29:07.489997 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tgxn8" podUID="8ce054d9-d9c1-44ef-9e32-5a4c502e0ced" Jul 27 18:29:07.669342 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:07.669123 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-pgzw4" event={"ID":"7d279661-f5f8-49ca-86c1-3c6f863340c8","Type":"ContainerStarted","Data":"2f31b1b7148ca60a417e0335ac8fca5775bd76f64f6affe29d2c89b584f09df2"} Jul 27 18:29:07.670901 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:07.670875 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-sxj58" event={"ID":"3706a9c3-0323-4665-99b3-8bd8889dcd85","Type":"ContainerStarted","Data":"fb450fb355e39579bf5ac5a8eebeb91efd98ee0b945aaa72b7a70ed6aa7603e6"} Jul 27 18:29:07.673198 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:07.673175 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vnbl8" event={"ID":"4f79aa08-0ec5-4fd9-a5a2-06c0d9a755ef","Type":"ContainerStarted","Data":"834046bc936607a1b8a1657e24400a44f018fceabb56fc89e222e3b89c7ee43f"} Jul 27 18:29:07.673323 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:07.673257 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vnbl8" event={"ID":"4f79aa08-0ec5-4fd9-a5a2-06c0d9a755ef","Type":"ContainerStarted","Data":"8713088397f62ed5d1419e0dcb15960787bcb554c048157e76524b808fe30411"} Jul 27 18:29:07.675072 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:07.675052 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-32.ec2.internal" event={"ID":"8aa0e86aff493755e5519df8f9d22c9f","Type":"ContainerStarted","Data":"6107c39b5b363f21db8268b32667da14a9fa1f1e950f4644da60162233808b57"} Jul 27 18:29:07.688587 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:07.688544 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-pgzw4" podStartSLOduration=1.884873623 podStartE2EDuration="22.688530115s" totalImagesPullingTime="20.803656492s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 18:28:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:29:07.687631671 +0000 UTC m=+22.749320703" watchObservedRunningTime="2026-07-27 18:29:07.688530115 +0000 UTC m=+22.750219146" Jul 27 18:29:07.701718 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:07.701670 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-32.ec2.internal" podStartSLOduration=21.701658987 podStartE2EDuration="21.701658987s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 18:28:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:29:07.701348087 +0000 UTC m=+22.763037118" watchObservedRunningTime="2026-07-27 18:29:07.701658987 +0000 UTC m=+22.763348018" Jul 27 18:29:08.487119 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:08.486904 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pr8j9" Jul 27 18:29:08.487310 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:29:08.487217 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pr8j9" podUID="bd3d598d-7458-4407-ba85-49402348da87" Jul 27 18:29:08.679469 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:08.679432 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-fdsdr" event={"ID":"8bbbb432-a482-444d-99d3-f2615e9b99bb","Type":"ContainerStarted","Data":"65f0c3277f01f1aa0833e9b3ed9155d9968c5aedc2d1012c4bbf92a66fd0ca66"} Jul 27 18:29:08.681639 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:08.681607 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-ttprc" event={"ID":"52bcf393-1ae7-4102-b762-569badf3538c","Type":"ContainerStarted","Data":"9adbb2bc939fc75dba2a248d34297bc3c3ae11c1623e99352c03a1817601f4fb"} Jul 27 18:29:08.683261 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:08.683210 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h9d4j" event={"ID":"af18a2fc-71f3-4f54-9907-89e31bd65bc9","Type":"ContainerStarted","Data":"03ee25af99da459737f9302c1265ec992e110dbf4415ff60507ef8601e7bed9f"} Jul 27 18:29:08.685128 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:08.685103 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-8phm4" event={"ID":"83c21902-488a-44d1-b494-7000443a9202","Type":"ContainerStarted","Data":"54188c57394a4665fdb4a8993e43a46e08ad613a4d4e6ba985f28869e03b409c"} Jul 27 18:29:08.686829 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:08.686806 2579 generic.go:356] "Generic (PLEG): container finished" podID="5ba4cb7b-80f6-470b-8ab2-7e22c50af412" containerID="753211d8a0ff2bcb0969f72de91c86c2f02429018f54b8560f0a39e5a30b43aa" exitCode=0 Jul 27 18:29:08.686948 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:08.686879 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6jzx9" event={"ID":"5ba4cb7b-80f6-470b-8ab2-7e22c50af412","Type":"ContainerDied","Data":"753211d8a0ff2bcb0969f72de91c86c2f02429018f54b8560f0a39e5a30b43aa"} Jul 27 18:29:08.688436 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:08.688396 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-67t26" event={"ID":"9c37bc5b-9ef7-4e2b-b555-52de0d9861b8","Type":"ContainerStarted","Data":"d1cc2611df3812ca4515ad1fbb6e45d064cb72019ababab2984a17a29d7edccc"} Jul 27 18:29:08.691896 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:08.691875 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vnbl8" event={"ID":"4f79aa08-0ec5-4fd9-a5a2-06c0d9a755ef","Type":"ContainerStarted","Data":"115368bfa089f0c649590df745ecec337ece4efeee6c11858780f417f02c0b61"} Jul 27 18:29:08.691997 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:08.691941 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vnbl8" event={"ID":"4f79aa08-0ec5-4fd9-a5a2-06c0d9a755ef","Type":"ContainerStarted","Data":"42fa1c4cddfc53fa91b842624b42cc3c89dc2b6396bcd27b12fc15ba38d7d925"} Jul 27 18:29:08.691997 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:08.691978 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vnbl8" event={"ID":"4f79aa08-0ec5-4fd9-a5a2-06c0d9a755ef","Type":"ContainerStarted","Data":"c0c0e9300d3c3ac1c5b1c71c41f6cce9a80c701bf744b8ab90624c782cfb1a3b"} Jul 27 18:29:08.692106 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:08.692023 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vnbl8" event={"ID":"4f79aa08-0ec5-4fd9-a5a2-06c0d9a755ef","Type":"ContainerStarted","Data":"387c6390b3c924d5430f9dca46cb6d32aaef0dbbeca01ac0d880058d89c52b61"} Jul 27 18:29:08.693455 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:08.693434 2579 generic.go:356] "Generic (PLEG): container finished" podID="8bf56391e54c962db9801fd2ce6f6f03" containerID="15c9fb8b389217f0fd962aa9305574b8bd7e53bd89992c20c4534e7cbd596a44" exitCode=0 Jul 27 18:29:08.693562 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:08.693523 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-32.ec2.internal" event={"ID":"8bf56391e54c962db9801fd2ce6f6f03","Type":"ContainerDied","Data":"15c9fb8b389217f0fd962aa9305574b8bd7e53bd89992c20c4534e7cbd596a44"} Jul 27 18:29:08.695608 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:08.695573 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-fdsdr" podStartSLOduration=3.229900889 podStartE2EDuration="23.695562576s" totalImagesPullingTime="20.465661687s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 18:28:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:29:08.695338993 +0000 UTC m=+23.757028025" watchObservedRunningTime="2026-07-27 18:29:08.695562576 +0000 UTC m=+23.757251607" Jul 27 18:29:08.696044 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:08.696018 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-sxj58" podStartSLOduration=2.923194224 podStartE2EDuration="23.696011058s" totalImagesPullingTime="20.772816834s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 18:28:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:29:07.719123567 +0000 UTC m=+22.780812607" watchObservedRunningTime="2026-07-27 18:29:08.696011058 +0000 UTC m=+23.757700089" Jul 27 18:29:08.711504 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:08.711456 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-67t26" podStartSLOduration=2.994822301 podStartE2EDuration="23.711436553s" totalImagesPullingTime="20.716614252s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:28:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:29:08.71069455 +0000 UTC m=+23.772383581" watchObservedRunningTime="2026-07-27 18:29:08.711436553 +0000 UTC m=+23.773125585" Jul 27 18:29:08.766977 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:08.766932 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-8phm4" podStartSLOduration=3.186917221 podStartE2EDuration="23.766918874s" totalImagesPullingTime="20.580001653s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 18:28:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:29:08.766793595 +0000 UTC m=+23.828482626" watchObservedRunningTime="2026-07-27 18:29:08.766918874 +0000 UTC m=+23.828607905" Jul 27 18:29:08.781954 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:08.781917 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-ttprc" podStartSLOduration=3.276271149 podStartE2EDuration="23.781905414s" totalImagesPullingTime="20.505634265s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 18:28:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:29:08.781689576 +0000 UTC m=+23.843378606" watchObservedRunningTime="2026-07-27 18:29:08.781905414 +0000 UTC m=+23.843594445" Jul 27 18:29:09.107070 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:09.107045 2579 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 27 18:29:09.418565 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:09.418473 2579 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-27T18:29:09.107065005Z","UUID":"4c7749e2-35a5-46a8-a523-1246d5767989","Handler":null,"Name":"","Endpoint":""} Jul 27 18:29:09.420105 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:09.420075 2579 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 27 18:29:09.420246 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:09.420114 2579 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 27 18:29:09.486697 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:09.486667 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6htnl" Jul 27 18:29:09.486922 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:29:09.486797 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6htnl" podUID="270a2080-a3dd-4227-9abc-c093f70df909" Jul 27 18:29:09.487146 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:09.487117 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgxn8" Jul 27 18:29:09.487418 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:29:09.487388 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tgxn8" podUID="8ce054d9-d9c1-44ef-9e32-5a4c502e0ced" Jul 27 18:29:09.699004 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:09.698883 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h9d4j" event={"ID":"af18a2fc-71f3-4f54-9907-89e31bd65bc9","Type":"ContainerStarted","Data":"11be9d336ac8ec5dbfdcd69dda12fb3a1310882e39abfe69725a882eafd6b2bb"} Jul 27 18:29:09.702775 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:09.702743 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-32.ec2.internal" event={"ID":"8bf56391e54c962db9801fd2ce6f6f03","Type":"ContainerStarted","Data":"16ae9bb132be31dac43ff89e616ecbb81bcac292252f01e5781089e15e329c31"} Jul 27 18:29:10.486857 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:10.486784 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pr8j9" Jul 27 18:29:10.487082 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:29:10.486916 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pr8j9" podUID="bd3d598d-7458-4407-ba85-49402348da87" Jul 27 18:29:10.707992 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:10.707952 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h9d4j" event={"ID":"af18a2fc-71f3-4f54-9907-89e31bd65bc9","Type":"ContainerStarted","Data":"3d98a53797b389e8177ec7ce3f1c1ac5947abb88f3da9e9f61a490b1d6863e76"} Jul 27 18:29:10.711965 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:10.711930 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vnbl8" event={"ID":"4f79aa08-0ec5-4fd9-a5a2-06c0d9a755ef","Type":"ContainerStarted","Data":"c78b246ba0e05ac891a97f2709903e72553454f2bb912eb45e43d38066498874"} Jul 27 18:29:10.729036 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:10.728982 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h9d4j" podStartSLOduration=3.021528966 podStartE2EDuration="25.728965525s" totalImagesPullingTime="22.707436559s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 18:28:45 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:29:10.728487131 +0000 UTC m=+25.790176155" watchObservedRunningTime="2026-07-27 18:29:10.728965525 +0000 UTC m=+25.790654555" Jul 27 18:29:10.729197 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:10.729159 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-32.ec2.internal" podStartSLOduration=24.729151986 podStartE2EDuration="24.729151986s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 18:28:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:29:09.72078465 +0000 UTC m=+24.782473683" watchObservedRunningTime="2026-07-27 18:29:10.729151986 +0000 UTC m=+25.790841017" Jul 27 18:29:11.487140 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:11.487095 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6htnl" Jul 27 18:29:11.487367 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:11.487124 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgxn8" Jul 27 18:29:11.487367 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:29:11.487261 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6htnl" podUID="270a2080-a3dd-4227-9abc-c093f70df909" Jul 27 18:29:11.487367 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:29:11.487283 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tgxn8" podUID="8ce054d9-d9c1-44ef-9e32-5a4c502e0ced" Jul 27 18:29:12.339843 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:12.339813 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-67t26" Jul 27 18:29:12.340453 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:12.340430 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-67t26" Jul 27 18:29:12.487287 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:12.487250 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pr8j9" Jul 27 18:29:12.487442 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:29:12.487384 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pr8j9" podUID="bd3d598d-7458-4407-ba85-49402348da87" Jul 27 18:29:12.717444 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:12.717359 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-67t26" Jul 27 18:29:12.718018 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:12.717998 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-67t26" Jul 27 18:29:13.487084 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:13.486848 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6htnl" Jul 27 18:29:13.487788 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:13.486889 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgxn8" Jul 27 18:29:13.487788 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:29:13.487159 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6htnl" podUID="270a2080-a3dd-4227-9abc-c093f70df909" Jul 27 18:29:13.487788 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:29:13.487184 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tgxn8" podUID="8ce054d9-d9c1-44ef-9e32-5a4c502e0ced" Jul 27 18:29:13.722795 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:13.722757 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vnbl8" event={"ID":"4f79aa08-0ec5-4fd9-a5a2-06c0d9a755ef","Type":"ContainerStarted","Data":"4dca62e4649df7debd3a8cbf1833f470d83905a70017d16b8a0dab7f0478be91"} Jul 27 18:29:13.723007 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:13.722990 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vnbl8" Jul 27 18:29:13.724656 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:13.724629 2579 generic.go:356] "Generic (PLEG): container finished" podID="5ba4cb7b-80f6-470b-8ab2-7e22c50af412" containerID="c68b8a3b254c983ec4d06e433c2a30a88f37f40b0332a6ccec43a55169308144" exitCode=0 Jul 27 18:29:13.724770 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:13.724718 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6jzx9" event={"ID":"5ba4cb7b-80f6-470b-8ab2-7e22c50af412","Type":"ContainerDied","Data":"c68b8a3b254c983ec4d06e433c2a30a88f37f40b0332a6ccec43a55169308144"} Jul 27 18:29:13.739632 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:13.739580 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vnbl8" Jul 27 18:29:13.752022 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:13.751983 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-vnbl8" podStartSLOduration=8.15037024 podStartE2EDuration="28.751969117s" totalImagesPullingTime="20.601598877s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 18:28:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:29:13.751353014 +0000 UTC m=+28.813042046" watchObservedRunningTime="2026-07-27 18:29:13.751969117 +0000 UTC m=+28.813658515" Jul 27 18:29:14.486989 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:14.486963 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pr8j9" Jul 27 18:29:14.487278 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:29:14.487075 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pr8j9" podUID="bd3d598d-7458-4407-ba85-49402348da87" Jul 27 18:29:14.727626 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:14.727434 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vnbl8" Jul 27 18:29:14.727805 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:14.727793 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vnbl8" Jul 27 18:29:14.743371 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:14.743288 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vnbl8" Jul 27 18:29:14.880497 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:14.880463 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-pr8j9"] Jul 27 18:29:14.880681 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:14.880594 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pr8j9" Jul 27 18:29:14.880746 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:29:14.880696 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pr8j9" podUID="bd3d598d-7458-4407-ba85-49402348da87" Jul 27 18:29:14.883029 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:14.883002 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-tgxn8"] Jul 27 18:29:14.883175 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:14.883089 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgxn8" Jul 27 18:29:14.883249 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:29:14.883173 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tgxn8" podUID="8ce054d9-d9c1-44ef-9e32-5a4c502e0ced" Jul 27 18:29:14.885779 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:14.885755 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-6htnl"] Jul 27 18:29:14.885888 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:14.885853 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6htnl" Jul 27 18:29:14.885952 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:29:14.885935 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6htnl" podUID="270a2080-a3dd-4227-9abc-c093f70df909" Jul 27 18:29:15.731104 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:15.731015 2579 generic.go:356] "Generic (PLEG): container finished" podID="5ba4cb7b-80f6-470b-8ab2-7e22c50af412" containerID="61f0f62859caacc0f0ae62e1b7e83b4838b8589b46931489fc8159876e4eb6a0" exitCode=0 Jul 27 18:29:15.731435 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:15.731086 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6jzx9" event={"ID":"5ba4cb7b-80f6-470b-8ab2-7e22c50af412","Type":"ContainerDied","Data":"61f0f62859caacc0f0ae62e1b7e83b4838b8589b46931489fc8159876e4eb6a0"} Jul 27 18:29:16.487545 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:16.487517 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgxn8" Jul 27 18:29:16.487671 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:16.487571 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pr8j9" Jul 27 18:29:16.487671 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:16.487658 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6htnl" Jul 27 18:29:16.487815 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:29:16.487667 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tgxn8" podUID="8ce054d9-d9c1-44ef-9e32-5a4c502e0ced" Jul 27 18:29:16.487815 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:29:16.487783 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6htnl" podUID="270a2080-a3dd-4227-9abc-c093f70df909" Jul 27 18:29:16.487928 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:29:16.487850 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pr8j9" podUID="bd3d598d-7458-4407-ba85-49402348da87" Jul 27 18:29:17.737870 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:17.737835 2579 generic.go:356] "Generic (PLEG): container finished" podID="5ba4cb7b-80f6-470b-8ab2-7e22c50af412" containerID="be849b4d0bc30742efede37293adb71de04309955cbbb012bea8e660a85f321e" exitCode=0 Jul 27 18:29:17.738393 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:17.737883 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6jzx9" event={"ID":"5ba4cb7b-80f6-470b-8ab2-7e22c50af412","Type":"ContainerDied","Data":"be849b4d0bc30742efede37293adb71de04309955cbbb012bea8e660a85f321e"} Jul 27 18:29:18.091146 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:18.091112 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bd3d598d-7458-4407-ba85-49402348da87-original-pull-secret\") pod \"global-pull-secret-syncer-pr8j9\" (UID: \"bd3d598d-7458-4407-ba85-49402348da87\") " pod="kube-system/global-pull-secret-syncer-pr8j9" Jul 27 18:29:18.091352 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:29:18.091276 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 27 18:29:18.091352 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:29:18.091335 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd3d598d-7458-4407-ba85-49402348da87-original-pull-secret podName:bd3d598d-7458-4407-ba85-49402348da87 nodeName:}" failed. No retries permitted until 2026-07-27 18:29:50.091320694 +0000 UTC m=+65.153009708 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bd3d598d-7458-4407-ba85-49402348da87-original-pull-secret") pod "global-pull-secret-syncer-pr8j9" (UID: "bd3d598d-7458-4407-ba85-49402348da87") : object "kube-system"/"original-pull-secret" not registered Jul 27 18:29:18.192328 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:18.192296 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/270a2080-a3dd-4227-9abc-c093f70df909-metrics-certs\") pod \"network-metrics-daemon-6htnl\" (UID: \"270a2080-a3dd-4227-9abc-c093f70df909\") " pod="openshift-multus/network-metrics-daemon-6htnl" Jul 27 18:29:18.192600 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:18.192578 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rr8z7\" (UniqueName: \"kubernetes.io/projected/8ce054d9-d9c1-44ef-9e32-5a4c502e0ced-kube-api-access-rr8z7\") pod \"network-check-target-tgxn8\" (UID: \"8ce054d9-d9c1-44ef-9e32-5a4c502e0ced\") " pod="openshift-network-diagnostics/network-check-target-tgxn8" Jul 27 18:29:18.192714 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:29:18.192490 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 18:29:18.192714 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:29:18.192670 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 27 18:29:18.192714 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:29:18.192689 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/270a2080-a3dd-4227-9abc-c093f70df909-metrics-certs podName:270a2080-a3dd-4227-9abc-c093f70df909 nodeName:}" failed. No retries permitted until 2026-07-27 18:29:50.192665447 +0000 UTC m=+65.254354457 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/270a2080-a3dd-4227-9abc-c093f70df909-metrics-certs") pod "network-metrics-daemon-6htnl" (UID: "270a2080-a3dd-4227-9abc-c093f70df909") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 18:29:18.192714 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:29:18.192693 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 27 18:29:18.192714 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:29:18.192709 2579 projected.go:196] Error preparing data for projected volume kube-api-access-rr8z7 for pod openshift-network-diagnostics/network-check-target-tgxn8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 18:29:18.192918 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:29:18.192771 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8ce054d9-d9c1-44ef-9e32-5a4c502e0ced-kube-api-access-rr8z7 podName:8ce054d9-d9c1-44ef-9e32-5a4c502e0ced nodeName:}" failed. No retries permitted until 2026-07-27 18:29:50.19275141 +0000 UTC m=+65.254440442 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-rr8z7" (UniqueName: "kubernetes.io/projected/8ce054d9-d9c1-44ef-9e32-5a4c502e0ced-kube-api-access-rr8z7") pod "network-check-target-tgxn8" (UID: "8ce054d9-d9c1-44ef-9e32-5a4c502e0ced") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 18:29:18.487441 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:18.487356 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6htnl" Jul 27 18:29:18.487441 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:18.487400 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgxn8" Jul 27 18:29:18.487654 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:18.487356 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pr8j9" Jul 27 18:29:18.487654 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:29:18.487503 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6htnl" podUID="270a2080-a3dd-4227-9abc-c093f70df909" Jul 27 18:29:18.487654 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:29:18.487606 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pr8j9" podUID="bd3d598d-7458-4407-ba85-49402348da87" Jul 27 18:29:18.487816 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:29:18.487645 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tgxn8" podUID="8ce054d9-d9c1-44ef-9e32-5a4c502e0ced" Jul 27 18:29:20.249849 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.249775 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-32.ec2.internal" event="NodeReady" Jul 27 18:29:20.250273 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.249946 2579 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 27 18:29:20.286379 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.286346 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-85c5fdc4b8-rwfwz"] Jul 27 18:29:20.288618 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.288595 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-85c5fdc4b8-rwfwz" Jul 27 18:29:20.294396 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.293705 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-g9cd9"] Jul 27 18:29:20.294396 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.293965 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 27 18:29:20.294396 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.294062 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 27 18:29:20.294396 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.294197 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-fwznz\"" Jul 27 18:29:20.294396 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.294293 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 27 18:29:20.296261 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.296218 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-79snb"] Jul 27 18:29:20.296407 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.296389 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-g9cd9" Jul 27 18:29:20.298213 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.298196 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5885fdb759-mhdxk"] Jul 27 18:29:20.298342 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.298329 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-79snb" Jul 27 18:29:20.300022 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.300002 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 27 18:29:20.300321 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.300298 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-5c874b7d95-95vlk"] Jul 27 18:29:20.300444 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.300421 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5885fdb759-mhdxk" Jul 27 18:29:20.302038 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.302009 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-9pgxt"] Jul 27 18:29:20.302326 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.302295 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 27 18:29:20.302405 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.302377 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 27 18:29:20.302478 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.302295 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-d4rxx\"" Jul 27 18:29:20.302584 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.302567 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 27 18:29:20.302859 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.302738 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5c874b7d95-95vlk" Jul 27 18:29:20.302859 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.302752 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-nc52h\"" Jul 27 18:29:20.303663 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.303645 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 27 18:29:20.303757 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.303687 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 27 18:29:20.303757 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.303695 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 27 18:29:20.303757 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.303747 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 27 18:29:20.304002 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.303983 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-6dl5t"] Jul 27 18:29:20.304061 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.304003 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 27 18:29:20.304061 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.304023 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-dxzx5\"" Jul 27 18:29:20.304144 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.304092 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-9pgxt" Jul 27 18:29:20.304209 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.304194 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 27 18:29:20.304475 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.304458 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 27 18:29:20.305579 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.305558 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 27 18:29:20.305863 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.305847 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-mv4hb"] Jul 27 18:29:20.306011 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.305989 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 27 18:29:20.306299 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.306285 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-6dl5t" Jul 27 18:29:20.306733 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.306713 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 27 18:29:20.306934 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.306744 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 27 18:29:20.306934 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.306716 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 27 18:29:20.306934 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.306932 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-wvq8l\"" Jul 27 18:29:20.307104 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.306942 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 27 18:29:20.307315 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.307297 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-v85bm\"" Jul 27 18:29:20.307517 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.307443 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Jul 27 18:29:20.307517 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.307443 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Jul 27 18:29:20.308081 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.308063 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-7lqlw"] Jul 27 18:29:20.308766 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.308699 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-mv4hb" Jul 27 18:29:20.310415 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.310398 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-79snb"] Jul 27 18:29:20.310534 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.310522 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-g9cd9"] Jul 27 18:29:20.310701 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.310688 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-7lqlw" Jul 27 18:29:20.318787 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.318766 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 27 18:29:20.319547 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.319528 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 27 18:29:20.319981 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.319960 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-grgzp\"" Jul 27 18:29:20.320478 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.320459 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 27 18:29:20.321429 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.321405 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-8dt4n\"" Jul 27 18:29:20.321820 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.321799 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 27 18:29:20.322740 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.322057 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 27 18:29:20.322740 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.322218 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 27 18:29:20.322740 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.322512 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 27 18:29:20.323468 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.323446 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-85c5fdc4b8-rwfwz"] Jul 27 18:29:20.324568 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.324130 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 27 18:29:20.324568 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.324262 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 27 18:29:20.324568 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.324511 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5885fdb759-mhdxk"] Jul 27 18:29:20.324873 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.324842 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 27 18:29:20.324981 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.324963 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-6dl5t"] Jul 27 18:29:20.325633 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.325613 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-mv4hb"] Jul 27 18:29:20.326051 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.326030 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 27 18:29:20.326338 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.326321 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-swtr8\"" Jul 27 18:29:20.326529 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.326341 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5c874b7d95-95vlk"] Jul 27 18:29:20.327004 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.326985 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-5m9rh"] Jul 27 18:29:20.328016 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.327992 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 27 18:29:20.329318 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.329300 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 27 18:29:20.329453 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.329435 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-f8hjw"] Jul 27 18:29:20.329880 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.329864 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-5m9rh" Jul 27 18:29:20.331285 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.331267 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-pqqm5"] Jul 27 18:29:20.331601 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.331583 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-f8hjw" Jul 27 18:29:20.335890 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.335873 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-nrk2j\"" Jul 27 18:29:20.336090 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.336072 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 27 18:29:20.336090 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.336082 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 27 18:29:20.336305 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.336290 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 27 18:29:20.336482 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.336466 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 27 18:29:20.336648 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.336633 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 27 18:29:20.336845 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.336827 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 27 18:29:20.337014 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.337002 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-5w26j\"" Jul 27 18:29:20.337387 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.337368 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-6hbg6"] Jul 27 18:29:20.337783 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.337766 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pqqm5" Jul 27 18:29:20.339220 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.339202 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-5m9rh"] Jul 27 18:29:20.339342 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.339266 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-f8hjw"] Jul 27 18:29:20.339342 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.339288 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-7lqlw"] Jul 27 18:29:20.339342 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.339330 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-9pgxt"] Jul 27 18:29:20.339505 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.339330 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-6hbg6" Jul 27 18:29:20.339785 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.339762 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-6hbg6"] Jul 27 18:29:20.340404 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.340386 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 27 18:29:20.340495 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.340440 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 27 18:29:20.340721 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.340704 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-9pm47\"" Jul 27 18:29:20.342215 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.342193 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 27 18:29:20.343462 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.343445 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 27 18:29:20.343662 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.343645 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 27 18:29:20.343812 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.343788 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-xfn8t\"" Jul 27 18:29:20.349127 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.349110 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-pqqm5"] Jul 27 18:29:20.405017 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.404990 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e660267b-cf0d-4e4e-bb05-1769ff224bb2-registry-tls\") pod \"image-registry-85c5fdc4b8-rwfwz\" (UID: \"e660267b-cf0d-4e4e-bb05-1769ff224bb2\") " pod="openshift-image-registry/image-registry-85c5fdc4b8-rwfwz" Jul 27 18:29:20.405017 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.405025 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e660267b-cf0d-4e4e-bb05-1769ff224bb2-ca-trust-extracted\") pod \"image-registry-85c5fdc4b8-rwfwz\" (UID: \"e660267b-cf0d-4e4e-bb05-1769ff224bb2\") " pod="openshift-image-registry/image-registry-85c5fdc4b8-rwfwz" Jul 27 18:29:20.405179 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.405042 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b61c6434-d128-4ec0-920d-a00b085ae9c6-serving-cert\") pod \"insights-operator-5885fdb759-mhdxk\" (UID: \"b61c6434-d128-4ec0-920d-a00b085ae9c6\") " pod="openshift-insights/insights-operator-5885fdb759-mhdxk" Jul 27 18:29:20.405179 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.405071 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b41bffe-08d5-4b52-ae91-dcfebab44a99-service-ca-bundle\") pod \"router-default-5c874b7d95-95vlk\" (UID: \"2b41bffe-08d5-4b52-ae91-dcfebab44a99\") " pod="openshift-ingress/router-default-5c874b7d95-95vlk" Jul 27 18:29:20.405179 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.405091 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcp7s\" (UniqueName: \"kubernetes.io/projected/d4bd05a3-ebb5-4c2e-9802-5c0b15f2bdba-kube-api-access-qcp7s\") pod \"service-ca-operator-5d6c75f84f-6dl5t\" (UID: \"d4bd05a3-ebb5-4c2e-9802-5c0b15f2bdba\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-6dl5t" Jul 27 18:29:20.405179 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.405111 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/715b2084-d79e-4cbf-a057-f2595b4e6b9b-trusted-ca\") pod \"console-operator-584dcb6b54-mv4hb\" (UID: \"715b2084-d79e-4cbf-a057-f2595b4e6b9b\") " pod="openshift-console-operator/console-operator-584dcb6b54-mv4hb" Jul 27 18:29:20.405179 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.405127 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/06e8b4ad-8ea1-4fa8-ab2c-3d6c24fafc88-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-5m9rh\" (UID: \"06e8b4ad-8ea1-4fa8-ab2c-3d6c24fafc88\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-5m9rh" Jul 27 18:29:20.405179 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.405144 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4e16cd40-8590-4739-a49c-b67d0f1e37d7-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-7lqlw\" (UID: \"4e16cd40-8590-4739-a49c-b67d0f1e37d7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-7lqlw" Jul 27 18:29:20.405179 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.405164 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/b61c6434-d128-4ec0-920d-a00b085ae9c6-snapshots\") pod \"insights-operator-5885fdb759-mhdxk\" (UID: \"b61c6434-d128-4ec0-920d-a00b085ae9c6\") " pod="openshift-insights/insights-operator-5885fdb759-mhdxk" Jul 27 18:29:20.405179 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.405179 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/2b41bffe-08d5-4b52-ae91-dcfebab44a99-default-certificate\") pod \"router-default-5c874b7d95-95vlk\" (UID: \"2b41bffe-08d5-4b52-ae91-dcfebab44a99\") " pod="openshift-ingress/router-default-5c874b7d95-95vlk" Jul 27 18:29:20.405462 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.405197 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/715b2084-d79e-4cbf-a057-f2595b4e6b9b-config\") pod \"console-operator-584dcb6b54-mv4hb\" (UID: \"715b2084-d79e-4cbf-a057-f2595b4e6b9b\") " pod="openshift-console-operator/console-operator-584dcb6b54-mv4hb" Jul 27 18:29:20.405462 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.405249 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e660267b-cf0d-4e4e-bb05-1769ff224bb2-registry-certificates\") pod \"image-registry-85c5fdc4b8-rwfwz\" (UID: \"e660267b-cf0d-4e4e-bb05-1769ff224bb2\") " pod="openshift-image-registry/image-registry-85c5fdc4b8-rwfwz" Jul 27 18:29:20.405462 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.405279 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/2b41bffe-08d5-4b52-ae91-dcfebab44a99-stats-auth\") pod \"router-default-5c874b7d95-95vlk\" (UID: \"2b41bffe-08d5-4b52-ae91-dcfebab44a99\") " pod="openshift-ingress/router-default-5c874b7d95-95vlk" Jul 27 18:29:20.405462 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.405312 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nxjl\" (UniqueName: \"kubernetes.io/projected/b61c6434-d128-4ec0-920d-a00b085ae9c6-kube-api-access-7nxjl\") pod \"insights-operator-5885fdb759-mhdxk\" (UID: \"b61c6434-d128-4ec0-920d-a00b085ae9c6\") " pod="openshift-insights/insights-operator-5885fdb759-mhdxk" Jul 27 18:29:20.405462 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.405345 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/06e8b4ad-8ea1-4fa8-ab2c-3d6c24fafc88-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-5m9rh\" (UID: \"06e8b4ad-8ea1-4fa8-ab2c-3d6c24fafc88\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-5m9rh" Jul 27 18:29:20.405462 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.405368 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e660267b-cf0d-4e4e-bb05-1769ff224bb2-trusted-ca\") pod \"image-registry-85c5fdc4b8-rwfwz\" (UID: \"e660267b-cf0d-4e4e-bb05-1769ff224bb2\") " pod="openshift-image-registry/image-registry-85c5fdc4b8-rwfwz" Jul 27 18:29:20.405462 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.405407 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcfqt\" (UniqueName: \"kubernetes.io/projected/e660267b-cf0d-4e4e-bb05-1769ff224bb2-kube-api-access-hcfqt\") pod \"image-registry-85c5fdc4b8-rwfwz\" (UID: \"e660267b-cf0d-4e4e-bb05-1769ff224bb2\") " pod="openshift-image-registry/image-registry-85c5fdc4b8-rwfwz" Jul 27 18:29:20.405462 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.405432 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/1e0a413d-db6c-4206-928b-6d95b365c39b-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-g9cd9\" (UID: \"1e0a413d-db6c-4206-928b-6d95b365c39b\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-g9cd9" Jul 27 18:29:20.405462 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.405454 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1e0a413d-db6c-4206-928b-6d95b365c39b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-g9cd9\" (UID: \"1e0a413d-db6c-4206-928b-6d95b365c39b\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-g9cd9" Jul 27 18:29:20.405866 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.405486 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b61c6434-d128-4ec0-920d-a00b085ae9c6-service-ca-bundle\") pod \"insights-operator-5885fdb759-mhdxk\" (UID: \"b61c6434-d128-4ec0-920d-a00b085ae9c6\") " pod="openshift-insights/insights-operator-5885fdb759-mhdxk" Jul 27 18:29:20.405866 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.405513 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/715b2084-d79e-4cbf-a057-f2595b4e6b9b-serving-cert\") pod \"console-operator-584dcb6b54-mv4hb\" (UID: \"715b2084-d79e-4cbf-a057-f2595b4e6b9b\") " pod="openshift-console-operator/console-operator-584dcb6b54-mv4hb" Jul 27 18:29:20.405866 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.405538 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b61c6434-d128-4ec0-920d-a00b085ae9c6-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-mhdxk\" (UID: \"b61c6434-d128-4ec0-920d-a00b085ae9c6\") " pod="openshift-insights/insights-operator-5885fdb759-mhdxk" Jul 27 18:29:20.405866 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.405568 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e4fef94e-e2c7-4725-9da9-ddb80b15149d-cert\") pod \"ingress-canary-6hbg6\" (UID: \"e4fef94e-e2c7-4725-9da9-ddb80b15149d\") " pod="openshift-ingress-canary/ingress-canary-6hbg6" Jul 27 18:29:20.405866 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.405593 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8h5v\" (UniqueName: \"kubernetes.io/projected/1e0a413d-db6c-4206-928b-6d95b365c39b-kube-api-access-k8h5v\") pod \"cluster-monitoring-operator-548b489cb8-g9cd9\" (UID: \"1e0a413d-db6c-4206-928b-6d95b365c39b\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-g9cd9" Jul 27 18:29:20.405866 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.405620 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgfdj\" (UniqueName: \"kubernetes.io/projected/2b41bffe-08d5-4b52-ae91-dcfebab44a99-kube-api-access-tgfdj\") pod \"router-default-5c874b7d95-95vlk\" (UID: \"2b41bffe-08d5-4b52-ae91-dcfebab44a99\") " pod="openshift-ingress/router-default-5c874b7d95-95vlk" Jul 27 18:29:20.405866 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.405646 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b61c6434-d128-4ec0-920d-a00b085ae9c6-tmp\") pod \"insights-operator-5885fdb759-mhdxk\" (UID: \"b61c6434-d128-4ec0-920d-a00b085ae9c6\") " pod="openshift-insights/insights-operator-5885fdb759-mhdxk" Jul 27 18:29:20.405866 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.405671 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b41bffe-08d5-4b52-ae91-dcfebab44a99-metrics-certs\") pod \"router-default-5c874b7d95-95vlk\" (UID: \"2b41bffe-08d5-4b52-ae91-dcfebab44a99\") " pod="openshift-ingress/router-default-5c874b7d95-95vlk" Jul 27 18:29:20.405866 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.405700 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d4bd05a3-ebb5-4c2e-9802-5c0b15f2bdba-serving-cert\") pod \"service-ca-operator-5d6c75f84f-6dl5t\" (UID: \"d4bd05a3-ebb5-4c2e-9802-5c0b15f2bdba\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-6dl5t" Jul 27 18:29:20.405866 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.405727 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-md2gw\" (UniqueName: \"kubernetes.io/projected/e4fef94e-e2c7-4725-9da9-ddb80b15149d-kube-api-access-md2gw\") pod \"ingress-canary-6hbg6\" (UID: \"e4fef94e-e2c7-4725-9da9-ddb80b15149d\") " pod="openshift-ingress-canary/ingress-canary-6hbg6" Jul 27 18:29:20.405866 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.405759 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2cgj\" (UniqueName: \"kubernetes.io/projected/715b2084-d79e-4cbf-a057-f2595b4e6b9b-kube-api-access-q2cgj\") pod \"console-operator-584dcb6b54-mv4hb\" (UID: \"715b2084-d79e-4cbf-a057-f2595b4e6b9b\") " pod="openshift-console-operator/console-operator-584dcb6b54-mv4hb" Jul 27 18:29:20.405866 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.405789 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/bfcc4ba6-b269-49a5-847a-2175dc0881fb-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-9pgxt\" (UID: \"bfcc4ba6-b269-49a5-847a-2175dc0881fb\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-9pgxt" Jul 27 18:29:20.405866 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.405834 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e660267b-cf0d-4e4e-bb05-1769ff224bb2-installation-pull-secrets\") pod \"image-registry-85c5fdc4b8-rwfwz\" (UID: \"e660267b-cf0d-4e4e-bb05-1769ff224bb2\") " pod="openshift-image-registry/image-registry-85c5fdc4b8-rwfwz" Jul 27 18:29:20.405866 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.405862 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pct8m\" (UniqueName: \"kubernetes.io/projected/74af2906-b939-4580-b91f-5d42c1805d74-kube-api-access-pct8m\") pod \"volume-data-source-validator-549c6d54cf-79snb\" (UID: \"74af2906-b939-4580-b91f-5d42c1805d74\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-79snb" Jul 27 18:29:20.406562 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.405879 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgshz\" (UniqueName: \"kubernetes.io/projected/06e8b4ad-8ea1-4fa8-ab2c-3d6c24fafc88-kube-api-access-pgshz\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-5m9rh\" (UID: \"06e8b4ad-8ea1-4fa8-ab2c-3d6c24fafc88\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-5m9rh" Jul 27 18:29:20.406562 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.405901 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf48a4f7-eedd-41cc-9d53-93229f167d40-tmp-dir\") pod \"dns-default-pqqm5\" (UID: \"bf48a4f7-eedd-41cc-9d53-93229f167d40\") " pod="openshift-dns/dns-default-pqqm5" Jul 27 18:29:20.406562 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.405929 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e660267b-cf0d-4e4e-bb05-1769ff224bb2-image-registry-private-configuration\") pod \"image-registry-85c5fdc4b8-rwfwz\" (UID: \"e660267b-cf0d-4e4e-bb05-1769ff224bb2\") " pod="openshift-image-registry/image-registry-85c5fdc4b8-rwfwz" Jul 27 18:29:20.406562 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.405961 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bf48a4f7-eedd-41cc-9d53-93229f167d40-config-volume\") pod \"dns-default-pqqm5\" (UID: \"bf48a4f7-eedd-41cc-9d53-93229f167d40\") " pod="openshift-dns/dns-default-pqqm5" Jul 27 18:29:20.406562 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.405991 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkz7r\" (UniqueName: \"kubernetes.io/projected/bf48a4f7-eedd-41cc-9d53-93229f167d40-kube-api-access-hkz7r\") pod \"dns-default-pqqm5\" (UID: \"bf48a4f7-eedd-41cc-9d53-93229f167d40\") " pod="openshift-dns/dns-default-pqqm5" Jul 27 18:29:20.406562 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.406026 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d4bd05a3-ebb5-4c2e-9802-5c0b15f2bdba-config\") pod \"service-ca-operator-5d6c75f84f-6dl5t\" (UID: \"d4bd05a3-ebb5-4c2e-9802-5c0b15f2bdba\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-6dl5t" Jul 27 18:29:20.406562 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.406055 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bf48a4f7-eedd-41cc-9d53-93229f167d40-metrics-tls\") pod \"dns-default-pqqm5\" (UID: \"bf48a4f7-eedd-41cc-9d53-93229f167d40\") " pod="openshift-dns/dns-default-pqqm5" Jul 27 18:29:20.406562 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.406077 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e660267b-cf0d-4e4e-bb05-1769ff224bb2-bound-sa-token\") pod \"image-registry-85c5fdc4b8-rwfwz\" (UID: \"e660267b-cf0d-4e4e-bb05-1769ff224bb2\") " pod="openshift-image-registry/image-registry-85c5fdc4b8-rwfwz" Jul 27 18:29:20.406562 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.406105 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ht7gm\" (UniqueName: \"kubernetes.io/projected/4e16cd40-8590-4739-a49c-b67d0f1e37d7-kube-api-access-ht7gm\") pod \"cluster-samples-operator-6748489656-7lqlw\" (UID: \"4e16cd40-8590-4739-a49c-b67d0f1e37d7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-7lqlw" Jul 27 18:29:20.406562 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.406134 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bfcc4ba6-b269-49a5-847a-2175dc0881fb-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-9pgxt\" (UID: \"bfcc4ba6-b269-49a5-847a-2175dc0881fb\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-9pgxt" Jul 27 18:29:20.406562 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.406158 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpb62\" (UniqueName: \"kubernetes.io/projected/bd88c5eb-6a78-42bb-9b53-e5eb5f6489c6-kube-api-access-dpb62\") pod \"network-check-source-7d88cb55d8-f8hjw\" (UID: \"bd88c5eb-6a78-42bb-9b53-e5eb5f6489c6\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-f8hjw" Jul 27 18:29:20.486908 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.486878 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgxn8" Jul 27 18:29:20.487099 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.486877 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pr8j9" Jul 27 18:29:20.487158 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.486877 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6htnl" Jul 27 18:29:20.489982 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.489762 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-fn8tb\"" Jul 27 18:29:20.489982 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.489787 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 27 18:29:20.489982 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.489830 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-bwbqc\"" Jul 27 18:29:20.489982 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.489769 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 27 18:29:20.507482 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.507463 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pct8m\" (UniqueName: \"kubernetes.io/projected/74af2906-b939-4580-b91f-5d42c1805d74-kube-api-access-pct8m\") pod \"volume-data-source-validator-549c6d54cf-79snb\" (UID: \"74af2906-b939-4580-b91f-5d42c1805d74\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-79snb" Jul 27 18:29:20.507586 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.507492 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgshz\" (UniqueName: \"kubernetes.io/projected/06e8b4ad-8ea1-4fa8-ab2c-3d6c24fafc88-kube-api-access-pgshz\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-5m9rh\" (UID: \"06e8b4ad-8ea1-4fa8-ab2c-3d6c24fafc88\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-5m9rh" Jul 27 18:29:20.507586 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.507512 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf48a4f7-eedd-41cc-9d53-93229f167d40-tmp-dir\") pod \"dns-default-pqqm5\" (UID: \"bf48a4f7-eedd-41cc-9d53-93229f167d40\") " pod="openshift-dns/dns-default-pqqm5" Jul 27 18:29:20.507586 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.507545 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e660267b-cf0d-4e4e-bb05-1769ff224bb2-image-registry-private-configuration\") pod \"image-registry-85c5fdc4b8-rwfwz\" (UID: \"e660267b-cf0d-4e4e-bb05-1769ff224bb2\") " pod="openshift-image-registry/image-registry-85c5fdc4b8-rwfwz" Jul 27 18:29:20.507586 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.507569 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bf48a4f7-eedd-41cc-9d53-93229f167d40-config-volume\") pod \"dns-default-pqqm5\" (UID: \"bf48a4f7-eedd-41cc-9d53-93229f167d40\") " pod="openshift-dns/dns-default-pqqm5" Jul 27 18:29:20.507789 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.507592 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkz7r\" (UniqueName: \"kubernetes.io/projected/bf48a4f7-eedd-41cc-9d53-93229f167d40-kube-api-access-hkz7r\") pod \"dns-default-pqqm5\" (UID: \"bf48a4f7-eedd-41cc-9d53-93229f167d40\") " pod="openshift-dns/dns-default-pqqm5" Jul 27 18:29:20.507789 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.507616 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d4bd05a3-ebb5-4c2e-9802-5c0b15f2bdba-config\") pod \"service-ca-operator-5d6c75f84f-6dl5t\" (UID: \"d4bd05a3-ebb5-4c2e-9802-5c0b15f2bdba\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-6dl5t" Jul 27 18:29:20.507789 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.507653 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bf48a4f7-eedd-41cc-9d53-93229f167d40-metrics-tls\") pod \"dns-default-pqqm5\" (UID: \"bf48a4f7-eedd-41cc-9d53-93229f167d40\") " pod="openshift-dns/dns-default-pqqm5" Jul 27 18:29:20.507789 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.507681 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e660267b-cf0d-4e4e-bb05-1769ff224bb2-bound-sa-token\") pod \"image-registry-85c5fdc4b8-rwfwz\" (UID: \"e660267b-cf0d-4e4e-bb05-1769ff224bb2\") " pod="openshift-image-registry/image-registry-85c5fdc4b8-rwfwz" Jul 27 18:29:20.507789 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.507705 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ht7gm\" (UniqueName: \"kubernetes.io/projected/4e16cd40-8590-4739-a49c-b67d0f1e37d7-kube-api-access-ht7gm\") pod \"cluster-samples-operator-6748489656-7lqlw\" (UID: \"4e16cd40-8590-4739-a49c-b67d0f1e37d7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-7lqlw" Jul 27 18:29:20.507789 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.507742 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bfcc4ba6-b269-49a5-847a-2175dc0881fb-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-9pgxt\" (UID: \"bfcc4ba6-b269-49a5-847a-2175dc0881fb\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-9pgxt" Jul 27 18:29:20.507789 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.507770 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpb62\" (UniqueName: \"kubernetes.io/projected/bd88c5eb-6a78-42bb-9b53-e5eb5f6489c6-kube-api-access-dpb62\") pod \"network-check-source-7d88cb55d8-f8hjw\" (UID: \"bd88c5eb-6a78-42bb-9b53-e5eb5f6489c6\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-f8hjw" Jul 27 18:29:20.508126 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:29:20.507848 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 27 18:29:20.508126 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:29:20.507927 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf48a4f7-eedd-41cc-9d53-93229f167d40-metrics-tls podName:bf48a4f7-eedd-41cc-9d53-93229f167d40 nodeName:}" failed. No retries permitted until 2026-07-27 18:29:21.007909505 +0000 UTC m=+36.069598519 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bf48a4f7-eedd-41cc-9d53-93229f167d40-metrics-tls") pod "dns-default-pqqm5" (UID: "bf48a4f7-eedd-41cc-9d53-93229f167d40") : secret "dns-default-metrics-tls" not found Jul 27 18:29:20.508126 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:29:20.508113 2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 27 18:29:20.508320 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:29:20.508171 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bfcc4ba6-b269-49a5-847a-2175dc0881fb-networking-console-plugin-cert podName:bfcc4ba6-b269-49a5-847a-2175dc0881fb nodeName:}" failed. No retries permitted until 2026-07-27 18:29:21.00815658 +0000 UTC m=+36.069845607 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/bfcc4ba6-b269-49a5-847a-2175dc0881fb-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-9pgxt" (UID: "bfcc4ba6-b269-49a5-847a-2175dc0881fb") : secret "networking-console-plugin-cert" not found Jul 27 18:29:20.508320 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.508223 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bf48a4f7-eedd-41cc-9d53-93229f167d40-config-volume\") pod \"dns-default-pqqm5\" (UID: \"bf48a4f7-eedd-41cc-9d53-93229f167d40\") " pod="openshift-dns/dns-default-pqqm5" Jul 27 18:29:20.508320 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.508301 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e660267b-cf0d-4e4e-bb05-1769ff224bb2-registry-tls\") pod \"image-registry-85c5fdc4b8-rwfwz\" (UID: \"e660267b-cf0d-4e4e-bb05-1769ff224bb2\") " pod="openshift-image-registry/image-registry-85c5fdc4b8-rwfwz" Jul 27 18:29:20.508474 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.508328 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf48a4f7-eedd-41cc-9d53-93229f167d40-tmp-dir\") pod \"dns-default-pqqm5\" (UID: \"bf48a4f7-eedd-41cc-9d53-93229f167d40\") " pod="openshift-dns/dns-default-pqqm5" Jul 27 18:29:20.508474 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.508341 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e660267b-cf0d-4e4e-bb05-1769ff224bb2-ca-trust-extracted\") pod \"image-registry-85c5fdc4b8-rwfwz\" (UID: \"e660267b-cf0d-4e4e-bb05-1769ff224bb2\") " pod="openshift-image-registry/image-registry-85c5fdc4b8-rwfwz" Jul 27 18:29:20.508474 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.508369 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b61c6434-d128-4ec0-920d-a00b085ae9c6-serving-cert\") pod \"insights-operator-5885fdb759-mhdxk\" (UID: \"b61c6434-d128-4ec0-920d-a00b085ae9c6\") " pod="openshift-insights/insights-operator-5885fdb759-mhdxk" Jul 27 18:29:20.508474 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:29:20.508402 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 27 18:29:20.508474 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:29:20.508415 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-85c5fdc4b8-rwfwz: secret "image-registry-tls" not found Jul 27 18:29:20.508474 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:29:20.508459 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e660267b-cf0d-4e4e-bb05-1769ff224bb2-registry-tls podName:e660267b-cf0d-4e4e-bb05-1769ff224bb2 nodeName:}" failed. No retries permitted until 2026-07-27 18:29:21.0084447 +0000 UTC m=+36.070133708 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e660267b-cf0d-4e4e-bb05-1769ff224bb2-registry-tls") pod "image-registry-85c5fdc4b8-rwfwz" (UID: "e660267b-cf0d-4e4e-bb05-1769ff224bb2") : secret "image-registry-tls" not found Jul 27 18:29:20.508474 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.508401 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b41bffe-08d5-4b52-ae91-dcfebab44a99-service-ca-bundle\") pod \"router-default-5c874b7d95-95vlk\" (UID: \"2b41bffe-08d5-4b52-ae91-dcfebab44a99\") " pod="openshift-ingress/router-default-5c874b7d95-95vlk" Jul 27 18:29:20.508794 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.508490 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcp7s\" (UniqueName: \"kubernetes.io/projected/d4bd05a3-ebb5-4c2e-9802-5c0b15f2bdba-kube-api-access-qcp7s\") pod \"service-ca-operator-5d6c75f84f-6dl5t\" (UID: \"d4bd05a3-ebb5-4c2e-9802-5c0b15f2bdba\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-6dl5t" Jul 27 18:29:20.508794 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:29:20.508503 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2b41bffe-08d5-4b52-ae91-dcfebab44a99-service-ca-bundle podName:2b41bffe-08d5-4b52-ae91-dcfebab44a99 nodeName:}" failed. No retries permitted until 2026-07-27 18:29:21.008487042 +0000 UTC m=+36.070176054 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/2b41bffe-08d5-4b52-ae91-dcfebab44a99-service-ca-bundle") pod "router-default-5c874b7d95-95vlk" (UID: "2b41bffe-08d5-4b52-ae91-dcfebab44a99") : configmap references non-existent config key: service-ca.crt Jul 27 18:29:20.508794 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.508536 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/715b2084-d79e-4cbf-a057-f2595b4e6b9b-trusted-ca\") pod \"console-operator-584dcb6b54-mv4hb\" (UID: \"715b2084-d79e-4cbf-a057-f2595b4e6b9b\") " pod="openshift-console-operator/console-operator-584dcb6b54-mv4hb" Jul 27 18:29:20.508794 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.508562 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/06e8b4ad-8ea1-4fa8-ab2c-3d6c24fafc88-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-5m9rh\" (UID: \"06e8b4ad-8ea1-4fa8-ab2c-3d6c24fafc88\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-5m9rh" Jul 27 18:29:20.508794 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.508587 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4e16cd40-8590-4739-a49c-b67d0f1e37d7-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-7lqlw\" (UID: \"4e16cd40-8590-4739-a49c-b67d0f1e37d7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-7lqlw" Jul 27 18:29:20.508794 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.508618 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/b61c6434-d128-4ec0-920d-a00b085ae9c6-snapshots\") pod \"insights-operator-5885fdb759-mhdxk\" (UID: \"b61c6434-d128-4ec0-920d-a00b085ae9c6\") " pod="openshift-insights/insights-operator-5885fdb759-mhdxk" Jul 27 18:29:20.508794 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.508645 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/2b41bffe-08d5-4b52-ae91-dcfebab44a99-default-certificate\") pod \"router-default-5c874b7d95-95vlk\" (UID: \"2b41bffe-08d5-4b52-ae91-dcfebab44a99\") " pod="openshift-ingress/router-default-5c874b7d95-95vlk" Jul 27 18:29:20.508794 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.508678 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/715b2084-d79e-4cbf-a057-f2595b4e6b9b-config\") pod \"console-operator-584dcb6b54-mv4hb\" (UID: \"715b2084-d79e-4cbf-a057-f2595b4e6b9b\") " pod="openshift-console-operator/console-operator-584dcb6b54-mv4hb" Jul 27 18:29:20.508794 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.508704 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e660267b-cf0d-4e4e-bb05-1769ff224bb2-registry-certificates\") pod \"image-registry-85c5fdc4b8-rwfwz\" (UID: \"e660267b-cf0d-4e4e-bb05-1769ff224bb2\") " pod="openshift-image-registry/image-registry-85c5fdc4b8-rwfwz" Jul 27 18:29:20.508794 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.508729 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/2b41bffe-08d5-4b52-ae91-dcfebab44a99-stats-auth\") pod \"router-default-5c874b7d95-95vlk\" (UID: \"2b41bffe-08d5-4b52-ae91-dcfebab44a99\") " pod="openshift-ingress/router-default-5c874b7d95-95vlk" Jul 27 18:29:20.508794 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.508730 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d4bd05a3-ebb5-4c2e-9802-5c0b15f2bdba-config\") pod \"service-ca-operator-5d6c75f84f-6dl5t\" (UID: \"d4bd05a3-ebb5-4c2e-9802-5c0b15f2bdba\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-6dl5t" Jul 27 18:29:20.509591 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.508758 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nxjl\" (UniqueName: \"kubernetes.io/projected/b61c6434-d128-4ec0-920d-a00b085ae9c6-kube-api-access-7nxjl\") pod \"insights-operator-5885fdb759-mhdxk\" (UID: \"b61c6434-d128-4ec0-920d-a00b085ae9c6\") " pod="openshift-insights/insights-operator-5885fdb759-mhdxk" Jul 27 18:29:20.509591 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.509491 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/06e8b4ad-8ea1-4fa8-ab2c-3d6c24fafc88-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-5m9rh\" (UID: \"06e8b4ad-8ea1-4fa8-ab2c-3d6c24fafc88\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-5m9rh" Jul 27 18:29:20.509591 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.509523 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e660267b-cf0d-4e4e-bb05-1769ff224bb2-trusted-ca\") pod \"image-registry-85c5fdc4b8-rwfwz\" (UID: \"e660267b-cf0d-4e4e-bb05-1769ff224bb2\") " pod="openshift-image-registry/image-registry-85c5fdc4b8-rwfwz" Jul 27 18:29:20.509591 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.509551 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcfqt\" (UniqueName: \"kubernetes.io/projected/e660267b-cf0d-4e4e-bb05-1769ff224bb2-kube-api-access-hcfqt\") pod \"image-registry-85c5fdc4b8-rwfwz\" (UID: \"e660267b-cf0d-4e4e-bb05-1769ff224bb2\") " pod="openshift-image-registry/image-registry-85c5fdc4b8-rwfwz" Jul 27 18:29:20.509591 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.509574 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/1e0a413d-db6c-4206-928b-6d95b365c39b-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-g9cd9\" (UID: \"1e0a413d-db6c-4206-928b-6d95b365c39b\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-g9cd9" Jul 27 18:29:20.509841 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.509600 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1e0a413d-db6c-4206-928b-6d95b365c39b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-g9cd9\" (UID: \"1e0a413d-db6c-4206-928b-6d95b365c39b\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-g9cd9" Jul 27 18:29:20.509841 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.509625 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b61c6434-d128-4ec0-920d-a00b085ae9c6-service-ca-bundle\") pod \"insights-operator-5885fdb759-mhdxk\" (UID: \"b61c6434-d128-4ec0-920d-a00b085ae9c6\") " pod="openshift-insights/insights-operator-5885fdb759-mhdxk" Jul 27 18:29:20.510173 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.510152 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b61c6434-d128-4ec0-920d-a00b085ae9c6-service-ca-bundle\") pod \"insights-operator-5885fdb759-mhdxk\" (UID: \"b61c6434-d128-4ec0-920d-a00b085ae9c6\") " pod="openshift-insights/insights-operator-5885fdb759-mhdxk" Jul 27 18:29:20.510451 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:29:20.508871 2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 27 18:29:20.510538 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:29:20.510492 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e16cd40-8590-4739-a49c-b67d0f1e37d7-samples-operator-tls podName:4e16cd40-8590-4739-a49c-b67d0f1e37d7 nodeName:}" failed. No retries permitted until 2026-07-27 18:29:21.010475285 +0000 UTC m=+36.072164306 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/4e16cd40-8590-4739-a49c-b67d0f1e37d7-samples-operator-tls") pod "cluster-samples-operator-6748489656-7lqlw" (UID: "4e16cd40-8590-4739-a49c-b67d0f1e37d7") : secret "samples-operator-tls" not found Jul 27 18:29:20.510538 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.510531 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/715b2084-d79e-4cbf-a057-f2595b4e6b9b-trusted-ca\") pod \"console-operator-584dcb6b54-mv4hb\" (UID: \"715b2084-d79e-4cbf-a057-f2595b4e6b9b\") " pod="openshift-console-operator/console-operator-584dcb6b54-mv4hb" Jul 27 18:29:20.510653 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.510575 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/715b2084-d79e-4cbf-a057-f2595b4e6b9b-serving-cert\") pod \"console-operator-584dcb6b54-mv4hb\" (UID: \"715b2084-d79e-4cbf-a057-f2595b4e6b9b\") " pod="openshift-console-operator/console-operator-584dcb6b54-mv4hb" Jul 27 18:29:20.510653 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.510596 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/b61c6434-d128-4ec0-920d-a00b085ae9c6-snapshots\") pod \"insights-operator-5885fdb759-mhdxk\" (UID: \"b61c6434-d128-4ec0-920d-a00b085ae9c6\") " pod="openshift-insights/insights-operator-5885fdb759-mhdxk" Jul 27 18:29:20.510653 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.510609 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b61c6434-d128-4ec0-920d-a00b085ae9c6-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-mhdxk\" (UID: \"b61c6434-d128-4ec0-920d-a00b085ae9c6\") " pod="openshift-insights/insights-operator-5885fdb759-mhdxk" Jul 27 18:29:20.510653 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.510642 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e4fef94e-e2c7-4725-9da9-ddb80b15149d-cert\") pod \"ingress-canary-6hbg6\" (UID: \"e4fef94e-e2c7-4725-9da9-ddb80b15149d\") " pod="openshift-ingress-canary/ingress-canary-6hbg6" Jul 27 18:29:20.510860 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.510660 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8h5v\" (UniqueName: \"kubernetes.io/projected/1e0a413d-db6c-4206-928b-6d95b365c39b-kube-api-access-k8h5v\") pod \"cluster-monitoring-operator-548b489cb8-g9cd9\" (UID: \"1e0a413d-db6c-4206-928b-6d95b365c39b\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-g9cd9" Jul 27 18:29:20.510860 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.510679 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgfdj\" (UniqueName: \"kubernetes.io/projected/2b41bffe-08d5-4b52-ae91-dcfebab44a99-kube-api-access-tgfdj\") pod \"router-default-5c874b7d95-95vlk\" (UID: \"2b41bffe-08d5-4b52-ae91-dcfebab44a99\") " pod="openshift-ingress/router-default-5c874b7d95-95vlk" Jul 27 18:29:20.510860 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.510701 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b61c6434-d128-4ec0-920d-a00b085ae9c6-tmp\") pod \"insights-operator-5885fdb759-mhdxk\" (UID: \"b61c6434-d128-4ec0-920d-a00b085ae9c6\") " pod="openshift-insights/insights-operator-5885fdb759-mhdxk" Jul 27 18:29:20.510860 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.510717 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b41bffe-08d5-4b52-ae91-dcfebab44a99-metrics-certs\") pod \"router-default-5c874b7d95-95vlk\" (UID: \"2b41bffe-08d5-4b52-ae91-dcfebab44a99\") " pod="openshift-ingress/router-default-5c874b7d95-95vlk" Jul 27 18:29:20.510860 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.510732 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d4bd05a3-ebb5-4c2e-9802-5c0b15f2bdba-serving-cert\") pod \"service-ca-operator-5d6c75f84f-6dl5t\" (UID: \"d4bd05a3-ebb5-4c2e-9802-5c0b15f2bdba\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-6dl5t" Jul 27 18:29:20.510860 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.510751 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-md2gw\" (UniqueName: \"kubernetes.io/projected/e4fef94e-e2c7-4725-9da9-ddb80b15149d-kube-api-access-md2gw\") pod \"ingress-canary-6hbg6\" (UID: \"e4fef94e-e2c7-4725-9da9-ddb80b15149d\") " pod="openshift-ingress-canary/ingress-canary-6hbg6" Jul 27 18:29:20.510860 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.510785 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2cgj\" (UniqueName: \"kubernetes.io/projected/715b2084-d79e-4cbf-a057-f2595b4e6b9b-kube-api-access-q2cgj\") pod \"console-operator-584dcb6b54-mv4hb\" (UID: \"715b2084-d79e-4cbf-a057-f2595b4e6b9b\") " pod="openshift-console-operator/console-operator-584dcb6b54-mv4hb" Jul 27 18:29:20.510860 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.510802 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/bfcc4ba6-b269-49a5-847a-2175dc0881fb-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-9pgxt\" (UID: \"bfcc4ba6-b269-49a5-847a-2175dc0881fb\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-9pgxt" Jul 27 18:29:20.510860 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.510829 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e660267b-cf0d-4e4e-bb05-1769ff224bb2-installation-pull-secrets\") pod \"image-registry-85c5fdc4b8-rwfwz\" (UID: \"e660267b-cf0d-4e4e-bb05-1769ff224bb2\") " pod="openshift-image-registry/image-registry-85c5fdc4b8-rwfwz" Jul 27 18:29:20.511696 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.511477 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e660267b-cf0d-4e4e-bb05-1769ff224bb2-trusted-ca\") pod \"image-registry-85c5fdc4b8-rwfwz\" (UID: \"e660267b-cf0d-4e4e-bb05-1769ff224bb2\") " pod="openshift-image-registry/image-registry-85c5fdc4b8-rwfwz" Jul 27 18:29:20.511797 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:29:20.511752 2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 27 18:29:20.511862 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:29:20.511800 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e0a413d-db6c-4206-928b-6d95b365c39b-cluster-monitoring-operator-tls podName:1e0a413d-db6c-4206-928b-6d95b365c39b nodeName:}" failed. No retries permitted until 2026-07-27 18:29:21.011785411 +0000 UTC m=+36.073474423 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/1e0a413d-db6c-4206-928b-6d95b365c39b-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-g9cd9" (UID: "1e0a413d-db6c-4206-928b-6d95b365c39b") : secret "cluster-monitoring-operator-tls" not found Jul 27 18:29:20.511956 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.511930 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/1e0a413d-db6c-4206-928b-6d95b365c39b-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-g9cd9\" (UID: \"1e0a413d-db6c-4206-928b-6d95b365c39b\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-g9cd9" Jul 27 18:29:20.512096 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.512073 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/06e8b4ad-8ea1-4fa8-ab2c-3d6c24fafc88-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-5m9rh\" (UID: \"06e8b4ad-8ea1-4fa8-ab2c-3d6c24fafc88\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-5m9rh" Jul 27 18:29:20.512644 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.512609 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e660267b-cf0d-4e4e-bb05-1769ff224bb2-ca-trust-extracted\") pod \"image-registry-85c5fdc4b8-rwfwz\" (UID: \"e660267b-cf0d-4e4e-bb05-1769ff224bb2\") " pod="openshift-image-registry/image-registry-85c5fdc4b8-rwfwz" Jul 27 18:29:20.513918 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.512919 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b61c6434-d128-4ec0-920d-a00b085ae9c6-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-mhdxk\" (UID: \"b61c6434-d128-4ec0-920d-a00b085ae9c6\") " pod="openshift-insights/insights-operator-5885fdb759-mhdxk" Jul 27 18:29:20.513918 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.513313 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b61c6434-d128-4ec0-920d-a00b085ae9c6-tmp\") pod \"insights-operator-5885fdb759-mhdxk\" (UID: \"b61c6434-d128-4ec0-920d-a00b085ae9c6\") " pod="openshift-insights/insights-operator-5885fdb759-mhdxk" Jul 27 18:29:20.513918 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.513521 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/715b2084-d79e-4cbf-a057-f2595b4e6b9b-config\") pod \"console-operator-584dcb6b54-mv4hb\" (UID: \"715b2084-d79e-4cbf-a057-f2595b4e6b9b\") " pod="openshift-console-operator/console-operator-584dcb6b54-mv4hb" Jul 27 18:29:20.513918 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.513681 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/bfcc4ba6-b269-49a5-847a-2175dc0881fb-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-9pgxt\" (UID: \"bfcc4ba6-b269-49a5-847a-2175dc0881fb\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-9pgxt" Jul 27 18:29:20.513918 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:29:20.513835 2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 27 18:29:20.513918 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:29:20.513887 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b41bffe-08d5-4b52-ae91-dcfebab44a99-metrics-certs podName:2b41bffe-08d5-4b52-ae91-dcfebab44a99 nodeName:}" failed. No retries permitted until 2026-07-27 18:29:21.013871943 +0000 UTC m=+36.075560953 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2b41bffe-08d5-4b52-ae91-dcfebab44a99-metrics-certs") pod "router-default-5c874b7d95-95vlk" (UID: "2b41bffe-08d5-4b52-ae91-dcfebab44a99") : secret "router-metrics-certs-default" not found Jul 27 18:29:20.514292 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.514144 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e660267b-cf0d-4e4e-bb05-1769ff224bb2-registry-certificates\") pod \"image-registry-85c5fdc4b8-rwfwz\" (UID: \"e660267b-cf0d-4e4e-bb05-1769ff224bb2\") " pod="openshift-image-registry/image-registry-85c5fdc4b8-rwfwz" Jul 27 18:29:20.515270 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.515245 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/715b2084-d79e-4cbf-a057-f2595b4e6b9b-serving-cert\") pod \"console-operator-584dcb6b54-mv4hb\" (UID: \"715b2084-d79e-4cbf-a057-f2595b4e6b9b\") " pod="openshift-console-operator/console-operator-584dcb6b54-mv4hb" Jul 27 18:29:20.515791 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.515765 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/06e8b4ad-8ea1-4fa8-ab2c-3d6c24fafc88-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-5m9rh\" (UID: \"06e8b4ad-8ea1-4fa8-ab2c-3d6c24fafc88\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-5m9rh" Jul 27 18:29:20.516220 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.516193 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/2b41bffe-08d5-4b52-ae91-dcfebab44a99-default-certificate\") pod \"router-default-5c874b7d95-95vlk\" (UID: \"2b41bffe-08d5-4b52-ae91-dcfebab44a99\") " pod="openshift-ingress/router-default-5c874b7d95-95vlk" Jul 27 18:29:20.516343 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.516223 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b61c6434-d128-4ec0-920d-a00b085ae9c6-serving-cert\") pod \"insights-operator-5885fdb759-mhdxk\" (UID: \"b61c6434-d128-4ec0-920d-a00b085ae9c6\") " pod="openshift-insights/insights-operator-5885fdb759-mhdxk" Jul 27 18:29:20.516343 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.516295 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e660267b-cf0d-4e4e-bb05-1769ff224bb2-image-registry-private-configuration\") pod \"image-registry-85c5fdc4b8-rwfwz\" (UID: \"e660267b-cf0d-4e4e-bb05-1769ff224bb2\") " pod="openshift-image-registry/image-registry-85c5fdc4b8-rwfwz" Jul 27 18:29:20.516994 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.516947 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d4bd05a3-ebb5-4c2e-9802-5c0b15f2bdba-serving-cert\") pod \"service-ca-operator-5d6c75f84f-6dl5t\" (UID: \"d4bd05a3-ebb5-4c2e-9802-5c0b15f2bdba\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-6dl5t" Jul 27 18:29:20.517955 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.517903 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e660267b-cf0d-4e4e-bb05-1769ff224bb2-installation-pull-secrets\") pod \"image-registry-85c5fdc4b8-rwfwz\" (UID: \"e660267b-cf0d-4e4e-bb05-1769ff224bb2\") " pod="openshift-image-registry/image-registry-85c5fdc4b8-rwfwz" Jul 27 18:29:20.518506 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.518464 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/2b41bffe-08d5-4b52-ae91-dcfebab44a99-stats-auth\") pod \"router-default-5c874b7d95-95vlk\" (UID: \"2b41bffe-08d5-4b52-ae91-dcfebab44a99\") " pod="openshift-ingress/router-default-5c874b7d95-95vlk" Jul 27 18:29:20.519374 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.519210 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e4fef94e-e2c7-4725-9da9-ddb80b15149d-cert\") pod \"ingress-canary-6hbg6\" (UID: \"e4fef94e-e2c7-4725-9da9-ddb80b15149d\") " pod="openshift-ingress-canary/ingress-canary-6hbg6" Jul 27 18:29:20.519975 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.519942 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e660267b-cf0d-4e4e-bb05-1769ff224bb2-bound-sa-token\") pod \"image-registry-85c5fdc4b8-rwfwz\" (UID: \"e660267b-cf0d-4e4e-bb05-1769ff224bb2\") " pod="openshift-image-registry/image-registry-85c5fdc4b8-rwfwz" Jul 27 18:29:20.521381 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.521355 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgfdj\" (UniqueName: \"kubernetes.io/projected/2b41bffe-08d5-4b52-ae91-dcfebab44a99-kube-api-access-tgfdj\") pod \"router-default-5c874b7d95-95vlk\" (UID: \"2b41bffe-08d5-4b52-ae91-dcfebab44a99\") " pod="openshift-ingress/router-default-5c874b7d95-95vlk" Jul 27 18:29:20.521852 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.521800 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgshz\" (UniqueName: \"kubernetes.io/projected/06e8b4ad-8ea1-4fa8-ab2c-3d6c24fafc88-kube-api-access-pgshz\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-5m9rh\" (UID: \"06e8b4ad-8ea1-4fa8-ab2c-3d6c24fafc88\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-5m9rh" Jul 27 18:29:20.522664 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.522640 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcp7s\" (UniqueName: \"kubernetes.io/projected/d4bd05a3-ebb5-4c2e-9802-5c0b15f2bdba-kube-api-access-qcp7s\") pod \"service-ca-operator-5d6c75f84f-6dl5t\" (UID: \"d4bd05a3-ebb5-4c2e-9802-5c0b15f2bdba\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-6dl5t" Jul 27 18:29:20.523202 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.523146 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8h5v\" (UniqueName: \"kubernetes.io/projected/1e0a413d-db6c-4206-928b-6d95b365c39b-kube-api-access-k8h5v\") pod \"cluster-monitoring-operator-548b489cb8-g9cd9\" (UID: \"1e0a413d-db6c-4206-928b-6d95b365c39b\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-g9cd9" Jul 27 18:29:20.523202 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.523148 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nxjl\" (UniqueName: \"kubernetes.io/projected/b61c6434-d128-4ec0-920d-a00b085ae9c6-kube-api-access-7nxjl\") pod \"insights-operator-5885fdb759-mhdxk\" (UID: \"b61c6434-d128-4ec0-920d-a00b085ae9c6\") " pod="openshift-insights/insights-operator-5885fdb759-mhdxk" Jul 27 18:29:20.523363 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.523347 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ht7gm\" (UniqueName: \"kubernetes.io/projected/4e16cd40-8590-4739-a49c-b67d0f1e37d7-kube-api-access-ht7gm\") pod \"cluster-samples-operator-6748489656-7lqlw\" (UID: \"4e16cd40-8590-4739-a49c-b67d0f1e37d7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-7lqlw" Jul 27 18:29:20.523983 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.523952 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcfqt\" (UniqueName: \"kubernetes.io/projected/e660267b-cf0d-4e4e-bb05-1769ff224bb2-kube-api-access-hcfqt\") pod \"image-registry-85c5fdc4b8-rwfwz\" (UID: \"e660267b-cf0d-4e4e-bb05-1769ff224bb2\") " pod="openshift-image-registry/image-registry-85c5fdc4b8-rwfwz" Jul 27 18:29:20.524332 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.524311 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2cgj\" (UniqueName: \"kubernetes.io/projected/715b2084-d79e-4cbf-a057-f2595b4e6b9b-kube-api-access-q2cgj\") pod \"console-operator-584dcb6b54-mv4hb\" (UID: \"715b2084-d79e-4cbf-a057-f2595b4e6b9b\") " pod="openshift-console-operator/console-operator-584dcb6b54-mv4hb" Jul 27 18:29:20.524908 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.524887 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpb62\" (UniqueName: \"kubernetes.io/projected/bd88c5eb-6a78-42bb-9b53-e5eb5f6489c6-kube-api-access-dpb62\") pod \"network-check-source-7d88cb55d8-f8hjw\" (UID: \"bd88c5eb-6a78-42bb-9b53-e5eb5f6489c6\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-f8hjw" Jul 27 18:29:20.525349 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.525327 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkz7r\" (UniqueName: \"kubernetes.io/projected/bf48a4f7-eedd-41cc-9d53-93229f167d40-kube-api-access-hkz7r\") pod \"dns-default-pqqm5\" (UID: \"bf48a4f7-eedd-41cc-9d53-93229f167d40\") " pod="openshift-dns/dns-default-pqqm5" Jul 27 18:29:20.525868 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.525848 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pct8m\" (UniqueName: \"kubernetes.io/projected/74af2906-b939-4580-b91f-5d42c1805d74-kube-api-access-pct8m\") pod \"volume-data-source-validator-549c6d54cf-79snb\" (UID: \"74af2906-b939-4580-b91f-5d42c1805d74\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-79snb" Jul 27 18:29:20.527285 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.527262 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-md2gw\" (UniqueName: \"kubernetes.io/projected/e4fef94e-e2c7-4725-9da9-ddb80b15149d-kube-api-access-md2gw\") pod \"ingress-canary-6hbg6\" (UID: \"e4fef94e-e2c7-4725-9da9-ddb80b15149d\") " pod="openshift-ingress-canary/ingress-canary-6hbg6" Jul 27 18:29:20.645779 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.645741 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-79snb" Jul 27 18:29:20.646902 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.646875 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5885fdb759-mhdxk" Jul 27 18:29:20.671609 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.671565 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-6dl5t" Jul 27 18:29:20.677698 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.677674 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-mv4hb" Jul 27 18:29:20.693520 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.693487 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-5m9rh" Jul 27 18:29:20.701020 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.701001 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-f8hjw" Jul 27 18:29:20.714020 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:20.713997 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-6hbg6" Jul 27 18:29:21.015559 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:21.015520 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1e0a413d-db6c-4206-928b-6d95b365c39b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-g9cd9\" (UID: \"1e0a413d-db6c-4206-928b-6d95b365c39b\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-g9cd9" Jul 27 18:29:21.015726 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:21.015580 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b41bffe-08d5-4b52-ae91-dcfebab44a99-metrics-certs\") pod \"router-default-5c874b7d95-95vlk\" (UID: \"2b41bffe-08d5-4b52-ae91-dcfebab44a99\") " pod="openshift-ingress/router-default-5c874b7d95-95vlk" Jul 27 18:29:21.015726 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:21.015655 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bf48a4f7-eedd-41cc-9d53-93229f167d40-metrics-tls\") pod \"dns-default-pqqm5\" (UID: \"bf48a4f7-eedd-41cc-9d53-93229f167d40\") " pod="openshift-dns/dns-default-pqqm5" Jul 27 18:29:21.015726 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:29:21.015691 2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 27 18:29:21.015880 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:29:21.015765 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e0a413d-db6c-4206-928b-6d95b365c39b-cluster-monitoring-operator-tls podName:1e0a413d-db6c-4206-928b-6d95b365c39b nodeName:}" failed. No retries permitted until 2026-07-27 18:29:22.015745861 +0000 UTC m=+37.077434871 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/1e0a413d-db6c-4206-928b-6d95b365c39b-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-g9cd9" (UID: "1e0a413d-db6c-4206-928b-6d95b365c39b") : secret "cluster-monitoring-operator-tls" not found Jul 27 18:29:21.015880 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:29:21.015771 2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 27 18:29:21.015880 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:29:21.015773 2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 27 18:29:21.015880 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:29:21.015818 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 27 18:29:21.015880 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:21.015697 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bfcc4ba6-b269-49a5-847a-2175dc0881fb-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-9pgxt\" (UID: \"bfcc4ba6-b269-49a5-847a-2175dc0881fb\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-9pgxt" Jul 27 18:29:21.015880 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:29:21.015824 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bfcc4ba6-b269-49a5-847a-2175dc0881fb-networking-console-plugin-cert podName:bfcc4ba6-b269-49a5-847a-2175dc0881fb nodeName:}" failed. No retries permitted until 2026-07-27 18:29:22.015808482 +0000 UTC m=+37.077497510 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/bfcc4ba6-b269-49a5-847a-2175dc0881fb-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-9pgxt" (UID: "bfcc4ba6-b269-49a5-847a-2175dc0881fb") : secret "networking-console-plugin-cert" not found Jul 27 18:29:21.015880 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:29:21.015886 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b41bffe-08d5-4b52-ae91-dcfebab44a99-metrics-certs podName:2b41bffe-08d5-4b52-ae91-dcfebab44a99 nodeName:}" failed. No retries permitted until 2026-07-27 18:29:22.015875157 +0000 UTC m=+37.077564166 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2b41bffe-08d5-4b52-ae91-dcfebab44a99-metrics-certs") pod "router-default-5c874b7d95-95vlk" (UID: "2b41bffe-08d5-4b52-ae91-dcfebab44a99") : secret "router-metrics-certs-default" not found Jul 27 18:29:21.016203 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:29:21.015904 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf48a4f7-eedd-41cc-9d53-93229f167d40-metrics-tls podName:bf48a4f7-eedd-41cc-9d53-93229f167d40 nodeName:}" failed. No retries permitted until 2026-07-27 18:29:22.015894567 +0000 UTC m=+37.077583578 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bf48a4f7-eedd-41cc-9d53-93229f167d40-metrics-tls") pod "dns-default-pqqm5" (UID: "bf48a4f7-eedd-41cc-9d53-93229f167d40") : secret "dns-default-metrics-tls" not found Jul 27 18:29:21.016203 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:21.015944 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e660267b-cf0d-4e4e-bb05-1769ff224bb2-registry-tls\") pod \"image-registry-85c5fdc4b8-rwfwz\" (UID: \"e660267b-cf0d-4e4e-bb05-1769ff224bb2\") " pod="openshift-image-registry/image-registry-85c5fdc4b8-rwfwz" Jul 27 18:29:21.016203 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:21.015975 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b41bffe-08d5-4b52-ae91-dcfebab44a99-service-ca-bundle\") pod \"router-default-5c874b7d95-95vlk\" (UID: \"2b41bffe-08d5-4b52-ae91-dcfebab44a99\") " pod="openshift-ingress/router-default-5c874b7d95-95vlk" Jul 27 18:29:21.016203 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:21.016014 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4e16cd40-8590-4739-a49c-b67d0f1e37d7-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-7lqlw\" (UID: \"4e16cd40-8590-4739-a49c-b67d0f1e37d7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-7lqlw" Jul 27 18:29:21.016203 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:29:21.016093 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 27 18:29:21.016203 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:29:21.016101 2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 27 18:29:21.016203 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:29:21.016111 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-85c5fdc4b8-rwfwz: secret "image-registry-tls" not found Jul 27 18:29:21.016203 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:29:21.016106 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2b41bffe-08d5-4b52-ae91-dcfebab44a99-service-ca-bundle podName:2b41bffe-08d5-4b52-ae91-dcfebab44a99 nodeName:}" failed. No retries permitted until 2026-07-27 18:29:22.016092531 +0000 UTC m=+37.077781540 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/2b41bffe-08d5-4b52-ae91-dcfebab44a99-service-ca-bundle") pod "router-default-5c874b7d95-95vlk" (UID: "2b41bffe-08d5-4b52-ae91-dcfebab44a99") : configmap references non-existent config key: service-ca.crt Jul 27 18:29:21.016203 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:29:21.016148 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e16cd40-8590-4739-a49c-b67d0f1e37d7-samples-operator-tls podName:4e16cd40-8590-4739-a49c-b67d0f1e37d7 nodeName:}" failed. No retries permitted until 2026-07-27 18:29:22.016136941 +0000 UTC m=+37.077825953 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/4e16cd40-8590-4739-a49c-b67d0f1e37d7-samples-operator-tls") pod "cluster-samples-operator-6748489656-7lqlw" (UID: "4e16cd40-8590-4739-a49c-b67d0f1e37d7") : secret "samples-operator-tls" not found Jul 27 18:29:21.016203 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:29:21.016167 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e660267b-cf0d-4e4e-bb05-1769ff224bb2-registry-tls podName:e660267b-cf0d-4e4e-bb05-1769ff224bb2 nodeName:}" failed. No retries permitted until 2026-07-27 18:29:22.01615604 +0000 UTC m=+37.077845049 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e660267b-cf0d-4e4e-bb05-1769ff224bb2-registry-tls") pod "image-registry-85c5fdc4b8-rwfwz" (UID: "e660267b-cf0d-4e4e-bb05-1769ff224bb2") : secret "image-registry-tls" not found Jul 27 18:29:22.027865 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:22.026357 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b41bffe-08d5-4b52-ae91-dcfebab44a99-service-ca-bundle\") pod \"router-default-5c874b7d95-95vlk\" (UID: \"2b41bffe-08d5-4b52-ae91-dcfebab44a99\") " pod="openshift-ingress/router-default-5c874b7d95-95vlk" Jul 27 18:29:22.027865 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:22.026419 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4e16cd40-8590-4739-a49c-b67d0f1e37d7-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-7lqlw\" (UID: \"4e16cd40-8590-4739-a49c-b67d0f1e37d7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-7lqlw" Jul 27 18:29:22.027865 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:22.026474 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1e0a413d-db6c-4206-928b-6d95b365c39b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-g9cd9\" (UID: \"1e0a413d-db6c-4206-928b-6d95b365c39b\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-g9cd9" Jul 27 18:29:22.027865 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:22.026513 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b41bffe-08d5-4b52-ae91-dcfebab44a99-metrics-certs\") pod \"router-default-5c874b7d95-95vlk\" (UID: \"2b41bffe-08d5-4b52-ae91-dcfebab44a99\") " pod="openshift-ingress/router-default-5c874b7d95-95vlk" Jul 27 18:29:22.027865 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:22.026586 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bf48a4f7-eedd-41cc-9d53-93229f167d40-metrics-tls\") pod \"dns-default-pqqm5\" (UID: \"bf48a4f7-eedd-41cc-9d53-93229f167d40\") " pod="openshift-dns/dns-default-pqqm5" Jul 27 18:29:22.027865 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:22.026630 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bfcc4ba6-b269-49a5-847a-2175dc0881fb-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-9pgxt\" (UID: \"bfcc4ba6-b269-49a5-847a-2175dc0881fb\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-9pgxt" Jul 27 18:29:22.027865 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:22.026659 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e660267b-cf0d-4e4e-bb05-1769ff224bb2-registry-tls\") pod \"image-registry-85c5fdc4b8-rwfwz\" (UID: \"e660267b-cf0d-4e4e-bb05-1769ff224bb2\") " pod="openshift-image-registry/image-registry-85c5fdc4b8-rwfwz" Jul 27 18:29:22.027865 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:29:22.026812 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 27 18:29:22.027865 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:29:22.026824 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-85c5fdc4b8-rwfwz: secret "image-registry-tls" not found Jul 27 18:29:22.027865 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:29:22.026883 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e660267b-cf0d-4e4e-bb05-1769ff224bb2-registry-tls podName:e660267b-cf0d-4e4e-bb05-1769ff224bb2 nodeName:}" failed. No retries permitted until 2026-07-27 18:29:24.02686577 +0000 UTC m=+39.088554785 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e660267b-cf0d-4e4e-bb05-1769ff224bb2-registry-tls") pod "image-registry-85c5fdc4b8-rwfwz" (UID: "e660267b-cf0d-4e4e-bb05-1769ff224bb2") : secret "image-registry-tls" not found Jul 27 18:29:22.027865 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:29:22.027376 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2b41bffe-08d5-4b52-ae91-dcfebab44a99-service-ca-bundle podName:2b41bffe-08d5-4b52-ae91-dcfebab44a99 nodeName:}" failed. No retries permitted until 2026-07-27 18:29:24.027357219 +0000 UTC m=+39.089046237 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/2b41bffe-08d5-4b52-ae91-dcfebab44a99-service-ca-bundle") pod "router-default-5c874b7d95-95vlk" (UID: "2b41bffe-08d5-4b52-ae91-dcfebab44a99") : configmap references non-existent config key: service-ca.crt Jul 27 18:29:22.027865 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:29:22.027456 2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 27 18:29:22.027865 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:29:22.027487 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e16cd40-8590-4739-a49c-b67d0f1e37d7-samples-operator-tls podName:4e16cd40-8590-4739-a49c-b67d0f1e37d7 nodeName:}" failed. No retries permitted until 2026-07-27 18:29:24.027477286 +0000 UTC m=+39.089166297 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/4e16cd40-8590-4739-a49c-b67d0f1e37d7-samples-operator-tls") pod "cluster-samples-operator-6748489656-7lqlw" (UID: "4e16cd40-8590-4739-a49c-b67d0f1e37d7") : secret "samples-operator-tls" not found Jul 27 18:29:22.027865 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:29:22.027539 2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 27 18:29:22.027865 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:29:22.027568 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e0a413d-db6c-4206-928b-6d95b365c39b-cluster-monitoring-operator-tls podName:1e0a413d-db6c-4206-928b-6d95b365c39b nodeName:}" failed. No retries permitted until 2026-07-27 18:29:24.027559378 +0000 UTC m=+39.089248393 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/1e0a413d-db6c-4206-928b-6d95b365c39b-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-g9cd9" (UID: "1e0a413d-db6c-4206-928b-6d95b365c39b") : secret "cluster-monitoring-operator-tls" not found Jul 27 18:29:22.027865 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:29:22.027617 2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 27 18:29:22.029469 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:29:22.027645 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b41bffe-08d5-4b52-ae91-dcfebab44a99-metrics-certs podName:2b41bffe-08d5-4b52-ae91-dcfebab44a99 nodeName:}" failed. No retries permitted until 2026-07-27 18:29:24.027635916 +0000 UTC m=+39.089324931 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2b41bffe-08d5-4b52-ae91-dcfebab44a99-metrics-certs") pod "router-default-5c874b7d95-95vlk" (UID: "2b41bffe-08d5-4b52-ae91-dcfebab44a99") : secret "router-metrics-certs-default" not found Jul 27 18:29:22.029469 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:29:22.027696 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 27 18:29:22.029469 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:29:22.027720 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf48a4f7-eedd-41cc-9d53-93229f167d40-metrics-tls podName:bf48a4f7-eedd-41cc-9d53-93229f167d40 nodeName:}" failed. No retries permitted until 2026-07-27 18:29:24.027711365 +0000 UTC m=+39.089400379 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bf48a4f7-eedd-41cc-9d53-93229f167d40-metrics-tls") pod "dns-default-pqqm5" (UID: "bf48a4f7-eedd-41cc-9d53-93229f167d40") : secret "dns-default-metrics-tls" not found Jul 27 18:29:22.029469 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:29:22.027769 2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 27 18:29:22.029469 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:29:22.027796 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bfcc4ba6-b269-49a5-847a-2175dc0881fb-networking-console-plugin-cert podName:bfcc4ba6-b269-49a5-847a-2175dc0881fb nodeName:}" failed. No retries permitted until 2026-07-27 18:29:24.027787971 +0000 UTC m=+39.089476985 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/bfcc4ba6-b269-49a5-847a-2175dc0881fb-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-9pgxt" (UID: "bfcc4ba6-b269-49a5-847a-2175dc0881fb") : secret "networking-console-plugin-cert" not found Jul 27 18:29:22.090919 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:22.089964 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-79snb"] Jul 27 18:29:22.101751 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:22.101277 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-f8hjw"] Jul 27 18:29:22.103952 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:29:22.103896 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd88c5eb_6a78_42bb_9b53_e5eb5f6489c6.slice/crio-1271232a3aa637611f5fedafac134718b73cc9a4e30078214a856ff1407346f6 WatchSource:0}: Error finding container 1271232a3aa637611f5fedafac134718b73cc9a4e30078214a856ff1407346f6: Status 404 returned error can't find the container with id 1271232a3aa637611f5fedafac134718b73cc9a4e30078214a856ff1407346f6 Jul 27 18:29:22.107381 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:22.107340 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5885fdb759-mhdxk"] Jul 27 18:29:22.121512 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:22.121490 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-6dl5t"] Jul 27 18:29:22.122812 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:22.122795 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-6hbg6"] Jul 27 18:29:22.123900 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:29:22.123877 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4fef94e_e2c7_4725_9da9_ddb80b15149d.slice/crio-a9139b72ed2f72b59757e9ad344a4f4de335536bf533e858aca8e36e6bce4298 WatchSource:0}: Error finding container a9139b72ed2f72b59757e9ad344a4f4de335536bf533e858aca8e36e6bce4298: Status 404 returned error can't find the container with id a9139b72ed2f72b59757e9ad344a4f4de335536bf533e858aca8e36e6bce4298 Jul 27 18:29:22.124873 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:29:22.124834 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4bd05a3_ebb5_4c2e_9802_5c0b15f2bdba.slice/crio-abe17194c4f878d335b46cb214468dfc943a936b4e4566b89c6e98bcad561d8b WatchSource:0}: Error finding container abe17194c4f878d335b46cb214468dfc943a936b4e4566b89c6e98bcad561d8b: Status 404 returned error can't find the container with id abe17194c4f878d335b46cb214468dfc943a936b4e4566b89c6e98bcad561d8b Jul 27 18:29:22.132147 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:22.132124 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-5m9rh"] Jul 27 18:29:22.135641 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:29:22.135621 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06e8b4ad_8ea1_4fa8_ab2c_3d6c24fafc88.slice/crio-67a566a6dd13513fab3ba5e8b98934fbd68f79177297177fe7e492a1512aebf1 WatchSource:0}: Error finding container 67a566a6dd13513fab3ba5e8b98934fbd68f79177297177fe7e492a1512aebf1: Status 404 returned error can't find the container with id 67a566a6dd13513fab3ba5e8b98934fbd68f79177297177fe7e492a1512aebf1 Jul 27 18:29:22.138784 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:22.138765 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-mv4hb"] Jul 27 18:29:22.141017 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:29:22.140996 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod715b2084_d79e_4cbf_a057_f2595b4e6b9b.slice/crio-d6143ae0917fbdff85ac527cbb010cccb0e98f920daab361dd70e8c20c7c8bcd WatchSource:0}: Error finding container d6143ae0917fbdff85ac527cbb010cccb0e98f920daab361dd70e8c20c7c8bcd: Status 404 returned error can't find the container with id d6143ae0917fbdff85ac527cbb010cccb0e98f920daab361dd70e8c20c7c8bcd Jul 27 18:29:22.752494 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:22.752455 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-5m9rh" event={"ID":"06e8b4ad-8ea1-4fa8-ab2c-3d6c24fafc88","Type":"ContainerStarted","Data":"67a566a6dd13513fab3ba5e8b98934fbd68f79177297177fe7e492a1512aebf1"} Jul 27 18:29:22.753978 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:22.753944 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-6dl5t" event={"ID":"d4bd05a3-ebb5-4c2e-9802-5c0b15f2bdba","Type":"ContainerStarted","Data":"abe17194c4f878d335b46cb214468dfc943a936b4e4566b89c6e98bcad561d8b"} Jul 27 18:29:22.755296 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:22.755245 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-79snb" event={"ID":"74af2906-b939-4580-b91f-5d42c1805d74","Type":"ContainerStarted","Data":"0929d21f00fe3b44cbaf0810c61cde3059055eef5232bbd76f23346bde0c11e3"} Jul 27 18:29:22.756676 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:22.756652 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-mhdxk" event={"ID":"b61c6434-d128-4ec0-920d-a00b085ae9c6","Type":"ContainerStarted","Data":"f23130231815a1dbb1f24665b8e44eeb959f0e3347f040f857577756a931edd2"} Jul 27 18:29:22.758176 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:22.758152 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-mv4hb" event={"ID":"715b2084-d79e-4cbf-a057-f2595b4e6b9b","Type":"ContainerStarted","Data":"d6143ae0917fbdff85ac527cbb010cccb0e98f920daab361dd70e8c20c7c8bcd"} Jul 27 18:29:22.761953 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:22.761926 2579 generic.go:356] "Generic (PLEG): container finished" podID="5ba4cb7b-80f6-470b-8ab2-7e22c50af412" containerID="42e53fe25845880585cb383b1d0570b302767f23bbd429087b1853735d0f05d0" exitCode=0 Jul 27 18:29:22.762052 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:22.762001 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6jzx9" event={"ID":"5ba4cb7b-80f6-470b-8ab2-7e22c50af412","Type":"ContainerDied","Data":"42e53fe25845880585cb383b1d0570b302767f23bbd429087b1853735d0f05d0"} Jul 27 18:29:22.763935 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:22.763908 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-6hbg6" event={"ID":"e4fef94e-e2c7-4725-9da9-ddb80b15149d","Type":"ContainerStarted","Data":"a9139b72ed2f72b59757e9ad344a4f4de335536bf533e858aca8e36e6bce4298"} Jul 27 18:29:22.765138 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:22.765092 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-f8hjw" event={"ID":"bd88c5eb-6a78-42bb-9b53-e5eb5f6489c6","Type":"ContainerStarted","Data":"1271232a3aa637611f5fedafac134718b73cc9a4e30078214a856ff1407346f6"} Jul 27 18:29:23.780043 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:23.780002 2579 generic.go:356] "Generic (PLEG): container finished" podID="5ba4cb7b-80f6-470b-8ab2-7e22c50af412" containerID="1f10a8a92147ed054c6c073e4bbe2c3a0bfcdf35828d5910cd4e05fc38477f2a" exitCode=0 Jul 27 18:29:23.780564 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:23.780073 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6jzx9" event={"ID":"5ba4cb7b-80f6-470b-8ab2-7e22c50af412","Type":"ContainerDied","Data":"1f10a8a92147ed054c6c073e4bbe2c3a0bfcdf35828d5910cd4e05fc38477f2a"} Jul 27 18:29:23.905674 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:23.905635 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:29:24.000328 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:23.999969 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:29:24.049977 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:24.049867 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bf48a4f7-eedd-41cc-9d53-93229f167d40-metrics-tls\") pod \"dns-default-pqqm5\" (UID: \"bf48a4f7-eedd-41cc-9d53-93229f167d40\") " pod="openshift-dns/dns-default-pqqm5" Jul 27 18:29:24.049977 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:24.049933 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bfcc4ba6-b269-49a5-847a-2175dc0881fb-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-9pgxt\" (UID: \"bfcc4ba6-b269-49a5-847a-2175dc0881fb\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-9pgxt" Jul 27 18:29:24.049977 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:24.049967 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e660267b-cf0d-4e4e-bb05-1769ff224bb2-registry-tls\") pod \"image-registry-85c5fdc4b8-rwfwz\" (UID: \"e660267b-cf0d-4e4e-bb05-1769ff224bb2\") " pod="openshift-image-registry/image-registry-85c5fdc4b8-rwfwz" Jul 27 18:29:24.050288 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:24.050005 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b41bffe-08d5-4b52-ae91-dcfebab44a99-service-ca-bundle\") pod \"router-default-5c874b7d95-95vlk\" (UID: \"2b41bffe-08d5-4b52-ae91-dcfebab44a99\") " pod="openshift-ingress/router-default-5c874b7d95-95vlk" Jul 27 18:29:24.050288 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:24.050036 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4e16cd40-8590-4739-a49c-b67d0f1e37d7-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-7lqlw\" (UID: \"4e16cd40-8590-4739-a49c-b67d0f1e37d7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-7lqlw" Jul 27 18:29:24.050288 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:24.050089 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1e0a413d-db6c-4206-928b-6d95b365c39b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-g9cd9\" (UID: \"1e0a413d-db6c-4206-928b-6d95b365c39b\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-g9cd9" Jul 27 18:29:24.050288 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:24.050125 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b41bffe-08d5-4b52-ae91-dcfebab44a99-metrics-certs\") pod \"router-default-5c874b7d95-95vlk\" (UID: \"2b41bffe-08d5-4b52-ae91-dcfebab44a99\") " pod="openshift-ingress/router-default-5c874b7d95-95vlk" Jul 27 18:29:24.050505 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:29:24.050295 2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 27 18:29:24.050505 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:29:24.050354 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b41bffe-08d5-4b52-ae91-dcfebab44a99-metrics-certs podName:2b41bffe-08d5-4b52-ae91-dcfebab44a99 nodeName:}" failed. No retries permitted until 2026-07-27 18:29:28.050336701 +0000 UTC m=+43.112025724 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2b41bffe-08d5-4b52-ae91-dcfebab44a99-metrics-certs") pod "router-default-5c874b7d95-95vlk" (UID: "2b41bffe-08d5-4b52-ae91-dcfebab44a99") : secret "router-metrics-certs-default" not found Jul 27 18:29:24.051384 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:29:24.050823 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 27 18:29:24.051384 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:29:24.050874 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf48a4f7-eedd-41cc-9d53-93229f167d40-metrics-tls podName:bf48a4f7-eedd-41cc-9d53-93229f167d40 nodeName:}" failed. No retries permitted until 2026-07-27 18:29:28.050858423 +0000 UTC m=+43.112547436 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bf48a4f7-eedd-41cc-9d53-93229f167d40-metrics-tls") pod "dns-default-pqqm5" (UID: "bf48a4f7-eedd-41cc-9d53-93229f167d40") : secret "dns-default-metrics-tls" not found Jul 27 18:29:24.051384 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:29:24.050932 2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 27 18:29:24.051384 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:29:24.050965 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bfcc4ba6-b269-49a5-847a-2175dc0881fb-networking-console-plugin-cert podName:bfcc4ba6-b269-49a5-847a-2175dc0881fb nodeName:}" failed. No retries permitted until 2026-07-27 18:29:28.050954292 +0000 UTC m=+43.112643309 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/bfcc4ba6-b269-49a5-847a-2175dc0881fb-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-9pgxt" (UID: "bfcc4ba6-b269-49a5-847a-2175dc0881fb") : secret "networking-console-plugin-cert" not found Jul 27 18:29:24.051384 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:29:24.051037 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 27 18:29:24.051384 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:29:24.051048 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-85c5fdc4b8-rwfwz: secret "image-registry-tls" not found Jul 27 18:29:24.051384 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:29:24.051080 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e660267b-cf0d-4e4e-bb05-1769ff224bb2-registry-tls podName:e660267b-cf0d-4e4e-bb05-1769ff224bb2 nodeName:}" failed. No retries permitted until 2026-07-27 18:29:28.051067824 +0000 UTC m=+43.112756837 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e660267b-cf0d-4e4e-bb05-1769ff224bb2-registry-tls") pod "image-registry-85c5fdc4b8-rwfwz" (UID: "e660267b-cf0d-4e4e-bb05-1769ff224bb2") : secret "image-registry-tls" not found Jul 27 18:29:24.051384 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:29:24.051156 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2b41bffe-08d5-4b52-ae91-dcfebab44a99-service-ca-bundle podName:2b41bffe-08d5-4b52-ae91-dcfebab44a99 nodeName:}" failed. No retries permitted until 2026-07-27 18:29:28.051144965 +0000 UTC m=+43.112833981 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/2b41bffe-08d5-4b52-ae91-dcfebab44a99-service-ca-bundle") pod "router-default-5c874b7d95-95vlk" (UID: "2b41bffe-08d5-4b52-ae91-dcfebab44a99") : configmap references non-existent config key: service-ca.crt Jul 27 18:29:24.051384 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:29:24.051220 2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 27 18:29:24.051384 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:29:24.051270 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e16cd40-8590-4739-a49c-b67d0f1e37d7-samples-operator-tls podName:4e16cd40-8590-4739-a49c-b67d0f1e37d7 nodeName:}" failed. No retries permitted until 2026-07-27 18:29:28.051258301 +0000 UTC m=+43.112947314 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/4e16cd40-8590-4739-a49c-b67d0f1e37d7-samples-operator-tls") pod "cluster-samples-operator-6748489656-7lqlw" (UID: "4e16cd40-8590-4739-a49c-b67d0f1e37d7") : secret "samples-operator-tls" not found Jul 27 18:29:24.051384 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:29:24.051323 2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 27 18:29:24.051384 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:29:24.051352 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e0a413d-db6c-4206-928b-6d95b365c39b-cluster-monitoring-operator-tls podName:1e0a413d-db6c-4206-928b-6d95b365c39b nodeName:}" failed. No retries permitted until 2026-07-27 18:29:28.051343088 +0000 UTC m=+43.113032103 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/1e0a413d-db6c-4206-928b-6d95b365c39b-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-g9cd9" (UID: "1e0a413d-db6c-4206-928b-6d95b365c39b") : secret "cluster-monitoring-operator-tls" not found Jul 27 18:29:26.089047 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:26.088974 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:29:26.148847 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:26.148805 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:29:27.126139 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:27.126100 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:29:27.888214 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:27.888164 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:29:28.087620 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:28.087583 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1e0a413d-db6c-4206-928b-6d95b365c39b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-g9cd9\" (UID: \"1e0a413d-db6c-4206-928b-6d95b365c39b\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-g9cd9" Jul 27 18:29:28.087800 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:28.087636 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b41bffe-08d5-4b52-ae91-dcfebab44a99-metrics-certs\") pod \"router-default-5c874b7d95-95vlk\" (UID: \"2b41bffe-08d5-4b52-ae91-dcfebab44a99\") " pod="openshift-ingress/router-default-5c874b7d95-95vlk" Jul 27 18:29:28.087800 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:28.087711 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bf48a4f7-eedd-41cc-9d53-93229f167d40-metrics-tls\") pod \"dns-default-pqqm5\" (UID: \"bf48a4f7-eedd-41cc-9d53-93229f167d40\") " pod="openshift-dns/dns-default-pqqm5" Jul 27 18:29:28.087800 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:29:28.087730 2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 27 18:29:28.087800 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:28.087756 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bfcc4ba6-b269-49a5-847a-2175dc0881fb-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-9pgxt\" (UID: \"bfcc4ba6-b269-49a5-847a-2175dc0881fb\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-9pgxt" Jul 27 18:29:28.087800 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:29:28.087791 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e0a413d-db6c-4206-928b-6d95b365c39b-cluster-monitoring-operator-tls podName:1e0a413d-db6c-4206-928b-6d95b365c39b nodeName:}" failed. No retries permitted until 2026-07-27 18:29:36.087775491 +0000 UTC m=+51.149464501 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/1e0a413d-db6c-4206-928b-6d95b365c39b-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-g9cd9" (UID: "1e0a413d-db6c-4206-928b-6d95b365c39b") : secret "cluster-monitoring-operator-tls" not found Jul 27 18:29:28.088070 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:29:28.087803 2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 27 18:29:28.088070 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:28.087813 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e660267b-cf0d-4e4e-bb05-1769ff224bb2-registry-tls\") pod \"image-registry-85c5fdc4b8-rwfwz\" (UID: \"e660267b-cf0d-4e4e-bb05-1769ff224bb2\") " pod="openshift-image-registry/image-registry-85c5fdc4b8-rwfwz" Jul 27 18:29:28.088070 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:29:28.087849 2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 27 18:29:28.088070 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:28.087856 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b41bffe-08d5-4b52-ae91-dcfebab44a99-service-ca-bundle\") pod \"router-default-5c874b7d95-95vlk\" (UID: \"2b41bffe-08d5-4b52-ae91-dcfebab44a99\") " pod="openshift-ingress/router-default-5c874b7d95-95vlk" Jul 27 18:29:28.088070 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:29:28.087870 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 27 18:29:28.088070 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:28.087888 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4e16cd40-8590-4739-a49c-b67d0f1e37d7-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-7lqlw\" (UID: \"4e16cd40-8590-4739-a49c-b67d0f1e37d7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-7lqlw" Jul 27 18:29:28.088070 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:29:28.087898 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bfcc4ba6-b269-49a5-847a-2175dc0881fb-networking-console-plugin-cert podName:bfcc4ba6-b269-49a5-847a-2175dc0881fb nodeName:}" failed. No retries permitted until 2026-07-27 18:29:36.087881863 +0000 UTC m=+51.149570878 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/bfcc4ba6-b269-49a5-847a-2175dc0881fb-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-9pgxt" (UID: "bfcc4ba6-b269-49a5-847a-2175dc0881fb") : secret "networking-console-plugin-cert" not found Jul 27 18:29:28.088070 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:29:28.087920 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf48a4f7-eedd-41cc-9d53-93229f167d40-metrics-tls podName:bf48a4f7-eedd-41cc-9d53-93229f167d40 nodeName:}" failed. No retries permitted until 2026-07-27 18:29:36.08790978 +0000 UTC m=+51.149598794 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bf48a4f7-eedd-41cc-9d53-93229f167d40-metrics-tls") pod "dns-default-pqqm5" (UID: "bf48a4f7-eedd-41cc-9d53-93229f167d40") : secret "dns-default-metrics-tls" not found Jul 27 18:29:28.088070 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:29:28.087938 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2b41bffe-08d5-4b52-ae91-dcfebab44a99-service-ca-bundle podName:2b41bffe-08d5-4b52-ae91-dcfebab44a99 nodeName:}" failed. No retries permitted until 2026-07-27 18:29:36.087929254 +0000 UTC m=+51.149618272 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/2b41bffe-08d5-4b52-ae91-dcfebab44a99-service-ca-bundle") pod "router-default-5c874b7d95-95vlk" (UID: "2b41bffe-08d5-4b52-ae91-dcfebab44a99") : configmap references non-existent config key: service-ca.crt Jul 27 18:29:28.088070 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:29:28.087940 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 27 18:29:28.088070 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:29:28.087960 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b41bffe-08d5-4b52-ae91-dcfebab44a99-metrics-certs podName:2b41bffe-08d5-4b52-ae91-dcfebab44a99 nodeName:}" failed. No retries permitted until 2026-07-27 18:29:36.087950592 +0000 UTC m=+51.149639607 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2b41bffe-08d5-4b52-ae91-dcfebab44a99-metrics-certs") pod "router-default-5c874b7d95-95vlk" (UID: "2b41bffe-08d5-4b52-ae91-dcfebab44a99") : secret "router-metrics-certs-default" not found Jul 27 18:29:28.088070 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:29:28.087958 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-85c5fdc4b8-rwfwz: secret "image-registry-tls" not found Jul 27 18:29:28.088070 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:29:28.087944 2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 27 18:29:28.088070 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:29:28.088004 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e660267b-cf0d-4e4e-bb05-1769ff224bb2-registry-tls podName:e660267b-cf0d-4e4e-bb05-1769ff224bb2 nodeName:}" failed. No retries permitted until 2026-07-27 18:29:36.087992609 +0000 UTC m=+51.149681618 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e660267b-cf0d-4e4e-bb05-1769ff224bb2-registry-tls") pod "image-registry-85c5fdc4b8-rwfwz" (UID: "e660267b-cf0d-4e4e-bb05-1769ff224bb2") : secret "image-registry-tls" not found Jul 27 18:29:28.088070 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:29:28.088026 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e16cd40-8590-4739-a49c-b67d0f1e37d7-samples-operator-tls podName:4e16cd40-8590-4739-a49c-b67d0f1e37d7 nodeName:}" failed. No retries permitted until 2026-07-27 18:29:36.088017439 +0000 UTC m=+51.149706451 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/4e16cd40-8590-4739-a49c-b67d0f1e37d7-samples-operator-tls") pod "cluster-samples-operator-6748489656-7lqlw" (UID: "4e16cd40-8590-4739-a49c-b67d0f1e37d7") : secret "samples-operator-tls" not found Jul 27 18:29:30.815677 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:30.815650 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-mv4hb_715b2084-d79e-4cbf-a057-f2595b4e6b9b/console-operator/0.log" Jul 27 18:29:30.816166 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:30.815689 2579 generic.go:356] "Generic (PLEG): container finished" podID="715b2084-d79e-4cbf-a057-f2595b4e6b9b" containerID="6195fdaf7ae73c38988cac2974185c326ba30eef9fd4fe85d90c5c8a84ed2cea" exitCode=255 Jul 27 18:29:30.816166 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:30.815778 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-mv4hb" event={"ID":"715b2084-d79e-4cbf-a057-f2595b4e6b9b","Type":"ContainerDied","Data":"6195fdaf7ae73c38988cac2974185c326ba30eef9fd4fe85d90c5c8a84ed2cea"} Jul 27 18:29:30.816166 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:30.816032 2579 scope.go:117] "RemoveContainer" containerID="6195fdaf7ae73c38988cac2974185c326ba30eef9fd4fe85d90c5c8a84ed2cea" Jul 27 18:29:30.822555 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:30.822526 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6jzx9" event={"ID":"5ba4cb7b-80f6-470b-8ab2-7e22c50af412","Type":"ContainerStarted","Data":"f221015b59fa1256dd79585ba48fcae8593793413edd8726881745125d760a85"} Jul 27 18:29:30.824144 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:30.824117 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-6hbg6" event={"ID":"e4fef94e-e2c7-4725-9da9-ddb80b15149d","Type":"ContainerStarted","Data":"9d12de0f748d45b62e879efac1f450a64207c1bfdfc2be4fdb370038a1079221"} Jul 27 18:29:30.825778 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:30.825752 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-f8hjw" event={"ID":"bd88c5eb-6a78-42bb-9b53-e5eb5f6489c6","Type":"ContainerStarted","Data":"55a368a46947c77cbd1aaab68e90d28621f2aed84372c9b4abd438e381e232ec"} Jul 27 18:29:30.827481 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:30.827456 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-5m9rh" event={"ID":"06e8b4ad-8ea1-4fa8-ab2c-3d6c24fafc88","Type":"ContainerStarted","Data":"aa76be4766a1933c795802992ce6f5bb890ce2054f12ccd7cf4f5a10a9a3109a"} Jul 27 18:29:30.829259 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:30.829162 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-6dl5t" event={"ID":"d4bd05a3-ebb5-4c2e-9802-5c0b15f2bdba","Type":"ContainerStarted","Data":"cd5b52d400eb8248dde18adea882c98ab7c8f0243c1897987929f43b1700c4fd"} Jul 27 18:29:30.830983 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:30.830947 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-79snb" event={"ID":"74af2906-b939-4580-b91f-5d42c1805d74","Type":"ContainerStarted","Data":"bb616c53b4bf815f10cf65a7b298a0d3f0f6cdd4e62547763fb81e26baa5b97d"} Jul 27 18:29:30.832590 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:30.832564 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-mhdxk" event={"ID":"b61c6434-d128-4ec0-920d-a00b085ae9c6","Type":"ContainerStarted","Data":"fbd3e16bf0cafb85de56c6c68ddbb4dccad589a2cbaa5de17df511de7ea53109"} Jul 27 18:29:30.857081 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:30.857024 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-6dl5t" podStartSLOduration=14.011721049 podStartE2EDuration="21.857004272s" totalImagesPullingTime="7.845283223s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:29:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:29:30.85518179 +0000 UTC m=+45.916870823" watchObservedRunningTime="2026-07-27 18:29:30.857004272 +0000 UTC m=+45.918693304" Jul 27 18:29:30.871269 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:30.871191 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-79snb" podStartSLOduration=13.632482275 podStartE2EDuration="21.871172838s" totalImagesPullingTime="8.238690563s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:29:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:29:30.870977565 +0000 UTC m=+45.932666597" watchObservedRunningTime="2026-07-27 18:29:30.871172838 +0000 UTC m=+45.932861870" Jul 27 18:29:30.892415 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:30.892160 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5885fdb759-mhdxk" podStartSLOduration=14.037474253 podStartE2EDuration="21.892138965s" totalImagesPullingTime="7.854664712s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:29:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:29:30.890919747 +0000 UTC m=+45.952608802" watchObservedRunningTime="2026-07-27 18:29:30.892138965 +0000 UTC m=+45.953827997" Jul 27 18:29:30.941121 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:30.941069 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-5m9rh" podStartSLOduration=14.106161715 podStartE2EDuration="21.941046192s" totalImagesPullingTime="7.834884477s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:29:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:29:30.90905877 +0000 UTC m=+45.970747816" watchObservedRunningTime="2026-07-27 18:29:30.941046192 +0000 UTC m=+46.002735224" Jul 27 18:29:30.941823 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:30.941773 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-f8hjw" podStartSLOduration=14.066427945 podStartE2EDuration="21.941764015s" totalImagesPullingTime="7.87533607s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:29:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:29:30.94087095 +0000 UTC m=+46.002559983" watchObservedRunningTime="2026-07-27 18:29:30.941764015 +0000 UTC m=+46.003453048" Jul 27 18:29:30.975912 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:30.975858 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-6hbg6" podStartSLOduration=3.130273863 podStartE2EDuration="10.975838012s" totalImagesPullingTime="7.845564149s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:29:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:29:30.964706807 +0000 UTC m=+46.026395839" watchObservedRunningTime="2026-07-27 18:29:30.975838012 +0000 UTC m=+46.037527046" Jul 27 18:29:30.981907 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:30.981873 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:29:30.998787 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:30.998735 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-6jzx9" podStartSLOduration=13.985803787 podStartE2EDuration="45.998714451s" totalImagesPullingTime="32.012910664s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 18:28:45 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:29:30.996331752 +0000 UTC m=+46.058020784" watchObservedRunningTime="2026-07-27 18:29:30.998714451 +0000 UTC m=+46.060403484" Jul 27 18:29:31.080510 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:31.080396 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:29:31.185976 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:31.185939 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:29:31.284944 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:31.284916 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:29:31.394067 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:31.393987 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:29:31.529198 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:31.529155 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:29:31.623861 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:31.623823 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:29:31.625287 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:31.625256 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-dftr7"] Jul 27 18:29:31.660606 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:31.660518 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-dftr7"] Jul 27 18:29:31.660735 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:31.660664 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-dftr7" Jul 27 18:29:31.663677 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:31.663651 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-584h7\"" Jul 27 18:29:31.663811 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:31.663715 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 27 18:29:31.663811 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:31.663738 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 27 18:29:31.703550 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:31.703521 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:29:31.822805 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:31.822768 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8c36e039-1d59-4c42-8162-0f275315c3a2-data-volume\") pod \"insights-runtime-extractor-dftr7\" (UID: \"8c36e039-1d59-4c42-8162-0f275315c3a2\") " pod="openshift-insights/insights-runtime-extractor-dftr7" Jul 27 18:29:31.823171 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:31.822815 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8c36e039-1d59-4c42-8162-0f275315c3a2-crio-socket\") pod \"insights-runtime-extractor-dftr7\" (UID: \"8c36e039-1d59-4c42-8162-0f275315c3a2\") " pod="openshift-insights/insights-runtime-extractor-dftr7" Jul 27 18:29:31.823171 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:31.822861 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8c36e039-1d59-4c42-8162-0f275315c3a2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dftr7\" (UID: \"8c36e039-1d59-4c42-8162-0f275315c3a2\") " pod="openshift-insights/insights-runtime-extractor-dftr7" Jul 27 18:29:31.823171 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:31.823046 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8c36e039-1d59-4c42-8162-0f275315c3a2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dftr7\" (UID: \"8c36e039-1d59-4c42-8162-0f275315c3a2\") " pod="openshift-insights/insights-runtime-extractor-dftr7" Jul 27 18:29:31.823171 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:31.823073 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47kr8\" (UniqueName: \"kubernetes.io/projected/8c36e039-1d59-4c42-8162-0f275315c3a2-kube-api-access-47kr8\") pod \"insights-runtime-extractor-dftr7\" (UID: \"8c36e039-1d59-4c42-8162-0f275315c3a2\") " pod="openshift-insights/insights-runtime-extractor-dftr7" Jul 27 18:29:31.836825 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:31.836801 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:29:31.837943 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:31.837925 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-mv4hb_715b2084-d79e-4cbf-a057-f2595b4e6b9b/console-operator/1.log" Jul 27 18:29:31.838303 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:31.838290 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-mv4hb_715b2084-d79e-4cbf-a057-f2595b4e6b9b/console-operator/0.log" Jul 27 18:29:31.838380 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:31.838322 2579 generic.go:356] "Generic (PLEG): container finished" podID="715b2084-d79e-4cbf-a057-f2595b4e6b9b" containerID="500cf6669e9b32f9bfbaed857bebd570628cc16b01cd6517e2511021b5dc55c0" exitCode=255 Jul 27 18:29:31.838463 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:31.838441 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-mv4hb" event={"ID":"715b2084-d79e-4cbf-a057-f2595b4e6b9b","Type":"ContainerDied","Data":"500cf6669e9b32f9bfbaed857bebd570628cc16b01cd6517e2511021b5dc55c0"} Jul 27 18:29:31.838519 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:31.838512 2579 scope.go:117] "RemoveContainer" containerID="6195fdaf7ae73c38988cac2974185c326ba30eef9fd4fe85d90c5c8a84ed2cea" Jul 27 18:29:31.838722 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:31.838697 2579 scope.go:117] "RemoveContainer" containerID="500cf6669e9b32f9bfbaed857bebd570628cc16b01cd6517e2511021b5dc55c0" Jul 27 18:29:31.838902 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:29:31.838878 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-mv4hb_openshift-console-operator(715b2084-d79e-4cbf-a057-f2595b4e6b9b)\"" pod="openshift-console-operator/console-operator-584dcb6b54-mv4hb" podUID="715b2084-d79e-4cbf-a057-f2595b4e6b9b" Jul 27 18:29:31.922005 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:31.921925 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:29:31.923853 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:31.923825 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8c36e039-1d59-4c42-8162-0f275315c3a2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dftr7\" (UID: \"8c36e039-1d59-4c42-8162-0f275315c3a2\") " pod="openshift-insights/insights-runtime-extractor-dftr7" Jul 27 18:29:31.924005 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:29:31.923909 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 27 18:29:31.924005 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:29:31.923974 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c36e039-1d59-4c42-8162-0f275315c3a2-insights-runtime-extractor-tls podName:8c36e039-1d59-4c42-8162-0f275315c3a2 nodeName:}" failed. No retries permitted until 2026-07-27 18:29:32.423955156 +0000 UTC m=+47.485644165 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8c36e039-1d59-4c42-8162-0f275315c3a2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dftr7" (UID: "8c36e039-1d59-4c42-8162-0f275315c3a2") : secret "insights-runtime-extractor-tls" not found Jul 27 18:29:31.924005 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:31.924003 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47kr8\" (UniqueName: \"kubernetes.io/projected/8c36e039-1d59-4c42-8162-0f275315c3a2-kube-api-access-47kr8\") pod \"insights-runtime-extractor-dftr7\" (UID: \"8c36e039-1d59-4c42-8162-0f275315c3a2\") " pod="openshift-insights/insights-runtime-extractor-dftr7" Jul 27 18:29:31.924753 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:31.924335 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8c36e039-1d59-4c42-8162-0f275315c3a2-data-volume\") pod \"insights-runtime-extractor-dftr7\" (UID: \"8c36e039-1d59-4c42-8162-0f275315c3a2\") " pod="openshift-insights/insights-runtime-extractor-dftr7" Jul 27 18:29:31.924753 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:31.924472 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8c36e039-1d59-4c42-8162-0f275315c3a2-crio-socket\") pod \"insights-runtime-extractor-dftr7\" (UID: \"8c36e039-1d59-4c42-8162-0f275315c3a2\") " pod="openshift-insights/insights-runtime-extractor-dftr7" Jul 27 18:29:31.924753 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:31.924513 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8c36e039-1d59-4c42-8162-0f275315c3a2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dftr7\" (UID: \"8c36e039-1d59-4c42-8162-0f275315c3a2\") " pod="openshift-insights/insights-runtime-extractor-dftr7" Jul 27 18:29:31.924753 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:31.924565 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8c36e039-1d59-4c42-8162-0f275315c3a2-crio-socket\") pod \"insights-runtime-extractor-dftr7\" (UID: \"8c36e039-1d59-4c42-8162-0f275315c3a2\") " pod="openshift-insights/insights-runtime-extractor-dftr7" Jul 27 18:29:31.924753 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:29:31.924636 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 27 18:29:31.924753 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:29:31.924698 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8c36e039-1d59-4c42-8162-0f275315c3a2-kube-rbac-proxy-cm podName:8c36e039-1d59-4c42-8162-0f275315c3a2 nodeName:}" failed. No retries permitted until 2026-07-27 18:29:32.424683536 +0000 UTC m=+47.486372547 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8c36e039-1d59-4c42-8162-0f275315c3a2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dftr7" (UID: "8c36e039-1d59-4c42-8162-0f275315c3a2") : configmap "kube-rbac-proxy" not found Jul 27 18:29:31.924753 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:31.924639 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8c36e039-1d59-4c42-8162-0f275315c3a2-data-volume\") pod \"insights-runtime-extractor-dftr7\" (UID: \"8c36e039-1d59-4c42-8162-0f275315c3a2\") " pod="openshift-insights/insights-runtime-extractor-dftr7" Jul 27 18:29:31.934946 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:31.934925 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-47kr8\" (UniqueName: \"kubernetes.io/projected/8c36e039-1d59-4c42-8162-0f275315c3a2-kube-api-access-47kr8\") pod \"insights-runtime-extractor-dftr7\" (UID: \"8c36e039-1d59-4c42-8162-0f275315c3a2\") " pod="openshift-insights/insights-runtime-extractor-dftr7" Jul 27 18:29:32.105180 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:32.105140 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:29:32.215009 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:32.214904 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:29:32.428651 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:32.428610 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8c36e039-1d59-4c42-8162-0f275315c3a2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dftr7\" (UID: \"8c36e039-1d59-4c42-8162-0f275315c3a2\") " pod="openshift-insights/insights-runtime-extractor-dftr7" Jul 27 18:29:32.428825 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:32.428685 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8c36e039-1d59-4c42-8162-0f275315c3a2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dftr7\" (UID: \"8c36e039-1d59-4c42-8162-0f275315c3a2\") " pod="openshift-insights/insights-runtime-extractor-dftr7" Jul 27 18:29:32.428825 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:29:32.428771 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 27 18:29:32.428825 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:29:32.428804 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 27 18:29:32.428926 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:29:32.428839 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c36e039-1d59-4c42-8162-0f275315c3a2-insights-runtime-extractor-tls podName:8c36e039-1d59-4c42-8162-0f275315c3a2 nodeName:}" failed. No retries permitted until 2026-07-27 18:29:33.428823005 +0000 UTC m=+48.490512014 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8c36e039-1d59-4c42-8162-0f275315c3a2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dftr7" (UID: "8c36e039-1d59-4c42-8162-0f275315c3a2") : secret "insights-runtime-extractor-tls" not found Jul 27 18:29:32.428926 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:29:32.428858 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8c36e039-1d59-4c42-8162-0f275315c3a2-kube-rbac-proxy-cm podName:8c36e039-1d59-4c42-8162-0f275315c3a2 nodeName:}" failed. No retries permitted until 2026-07-27 18:29:33.428845218 +0000 UTC m=+48.490534227 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8c36e039-1d59-4c42-8162-0f275315c3a2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dftr7" (UID: "8c36e039-1d59-4c42-8162-0f275315c3a2") : configmap "kube-rbac-proxy" not found Jul 27 18:29:32.519758 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:32.519655 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:29:32.611926 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:32.611888 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:29:32.843449 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:32.843423 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-mv4hb_715b2084-d79e-4cbf-a057-f2595b4e6b9b/console-operator/1.log" Jul 27 18:29:32.843822 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:32.843741 2579 scope.go:117] "RemoveContainer" containerID="500cf6669e9b32f9bfbaed857bebd570628cc16b01cd6517e2511021b5dc55c0" Jul 27 18:29:32.843931 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:29:32.843907 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-mv4hb_openshift-console-operator(715b2084-d79e-4cbf-a057-f2595b4e6b9b)\"" pod="openshift-console-operator/console-operator-584dcb6b54-mv4hb" podUID="715b2084-d79e-4cbf-a057-f2595b4e6b9b" Jul 27 18:29:33.418440 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:33.418412 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-ttprc_52bcf393-1ae7-4102-b762-569badf3538c/dns-node-resolver/0.log" Jul 27 18:29:33.437892 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:33.437860 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8c36e039-1d59-4c42-8162-0f275315c3a2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dftr7\" (UID: \"8c36e039-1d59-4c42-8162-0f275315c3a2\") " pod="openshift-insights/insights-runtime-extractor-dftr7" Jul 27 18:29:33.438036 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:29:33.437981 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 27 18:29:33.438036 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:33.438001 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8c36e039-1d59-4c42-8162-0f275315c3a2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dftr7\" (UID: \"8c36e039-1d59-4c42-8162-0f275315c3a2\") " pod="openshift-insights/insights-runtime-extractor-dftr7" Jul 27 18:29:33.438104 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:29:33.438043 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8c36e039-1d59-4c42-8162-0f275315c3a2-kube-rbac-proxy-cm podName:8c36e039-1d59-4c42-8162-0f275315c3a2 nodeName:}" failed. No retries permitted until 2026-07-27 18:29:35.438024993 +0000 UTC m=+50.499714018 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8c36e039-1d59-4c42-8162-0f275315c3a2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dftr7" (UID: "8c36e039-1d59-4c42-8162-0f275315c3a2") : configmap "kube-rbac-proxy" not found Jul 27 18:29:33.438104 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:29:33.438098 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 27 18:29:33.438170 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:29:33.438142 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c36e039-1d59-4c42-8162-0f275315c3a2-insights-runtime-extractor-tls podName:8c36e039-1d59-4c42-8162-0f275315c3a2 nodeName:}" failed. No retries permitted until 2026-07-27 18:29:35.438130147 +0000 UTC m=+50.499819158 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8c36e039-1d59-4c42-8162-0f275315c3a2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dftr7" (UID: "8c36e039-1d59-4c42-8162-0f275315c3a2") : secret "insights-runtime-extractor-tls" not found Jul 27 18:29:33.750151 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:33.750068 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:29:34.127279 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:34.127241 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:29:34.220802 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:34.220775 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-fdsdr_8bbbb432-a482-444d-99d3-f2615e9b99bb/node-ca/0.log" Jul 27 18:29:34.234942 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:34.234913 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:29:34.622179 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:34.622143 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:29:34.820864 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:34.820832 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-6hbg6_e4fef94e-e2c7-4725-9da9-ddb80b15149d/serve-healthcheck-canary/0.log" Jul 27 18:29:35.453902 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:35.453870 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8c36e039-1d59-4c42-8162-0f275315c3a2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dftr7\" (UID: \"8c36e039-1d59-4c42-8162-0f275315c3a2\") " pod="openshift-insights/insights-runtime-extractor-dftr7" Jul 27 18:29:35.454332 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:35.453939 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8c36e039-1d59-4c42-8162-0f275315c3a2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dftr7\" (UID: \"8c36e039-1d59-4c42-8162-0f275315c3a2\") " pod="openshift-insights/insights-runtime-extractor-dftr7" Jul 27 18:29:35.454332 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:29:35.454018 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 27 18:29:35.454332 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:29:35.454038 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 27 18:29:35.454332 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:29:35.454087 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c36e039-1d59-4c42-8162-0f275315c3a2-insights-runtime-extractor-tls podName:8c36e039-1d59-4c42-8162-0f275315c3a2 nodeName:}" failed. No retries permitted until 2026-07-27 18:29:39.454072509 +0000 UTC m=+54.515761518 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8c36e039-1d59-4c42-8162-0f275315c3a2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dftr7" (UID: "8c36e039-1d59-4c42-8162-0f275315c3a2") : secret "insights-runtime-extractor-tls" not found Jul 27 18:29:35.454332 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:29:35.454102 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8c36e039-1d59-4c42-8162-0f275315c3a2-kube-rbac-proxy-cm podName:8c36e039-1d59-4c42-8162-0f275315c3a2 nodeName:}" failed. No retries permitted until 2026-07-27 18:29:39.454096005 +0000 UTC m=+54.515785014 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8c36e039-1d59-4c42-8162-0f275315c3a2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dftr7" (UID: "8c36e039-1d59-4c42-8162-0f275315c3a2") : configmap "kube-rbac-proxy" not found Jul 27 18:29:36.161315 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:36.161278 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1e0a413d-db6c-4206-928b-6d95b365c39b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-g9cd9\" (UID: \"1e0a413d-db6c-4206-928b-6d95b365c39b\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-g9cd9" Jul 27 18:29:36.161315 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:36.161320 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b41bffe-08d5-4b52-ae91-dcfebab44a99-metrics-certs\") pod \"router-default-5c874b7d95-95vlk\" (UID: \"2b41bffe-08d5-4b52-ae91-dcfebab44a99\") " pod="openshift-ingress/router-default-5c874b7d95-95vlk" Jul 27 18:29:36.161519 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:29:36.161416 2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 27 18:29:36.161519 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:29:36.161418 2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 27 18:29:36.161519 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:29:36.161469 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e0a413d-db6c-4206-928b-6d95b365c39b-cluster-monitoring-operator-tls podName:1e0a413d-db6c-4206-928b-6d95b365c39b nodeName:}" failed. No retries permitted until 2026-07-27 18:29:52.161454044 +0000 UTC m=+67.223143053 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/1e0a413d-db6c-4206-928b-6d95b365c39b-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-g9cd9" (UID: "1e0a413d-db6c-4206-928b-6d95b365c39b") : secret "cluster-monitoring-operator-tls" not found Jul 27 18:29:36.161519 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:29:36.161482 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b41bffe-08d5-4b52-ae91-dcfebab44a99-metrics-certs podName:2b41bffe-08d5-4b52-ae91-dcfebab44a99 nodeName:}" failed. No retries permitted until 2026-07-27 18:29:52.161476644 +0000 UTC m=+67.223165653 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2b41bffe-08d5-4b52-ae91-dcfebab44a99-metrics-certs") pod "router-default-5c874b7d95-95vlk" (UID: "2b41bffe-08d5-4b52-ae91-dcfebab44a99") : secret "router-metrics-certs-default" not found Jul 27 18:29:36.161650 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:36.161520 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bf48a4f7-eedd-41cc-9d53-93229f167d40-metrics-tls\") pod \"dns-default-pqqm5\" (UID: \"bf48a4f7-eedd-41cc-9d53-93229f167d40\") " pod="openshift-dns/dns-default-pqqm5" Jul 27 18:29:36.161650 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:36.161550 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bfcc4ba6-b269-49a5-847a-2175dc0881fb-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-9pgxt\" (UID: \"bfcc4ba6-b269-49a5-847a-2175dc0881fb\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-9pgxt" Jul 27 18:29:36.161650 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:36.161571 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e660267b-cf0d-4e4e-bb05-1769ff224bb2-registry-tls\") pod \"image-registry-85c5fdc4b8-rwfwz\" (UID: \"e660267b-cf0d-4e4e-bb05-1769ff224bb2\") " pod="openshift-image-registry/image-registry-85c5fdc4b8-rwfwz" Jul 27 18:29:36.161650 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:36.161597 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b41bffe-08d5-4b52-ae91-dcfebab44a99-service-ca-bundle\") pod \"router-default-5c874b7d95-95vlk\" (UID: \"2b41bffe-08d5-4b52-ae91-dcfebab44a99\") " pod="openshift-ingress/router-default-5c874b7d95-95vlk" Jul 27 18:29:36.161650 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:29:36.161645 2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 27 18:29:36.161794 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:29:36.161676 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2b41bffe-08d5-4b52-ae91-dcfebab44a99-service-ca-bundle podName:2b41bffe-08d5-4b52-ae91-dcfebab44a99 nodeName:}" failed. No retries permitted until 2026-07-27 18:29:52.161664308 +0000 UTC m=+67.223353320 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/2b41bffe-08d5-4b52-ae91-dcfebab44a99-service-ca-bundle") pod "router-default-5c874b7d95-95vlk" (UID: "2b41bffe-08d5-4b52-ae91-dcfebab44a99") : configmap references non-existent config key: service-ca.crt Jul 27 18:29:36.161794 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:29:36.161680 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 27 18:29:36.161794 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:29:36.161693 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bfcc4ba6-b269-49a5-847a-2175dc0881fb-networking-console-plugin-cert podName:bfcc4ba6-b269-49a5-847a-2175dc0881fb nodeName:}" failed. No retries permitted until 2026-07-27 18:29:52.161684033 +0000 UTC m=+67.223373042 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/bfcc4ba6-b269-49a5-847a-2175dc0881fb-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-9pgxt" (UID: "bfcc4ba6-b269-49a5-847a-2175dc0881fb") : secret "networking-console-plugin-cert" not found Jul 27 18:29:36.161794 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:36.161710 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4e16cd40-8590-4739-a49c-b67d0f1e37d7-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-7lqlw\" (UID: \"4e16cd40-8590-4739-a49c-b67d0f1e37d7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-7lqlw" Jul 27 18:29:36.161794 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:29:36.161718 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf48a4f7-eedd-41cc-9d53-93229f167d40-metrics-tls podName:bf48a4f7-eedd-41cc-9d53-93229f167d40 nodeName:}" failed. No retries permitted until 2026-07-27 18:29:52.161706889 +0000 UTC m=+67.223395899 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bf48a4f7-eedd-41cc-9d53-93229f167d40-metrics-tls") pod "dns-default-pqqm5" (UID: "bf48a4f7-eedd-41cc-9d53-93229f167d40") : secret "dns-default-metrics-tls" not found Jul 27 18:29:36.161794 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:29:36.161720 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 27 18:29:36.161794 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:29:36.161738 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-85c5fdc4b8-rwfwz: secret "image-registry-tls" not found Jul 27 18:29:36.161794 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:29:36.161764 2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 27 18:29:36.162073 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:29:36.161801 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e660267b-cf0d-4e4e-bb05-1769ff224bb2-registry-tls podName:e660267b-cf0d-4e4e-bb05-1769ff224bb2 nodeName:}" failed. No retries permitted until 2026-07-27 18:29:52.161773755 +0000 UTC m=+67.223462766 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e660267b-cf0d-4e4e-bb05-1769ff224bb2-registry-tls") pod "image-registry-85c5fdc4b8-rwfwz" (UID: "e660267b-cf0d-4e4e-bb05-1769ff224bb2") : secret "image-registry-tls" not found Jul 27 18:29:36.162073 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:29:36.161816 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e16cd40-8590-4739-a49c-b67d0f1e37d7-samples-operator-tls podName:4e16cd40-8590-4739-a49c-b67d0f1e37d7 nodeName:}" failed. No retries permitted until 2026-07-27 18:29:52.16180927 +0000 UTC m=+67.223498278 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/4e16cd40-8590-4739-a49c-b67d0f1e37d7-samples-operator-tls") pod "cluster-samples-operator-6748489656-7lqlw" (UID: "4e16cd40-8590-4739-a49c-b67d0f1e37d7") : secret "samples-operator-tls" not found Jul 27 18:29:36.315589 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:36.315557 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:29:38.333289 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:38.333251 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:29:38.945838 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:38.945803 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:29:39.046794 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:39.046756 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:29:39.199582 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:39.199491 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:29:39.369316 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:39.369282 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:29:39.470620 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:39.470520 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:29:39.490915 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:39.490883 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8c36e039-1d59-4c42-8162-0f275315c3a2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dftr7\" (UID: \"8c36e039-1d59-4c42-8162-0f275315c3a2\") " pod="openshift-insights/insights-runtime-extractor-dftr7" Jul 27 18:29:39.491048 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:29:39.490986 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 27 18:29:39.491048 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:29:39.491042 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c36e039-1d59-4c42-8162-0f275315c3a2-insights-runtime-extractor-tls podName:8c36e039-1d59-4c42-8162-0f275315c3a2 nodeName:}" failed. No retries permitted until 2026-07-27 18:29:47.491026745 +0000 UTC m=+62.552715754 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8c36e039-1d59-4c42-8162-0f275315c3a2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dftr7" (UID: "8c36e039-1d59-4c42-8162-0f275315c3a2") : secret "insights-runtime-extractor-tls" not found Jul 27 18:29:39.491153 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:39.490986 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8c36e039-1d59-4c42-8162-0f275315c3a2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dftr7\" (UID: \"8c36e039-1d59-4c42-8162-0f275315c3a2\") " pod="openshift-insights/insights-runtime-extractor-dftr7" Jul 27 18:29:39.491519 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:39.491501 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8c36e039-1d59-4c42-8162-0f275315c3a2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dftr7\" (UID: \"8c36e039-1d59-4c42-8162-0f275315c3a2\") " pod="openshift-insights/insights-runtime-extractor-dftr7" Jul 27 18:29:39.569956 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:39.569918 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:29:39.670241 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:39.670194 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:29:40.678615 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:40.678582 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-mv4hb" Jul 27 18:29:40.679065 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:40.678676 2579 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-584dcb6b54-mv4hb" Jul 27 18:29:40.679065 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:40.678949 2579 scope.go:117] "RemoveContainer" containerID="500cf6669e9b32f9bfbaed857bebd570628cc16b01cd6517e2511021b5dc55c0" Jul 27 18:29:40.679144 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:29:40.679100 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-mv4hb_openshift-console-operator(715b2084-d79e-4cbf-a057-f2595b4e6b9b)\"" pod="openshift-console-operator/console-operator-584dcb6b54-mv4hb" podUID="715b2084-d79e-4cbf-a057-f2595b4e6b9b" Jul 27 18:29:40.867203 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:40.867175 2579 scope.go:117] "RemoveContainer" containerID="500cf6669e9b32f9bfbaed857bebd570628cc16b01cd6517e2511021b5dc55c0" Jul 27 18:29:40.867400 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:29:40.867380 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-mv4hb_openshift-console-operator(715b2084-d79e-4cbf-a057-f2595b4e6b9b)\"" pod="openshift-console-operator/console-operator-584dcb6b54-mv4hb" podUID="715b2084-d79e-4cbf-a057-f2595b4e6b9b" Jul 27 18:29:41.340015 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:41.339976 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:29:41.440538 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:41.440504 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:29:41.627823 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:41.627737 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:29:41.828355 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:41.828311 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:29:42.120395 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:42.120356 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:29:42.221731 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:42.221679 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:29:42.282599 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:42.282562 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:29:42.539319 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:42.539288 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:29:42.639439 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:42.639400 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:29:43.230520 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:43.230488 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:29:43.284641 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:43.284604 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:29:43.626445 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:43.626404 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:29:44.024275 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:44.024239 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:29:44.128633 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:44.128593 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:29:44.234498 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:44.234468 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:29:44.428493 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:44.428388 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:29:44.828745 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:44.828712 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:29:44.930030 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:44.929984 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:29:45.029828 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:45.029787 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:29:45.447396 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:45.447364 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:29:46.154330 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:46.154300 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:29:46.254618 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:46.254586 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:29:46.639641 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:46.639610 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:29:46.745527 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:46.745503 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vnbl8" Jul 27 18:29:46.837693 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:46.837661 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:29:47.047674 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:47.047640 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:29:47.228436 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:47.228398 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:29:47.563399 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:47.563373 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8c36e039-1d59-4c42-8162-0f275315c3a2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dftr7\" (UID: \"8c36e039-1d59-4c42-8162-0f275315c3a2\") " pod="openshift-insights/insights-runtime-extractor-dftr7" Jul 27 18:29:47.565757 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:47.565727 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8c36e039-1d59-4c42-8162-0f275315c3a2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dftr7\" (UID: \"8c36e039-1d59-4c42-8162-0f275315c3a2\") " pod="openshift-insights/insights-runtime-extractor-dftr7" Jul 27 18:29:47.573361 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:47.573339 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-584h7\"" Jul 27 18:29:47.581579 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:47.581565 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-dftr7" Jul 27 18:29:47.706859 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:47.706826 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-dftr7"] Jul 27 18:29:47.709864 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:29:47.709830 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c36e039_1d59_4c42_8162_0f275315c3a2.slice/crio-0d119ac552b9bd047f315547332bdd02bdef571d14dddd9e569a775984adff4d WatchSource:0}: Error finding container 0d119ac552b9bd047f315547332bdd02bdef571d14dddd9e569a775984adff4d: Status 404 returned error can't find the container with id 0d119ac552b9bd047f315547332bdd02bdef571d14dddd9e569a775984adff4d Jul 27 18:29:47.868953 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:47.868858 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:29:47.892295 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:47.892259 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dftr7" event={"ID":"8c36e039-1d59-4c42-8162-0f275315c3a2","Type":"ContainerStarted","Data":"0d119ac552b9bd047f315547332bdd02bdef571d14dddd9e569a775984adff4d"} Jul 27 18:29:48.029002 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:48.028973 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:29:50.098434 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:50.098325 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:29:50.169675 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:50.169642 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:29:50.190888 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:50.190855 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bd3d598d-7458-4407-ba85-49402348da87-original-pull-secret\") pod \"global-pull-secret-syncer-pr8j9\" (UID: \"bd3d598d-7458-4407-ba85-49402348da87\") " pod="kube-system/global-pull-secret-syncer-pr8j9" Jul 27 18:29:50.193898 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:50.193874 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 27 18:29:50.207636 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:50.207606 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bd3d598d-7458-4407-ba85-49402348da87-original-pull-secret\") pod \"global-pull-secret-syncer-pr8j9\" (UID: \"bd3d598d-7458-4407-ba85-49402348da87\") " pod="kube-system/global-pull-secret-syncer-pr8j9" Jul 27 18:29:50.207951 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:50.207928 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pr8j9" Jul 27 18:29:50.291964 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:50.291925 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/270a2080-a3dd-4227-9abc-c093f70df909-metrics-certs\") pod \"network-metrics-daemon-6htnl\" (UID: \"270a2080-a3dd-4227-9abc-c093f70df909\") " pod="openshift-multus/network-metrics-daemon-6htnl" Jul 27 18:29:50.292148 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:50.292017 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rr8z7\" (UniqueName: \"kubernetes.io/projected/8ce054d9-d9c1-44ef-9e32-5a4c502e0ced-kube-api-access-rr8z7\") pod \"network-check-target-tgxn8\" (UID: \"8ce054d9-d9c1-44ef-9e32-5a4c502e0ced\") " pod="openshift-network-diagnostics/network-check-target-tgxn8" Jul 27 18:29:50.295246 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:50.295197 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 27 18:29:50.295380 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:50.295251 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rr8z7\" (UniqueName: \"kubernetes.io/projected/8ce054d9-d9c1-44ef-9e32-5a4c502e0ced-kube-api-access-rr8z7\") pod \"network-check-target-tgxn8\" (UID: \"8ce054d9-d9c1-44ef-9e32-5a4c502e0ced\") " pod="openshift-network-diagnostics/network-check-target-tgxn8" Jul 27 18:29:50.305220 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:50.305194 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/270a2080-a3dd-4227-9abc-c093f70df909-metrics-certs\") pod \"network-metrics-daemon-6htnl\" (UID: \"270a2080-a3dd-4227-9abc-c093f70df909\") " pod="openshift-multus/network-metrics-daemon-6htnl" Jul 27 18:29:50.504496 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:50.504421 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-fn8tb\"" Jul 27 18:29:50.512110 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:50.512085 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgxn8" Jul 27 18:29:50.554499 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:50.554465 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-bwbqc\"" Jul 27 18:29:50.562910 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:50.562889 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6htnl" Jul 27 18:29:52.207139 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:52.207100 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bf48a4f7-eedd-41cc-9d53-93229f167d40-metrics-tls\") pod \"dns-default-pqqm5\" (UID: \"bf48a4f7-eedd-41cc-9d53-93229f167d40\") " pod="openshift-dns/dns-default-pqqm5" Jul 27 18:29:52.207700 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:52.207161 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bfcc4ba6-b269-49a5-847a-2175dc0881fb-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-9pgxt\" (UID: \"bfcc4ba6-b269-49a5-847a-2175dc0881fb\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-9pgxt" Jul 27 18:29:52.207700 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:52.207196 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e660267b-cf0d-4e4e-bb05-1769ff224bb2-registry-tls\") pod \"image-registry-85c5fdc4b8-rwfwz\" (UID: \"e660267b-cf0d-4e4e-bb05-1769ff224bb2\") " pod="openshift-image-registry/image-registry-85c5fdc4b8-rwfwz" Jul 27 18:29:52.207700 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:52.207253 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b41bffe-08d5-4b52-ae91-dcfebab44a99-service-ca-bundle\") pod \"router-default-5c874b7d95-95vlk\" (UID: \"2b41bffe-08d5-4b52-ae91-dcfebab44a99\") " pod="openshift-ingress/router-default-5c874b7d95-95vlk" Jul 27 18:29:52.207700 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:52.207282 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4e16cd40-8590-4739-a49c-b67d0f1e37d7-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-7lqlw\" (UID: \"4e16cd40-8590-4739-a49c-b67d0f1e37d7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-7lqlw" Jul 27 18:29:52.207700 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:52.207331 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1e0a413d-db6c-4206-928b-6d95b365c39b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-g9cd9\" (UID: \"1e0a413d-db6c-4206-928b-6d95b365c39b\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-g9cd9" Jul 27 18:29:52.207700 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:52.207365 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b41bffe-08d5-4b52-ae91-dcfebab44a99-metrics-certs\") pod \"router-default-5c874b7d95-95vlk\" (UID: \"2b41bffe-08d5-4b52-ae91-dcfebab44a99\") " pod="openshift-ingress/router-default-5c874b7d95-95vlk" Jul 27 18:29:52.208763 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:52.208553 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b41bffe-08d5-4b52-ae91-dcfebab44a99-service-ca-bundle\") pod \"router-default-5c874b7d95-95vlk\" (UID: \"2b41bffe-08d5-4b52-ae91-dcfebab44a99\") " pod="openshift-ingress/router-default-5c874b7d95-95vlk" Jul 27 18:29:52.210249 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:52.210210 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bf48a4f7-eedd-41cc-9d53-93229f167d40-metrics-tls\") pod \"dns-default-pqqm5\" (UID: \"bf48a4f7-eedd-41cc-9d53-93229f167d40\") " pod="openshift-dns/dns-default-pqqm5" Jul 27 18:29:52.210373 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:52.210282 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bfcc4ba6-b269-49a5-847a-2175dc0881fb-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-9pgxt\" (UID: \"bfcc4ba6-b269-49a5-847a-2175dc0881fb\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-9pgxt" Jul 27 18:29:52.210373 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:52.210356 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e660267b-cf0d-4e4e-bb05-1769ff224bb2-registry-tls\") pod \"image-registry-85c5fdc4b8-rwfwz\" (UID: \"e660267b-cf0d-4e4e-bb05-1769ff224bb2\") " pod="openshift-image-registry/image-registry-85c5fdc4b8-rwfwz" Jul 27 18:29:52.210537 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:52.210514 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4e16cd40-8590-4739-a49c-b67d0f1e37d7-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-7lqlw\" (UID: \"4e16cd40-8590-4739-a49c-b67d0f1e37d7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-7lqlw" Jul 27 18:29:52.210714 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:52.210692 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b41bffe-08d5-4b52-ae91-dcfebab44a99-metrics-certs\") pod \"router-default-5c874b7d95-95vlk\" (UID: \"2b41bffe-08d5-4b52-ae91-dcfebab44a99\") " pod="openshift-ingress/router-default-5c874b7d95-95vlk" Jul 27 18:29:52.210905 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:52.210886 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1e0a413d-db6c-4206-928b-6d95b365c39b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-g9cd9\" (UID: \"1e0a413d-db6c-4206-928b-6d95b365c39b\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-g9cd9" Jul 27 18:29:52.410259 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:52.410216 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-fwznz\"" Jul 27 18:29:52.417747 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:52.417709 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-85c5fdc4b8-rwfwz" Jul 27 18:29:52.431587 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:52.431552 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-d4rxx\"" Jul 27 18:29:52.438630 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:52.438600 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-g9cd9" Jul 27 18:29:52.460146 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:52.460067 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-wvq8l\"" Jul 27 18:29:52.467680 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:52.467658 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5c874b7d95-95vlk" Jul 27 18:29:52.467959 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:52.467937 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-v85bm\"" Jul 27 18:29:52.475430 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:52.475412 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-9pgxt" Jul 27 18:29:52.488322 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:52.488301 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-swtr8\"" Jul 27 18:29:52.495362 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:52.495342 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-7lqlw" Jul 27 18:29:52.511513 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:52.511489 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-9pm47\"" Jul 27 18:29:52.519625 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:52.519606 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pqqm5" Jul 27 18:29:53.322345 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:53.321779 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-6htnl"] Jul 27 18:29:53.325684 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:29:53.325654 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod270a2080_a3dd_4227_9abc_c093f70df909.slice/crio-52697c29c56970fb0f6691746de18c1a63829b1b33e41576a4a23b5b12c51a5c WatchSource:0}: Error finding container 52697c29c56970fb0f6691746de18c1a63829b1b33e41576a4a23b5b12c51a5c: Status 404 returned error can't find the container with id 52697c29c56970fb0f6691746de18c1a63829b1b33e41576a4a23b5b12c51a5c Jul 27 18:29:53.362117 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:53.362016 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-7lqlw"] Jul 27 18:29:53.372502 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:53.372295 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-85c5fdc4b8-rwfwz"] Jul 27 18:29:53.375431 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:29:53.375405 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode660267b_cf0d_4e4e_bb05_1769ff224bb2.slice/crio-86177636677ce4e632674afaccfa4ebf6df6fe0c16acbfa1efea3a2986601278 WatchSource:0}: Error finding container 86177636677ce4e632674afaccfa4ebf6df6fe0c16acbfa1efea3a2986601278: Status 404 returned error can't find the container with id 86177636677ce4e632674afaccfa4ebf6df6fe0c16acbfa1efea3a2986601278 Jul 27 18:29:53.472729 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:53.472709 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-pqqm5"] Jul 27 18:29:53.475041 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:29:53.475005 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf48a4f7_eedd_41cc_9d53_93229f167d40.slice/crio-77e6e4d3ed33e90336e85d8299c82e7b66428ba4c13d50defde9f68ebedc4cb1 WatchSource:0}: Error finding container 77e6e4d3ed33e90336e85d8299c82e7b66428ba4c13d50defde9f68ebedc4cb1: Status 404 returned error can't find the container with id 77e6e4d3ed33e90336e85d8299c82e7b66428ba4c13d50defde9f68ebedc4cb1 Jul 27 18:29:53.607108 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:53.607031 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-pr8j9"] Jul 27 18:29:53.608378 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:53.608338 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-tgxn8"] Jul 27 18:29:53.612629 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:53.612603 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-g9cd9"] Jul 27 18:29:53.614612 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:29:53.614569 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd3d598d_7458_4407_ba85_49402348da87.slice/crio-0d028f4866e5e861a916c6d83f801ff7338c3f8c0ca61461e71224e06a4fd0a0 WatchSource:0}: Error finding container 0d028f4866e5e861a916c6d83f801ff7338c3f8c0ca61461e71224e06a4fd0a0: Status 404 returned error can't find the container with id 0d028f4866e5e861a916c6d83f801ff7338c3f8c0ca61461e71224e06a4fd0a0 Jul 27 18:29:53.617646 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:53.616630 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-9pgxt"] Jul 27 18:29:53.617646 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:29:53.616721 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ce054d9_d9c1_44ef_9e32_5a4c502e0ced.slice/crio-293444e1bbb9b81b81ea20965be74ce8a2b5802d254b5af7ec48826ab1b01a1b WatchSource:0}: Error finding container 293444e1bbb9b81b81ea20965be74ce8a2b5802d254b5af7ec48826ab1b01a1b: Status 404 returned error can't find the container with id 293444e1bbb9b81b81ea20965be74ce8a2b5802d254b5af7ec48826ab1b01a1b Jul 27 18:29:53.617646 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:53.617463 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5c874b7d95-95vlk"] Jul 27 18:29:53.620286 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:29:53.620193 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e0a413d_db6c_4206_928b_6d95b365c39b.slice/crio-46f370b8bf4af014c09b90f2f49f2acbcc79935f9de944490d06cb63a3998274 WatchSource:0}: Error finding container 46f370b8bf4af014c09b90f2f49f2acbcc79935f9de944490d06cb63a3998274: Status 404 returned error can't find the container with id 46f370b8bf4af014c09b90f2f49f2acbcc79935f9de944490d06cb63a3998274 Jul 27 18:29:53.620584 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:29:53.620535 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfcc4ba6_b269_49a5_847a_2175dc0881fb.slice/crio-4ab3e33b32691008e4e3e6de5e2831a59f60298d50eac9c0c4df1282ef73d9a3 WatchSource:0}: Error finding container 4ab3e33b32691008e4e3e6de5e2831a59f60298d50eac9c0c4df1282ef73d9a3: Status 404 returned error can't find the container with id 4ab3e33b32691008e4e3e6de5e2831a59f60298d50eac9c0c4df1282ef73d9a3 Jul 27 18:29:53.622851 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:29:53.622834 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b41bffe_08d5_4b52_ae91_dcfebab44a99.slice/crio-deb7bc39d752214b5392bca861c39f54c452b769f7b36f0338dedca4eccb8d62 WatchSource:0}: Error finding container deb7bc39d752214b5392bca861c39f54c452b769f7b36f0338dedca4eccb8d62: Status 404 returned error can't find the container with id deb7bc39d752214b5392bca861c39f54c452b769f7b36f0338dedca4eccb8d62 Jul 27 18:29:53.916479 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:53.916336 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-7lqlw" event={"ID":"4e16cd40-8590-4739-a49c-b67d0f1e37d7","Type":"ContainerStarted","Data":"bb65a49ec4fe596d15773c35eef77ceb06baf4f457acfd2a571e4fcca329aa1e"} Jul 27 18:29:53.919593 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:53.919559 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-g9cd9" event={"ID":"1e0a413d-db6c-4206-928b-6d95b365c39b","Type":"ContainerStarted","Data":"46f370b8bf4af014c09b90f2f49f2acbcc79935f9de944490d06cb63a3998274"} Jul 27 18:29:53.922921 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:53.922322 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5c874b7d95-95vlk" event={"ID":"2b41bffe-08d5-4b52-ae91-dcfebab44a99","Type":"ContainerStarted","Data":"adb95c09028a418d37900654edecd2b2757d7ce57f6efaa30fa3cdcf4a0719e2"} Jul 27 18:29:53.922921 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:53.922369 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5c874b7d95-95vlk" event={"ID":"2b41bffe-08d5-4b52-ae91-dcfebab44a99","Type":"ContainerStarted","Data":"deb7bc39d752214b5392bca861c39f54c452b769f7b36f0338dedca4eccb8d62"} Jul 27 18:29:53.925572 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:53.925544 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-tgxn8" event={"ID":"8ce054d9-d9c1-44ef-9e32-5a4c502e0ced","Type":"ContainerStarted","Data":"f514df39ce445fc2f86673422ec05da34bed42ef0e56dfb261178669214e7a98"} Jul 27 18:29:53.925660 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:53.925597 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-tgxn8" event={"ID":"8ce054d9-d9c1-44ef-9e32-5a4c502e0ced","Type":"ContainerStarted","Data":"293444e1bbb9b81b81ea20965be74ce8a2b5802d254b5af7ec48826ab1b01a1b"} Jul 27 18:29:53.926012 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:53.925991 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-tgxn8" Jul 27 18:29:53.928633 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:53.928561 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6htnl" event={"ID":"270a2080-a3dd-4227-9abc-c093f70df909","Type":"ContainerStarted","Data":"52697c29c56970fb0f6691746de18c1a63829b1b33e41576a4a23b5b12c51a5c"} Jul 27 18:29:53.930680 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:53.930612 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dftr7" event={"ID":"8c36e039-1d59-4c42-8162-0f275315c3a2","Type":"ContainerStarted","Data":"0bbd8c8d4abfa333ac21261d81dfbde9eed742fc6c1fff949e22c267d5ea76d6"} Jul 27 18:29:53.932093 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:53.932032 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-9pgxt" event={"ID":"bfcc4ba6-b269-49a5-847a-2175dc0881fb","Type":"ContainerStarted","Data":"4ab3e33b32691008e4e3e6de5e2831a59f60298d50eac9c0c4df1282ef73d9a3"} Jul 27 18:29:53.933434 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:53.933378 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pqqm5" event={"ID":"bf48a4f7-eedd-41cc-9d53-93229f167d40","Type":"ContainerStarted","Data":"77e6e4d3ed33e90336e85d8299c82e7b66428ba4c13d50defde9f68ebedc4cb1"} Jul 27 18:29:53.935402 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:53.935379 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-85c5fdc4b8-rwfwz" event={"ID":"e660267b-cf0d-4e4e-bb05-1769ff224bb2","Type":"ContainerStarted","Data":"43efca213fe65e17b1478f19fb6910613e71bfbf712cc16a08e219b56f874eff"} Jul 27 18:29:53.935485 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:53.935413 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-85c5fdc4b8-rwfwz" event={"ID":"e660267b-cf0d-4e4e-bb05-1769ff224bb2","Type":"ContainerStarted","Data":"86177636677ce4e632674afaccfa4ebf6df6fe0c16acbfa1efea3a2986601278"} Jul 27 18:29:53.935685 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:53.935637 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-85c5fdc4b8-rwfwz" Jul 27 18:29:53.938539 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:53.938517 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-pr8j9" event={"ID":"bd3d598d-7458-4407-ba85-49402348da87","Type":"ContainerStarted","Data":"0d028f4866e5e861a916c6d83f801ff7338c3f8c0ca61461e71224e06a4fd0a0"} Jul 27 18:29:53.945378 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:53.945107 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-5c874b7d95-95vlk" podStartSLOduration=44.945089985 podStartE2EDuration="44.945089985s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:29:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:29:53.943685126 +0000 UTC m=+69.005374168" watchObservedRunningTime="2026-07-27 18:29:53.945089985 +0000 UTC m=+69.006779017" Jul 27 18:29:53.985091 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:53.984951 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-85c5fdc4b8-rwfwz" podStartSLOduration=67.984928819 podStartE2EDuration="1m7.984928819s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:28:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:29:53.984457749 +0000 UTC m=+69.046146781" watchObservedRunningTime="2026-07-27 18:29:53.984928819 +0000 UTC m=+69.046617832" Jul 27 18:29:53.985736 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:53.985533 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-tgxn8" podStartSLOduration=68.985518639 podStartE2EDuration="1m8.985518639s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:28:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:29:53.96118069 +0000 UTC m=+69.022869728" watchObservedRunningTime="2026-07-27 18:29:53.985518639 +0000 UTC m=+69.047207673" Jul 27 18:29:54.469675 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:54.468595 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-5c874b7d95-95vlk" Jul 27 18:29:54.472263 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:54.472049 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-5c874b7d95-95vlk" Jul 27 18:29:54.952293 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:54.951307 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6htnl" event={"ID":"270a2080-a3dd-4227-9abc-c093f70df909","Type":"ContainerStarted","Data":"e2c22383de37728d5c55cbc3331b990b9183e89acc5f8b8a2b1ce902ca498820"} Jul 27 18:29:54.952293 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:54.951435 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6htnl" event={"ID":"270a2080-a3dd-4227-9abc-c093f70df909","Type":"ContainerStarted","Data":"4c2f87876e0440ea54f8e7f2efc1dbf6629de290c6ab752e68649379dd128627"} Jul 27 18:29:54.953479 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:54.952805 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-5c874b7d95-95vlk" Jul 27 18:29:54.953716 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:54.953693 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-5c874b7d95-95vlk" Jul 27 18:29:54.973122 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:54.972521 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-6htnl" podStartSLOduration=68.740851264 podStartE2EDuration="1m9.972502818s" totalImagesPullingTime="1.231651554s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:28:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:29:54.97119858 +0000 UTC m=+70.032887612" watchObservedRunningTime="2026-07-27 18:29:54.972502818 +0000 UTC m=+70.034191852" Jul 27 18:29:55.489781 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:55.489372 2579 scope.go:117] "RemoveContainer" containerID="500cf6669e9b32f9bfbaed857bebd570628cc16b01cd6517e2511021b5dc55c0" Jul 27 18:29:56.049215 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:56.049156 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:29:56.139592 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:56.139557 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:29:56.242725 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:56.242685 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:29:56.414902 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:56.414812 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:29:56.994313 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:56.994273 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:29:57.145626 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:57.145564 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:29:57.204438 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:57.204358 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:29:57.383087 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:57.383052 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:29:57.492441 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:57.492407 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:29:59.776632 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:59.776591 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:29:59.881790 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:29:59.881687 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:30:01.993780 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:01.993073 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dftr7" event={"ID":"8c36e039-1d59-4c42-8162-0f275315c3a2","Type":"ContainerStarted","Data":"ea9448057f24aa4819c9fa1df3a06da86f15920581075a9d58e2618695333a72"} Jul 27 18:30:01.994889 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:01.994828 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-9pgxt" event={"ID":"bfcc4ba6-b269-49a5-847a-2175dc0881fb","Type":"ContainerStarted","Data":"eca4e7ed49055b7b982bd227d114a14d91b952e6825db294bf26cd3982e263cc"} Jul 27 18:30:01.996995 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:01.996881 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pqqm5" event={"ID":"bf48a4f7-eedd-41cc-9d53-93229f167d40","Type":"ContainerStarted","Data":"cc2623781c7baa3261c684eeb0d0036d64bc2c102983a8daa176a52f6a10b230"} Jul 27 18:30:01.998593 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:01.998571 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-pr8j9" event={"ID":"bd3d598d-7458-4407-ba85-49402348da87","Type":"ContainerStarted","Data":"11124b87d57ef825af82b33711f47dc89d8742f17e8cc192193f1182bee54f0e"} Jul 27 18:30:02.001078 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:02.001057 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-mv4hb_715b2084-d79e-4cbf-a057-f2595b4e6b9b/console-operator/1.log" Jul 27 18:30:02.001191 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:02.001175 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-mv4hb" event={"ID":"715b2084-d79e-4cbf-a057-f2595b4e6b9b","Type":"ContainerStarted","Data":"c87927d8428dc7adca8ca2ac52f1ef1e7e26aa9db5d2bf4ea669a29fcb097664"} Jul 27 18:30:02.001923 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:02.001856 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-mv4hb" Jul 27 18:30:02.004170 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:02.004134 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-7lqlw" event={"ID":"4e16cd40-8590-4739-a49c-b67d0f1e37d7","Type":"ContainerStarted","Data":"22e61bab543e617bcdc94744ec2d7042b280aa6ea0e9ca25e422fee884ded054"} Jul 27 18:30:02.004312 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:02.004190 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-7lqlw" event={"ID":"4e16cd40-8590-4739-a49c-b67d0f1e37d7","Type":"ContainerStarted","Data":"7fee12566dcc1d030c21ebbe1a60efd4af76f88a33326035120fa3000eafa52f"} Jul 27 18:30:02.006048 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:02.006027 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-g9cd9" event={"ID":"1e0a413d-db6c-4206-928b-6d95b365c39b","Type":"ContainerStarted","Data":"d376ce4586502db22631c4f6d8e491a7419439efcb81a323b705a145d08d1f58"} Jul 27 18:30:02.049260 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:02.048076 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-584dcb6b54-mv4hb" podStartSLOduration=45.219455958 podStartE2EDuration="53.048055384s" totalImagesPullingTime="7.828599426s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:29:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:30:02.046755716 +0000 UTC m=+77.108444749" watchObservedRunningTime="2026-07-27 18:30:02.048055384 +0000 UTC m=+77.109744416" Jul 27 18:30:02.049260 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:02.048557 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-74c9f884b7-9pgxt" podStartSLOduration=45.337764873 podStartE2EDuration="53.048544226s" totalImagesPullingTime="7.710779353s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:29:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:30:02.019866823 +0000 UTC m=+77.081555854" watchObservedRunningTime="2026-07-27 18:30:02.048544226 +0000 UTC m=+77.110233258" Jul 27 18:30:02.110139 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:02.109914 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-g9cd9" podStartSLOduration=45.322188271 podStartE2EDuration="53.109895183s" totalImagesPullingTime="7.787706912s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:29:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:30:02.068906039 +0000 UTC m=+77.130595071" watchObservedRunningTime="2026-07-27 18:30:02.109895183 +0000 UTC m=+77.171584215" Jul 27 18:30:02.110859 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:02.110825 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-7lqlw" podStartSLOduration=45.139475948 podStartE2EDuration="53.110814852s" totalImagesPullingTime="7.971338904s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:29:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:30:02.108557568 +0000 UTC m=+77.170246600" watchObservedRunningTime="2026-07-27 18:30:02.110814852 +0000 UTC m=+77.172503883" Jul 27 18:30:02.141868 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:02.141814 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-pr8j9" podStartSLOduration=69.322391587 podStartE2EDuration="1m17.141796985s" totalImagesPullingTime="7.819405398s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 18:28:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:30:02.140071435 +0000 UTC m=+77.201760465" watchObservedRunningTime="2026-07-27 18:30:02.141796985 +0000 UTC m=+77.203485997" Jul 27 18:30:02.158152 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:02.158118 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:30:02.187341 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:02.187318 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-584dcb6b54-mv4hb" Jul 27 18:30:02.238392 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:02.238292 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:30:02.502937 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:02.502847 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:30:02.579988 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:02.579950 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:30:02.708304 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:02.708269 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:30:02.846071 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:02.846037 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:30:02.906633 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:02.906602 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:30:03.012637 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:03.012602 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pqqm5" event={"ID":"bf48a4f7-eedd-41cc-9d53-93229f167d40","Type":"ContainerStarted","Data":"896c9aee07459e9188d503482e5481b417d44001fefc134a8bc226e0f46672fe"} Jul 27 18:30:03.034111 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:03.034059 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-pqqm5" podStartSLOduration=35.086589143 podStartE2EDuration="43.034039332s" totalImagesPullingTime="7.947450189s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:29:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:30:03.03273544 +0000 UTC m=+78.094424471" watchObservedRunningTime="2026-07-27 18:30:03.034039332 +0000 UTC m=+78.095728366" Jul 27 18:30:03.531322 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:03.531286 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:30:03.595011 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:03.594962 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:30:04.018531 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:04.018457 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-pqqm5" Jul 27 18:30:04.057342 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:04.057297 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:30:04.119048 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:04.119011 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:30:04.257453 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:04.257422 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:30:04.322198 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:04.322159 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:30:04.394364 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:04.394328 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:30:04.905058 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:04.905026 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:30:04.968525 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:04.968482 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:30:05.880502 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:05.880352 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:30:05.978048 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:05.978018 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:30:06.277009 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:06.276974 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:30:06.336097 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:06.336063 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:30:06.449351 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:06.449313 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:30:06.521876 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:06.521851 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:30:06.949079 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:06.949041 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:30:07.036909 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:07.036873 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dftr7" event={"ID":"8c36e039-1d59-4c42-8162-0f275315c3a2","Type":"ContainerStarted","Data":"9ea2c96ee596f1d4c57a5387a5e854e5d7275561267a9ca82553e0b34c289c5a"} Jul 27 18:30:07.058510 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:07.058458 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-dftr7" podStartSLOduration=18.07768101 podStartE2EDuration="36.058442002s" totalImagesPullingTime="17.980760992s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 18:29:31 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:30:07.057331021 +0000 UTC m=+82.119020052" watchObservedRunningTime="2026-07-27 18:30:07.058442002 +0000 UTC m=+82.120131034" Jul 27 18:30:08.800168 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:08.800135 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:30:11.025014 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:11.024986 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-pqqm5" Jul 27 18:30:11.086112 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:11.086079 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:30:11.187922 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:11.187886 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:30:12.192922 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:12.192882 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:30:12.264195 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:12.264158 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:30:12.446174 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:12.446083 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:30:13.501958 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:13.501920 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:30:13.596100 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:13.596062 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:30:13.697024 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:13.696982 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:30:13.983393 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:13.983350 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:30:14.194147 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:14.194089 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:30:14.956356 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:14.956328 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-85c5fdc4b8-rwfwz" Jul 27 18:30:15.113924 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:15.113871 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:30:15.177252 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:15.177189 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:30:18.736576 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:18.736543 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-85c5fdc4b8-rwfwz"] Jul 27 18:30:19.095859 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:19.095804 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:30:19.152619 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:19.152551 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:30:20.162254 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:20.162197 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:30:24.140724 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:24.140683 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:30:24.241502 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:24.241444 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:30:25.959360 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:25.959330 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-tgxn8" Jul 27 18:30:27.138637 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:27.138604 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:30:27.206881 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:27.206832 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:30:27.886742 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:27.886706 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:30:30.231409 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:30.231371 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:30:30.427288 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:30.427225 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:30:30.481913 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:30.481835 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-l5cnt"] Jul 27 18:30:30.512737 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:30.512711 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:30:30.580389 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:30.580358 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:30:30.683706 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:30.683671 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:30:30.782211 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:30.782178 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:30:30.789969 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:30.789943 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-l5cnt" Jul 27 18:30:30.793060 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:30.793034 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 27 18:30:30.793161 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:30.793034 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-6p8s5\"" Jul 27 18:30:30.793161 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:30.793037 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 27 18:30:30.794258 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:30.794242 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 27 18:30:30.794338 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:30.794242 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 27 18:30:30.882527 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:30.882485 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:30:30.922541 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:30.922508 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5b877858-8e86-4213-9a5b-10a18e69d669-node-exporter-tls\") pod \"node-exporter-l5cnt\" (UID: \"5b877858-8e86-4213-9a5b-10a18e69d669\") " pod="openshift-monitoring/node-exporter-l5cnt" Jul 27 18:30:30.922695 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:30.922550 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5b877858-8e86-4213-9a5b-10a18e69d669-node-exporter-textfile\") pod \"node-exporter-l5cnt\" (UID: \"5b877858-8e86-4213-9a5b-10a18e69d669\") " pod="openshift-monitoring/node-exporter-l5cnt" Jul 27 18:30:30.922695 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:30.922574 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5b877858-8e86-4213-9a5b-10a18e69d669-node-exporter-accelerators-collector-config\") pod \"node-exporter-l5cnt\" (UID: \"5b877858-8e86-4213-9a5b-10a18e69d669\") " pod="openshift-monitoring/node-exporter-l5cnt" Jul 27 18:30:30.922695 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:30.922605 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5b877858-8e86-4213-9a5b-10a18e69d669-node-exporter-wtmp\") pod \"node-exporter-l5cnt\" (UID: \"5b877858-8e86-4213-9a5b-10a18e69d669\") " pod="openshift-monitoring/node-exporter-l5cnt" Jul 27 18:30:30.922695 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:30.922664 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5b877858-8e86-4213-9a5b-10a18e69d669-metrics-client-ca\") pod \"node-exporter-l5cnt\" (UID: \"5b877858-8e86-4213-9a5b-10a18e69d669\") " pod="openshift-monitoring/node-exporter-l5cnt" Jul 27 18:30:30.922847 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:30.922717 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5b877858-8e86-4213-9a5b-10a18e69d669-sys\") pod \"node-exporter-l5cnt\" (UID: \"5b877858-8e86-4213-9a5b-10a18e69d669\") " pod="openshift-monitoring/node-exporter-l5cnt" Jul 27 18:30:30.922847 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:30.922738 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5b877858-8e86-4213-9a5b-10a18e69d669-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-l5cnt\" (UID: \"5b877858-8e86-4213-9a5b-10a18e69d669\") " pod="openshift-monitoring/node-exporter-l5cnt" Jul 27 18:30:30.922847 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:30.922765 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tkzt\" (UniqueName: \"kubernetes.io/projected/5b877858-8e86-4213-9a5b-10a18e69d669-kube-api-access-5tkzt\") pod \"node-exporter-l5cnt\" (UID: \"5b877858-8e86-4213-9a5b-10a18e69d669\") " pod="openshift-monitoring/node-exporter-l5cnt" Jul 27 18:30:30.922847 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:30.922788 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5b877858-8e86-4213-9a5b-10a18e69d669-root\") pod \"node-exporter-l5cnt\" (UID: \"5b877858-8e86-4213-9a5b-10a18e69d669\") " pod="openshift-monitoring/node-exporter-l5cnt" Jul 27 18:30:31.023962 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:31.023923 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5b877858-8e86-4213-9a5b-10a18e69d669-root\") pod \"node-exporter-l5cnt\" (UID: \"5b877858-8e86-4213-9a5b-10a18e69d669\") " pod="openshift-monitoring/node-exporter-l5cnt" Jul 27 18:30:31.024150 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:31.023968 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5b877858-8e86-4213-9a5b-10a18e69d669-node-exporter-tls\") pod \"node-exporter-l5cnt\" (UID: \"5b877858-8e86-4213-9a5b-10a18e69d669\") " pod="openshift-monitoring/node-exporter-l5cnt" Jul 27 18:30:31.024150 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:31.024003 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5b877858-8e86-4213-9a5b-10a18e69d669-node-exporter-textfile\") pod \"node-exporter-l5cnt\" (UID: \"5b877858-8e86-4213-9a5b-10a18e69d669\") " pod="openshift-monitoring/node-exporter-l5cnt" Jul 27 18:30:31.024150 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:31.024031 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5b877858-8e86-4213-9a5b-10a18e69d669-node-exporter-accelerators-collector-config\") pod \"node-exporter-l5cnt\" (UID: \"5b877858-8e86-4213-9a5b-10a18e69d669\") " pod="openshift-monitoring/node-exporter-l5cnt" Jul 27 18:30:31.024150 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:31.024044 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5b877858-8e86-4213-9a5b-10a18e69d669-root\") pod \"node-exporter-l5cnt\" (UID: \"5b877858-8e86-4213-9a5b-10a18e69d669\") " pod="openshift-monitoring/node-exporter-l5cnt" Jul 27 18:30:31.024150 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:31.024081 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5b877858-8e86-4213-9a5b-10a18e69d669-node-exporter-wtmp\") pod \"node-exporter-l5cnt\" (UID: \"5b877858-8e86-4213-9a5b-10a18e69d669\") " pod="openshift-monitoring/node-exporter-l5cnt" Jul 27 18:30:31.024150 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:31.024109 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5b877858-8e86-4213-9a5b-10a18e69d669-metrics-client-ca\") pod \"node-exporter-l5cnt\" (UID: \"5b877858-8e86-4213-9a5b-10a18e69d669\") " pod="openshift-monitoring/node-exporter-l5cnt" Jul 27 18:30:31.024501 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:31.024168 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5b877858-8e86-4213-9a5b-10a18e69d669-sys\") pod \"node-exporter-l5cnt\" (UID: \"5b877858-8e86-4213-9a5b-10a18e69d669\") " pod="openshift-monitoring/node-exporter-l5cnt" Jul 27 18:30:31.024501 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:31.024201 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5b877858-8e86-4213-9a5b-10a18e69d669-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-l5cnt\" (UID: \"5b877858-8e86-4213-9a5b-10a18e69d669\") " pod="openshift-monitoring/node-exporter-l5cnt" Jul 27 18:30:31.024501 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:31.024273 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tkzt\" (UniqueName: \"kubernetes.io/projected/5b877858-8e86-4213-9a5b-10a18e69d669-kube-api-access-5tkzt\") pod \"node-exporter-l5cnt\" (UID: \"5b877858-8e86-4213-9a5b-10a18e69d669\") " pod="openshift-monitoring/node-exporter-l5cnt" Jul 27 18:30:31.024501 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:31.024301 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5b877858-8e86-4213-9a5b-10a18e69d669-node-exporter-wtmp\") pod \"node-exporter-l5cnt\" (UID: \"5b877858-8e86-4213-9a5b-10a18e69d669\") " pod="openshift-monitoring/node-exporter-l5cnt" Jul 27 18:30:31.024501 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:31.024308 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5b877858-8e86-4213-9a5b-10a18e69d669-sys\") pod \"node-exporter-l5cnt\" (UID: \"5b877858-8e86-4213-9a5b-10a18e69d669\") " pod="openshift-monitoring/node-exporter-l5cnt" Jul 27 18:30:31.024709 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:31.024574 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5b877858-8e86-4213-9a5b-10a18e69d669-node-exporter-textfile\") pod \"node-exporter-l5cnt\" (UID: \"5b877858-8e86-4213-9a5b-10a18e69d669\") " pod="openshift-monitoring/node-exporter-l5cnt" Jul 27 18:30:31.024756 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:31.024740 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5b877858-8e86-4213-9a5b-10a18e69d669-node-exporter-accelerators-collector-config\") pod \"node-exporter-l5cnt\" (UID: \"5b877858-8e86-4213-9a5b-10a18e69d669\") " pod="openshift-monitoring/node-exporter-l5cnt" Jul 27 18:30:31.024975 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:31.024956 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5b877858-8e86-4213-9a5b-10a18e69d669-metrics-client-ca\") pod \"node-exporter-l5cnt\" (UID: \"5b877858-8e86-4213-9a5b-10a18e69d669\") " pod="openshift-monitoring/node-exporter-l5cnt" Jul 27 18:30:31.026578 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:31.026561 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5b877858-8e86-4213-9a5b-10a18e69d669-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-l5cnt\" (UID: \"5b877858-8e86-4213-9a5b-10a18e69d669\") " pod="openshift-monitoring/node-exporter-l5cnt" Jul 27 18:30:31.026659 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:31.026643 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5b877858-8e86-4213-9a5b-10a18e69d669-node-exporter-tls\") pod \"node-exporter-l5cnt\" (UID: \"5b877858-8e86-4213-9a5b-10a18e69d669\") " pod="openshift-monitoring/node-exporter-l5cnt" Jul 27 18:30:31.036192 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:31.036135 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tkzt\" (UniqueName: \"kubernetes.io/projected/5b877858-8e86-4213-9a5b-10a18e69d669-kube-api-access-5tkzt\") pod \"node-exporter-l5cnt\" (UID: \"5b877858-8e86-4213-9a5b-10a18e69d669\") " pod="openshift-monitoring/node-exporter-l5cnt" Jul 27 18:30:31.098719 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:31.098690 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-l5cnt" Jul 27 18:30:31.141254 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:30:31.141211 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b877858_8e86_4213_9a5b_10a18e69d669.slice/crio-f675ef51e98f53f0d8187087040c0ba667c8883365549644ec8f4275c07452b4 WatchSource:0}: Error finding container f675ef51e98f53f0d8187087040c0ba667c8883365549644ec8f4275c07452b4: Status 404 returned error can't find the container with id f675ef51e98f53f0d8187087040c0ba667c8883365549644ec8f4275c07452b4 Jul 27 18:30:31.296158 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:31.296056 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:30:31.453070 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:31.453026 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:30:31.544087 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:31.544053 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:30:31.621647 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:31.621550 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:30:31.685281 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:31.685247 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:30:32.131413 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:32.131378 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-l5cnt" event={"ID":"5b877858-8e86-4213-9a5b-10a18e69d669","Type":"ContainerStarted","Data":"f675ef51e98f53f0d8187087040c0ba667c8883365549644ec8f4275c07452b4"} Jul 27 18:30:32.715958 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:32.715909 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:30:32.777037 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:32.776993 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:30:33.425319 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:33.425212 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:30:33.526116 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:33.526077 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:30:33.631266 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:33.631211 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:30:33.933929 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:33.933885 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:30:34.140217 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:34.140180 2579 generic.go:356] "Generic (PLEG): container finished" podID="5b877858-8e86-4213-9a5b-10a18e69d669" containerID="187a5d72a3e25fd33f0edc0a242ff46d9efe1d9432f8d0907ac12478bf048024" exitCode=0 Jul 27 18:30:34.140393 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:34.140268 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-l5cnt" event={"ID":"5b877858-8e86-4213-9a5b-10a18e69d669","Type":"ContainerDied","Data":"187a5d72a3e25fd33f0edc0a242ff46d9efe1d9432f8d0907ac12478bf048024"} Jul 27 18:30:34.670443 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:34.670411 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:30:34.808452 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:34.808419 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:30:34.914343 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:34.914309 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:30:35.146522 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:35.146492 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-l5cnt" event={"ID":"5b877858-8e86-4213-9a5b-10a18e69d669","Type":"ContainerStarted","Data":"31cb5af01da1aee4fdf65924c7e091996175f831b97dc3f0f6864b9d66022961"} Jul 27 18:30:35.147010 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:35.146540 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-l5cnt" event={"ID":"5b877858-8e86-4213-9a5b-10a18e69d669","Type":"ContainerStarted","Data":"63f2cf19d5ab15bcb03bc0370553266a1dba17554d4db8990d3ef19bbf22734f"} Jul 27 18:30:35.169757 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:35.169710 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-l5cnt" podStartSLOduration=3.199001527 podStartE2EDuration="5.169689766s" totalImagesPullingTime="1.921037669s" totalInitContainerRuntime="49.65057ms" isStatefulPod=true podCreationTimestamp="2026-07-27 18:30:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:30:35.167834032 +0000 UTC m=+110.229523064" watchObservedRunningTime="2026-07-27 18:30:35.169689766 +0000 UTC m=+110.231378807" Jul 27 18:30:35.493773 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:35.493686 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:30:35.596637 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:35.596600 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:30:36.679106 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:36.679069 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:30:36.781969 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:36.781929 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:30:36.987262 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:36.987133 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:30:37.086678 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:37.086644 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:30:42.126736 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:42.126697 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:30:42.228640 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:42.228608 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:30:43.755682 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:43.755618 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-85c5fdc4b8-rwfwz" podUID="e660267b-cf0d-4e4e-bb05-1769ff224bb2" containerName="registry" containerID="cri-o://43efca213fe65e17b1478f19fb6910613e71bfbf712cc16a08e219b56f874eff" gracePeriod=30 Jul 27 18:30:43.990723 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:43.990701 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-85c5fdc4b8-rwfwz" Jul 27 18:30:44.027290 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:44.027248 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e660267b-cf0d-4e4e-bb05-1769ff224bb2-registry-tls\") pod \"e660267b-cf0d-4e4e-bb05-1769ff224bb2\" (UID: \"e660267b-cf0d-4e4e-bb05-1769ff224bb2\") " Jul 27 18:30:44.027290 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:44.027287 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e660267b-cf0d-4e4e-bb05-1769ff224bb2-trusted-ca\") pod \"e660267b-cf0d-4e4e-bb05-1769ff224bb2\" (UID: \"e660267b-cf0d-4e4e-bb05-1769ff224bb2\") " Jul 27 18:30:44.027514 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:44.027307 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hcfqt\" (UniqueName: \"kubernetes.io/projected/e660267b-cf0d-4e4e-bb05-1769ff224bb2-kube-api-access-hcfqt\") pod \"e660267b-cf0d-4e4e-bb05-1769ff224bb2\" (UID: \"e660267b-cf0d-4e4e-bb05-1769ff224bb2\") " Jul 27 18:30:44.027514 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:44.027351 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e660267b-cf0d-4e4e-bb05-1769ff224bb2-ca-trust-extracted\") pod \"e660267b-cf0d-4e4e-bb05-1769ff224bb2\" (UID: \"e660267b-cf0d-4e4e-bb05-1769ff224bb2\") " Jul 27 18:30:44.027514 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:44.027383 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e660267b-cf0d-4e4e-bb05-1769ff224bb2-installation-pull-secrets\") pod \"e660267b-cf0d-4e4e-bb05-1769ff224bb2\" (UID: \"e660267b-cf0d-4e4e-bb05-1769ff224bb2\") " Jul 27 18:30:44.027514 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:44.027398 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e660267b-cf0d-4e4e-bb05-1769ff224bb2-bound-sa-token\") pod \"e660267b-cf0d-4e4e-bb05-1769ff224bb2\" (UID: \"e660267b-cf0d-4e4e-bb05-1769ff224bb2\") " Jul 27 18:30:44.027514 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:44.027440 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e660267b-cf0d-4e4e-bb05-1769ff224bb2-image-registry-private-configuration\") pod \"e660267b-cf0d-4e4e-bb05-1769ff224bb2\" (UID: \"e660267b-cf0d-4e4e-bb05-1769ff224bb2\") " Jul 27 18:30:44.027514 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:44.027493 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e660267b-cf0d-4e4e-bb05-1769ff224bb2-registry-certificates\") pod \"e660267b-cf0d-4e4e-bb05-1769ff224bb2\" (UID: \"e660267b-cf0d-4e4e-bb05-1769ff224bb2\") " Jul 27 18:30:44.027931 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:44.027785 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e660267b-cf0d-4e4e-bb05-1769ff224bb2-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "e660267b-cf0d-4e4e-bb05-1769ff224bb2" (UID: "e660267b-cf0d-4e4e-bb05-1769ff224bb2"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 18:30:44.028053 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:44.027969 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e660267b-cf0d-4e4e-bb05-1769ff224bb2-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "e660267b-cf0d-4e4e-bb05-1769ff224bb2" (UID: "e660267b-cf0d-4e4e-bb05-1769ff224bb2"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 18:30:44.030837 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:44.030764 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e660267b-cf0d-4e4e-bb05-1769ff224bb2-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "e660267b-cf0d-4e4e-bb05-1769ff224bb2" (UID: "e660267b-cf0d-4e4e-bb05-1769ff224bb2"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 18:30:44.030961 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:44.030844 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e660267b-cf0d-4e4e-bb05-1769ff224bb2-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "e660267b-cf0d-4e4e-bb05-1769ff224bb2" (UID: "e660267b-cf0d-4e4e-bb05-1769ff224bb2"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 18:30:44.030961 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:44.030847 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e660267b-cf0d-4e4e-bb05-1769ff224bb2-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "e660267b-cf0d-4e4e-bb05-1769ff224bb2" (UID: "e660267b-cf0d-4e4e-bb05-1769ff224bb2"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 18:30:44.030961 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:44.030926 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e660267b-cf0d-4e4e-bb05-1769ff224bb2-kube-api-access-hcfqt" (OuterVolumeSpecName: "kube-api-access-hcfqt") pod "e660267b-cf0d-4e4e-bb05-1769ff224bb2" (UID: "e660267b-cf0d-4e4e-bb05-1769ff224bb2"). InnerVolumeSpecName "kube-api-access-hcfqt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 18:30:44.031133 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:44.030995 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e660267b-cf0d-4e4e-bb05-1769ff224bb2-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "e660267b-cf0d-4e4e-bb05-1769ff224bb2" (UID: "e660267b-cf0d-4e4e-bb05-1769ff224bb2"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 18:30:44.036815 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:44.036781 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e660267b-cf0d-4e4e-bb05-1769ff224bb2-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "e660267b-cf0d-4e4e-bb05-1769ff224bb2" (UID: "e660267b-cf0d-4e4e-bb05-1769ff224bb2"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 18:30:44.128692 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:44.128650 2579 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e660267b-cf0d-4e4e-bb05-1769ff224bb2-installation-pull-secrets\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 18:30:44.128692 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:44.128688 2579 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e660267b-cf0d-4e4e-bb05-1769ff224bb2-bound-sa-token\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 18:30:44.128692 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:44.128703 2579 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e660267b-cf0d-4e4e-bb05-1769ff224bb2-image-registry-private-configuration\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 18:30:44.128901 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:44.128717 2579 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e660267b-cf0d-4e4e-bb05-1769ff224bb2-registry-certificates\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 18:30:44.128901 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:44.128729 2579 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e660267b-cf0d-4e4e-bb05-1769ff224bb2-registry-tls\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 18:30:44.128901 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:44.128741 2579 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e660267b-cf0d-4e4e-bb05-1769ff224bb2-trusted-ca\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 18:30:44.128901 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:44.128751 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hcfqt\" (UniqueName: \"kubernetes.io/projected/e660267b-cf0d-4e4e-bb05-1769ff224bb2-kube-api-access-hcfqt\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 18:30:44.128901 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:44.128763 2579 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e660267b-cf0d-4e4e-bb05-1769ff224bb2-ca-trust-extracted\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 18:30:44.178701 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:44.178669 2579 generic.go:356] "Generic (PLEG): container finished" podID="e660267b-cf0d-4e4e-bb05-1769ff224bb2" containerID="43efca213fe65e17b1478f19fb6910613e71bfbf712cc16a08e219b56f874eff" exitCode=0 Jul 27 18:30:44.178809 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:44.178721 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-85c5fdc4b8-rwfwz" Jul 27 18:30:44.178809 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:44.178763 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-85c5fdc4b8-rwfwz" event={"ID":"e660267b-cf0d-4e4e-bb05-1769ff224bb2","Type":"ContainerDied","Data":"43efca213fe65e17b1478f19fb6910613e71bfbf712cc16a08e219b56f874eff"} Jul 27 18:30:44.178894 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:44.178827 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-85c5fdc4b8-rwfwz" event={"ID":"e660267b-cf0d-4e4e-bb05-1769ff224bb2","Type":"ContainerDied","Data":"86177636677ce4e632674afaccfa4ebf6df6fe0c16acbfa1efea3a2986601278"} Jul 27 18:30:44.178894 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:44.178847 2579 scope.go:117] "RemoveContainer" containerID="43efca213fe65e17b1478f19fb6910613e71bfbf712cc16a08e219b56f874eff" Jul 27 18:30:44.187544 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:44.187525 2579 scope.go:117] "RemoveContainer" containerID="43efca213fe65e17b1478f19fb6910613e71bfbf712cc16a08e219b56f874eff" Jul 27 18:30:44.187824 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:30:44.187804 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43efca213fe65e17b1478f19fb6910613e71bfbf712cc16a08e219b56f874eff\": container with ID starting with 43efca213fe65e17b1478f19fb6910613e71bfbf712cc16a08e219b56f874eff not found: ID does not exist" containerID="43efca213fe65e17b1478f19fb6910613e71bfbf712cc16a08e219b56f874eff" Jul 27 18:30:44.187895 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:44.187831 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43efca213fe65e17b1478f19fb6910613e71bfbf712cc16a08e219b56f874eff"} err="failed to get container status \"43efca213fe65e17b1478f19fb6910613e71bfbf712cc16a08e219b56f874eff\": rpc error: code = NotFound desc = could not find container \"43efca213fe65e17b1478f19fb6910613e71bfbf712cc16a08e219b56f874eff\": container with ID starting with 43efca213fe65e17b1478f19fb6910613e71bfbf712cc16a08e219b56f874eff not found: ID does not exist" Jul 27 18:30:44.203454 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:44.203432 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-85c5fdc4b8-rwfwz"] Jul 27 18:30:44.209117 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:44.209096 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-85c5fdc4b8-rwfwz"] Jul 27 18:30:45.490727 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:45.490700 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e660267b-cf0d-4e4e-bb05-1769ff224bb2" path="/var/lib/kubelet/pods/e660267b-cf0d-4e4e-bb05-1769ff224bb2/volumes" Jul 27 18:30:46.187950 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:46.187852 2579 generic.go:356] "Generic (PLEG): container finished" podID="06e8b4ad-8ea1-4fa8-ab2c-3d6c24fafc88" containerID="aa76be4766a1933c795802992ce6f5bb890ce2054f12ccd7cf4f5a10a9a3109a" exitCode=0 Jul 27 18:30:46.187950 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:46.187909 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-5m9rh" event={"ID":"06e8b4ad-8ea1-4fa8-ab2c-3d6c24fafc88","Type":"ContainerDied","Data":"aa76be4766a1933c795802992ce6f5bb890ce2054f12ccd7cf4f5a10a9a3109a"} Jul 27 18:30:46.188324 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:46.188306 2579 scope.go:117] "RemoveContainer" containerID="aa76be4766a1933c795802992ce6f5bb890ce2054f12ccd7cf4f5a10a9a3109a" Jul 27 18:30:46.189419 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:46.189398 2579 generic.go:356] "Generic (PLEG): container finished" podID="d4bd05a3-ebb5-4c2e-9802-5c0b15f2bdba" containerID="cd5b52d400eb8248dde18adea882c98ab7c8f0243c1897987929f43b1700c4fd" exitCode=0 Jul 27 18:30:46.189662 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:46.189466 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-6dl5t" event={"ID":"d4bd05a3-ebb5-4c2e-9802-5c0b15f2bdba","Type":"ContainerDied","Data":"cd5b52d400eb8248dde18adea882c98ab7c8f0243c1897987929f43b1700c4fd"} Jul 27 18:30:46.189814 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:46.189797 2579 scope.go:117] "RemoveContainer" containerID="cd5b52d400eb8248dde18adea882c98ab7c8f0243c1897987929f43b1700c4fd" Jul 27 18:30:47.194750 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:47.194716 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-5m9rh" event={"ID":"06e8b4ad-8ea1-4fa8-ab2c-3d6c24fafc88","Type":"ContainerStarted","Data":"03642361de7c0435b6a6e765ba82d4fe065bad5710543151e33a1b4b8d1ae57a"} Jul 27 18:30:47.196547 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:47.196522 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-6dl5t" event={"ID":"d4bd05a3-ebb5-4c2e-9802-5c0b15f2bdba","Type":"ContainerStarted","Data":"a2bd10c54779da37d8da8e9eae43a75932bd9e1e881c116a036fa2adaf39df98"} Jul 27 18:30:47.828374 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:47.828343 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:30:47.928529 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:47.928483 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:30:47.989279 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:47.989216 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:30:57.124698 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:57.124659 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:30:57.191394 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:57.191359 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:30:57.238061 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:57.238025 2579 generic.go:356] "Generic (PLEG): container finished" podID="b61c6434-d128-4ec0-920d-a00b085ae9c6" containerID="fbd3e16bf0cafb85de56c6c68ddbb4dccad589a2cbaa5de17df511de7ea53109" exitCode=0 Jul 27 18:30:57.238224 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:57.238104 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-mhdxk" event={"ID":"b61c6434-d128-4ec0-920d-a00b085ae9c6","Type":"ContainerDied","Data":"fbd3e16bf0cafb85de56c6c68ddbb4dccad589a2cbaa5de17df511de7ea53109"} Jul 27 18:30:57.238444 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:57.238431 2579 scope.go:117] "RemoveContainer" containerID="fbd3e16bf0cafb85de56c6c68ddbb4dccad589a2cbaa5de17df511de7ea53109" Jul 27 18:30:58.243172 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:30:58.243137 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-mhdxk" event={"ID":"b61c6434-d128-4ec0-920d-a00b085ae9c6","Type":"ContainerStarted","Data":"6cf9f1e700198b678051d8f73cf24ae177c0a10a8a0f6696db9ba72545f80214"} Jul 27 18:31:49.707658 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:31:49.707612 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:31:50.100294 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:31:50.100256 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:31:59.154114 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:31:59.154075 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:31:59.255220 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:31:59.255177 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:31:59.659037 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:31:59.659004 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:32:10.198150 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:32:10.198110 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:32:10.299510 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:32:10.299479 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:32:10.405735 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:32:10.405701 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:32:28.262535 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:32:28.262500 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:32:28.453030 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:32:28.452993 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:33:04.942274 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:33:04.942211 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:33:05.042787 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:33:05.042755 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:33:09.473737 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:33:09.473646 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:33:09.675767 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:33:09.675728 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:33:09.737631 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:33:09.737541 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:33:10.207530 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:33:10.207499 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:33:10.262025 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:33:10.261991 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:33:10.505203 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:33:10.505121 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:33:10.586626 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:33:10.586590 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:33:16.947288 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:33:16.947246 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:33:17.049148 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:33:17.049114 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:33:20.459728 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:33:20.459694 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:33:20.752595 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:33:20.752493 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:33:20.810513 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:33:20.810475 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:33:21.957647 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:33:21.957608 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:33:22.012668 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:33:22.012635 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:33:27.137251 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:33:27.137194 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:33:27.235863 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:33:27.235818 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:33:45.350649 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:33:45.350610 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-mv4hb_715b2084-d79e-4cbf-a057-f2595b4e6b9b/console-operator/1.log" Jul 27 18:33:45.361763 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:33:45.361736 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-mv4hb_715b2084-d79e-4cbf-a057-f2595b4e6b9b/console-operator/1.log" Jul 27 18:33:45.367009 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:33:45.366984 2579 kubelet.go:1648] "Image garbage collection succeeded" Jul 27 18:33:51.778654 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:33:51.778603 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:33:53.980990 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:33:53.980949 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:34:01.248851 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:34:01.248818 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:34:08.023975 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:34:08.023942 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:34:08.046395 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:34:08.046365 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5ff59dcc95-wsnqr"] Jul 27 18:34:08.046639 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:34:08.046629 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e660267b-cf0d-4e4e-bb05-1769ff224bb2" containerName="registry" Jul 27 18:34:08.046680 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:34:08.046641 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e660267b-cf0d-4e4e-bb05-1769ff224bb2" containerName="registry" Jul 27 18:34:08.046712 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:34:08.046698 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e660267b-cf0d-4e4e-bb05-1769ff224bb2" containerName="registry" Jul 27 18:34:08.048650 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:34:08.048635 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5ff59dcc95-wsnqr" Jul 27 18:34:08.052726 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:34:08.052702 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 27 18:34:08.052726 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:34:08.052721 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 27 18:34:08.052919 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:34:08.052741 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 27 18:34:08.052919 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:34:08.052759 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 27 18:34:08.052919 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:34:08.052769 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 27 18:34:08.052919 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:34:08.052777 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 27 18:34:08.052919 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:34:08.052721 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 27 18:34:08.062534 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:34:08.062497 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5ff59dcc95-wsnqr"] Jul 27 18:34:08.092207 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:34:08.092173 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:34:08.122246 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:34:08.122210 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/90a7b82d-a62c-48be-a48d-53e8a2d8aefb-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5ff59dcc95-wsnqr\" (UID: \"90a7b82d-a62c-48be-a48d-53e8a2d8aefb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5ff59dcc95-wsnqr" Jul 27 18:34:08.122379 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:34:08.122264 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/90a7b82d-a62c-48be-a48d-53e8a2d8aefb-hub\") pod \"cluster-proxy-proxy-agent-5ff59dcc95-wsnqr\" (UID: \"90a7b82d-a62c-48be-a48d-53e8a2d8aefb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5ff59dcc95-wsnqr" Jul 27 18:34:08.122379 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:34:08.122314 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppmhd\" (UniqueName: \"kubernetes.io/projected/90a7b82d-a62c-48be-a48d-53e8a2d8aefb-kube-api-access-ppmhd\") pod \"cluster-proxy-proxy-agent-5ff59dcc95-wsnqr\" (UID: \"90a7b82d-a62c-48be-a48d-53e8a2d8aefb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5ff59dcc95-wsnqr" Jul 27 18:34:08.122379 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:34:08.122333 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/90a7b82d-a62c-48be-a48d-53e8a2d8aefb-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5ff59dcc95-wsnqr\" (UID: \"90a7b82d-a62c-48be-a48d-53e8a2d8aefb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5ff59dcc95-wsnqr" Jul 27 18:34:08.122526 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:34:08.122377 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/90a7b82d-a62c-48be-a48d-53e8a2d8aefb-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5ff59dcc95-wsnqr\" (UID: \"90a7b82d-a62c-48be-a48d-53e8a2d8aefb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5ff59dcc95-wsnqr" Jul 27 18:34:08.122526 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:34:08.122431 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/90a7b82d-a62c-48be-a48d-53e8a2d8aefb-ca\") pod \"cluster-proxy-proxy-agent-5ff59dcc95-wsnqr\" (UID: \"90a7b82d-a62c-48be-a48d-53e8a2d8aefb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5ff59dcc95-wsnqr" Jul 27 18:34:08.223559 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:34:08.223517 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/90a7b82d-a62c-48be-a48d-53e8a2d8aefb-ca\") pod \"cluster-proxy-proxy-agent-5ff59dcc95-wsnqr\" (UID: \"90a7b82d-a62c-48be-a48d-53e8a2d8aefb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5ff59dcc95-wsnqr" Jul 27 18:34:08.223738 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:34:08.223599 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/90a7b82d-a62c-48be-a48d-53e8a2d8aefb-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5ff59dcc95-wsnqr\" (UID: \"90a7b82d-a62c-48be-a48d-53e8a2d8aefb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5ff59dcc95-wsnqr" Jul 27 18:34:08.223738 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:34:08.223634 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/90a7b82d-a62c-48be-a48d-53e8a2d8aefb-hub\") pod \"cluster-proxy-proxy-agent-5ff59dcc95-wsnqr\" (UID: \"90a7b82d-a62c-48be-a48d-53e8a2d8aefb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5ff59dcc95-wsnqr" Jul 27 18:34:08.223738 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:34:08.223672 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppmhd\" (UniqueName: \"kubernetes.io/projected/90a7b82d-a62c-48be-a48d-53e8a2d8aefb-kube-api-access-ppmhd\") pod \"cluster-proxy-proxy-agent-5ff59dcc95-wsnqr\" (UID: \"90a7b82d-a62c-48be-a48d-53e8a2d8aefb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5ff59dcc95-wsnqr" Jul 27 18:34:08.223738 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:34:08.223697 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/90a7b82d-a62c-48be-a48d-53e8a2d8aefb-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5ff59dcc95-wsnqr\" (UID: \"90a7b82d-a62c-48be-a48d-53e8a2d8aefb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5ff59dcc95-wsnqr" Jul 27 18:34:08.223738 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:34:08.223721 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/90a7b82d-a62c-48be-a48d-53e8a2d8aefb-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5ff59dcc95-wsnqr\" (UID: \"90a7b82d-a62c-48be-a48d-53e8a2d8aefb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5ff59dcc95-wsnqr" Jul 27 18:34:08.224750 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:34:08.224717 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/90a7b82d-a62c-48be-a48d-53e8a2d8aefb-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5ff59dcc95-wsnqr\" (UID: \"90a7b82d-a62c-48be-a48d-53e8a2d8aefb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5ff59dcc95-wsnqr" Jul 27 18:34:08.226145 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:34:08.226125 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/90a7b82d-a62c-48be-a48d-53e8a2d8aefb-ca\") pod \"cluster-proxy-proxy-agent-5ff59dcc95-wsnqr\" (UID: \"90a7b82d-a62c-48be-a48d-53e8a2d8aefb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5ff59dcc95-wsnqr" Jul 27 18:34:08.226453 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:34:08.226430 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/90a7b82d-a62c-48be-a48d-53e8a2d8aefb-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5ff59dcc95-wsnqr\" (UID: \"90a7b82d-a62c-48be-a48d-53e8a2d8aefb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5ff59dcc95-wsnqr" Jul 27 18:34:08.226534 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:34:08.226439 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/90a7b82d-a62c-48be-a48d-53e8a2d8aefb-hub\") pod \"cluster-proxy-proxy-agent-5ff59dcc95-wsnqr\" (UID: \"90a7b82d-a62c-48be-a48d-53e8a2d8aefb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5ff59dcc95-wsnqr" Jul 27 18:34:08.226534 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:34:08.226503 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/90a7b82d-a62c-48be-a48d-53e8a2d8aefb-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5ff59dcc95-wsnqr\" (UID: \"90a7b82d-a62c-48be-a48d-53e8a2d8aefb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5ff59dcc95-wsnqr" Jul 27 18:34:08.231745 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:34:08.231717 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppmhd\" (UniqueName: \"kubernetes.io/projected/90a7b82d-a62c-48be-a48d-53e8a2d8aefb-kube-api-access-ppmhd\") pod \"cluster-proxy-proxy-agent-5ff59dcc95-wsnqr\" (UID: \"90a7b82d-a62c-48be-a48d-53e8a2d8aefb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5ff59dcc95-wsnqr" Jul 27 18:34:08.376093 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:34:08.375979 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5ff59dcc95-wsnqr" Jul 27 18:34:08.509826 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:34:08.509784 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5ff59dcc95-wsnqr"] Jul 27 18:34:08.512716 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:34:08.512680 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod90a7b82d_a62c_48be_a48d_53e8a2d8aefb.slice/crio-b0e26fac0ef3acd633f000151b427deb683451d5103552fe33a0f2b98b17f948 WatchSource:0}: Error finding container b0e26fac0ef3acd633f000151b427deb683451d5103552fe33a0f2b98b17f948: Status 404 returned error can't find the container with id b0e26fac0ef3acd633f000151b427deb683451d5103552fe33a0f2b98b17f948 Jul 27 18:34:08.514422 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:34:08.514404 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 27 18:34:08.904558 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:34:08.904524 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5ff59dcc95-wsnqr" event={"ID":"90a7b82d-a62c-48be-a48d-53e8a2d8aefb","Type":"ContainerStarted","Data":"b0e26fac0ef3acd633f000151b427deb683451d5103552fe33a0f2b98b17f948"} Jul 27 18:34:12.924444 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:34:12.924404 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5ff59dcc95-wsnqr" event={"ID":"90a7b82d-a62c-48be-a48d-53e8a2d8aefb","Type":"ContainerStarted","Data":"bb2e8212ab821ea2ba26d46ec231909ef083f4f4649a8d4c8af0fd09ec25ebe4"} Jul 27 18:34:12.924444 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:34:12.924448 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5ff59dcc95-wsnqr" event={"ID":"90a7b82d-a62c-48be-a48d-53e8a2d8aefb","Type":"ContainerStarted","Data":"496c9a8fce389ffbb79ab8267a8dcca3740cf6316b46209d5d6667035d13bc53"} Jul 27 18:34:12.924875 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:34:12.924462 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5ff59dcc95-wsnqr" event={"ID":"90a7b82d-a62c-48be-a48d-53e8a2d8aefb","Type":"ContainerStarted","Data":"2daafbcd5d98b67995b22c6e7d2729390b18d00d0c2289820ffed9aef671c575"} Jul 27 18:34:12.948997 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:34:12.948939 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5ff59dcc95-wsnqr" podStartSLOduration=1.279479762 podStartE2EDuration="4.948923418s" totalImagesPullingTime="3.669443656s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:34:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:34:12.945560209 +0000 UTC m=+328.007249241" watchObservedRunningTime="2026-07-27 18:34:12.948923418 +0000 UTC m=+328.010612448" Jul 27 18:34:23.296431 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:34:23.296389 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:34:23.394950 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:34:23.394911 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:34:39.347584 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:34:39.347487 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:34:50.609365 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:34:50.609316 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:34:51.358528 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:34:51.358498 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:34:54.502423 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:34:54.502380 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:34:54.604063 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:34:54.604028 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:34:55.189788 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:34:55.189728 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:34:55.340798 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:34:55.340764 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:34:55.658251 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:34:55.658198 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:34:55.754731 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:34:55.754702 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:34:57.454339 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:34:57.454306 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:34:58.054335 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:34:58.054299 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:35:00.926255 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:35:00.926197 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:35:00.982691 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:35:00.982661 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:35:03.457916 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:35:03.457870 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:35:03.586101 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:35:03.586071 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:35:06.299566 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:35:06.299532 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:35:06.845518 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:35:06.845477 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:35:11.351002 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:35:11.350975 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:35:11.638094 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:35:11.637997 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:35:17.515317 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:35:17.515274 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:35:17.617297 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:35:17.617251 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:35:20.185862 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:35:20.185813 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:35:31.549646 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:35:31.549612 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:35:31.665286 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:35:31.665217 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:35:32.538483 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:35:32.538451 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-57d99c84d4-5228k"] Jul 27 18:35:32.541690 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:35:32.541669 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-57d99c84d4-5228k" Jul 27 18:35:32.545323 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:35:32.545272 2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"opendatahub-operator-controller-manager-dockercfg-dvvg4\" is forbidden: User \"system:node:ip-10-0-128-32.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-32.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-dvvg4\"" type="*v1.Secret" Jul 27 18:35:32.545323 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:35:32.545308 2579 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"openshift-service-ca.crt\" is forbidden: User \"system:node:ip-10-0-128-32.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-32.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" Jul 27 18:35:32.545513 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:35:32.545423 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"opendatahub-operator-controller-manager-57d99c84d4-5228k\" is forbidden: User \"system:node:ip-10-0-128-32.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-32.ec2.internal' and this object" podUID="9f6495eb-fc10-4791-b32a-81e55f8b09c8" pod="opendatahub/opendatahub-operator-controller-manager-57d99c84d4-5228k" Jul 27 18:35:32.545832 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:35:32.545805 2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"opendatahub-operator-controller-webhook-cert\" is forbidden: User \"system:node:ip-10-0-128-32.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-32.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" type="*v1.Secret" Jul 27 18:35:32.545928 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:35:32.545833 2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"opendatahub-operator-controller-manager-service-cert\" is forbidden: User \"system:node:ip-10-0-128-32.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-32.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" type="*v1.Secret" Jul 27 18:35:32.545928 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:35:32.545900 2579 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"kube-root-ca.crt\" is forbidden: User \"system:node:ip-10-0-128-32.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-32.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" Jul 27 18:35:32.570269 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:35:32.570248 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-57d99c84d4-5228k"] Jul 27 18:35:32.646642 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:35:32.646617 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnmq4\" (UniqueName: \"kubernetes.io/projected/9f6495eb-fc10-4791-b32a-81e55f8b09c8-kube-api-access-vnmq4\") pod \"opendatahub-operator-controller-manager-57d99c84d4-5228k\" (UID: \"9f6495eb-fc10-4791-b32a-81e55f8b09c8\") " pod="opendatahub/opendatahub-operator-controller-manager-57d99c84d4-5228k" Jul 27 18:35:32.646758 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:35:32.646649 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9f6495eb-fc10-4791-b32a-81e55f8b09c8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-57d99c84d4-5228k\" (UID: \"9f6495eb-fc10-4791-b32a-81e55f8b09c8\") " pod="opendatahub/opendatahub-operator-controller-manager-57d99c84d4-5228k" Jul 27 18:35:32.646758 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:35:32.646677 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9f6495eb-fc10-4791-b32a-81e55f8b09c8-webhook-cert\") pod \"opendatahub-operator-controller-manager-57d99c84d4-5228k\" (UID: \"9f6495eb-fc10-4791-b32a-81e55f8b09c8\") " pod="opendatahub/opendatahub-operator-controller-manager-57d99c84d4-5228k" Jul 27 18:35:32.646758 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:35:32.646693 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9f6495eb-fc10-4791-b32a-81e55f8b09c8-manifests\") pod \"opendatahub-operator-controller-manager-57d99c84d4-5228k\" (UID: \"9f6495eb-fc10-4791-b32a-81e55f8b09c8\") " pod="opendatahub/opendatahub-operator-controller-manager-57d99c84d4-5228k" Jul 27 18:35:32.646882 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:35:32.646786 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9f6495eb-fc10-4791-b32a-81e55f8b09c8-tmp\") pod \"opendatahub-operator-controller-manager-57d99c84d4-5228k\" (UID: \"9f6495eb-fc10-4791-b32a-81e55f8b09c8\") " pod="opendatahub/opendatahub-operator-controller-manager-57d99c84d4-5228k" Jul 27 18:35:32.747441 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:35:32.747409 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9f6495eb-fc10-4791-b32a-81e55f8b09c8-tmp\") pod \"opendatahub-operator-controller-manager-57d99c84d4-5228k\" (UID: \"9f6495eb-fc10-4791-b32a-81e55f8b09c8\") " pod="opendatahub/opendatahub-operator-controller-manager-57d99c84d4-5228k" Jul 27 18:35:32.747555 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:35:32.747449 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnmq4\" (UniqueName: \"kubernetes.io/projected/9f6495eb-fc10-4791-b32a-81e55f8b09c8-kube-api-access-vnmq4\") pod \"opendatahub-operator-controller-manager-57d99c84d4-5228k\" (UID: \"9f6495eb-fc10-4791-b32a-81e55f8b09c8\") " pod="opendatahub/opendatahub-operator-controller-manager-57d99c84d4-5228k" Jul 27 18:35:32.747555 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:35:32.747475 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9f6495eb-fc10-4791-b32a-81e55f8b09c8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-57d99c84d4-5228k\" (UID: \"9f6495eb-fc10-4791-b32a-81e55f8b09c8\") " pod="opendatahub/opendatahub-operator-controller-manager-57d99c84d4-5228k" Jul 27 18:35:32.747555 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:35:32.747520 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9f6495eb-fc10-4791-b32a-81e55f8b09c8-webhook-cert\") pod \"opendatahub-operator-controller-manager-57d99c84d4-5228k\" (UID: \"9f6495eb-fc10-4791-b32a-81e55f8b09c8\") " pod="opendatahub/opendatahub-operator-controller-manager-57d99c84d4-5228k" Jul 27 18:35:32.747555 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:35:32.747544 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9f6495eb-fc10-4791-b32a-81e55f8b09c8-manifests\") pod \"opendatahub-operator-controller-manager-57d99c84d4-5228k\" (UID: \"9f6495eb-fc10-4791-b32a-81e55f8b09c8\") " pod="opendatahub/opendatahub-operator-controller-manager-57d99c84d4-5228k" Jul 27 18:35:32.747890 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:35:32.747873 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9f6495eb-fc10-4791-b32a-81e55f8b09c8-manifests\") pod \"opendatahub-operator-controller-manager-57d99c84d4-5228k\" (UID: \"9f6495eb-fc10-4791-b32a-81e55f8b09c8\") " pod="opendatahub/opendatahub-operator-controller-manager-57d99c84d4-5228k" Jul 27 18:35:32.749722 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:35:32.749695 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9f6495eb-fc10-4791-b32a-81e55f8b09c8-tmp\") pod \"opendatahub-operator-controller-manager-57d99c84d4-5228k\" (UID: \"9f6495eb-fc10-4791-b32a-81e55f8b09c8\") " pod="opendatahub/opendatahub-operator-controller-manager-57d99c84d4-5228k" Jul 27 18:35:32.929296 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:35:32.929213 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:35:33.022878 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:35:33.022837 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:35:33.082546 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:35:33.082517 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:35:33.184595 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:35:33.184473 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:35:33.627732 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:35:33.627700 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-dvvg4\"" Jul 27 18:35:33.634635 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:35:33.634615 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 27 18:35:33.670690 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:35:33.670669 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 27 18:35:33.680652 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:35:33.680625 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9f6495eb-fc10-4791-b32a-81e55f8b09c8-webhook-cert\") pod \"opendatahub-operator-controller-manager-57d99c84d4-5228k\" (UID: \"9f6495eb-fc10-4791-b32a-81e55f8b09c8\") " pod="opendatahub/opendatahub-operator-controller-manager-57d99c84d4-5228k" Jul 27 18:35:33.681037 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:35:33.681023 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9f6495eb-fc10-4791-b32a-81e55f8b09c8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-57d99c84d4-5228k\" (UID: \"9f6495eb-fc10-4791-b32a-81e55f8b09c8\") " pod="opendatahub/opendatahub-operator-controller-manager-57d99c84d4-5228k" Jul 27 18:35:33.737740 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:35:33.737711 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 27 18:35:33.755114 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:35:33.755096 2579 projected.go:291] Couldn't get configMap opendatahub/kube-root-ca.crt: failed to sync configmap cache: timed out waiting for the condition Jul 27 18:35:33.755211 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:35:33.755123 2579 projected.go:196] Error preparing data for projected volume kube-api-access-vnmq4 for pod opendatahub/opendatahub-operator-controller-manager-57d99c84d4-5228k: failed to sync configmap cache: timed out waiting for the condition Jul 27 18:35:33.755211 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:35:33.755187 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9f6495eb-fc10-4791-b32a-81e55f8b09c8-kube-api-access-vnmq4 podName:9f6495eb-fc10-4791-b32a-81e55f8b09c8 nodeName:}" failed. No retries permitted until 2026-07-27 18:35:34.255168929 +0000 UTC m=+409.316857938 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-vnmq4" (UniqueName: "kubernetes.io/projected/9f6495eb-fc10-4791-b32a-81e55f8b09c8-kube-api-access-vnmq4") pod "opendatahub-operator-controller-manager-57d99c84d4-5228k" (UID: "9f6495eb-fc10-4791-b32a-81e55f8b09c8") : failed to sync configmap cache: timed out waiting for the condition Jul 27 18:35:33.864277 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:35:33.864253 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 27 18:35:34.259638 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:35:34.259603 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnmq4\" (UniqueName: \"kubernetes.io/projected/9f6495eb-fc10-4791-b32a-81e55f8b09c8-kube-api-access-vnmq4\") pod \"opendatahub-operator-controller-manager-57d99c84d4-5228k\" (UID: \"9f6495eb-fc10-4791-b32a-81e55f8b09c8\") " pod="opendatahub/opendatahub-operator-controller-manager-57d99c84d4-5228k" Jul 27 18:35:34.262162 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:35:34.262136 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnmq4\" (UniqueName: \"kubernetes.io/projected/9f6495eb-fc10-4791-b32a-81e55f8b09c8-kube-api-access-vnmq4\") pod \"opendatahub-operator-controller-manager-57d99c84d4-5228k\" (UID: \"9f6495eb-fc10-4791-b32a-81e55f8b09c8\") " pod="opendatahub/opendatahub-operator-controller-manager-57d99c84d4-5228k" Jul 27 18:35:34.351648 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:35:34.351618 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-57d99c84d4-5228k" Jul 27 18:35:34.502541 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:35:34.502518 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-57d99c84d4-5228k"] Jul 27 18:35:34.505291 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:35:34.505267 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f6495eb_fc10_4791_b32a_81e55f8b09c8.slice/crio-48f902c56df840e9974e6738072c7fb9667865c8d31c1dfb4010b34227e52704 WatchSource:0}: Error finding container 48f902c56df840e9974e6738072c7fb9667865c8d31c1dfb4010b34227e52704: Status 404 returned error can't find the container with id 48f902c56df840e9974e6738072c7fb9667865c8d31c1dfb4010b34227e52704 Jul 27 18:35:35.241972 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:35:35.241929 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-57d99c84d4-5228k" event={"ID":"9f6495eb-fc10-4791-b32a-81e55f8b09c8","Type":"ContainerStarted","Data":"48f902c56df840e9974e6738072c7fb9667865c8d31c1dfb4010b34227e52704"} Jul 27 18:35:38.257370 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:35:38.257337 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-57d99c84d4-5228k" event={"ID":"9f6495eb-fc10-4791-b32a-81e55f8b09c8","Type":"ContainerStarted","Data":"e9a7da6d03ee4e88146e665faf02cdda1cd97be21a8f7234f5e43048e6493510"} Jul 27 18:35:39.263550 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:35:39.263513 2579 generic.go:356] "Generic (PLEG): container finished" podID="9f6495eb-fc10-4791-b32a-81e55f8b09c8" containerID="e9a7da6d03ee4e88146e665faf02cdda1cd97be21a8f7234f5e43048e6493510" exitCode=0 Jul 27 18:35:39.263954 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:35:39.263560 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-57d99c84d4-5228k" event={"ID":"9f6495eb-fc10-4791-b32a-81e55f8b09c8","Type":"ContainerDied","Data":"e9a7da6d03ee4e88146e665faf02cdda1cd97be21a8f7234f5e43048e6493510"} Jul 27 18:35:40.269194 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:35:40.269110 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-57d99c84d4-5228k" event={"ID":"9f6495eb-fc10-4791-b32a-81e55f8b09c8","Type":"ContainerStarted","Data":"5c81879c3c6749039ba41edff8c061cb042bfad89ca51c5e89ac5b2ee20fef54"} Jul 27 18:35:40.269595 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:35:40.269282 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-57d99c84d4-5228k" Jul 27 18:35:40.302754 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:35:40.302701 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-57d99c84d4-5228k" podStartSLOduration=3.882384699 podStartE2EDuration="8.302686035s" totalImagesPullingTime="3.337442654s" totalInitContainerRuntime="1.082858682s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:35:32 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:35:40.302672748 +0000 UTC m=+415.364361779" watchObservedRunningTime="2026-07-27 18:35:40.302686035 +0000 UTC m=+415.364375047" Jul 27 18:35:40.550872 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:35:40.550823 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:35:40.669195 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:35:40.669159 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:35:51.275004 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:35:51.274964 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-57d99c84d4-5228k" Jul 27 18:35:53.520400 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:35:53.520360 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7896769b64-tkbg9"] Jul 27 18:35:53.536314 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:35:53.536282 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7896769b64-tkbg9"] Jul 27 18:35:53.536476 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:35:53.536391 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7896769b64-tkbg9" Jul 27 18:35:53.539344 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:35:53.539314 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 27 18:35:53.539344 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:35:53.539331 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 27 18:35:53.539344 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:35:53.539329 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-rg85g\"" Jul 27 18:35:53.610913 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:35:53.610878 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdjxk\" (UniqueName: \"kubernetes.io/projected/2bfb0e76-8207-48bb-810a-6f865001eeb1-kube-api-access-kdjxk\") pod \"kube-auth-proxy-7896769b64-tkbg9\" (UID: \"2bfb0e76-8207-48bb-810a-6f865001eeb1\") " pod="openshift-ingress/kube-auth-proxy-7896769b64-tkbg9" Jul 27 18:35:53.611074 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:35:53.610934 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2bfb0e76-8207-48bb-810a-6f865001eeb1-tmp\") pod \"kube-auth-proxy-7896769b64-tkbg9\" (UID: \"2bfb0e76-8207-48bb-810a-6f865001eeb1\") " pod="openshift-ingress/kube-auth-proxy-7896769b64-tkbg9" Jul 27 18:35:53.611074 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:35:53.610962 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2bfb0e76-8207-48bb-810a-6f865001eeb1-tls-certs\") pod \"kube-auth-proxy-7896769b64-tkbg9\" (UID: \"2bfb0e76-8207-48bb-810a-6f865001eeb1\") " pod="openshift-ingress/kube-auth-proxy-7896769b64-tkbg9" Jul 27 18:35:53.711953 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:35:53.711917 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdjxk\" (UniqueName: \"kubernetes.io/projected/2bfb0e76-8207-48bb-810a-6f865001eeb1-kube-api-access-kdjxk\") pod \"kube-auth-proxy-7896769b64-tkbg9\" (UID: \"2bfb0e76-8207-48bb-810a-6f865001eeb1\") " pod="openshift-ingress/kube-auth-proxy-7896769b64-tkbg9" Jul 27 18:35:53.712094 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:35:53.711963 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2bfb0e76-8207-48bb-810a-6f865001eeb1-tmp\") pod \"kube-auth-proxy-7896769b64-tkbg9\" (UID: \"2bfb0e76-8207-48bb-810a-6f865001eeb1\") " pod="openshift-ingress/kube-auth-proxy-7896769b64-tkbg9" Jul 27 18:35:53.712094 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:35:53.711981 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2bfb0e76-8207-48bb-810a-6f865001eeb1-tls-certs\") pod \"kube-auth-proxy-7896769b64-tkbg9\" (UID: \"2bfb0e76-8207-48bb-810a-6f865001eeb1\") " pod="openshift-ingress/kube-auth-proxy-7896769b64-tkbg9" Jul 27 18:35:53.712094 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:35:53.712079 2579 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found Jul 27 18:35:53.712208 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:35:53.712132 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2bfb0e76-8207-48bb-810a-6f865001eeb1-tls-certs podName:2bfb0e76-8207-48bb-810a-6f865001eeb1 nodeName:}" failed. No retries permitted until 2026-07-27 18:35:54.21211642 +0000 UTC m=+429.273805429 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/2bfb0e76-8207-48bb-810a-6f865001eeb1-tls-certs") pod "kube-auth-proxy-7896769b64-tkbg9" (UID: "2bfb0e76-8207-48bb-810a-6f865001eeb1") : secret "kube-auth-proxy-tls" not found Jul 27 18:35:53.714367 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:35:53.714337 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2bfb0e76-8207-48bb-810a-6f865001eeb1-tmp\") pod \"kube-auth-proxy-7896769b64-tkbg9\" (UID: \"2bfb0e76-8207-48bb-810a-6f865001eeb1\") " pod="openshift-ingress/kube-auth-proxy-7896769b64-tkbg9" Jul 27 18:35:53.722857 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:35:53.722837 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdjxk\" (UniqueName: \"kubernetes.io/projected/2bfb0e76-8207-48bb-810a-6f865001eeb1-kube-api-access-kdjxk\") pod \"kube-auth-proxy-7896769b64-tkbg9\" (UID: \"2bfb0e76-8207-48bb-810a-6f865001eeb1\") " pod="openshift-ingress/kube-auth-proxy-7896769b64-tkbg9" Jul 27 18:35:53.754733 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:35:53.754701 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:35:53.805587 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:35:53.805514 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-sgg44"] Jul 27 18:35:53.808794 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:35:53.808778 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-sgg44" Jul 27 18:35:53.812575 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:35:53.812552 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-glqgk\"" Jul 27 18:35:53.812575 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:35:53.812569 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 27 18:35:53.835638 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:35:53.835612 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-sgg44"] Jul 27 18:35:53.891962 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:35:53.891928 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:35:53.913185 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:35:53.913154 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/87423332-edfc-4e36-a11b-56324b7db773-cert\") pod \"odh-model-controller-7998c67dc4-sgg44\" (UID: \"87423332-edfc-4e36-a11b-56324b7db773\") " pod="opendatahub/odh-model-controller-7998c67dc4-sgg44" Jul 27 18:35:53.913332 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:35:53.913209 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhd4b\" (UniqueName: \"kubernetes.io/projected/87423332-edfc-4e36-a11b-56324b7db773-kube-api-access-fhd4b\") pod \"odh-model-controller-7998c67dc4-sgg44\" (UID: \"87423332-edfc-4e36-a11b-56324b7db773\") " pod="opendatahub/odh-model-controller-7998c67dc4-sgg44" Jul 27 18:35:53.988890 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:35:53.988856 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:35:54.014436 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:35:54.014400 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/87423332-edfc-4e36-a11b-56324b7db773-cert\") pod \"odh-model-controller-7998c67dc4-sgg44\" (UID: \"87423332-edfc-4e36-a11b-56324b7db773\") " pod="opendatahub/odh-model-controller-7998c67dc4-sgg44" Jul 27 18:35:54.014596 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:35:54.014447 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhd4b\" (UniqueName: \"kubernetes.io/projected/87423332-edfc-4e36-a11b-56324b7db773-kube-api-access-fhd4b\") pod \"odh-model-controller-7998c67dc4-sgg44\" (UID: \"87423332-edfc-4e36-a11b-56324b7db773\") " pod="opendatahub/odh-model-controller-7998c67dc4-sgg44" Jul 27 18:35:54.014596 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:35:54.014564 2579 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Jul 27 18:35:54.014712 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:35:54.014636 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87423332-edfc-4e36-a11b-56324b7db773-cert podName:87423332-edfc-4e36-a11b-56324b7db773 nodeName:}" failed. No retries permitted until 2026-07-27 18:35:54.51461545 +0000 UTC m=+429.576304468 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/87423332-edfc-4e36-a11b-56324b7db773-cert") pod "odh-model-controller-7998c67dc4-sgg44" (UID: "87423332-edfc-4e36-a11b-56324b7db773") : secret "odh-model-controller-webhook-cert" not found Jul 27 18:35:54.035870 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:35:54.035841 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhd4b\" (UniqueName: \"kubernetes.io/projected/87423332-edfc-4e36-a11b-56324b7db773-kube-api-access-fhd4b\") pod \"odh-model-controller-7998c67dc4-sgg44\" (UID: \"87423332-edfc-4e36-a11b-56324b7db773\") " pod="opendatahub/odh-model-controller-7998c67dc4-sgg44" Jul 27 18:35:54.114582 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:35:54.114492 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:35:54.215940 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:35:54.215828 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2bfb0e76-8207-48bb-810a-6f865001eeb1-tls-certs\") pod \"kube-auth-proxy-7896769b64-tkbg9\" (UID: \"2bfb0e76-8207-48bb-810a-6f865001eeb1\") " pod="openshift-ingress/kube-auth-proxy-7896769b64-tkbg9" Jul 27 18:35:54.218365 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:35:54.218342 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2bfb0e76-8207-48bb-810a-6f865001eeb1-tls-certs\") pod \"kube-auth-proxy-7896769b64-tkbg9\" (UID: \"2bfb0e76-8207-48bb-810a-6f865001eeb1\") " pod="openshift-ingress/kube-auth-proxy-7896769b64-tkbg9" Jul 27 18:35:54.446490 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:35:54.446403 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7896769b64-tkbg9" Jul 27 18:35:54.518470 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:35:54.518441 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/87423332-edfc-4e36-a11b-56324b7db773-cert\") pod \"odh-model-controller-7998c67dc4-sgg44\" (UID: \"87423332-edfc-4e36-a11b-56324b7db773\") " pod="opendatahub/odh-model-controller-7998c67dc4-sgg44" Jul 27 18:35:54.518616 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:35:54.518594 2579 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Jul 27 18:35:54.518673 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:35:54.518663 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87423332-edfc-4e36-a11b-56324b7db773-cert podName:87423332-edfc-4e36-a11b-56324b7db773 nodeName:}" failed. No retries permitted until 2026-07-27 18:35:55.518648179 +0000 UTC m=+430.580337188 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/87423332-edfc-4e36-a11b-56324b7db773-cert") pod "odh-model-controller-7998c67dc4-sgg44" (UID: "87423332-edfc-4e36-a11b-56324b7db773") : secret "odh-model-controller-webhook-cert" not found Jul 27 18:35:54.579887 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:35:54.579863 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7896769b64-tkbg9"] Jul 27 18:35:54.582711 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:35:54.582683 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bfb0e76_8207_48bb_810a_6f865001eeb1.slice/crio-b7f423ded4eeb47635a615696b93b8c19d542c0d5a4f5299213b7e6af963fe30 WatchSource:0}: Error finding container b7f423ded4eeb47635a615696b93b8c19d542c0d5a4f5299213b7e6af963fe30: Status 404 returned error can't find the container with id b7f423ded4eeb47635a615696b93b8c19d542c0d5a4f5299213b7e6af963fe30 Jul 27 18:35:54.620316 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:35:54.620286 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vhv7p"] Jul 27 18:35:54.624673 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:35:54.624658 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vhv7p" Jul 27 18:35:54.627714 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:35:54.627689 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 27 18:35:54.627847 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:35:54.627724 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 27 18:35:54.627847 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:35:54.627783 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-mw2c8\"" Jul 27 18:35:54.628030 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:35:54.628011 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 27 18:35:54.628138 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:35:54.628054 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 27 18:35:54.642450 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:35:54.642425 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:35:54.645945 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:35:54.645924 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vhv7p"] Jul 27 18:35:54.719528 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:35:54.719448 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0979e81f-93d2-43c3-be01-8355c6c4ccc2-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-vhv7p\" (UID: \"0979e81f-93d2-43c3-be01-8355c6c4ccc2\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vhv7p" Jul 27 18:35:54.719528 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:35:54.719490 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0979e81f-93d2-43c3-be01-8355c6c4ccc2-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-vhv7p\" (UID: \"0979e81f-93d2-43c3-be01-8355c6c4ccc2\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vhv7p" Jul 27 18:35:54.719528 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:35:54.719539 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0979e81f-93d2-43c3-be01-8355c6c4ccc2-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-vhv7p\" (UID: \"0979e81f-93d2-43c3-be01-8355c6c4ccc2\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vhv7p" Jul 27 18:35:54.719790 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:35:54.719560 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0979e81f-93d2-43c3-be01-8355c6c4ccc2-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-vhv7p\" (UID: \"0979e81f-93d2-43c3-be01-8355c6c4ccc2\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vhv7p" Jul 27 18:35:54.719790 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:35:54.719584 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0979e81f-93d2-43c3-be01-8355c6c4ccc2-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-vhv7p\" (UID: \"0979e81f-93d2-43c3-be01-8355c6c4ccc2\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vhv7p" Jul 27 18:35:54.719790 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:35:54.719657 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0979e81f-93d2-43c3-be01-8355c6c4ccc2-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-vhv7p\" (UID: \"0979e81f-93d2-43c3-be01-8355c6c4ccc2\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vhv7p" Jul 27 18:35:54.719790 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:35:54.719726 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6g6fg\" (UniqueName: \"kubernetes.io/projected/0979e81f-93d2-43c3-be01-8355c6c4ccc2-kube-api-access-6g6fg\") pod \"istiod-openshift-gateway-7b7995d5cd-vhv7p\" (UID: \"0979e81f-93d2-43c3-be01-8355c6c4ccc2\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vhv7p" Jul 27 18:35:54.728006 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:35:54.727974 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:35:54.820129 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:35:54.820103 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0979e81f-93d2-43c3-be01-8355c6c4ccc2-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-vhv7p\" (UID: \"0979e81f-93d2-43c3-be01-8355c6c4ccc2\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vhv7p" Jul 27 18:35:54.820315 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:35:54.820146 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0979e81f-93d2-43c3-be01-8355c6c4ccc2-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-vhv7p\" (UID: \"0979e81f-93d2-43c3-be01-8355c6c4ccc2\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vhv7p" Jul 27 18:35:54.820315 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:35:54.820167 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0979e81f-93d2-43c3-be01-8355c6c4ccc2-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-vhv7p\" (UID: \"0979e81f-93d2-43c3-be01-8355c6c4ccc2\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vhv7p" Jul 27 18:35:54.820315 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:35:54.820245 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0979e81f-93d2-43c3-be01-8355c6c4ccc2-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-vhv7p\" (UID: \"0979e81f-93d2-43c3-be01-8355c6c4ccc2\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vhv7p" Jul 27 18:35:54.820315 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:35:54.820292 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0979e81f-93d2-43c3-be01-8355c6c4ccc2-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-vhv7p\" (UID: \"0979e81f-93d2-43c3-be01-8355c6c4ccc2\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vhv7p" Jul 27 18:35:54.820523 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:35:54.820332 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6g6fg\" (UniqueName: \"kubernetes.io/projected/0979e81f-93d2-43c3-be01-8355c6c4ccc2-kube-api-access-6g6fg\") pod \"istiod-openshift-gateway-7b7995d5cd-vhv7p\" (UID: \"0979e81f-93d2-43c3-be01-8355c6c4ccc2\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vhv7p" Jul 27 18:35:54.820523 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:35:54.820372 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0979e81f-93d2-43c3-be01-8355c6c4ccc2-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-vhv7p\" (UID: \"0979e81f-93d2-43c3-be01-8355c6c4ccc2\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vhv7p" Jul 27 18:35:54.821208 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:35:54.821180 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0979e81f-93d2-43c3-be01-8355c6c4ccc2-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-vhv7p\" (UID: \"0979e81f-93d2-43c3-be01-8355c6c4ccc2\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vhv7p" Jul 27 18:35:54.822736 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:35:54.822709 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0979e81f-93d2-43c3-be01-8355c6c4ccc2-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-vhv7p\" (UID: \"0979e81f-93d2-43c3-be01-8355c6c4ccc2\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vhv7p" Jul 27 18:35:54.822837 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:35:54.822756 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0979e81f-93d2-43c3-be01-8355c6c4ccc2-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-vhv7p\" (UID: \"0979e81f-93d2-43c3-be01-8355c6c4ccc2\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vhv7p" Jul 27 18:35:54.822837 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:35:54.822765 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0979e81f-93d2-43c3-be01-8355c6c4ccc2-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-vhv7p\" (UID: \"0979e81f-93d2-43c3-be01-8355c6c4ccc2\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vhv7p" Jul 27 18:35:54.822912 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:35:54.822830 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0979e81f-93d2-43c3-be01-8355c6c4ccc2-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-vhv7p\" (UID: \"0979e81f-93d2-43c3-be01-8355c6c4ccc2\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vhv7p" Jul 27 18:35:54.831789 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:35:54.831768 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6g6fg\" (UniqueName: \"kubernetes.io/projected/0979e81f-93d2-43c3-be01-8355c6c4ccc2-kube-api-access-6g6fg\") pod \"istiod-openshift-gateway-7b7995d5cd-vhv7p\" (UID: \"0979e81f-93d2-43c3-be01-8355c6c4ccc2\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vhv7p" Jul 27 18:35:54.832680 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:35:54.832662 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0979e81f-93d2-43c3-be01-8355c6c4ccc2-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-vhv7p\" (UID: \"0979e81f-93d2-43c3-be01-8355c6c4ccc2\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vhv7p" Jul 27 18:35:54.835660 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:35:54.835627 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:35:54.929284 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:35:54.929216 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:35:54.934151 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:35:54.934131 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vhv7p" Jul 27 18:35:55.069256 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:35:55.069203 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vhv7p"] Jul 27 18:35:55.072378 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:35:55.072345 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0979e81f_93d2_43c3_be01_8355c6c4ccc2.slice/crio-3b573709a6545648eaac4f3fdc4b4c8738094d0ffd2d848c33a68c629c5842ab WatchSource:0}: Error finding container 3b573709a6545648eaac4f3fdc4b4c8738094d0ffd2d848c33a68c629c5842ab: Status 404 returned error can't find the container with id 3b573709a6545648eaac4f3fdc4b4c8738094d0ffd2d848c33a68c629c5842ab Jul 27 18:35:55.172224 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:35:55.172196 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:35:55.336509 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:35:55.336473 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vhv7p" event={"ID":"0979e81f-93d2-43c3-be01-8355c6c4ccc2","Type":"ContainerStarted","Data":"3b573709a6545648eaac4f3fdc4b4c8738094d0ffd2d848c33a68c629c5842ab"} Jul 27 18:35:55.337509 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:35:55.337484 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7896769b64-tkbg9" event={"ID":"2bfb0e76-8207-48bb-810a-6f865001eeb1","Type":"ContainerStarted","Data":"b7f423ded4eeb47635a615696b93b8c19d542c0d5a4f5299213b7e6af963fe30"} Jul 27 18:35:55.526135 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:35:55.526101 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/87423332-edfc-4e36-a11b-56324b7db773-cert\") pod \"odh-model-controller-7998c67dc4-sgg44\" (UID: \"87423332-edfc-4e36-a11b-56324b7db773\") " pod="opendatahub/odh-model-controller-7998c67dc4-sgg44" Jul 27 18:35:55.528653 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:35:55.528633 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/87423332-edfc-4e36-a11b-56324b7db773-cert\") pod \"odh-model-controller-7998c67dc4-sgg44\" (UID: \"87423332-edfc-4e36-a11b-56324b7db773\") " pod="opendatahub/odh-model-controller-7998c67dc4-sgg44" Jul 27 18:35:55.624014 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:35:55.623805 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-sgg44" Jul 27 18:35:55.778089 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:35:55.778046 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-sgg44"] Jul 27 18:35:55.781594 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:35:55.781556 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87423332_edfc_4e36_a11b_56324b7db773.slice/crio-8ad7201e69bdcf9285a4f0ec9b6ab998a8e6ac92f7e6230b6dc039e215593fe3 WatchSource:0}: Error finding container 8ad7201e69bdcf9285a4f0ec9b6ab998a8e6ac92f7e6230b6dc039e215593fe3: Status 404 returned error can't find the container with id 8ad7201e69bdcf9285a4f0ec9b6ab998a8e6ac92f7e6230b6dc039e215593fe3 Jul 27 18:35:55.855078 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:35:55.855024 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:35:56.343964 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:35:56.343915 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-sgg44" event={"ID":"87423332-edfc-4e36-a11b-56324b7db773","Type":"ContainerStarted","Data":"8ad7201e69bdcf9285a4f0ec9b6ab998a8e6ac92f7e6230b6dc039e215593fe3"} Jul 27 18:35:58.358737 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:35:58.358699 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7896769b64-tkbg9" event={"ID":"2bfb0e76-8207-48bb-810a-6f865001eeb1","Type":"ContainerStarted","Data":"8fae3c0c9f7ff9153a477fc1321a6870e15eace5be940f12835bbc53ffc2f86a"} Jul 27 18:35:58.380483 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:35:58.380417 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7896769b64-tkbg9" podStartSLOduration=2.530509222 podStartE2EDuration="5.380394797s" totalImagesPullingTime="2.849885575s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:35:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:35:58.379968582 +0000 UTC m=+433.441657614" watchObservedRunningTime="2026-07-27 18:35:58.380394797 +0000 UTC m=+433.442083831" Jul 27 18:35:59.365898 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:35:59.365808 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-sgg44" event={"ID":"87423332-edfc-4e36-a11b-56324b7db773","Type":"ContainerStarted","Data":"f5bd80946eb383f1d43792a8cdddd48f758b8b2f5a7fa15dec4a83c958c3ce6a"} Jul 27 18:35:59.366374 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:35:59.366087 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-sgg44" Jul 27 18:35:59.386189 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:35:59.386132 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-sgg44" podStartSLOduration=3.08857591 podStartE2EDuration="6.386109234s" totalImagesPullingTime="3.297533324s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:35:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:35:59.385451951 +0000 UTC m=+434.447140981" watchObservedRunningTime="2026-07-27 18:35:59.386109234 +0000 UTC m=+434.447798282" Jul 27 18:36:03.789825 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:03.789765 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:36:03.868332 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:03.868296 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-z4tz5"] Jul 27 18:36:03.871953 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:03.871938 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-z4tz5" Jul 27 18:36:03.875739 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:03.875717 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 27 18:36:03.876013 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:03.875994 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-zlwxd\"" Jul 27 18:36:03.878724 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:03.878705 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-w9d4l"] Jul 27 18:36:03.881777 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:03.881757 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-w9d4l" Jul 27 18:36:03.882712 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:03.882693 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-z4tz5"] Jul 27 18:36:03.884561 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:03.884544 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 27 18:36:03.884753 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:03.884735 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-d5vgr\"" Jul 27 18:36:03.891642 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:03.891615 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:36:03.899662 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:03.899643 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-w9d4l"] Jul 27 18:36:04.000652 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:04.000615 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ebf77b9d-9e5d-47cd-aa09-ea05ff63234a-cert\") pod \"kserve-controller-manager-d6f4b65b8-z4tz5\" (UID: \"ebf77b9d-9e5d-47cd-aa09-ea05ff63234a\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-z4tz5" Jul 27 18:36:04.000810 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:04.000663 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d97hl\" (UniqueName: \"kubernetes.io/projected/52b6da00-246b-4954-adcc-5b482d1c41bc-kube-api-access-d97hl\") pod \"llmisvc-controller-manager-777c6bd7d5-w9d4l\" (UID: \"52b6da00-246b-4954-adcc-5b482d1c41bc\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-w9d4l" Jul 27 18:36:04.000810 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:04.000731 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/52b6da00-246b-4954-adcc-5b482d1c41bc-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-w9d4l\" (UID: \"52b6da00-246b-4954-adcc-5b482d1c41bc\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-w9d4l" Jul 27 18:36:04.000810 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:04.000759 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hd8nt\" (UniqueName: \"kubernetes.io/projected/ebf77b9d-9e5d-47cd-aa09-ea05ff63234a-kube-api-access-hd8nt\") pod \"kserve-controller-manager-d6f4b65b8-z4tz5\" (UID: \"ebf77b9d-9e5d-47cd-aa09-ea05ff63234a\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-z4tz5" Jul 27 18:36:04.102010 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:04.101925 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d97hl\" (UniqueName: \"kubernetes.io/projected/52b6da00-246b-4954-adcc-5b482d1c41bc-kube-api-access-d97hl\") pod \"llmisvc-controller-manager-777c6bd7d5-w9d4l\" (UID: \"52b6da00-246b-4954-adcc-5b482d1c41bc\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-w9d4l" Jul 27 18:36:04.102010 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:04.101992 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/52b6da00-246b-4954-adcc-5b482d1c41bc-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-w9d4l\" (UID: \"52b6da00-246b-4954-adcc-5b482d1c41bc\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-w9d4l" Jul 27 18:36:04.102177 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:04.102048 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hd8nt\" (UniqueName: \"kubernetes.io/projected/ebf77b9d-9e5d-47cd-aa09-ea05ff63234a-kube-api-access-hd8nt\") pod \"kserve-controller-manager-d6f4b65b8-z4tz5\" (UID: \"ebf77b9d-9e5d-47cd-aa09-ea05ff63234a\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-z4tz5" Jul 27 18:36:04.102177 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:04.102080 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ebf77b9d-9e5d-47cd-aa09-ea05ff63234a-cert\") pod \"kserve-controller-manager-d6f4b65b8-z4tz5\" (UID: \"ebf77b9d-9e5d-47cd-aa09-ea05ff63234a\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-z4tz5" Jul 27 18:36:04.102177 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:36:04.102092 2579 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found Jul 27 18:36:04.102177 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:36:04.102160 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/52b6da00-246b-4954-adcc-5b482d1c41bc-cert podName:52b6da00-246b-4954-adcc-5b482d1c41bc nodeName:}" failed. No retries permitted until 2026-07-27 18:36:04.602140824 +0000 UTC m=+439.663829835 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/52b6da00-246b-4954-adcc-5b482d1c41bc-cert") pod "llmisvc-controller-manager-777c6bd7d5-w9d4l" (UID: "52b6da00-246b-4954-adcc-5b482d1c41bc") : secret "llmisvc-webhook-server-cert" not found Jul 27 18:36:04.102357 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:36:04.102177 2579 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found Jul 27 18:36:04.102357 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:36:04.102222 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ebf77b9d-9e5d-47cd-aa09-ea05ff63234a-cert podName:ebf77b9d-9e5d-47cd-aa09-ea05ff63234a nodeName:}" failed. No retries permitted until 2026-07-27 18:36:04.602205957 +0000 UTC m=+439.663894966 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/ebf77b9d-9e5d-47cd-aa09-ea05ff63234a-cert") pod "kserve-controller-manager-d6f4b65b8-z4tz5" (UID: "ebf77b9d-9e5d-47cd-aa09-ea05ff63234a") : secret "kserve-webhook-server-cert" not found Jul 27 18:36:04.116011 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:04.115981 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d97hl\" (UniqueName: \"kubernetes.io/projected/52b6da00-246b-4954-adcc-5b482d1c41bc-kube-api-access-d97hl\") pod \"llmisvc-controller-manager-777c6bd7d5-w9d4l\" (UID: \"52b6da00-246b-4954-adcc-5b482d1c41bc\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-w9d4l" Jul 27 18:36:04.119675 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:04.119653 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hd8nt\" (UniqueName: \"kubernetes.io/projected/ebf77b9d-9e5d-47cd-aa09-ea05ff63234a-kube-api-access-hd8nt\") pod \"kserve-controller-manager-d6f4b65b8-z4tz5\" (UID: \"ebf77b9d-9e5d-47cd-aa09-ea05ff63234a\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-z4tz5" Jul 27 18:36:04.265117 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:04.265066 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:36:04.363528 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:04.363437 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:36:04.606324 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:04.606184 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/52b6da00-246b-4954-adcc-5b482d1c41bc-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-w9d4l\" (UID: \"52b6da00-246b-4954-adcc-5b482d1c41bc\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-w9d4l" Jul 27 18:36:04.606324 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:04.606286 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ebf77b9d-9e5d-47cd-aa09-ea05ff63234a-cert\") pod \"kserve-controller-manager-d6f4b65b8-z4tz5\" (UID: \"ebf77b9d-9e5d-47cd-aa09-ea05ff63234a\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-z4tz5" Jul 27 18:36:04.609348 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:04.609299 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/52b6da00-246b-4954-adcc-5b482d1c41bc-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-w9d4l\" (UID: \"52b6da00-246b-4954-adcc-5b482d1c41bc\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-w9d4l" Jul 27 18:36:04.609481 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:04.609364 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ebf77b9d-9e5d-47cd-aa09-ea05ff63234a-cert\") pod \"kserve-controller-manager-d6f4b65b8-z4tz5\" (UID: \"ebf77b9d-9e5d-47cd-aa09-ea05ff63234a\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-z4tz5" Jul 27 18:36:04.782413 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:04.782377 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-z4tz5" Jul 27 18:36:04.790970 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:04.790944 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-w9d4l" Jul 27 18:36:04.997820 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:04.997784 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-w9d4l"] Jul 27 18:36:05.001139 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:36:05.001106 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod52b6da00_246b_4954_adcc_5b482d1c41bc.slice/crio-4c33904051f5c6182efb882bb37c9f1ff2d8cec81ee051a447c31e379335f47d WatchSource:0}: Error finding container 4c33904051f5c6182efb882bb37c9f1ff2d8cec81ee051a447c31e379335f47d: Status 404 returned error can't find the container with id 4c33904051f5c6182efb882bb37c9f1ff2d8cec81ee051a447c31e379335f47d Jul 27 18:36:05.010634 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:05.010606 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-z4tz5"] Jul 27 18:36:05.012981 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:36:05.012956 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebf77b9d_9e5d_47cd_aa09_ea05ff63234a.slice/crio-b913351a48ae08e178254fa208341d4eb5704389d1bd67fee7bf7aba07d08556 WatchSource:0}: Error finding container b913351a48ae08e178254fa208341d4eb5704389d1bd67fee7bf7aba07d08556: Status 404 returned error can't find the container with id b913351a48ae08e178254fa208341d4eb5704389d1bd67fee7bf7aba07d08556 Jul 27 18:36:05.077315 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:05.077204 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:36:05.181278 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:05.181243 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:36:05.296873 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:05.296822 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:36:05.399843 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:05.399756 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-w9d4l" event={"ID":"52b6da00-246b-4954-adcc-5b482d1c41bc","Type":"ContainerStarted","Data":"4c33904051f5c6182efb882bb37c9f1ff2d8cec81ee051a447c31e379335f47d"} Jul 27 18:36:05.401138 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:05.401104 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-z4tz5" event={"ID":"ebf77b9d-9e5d-47cd-aa09-ea05ff63234a","Type":"ContainerStarted","Data":"b913351a48ae08e178254fa208341d4eb5704389d1bd67fee7bf7aba07d08556"} Jul 27 18:36:06.077337 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:06.077294 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 27 18:36:06.077613 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:06.077371 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 27 18:36:06.311538 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:06.311496 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:36:06.407871 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:06.407760 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vhv7p" event={"ID":"0979e81f-93d2-43c3-be01-8355c6c4ccc2","Type":"ContainerStarted","Data":"5fb80ed55f9e2ed75884cdb87fa515ce109e1ef3bc17c457ed542dc568b2c8e5"} Jul 27 18:36:06.408027 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:06.407908 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vhv7p" Jul 27 18:36:06.439858 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:06.439824 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:36:06.443635 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:06.443595 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vhv7p" podStartSLOduration=1.440915429 podStartE2EDuration="12.443581982s" totalImagesPullingTime="11.002666553s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:35:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:36:06.441912441 +0000 UTC m=+441.503601483" watchObservedRunningTime="2026-07-27 18:36:06.443581982 +0000 UTC m=+441.505271012" Jul 27 18:36:06.656410 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:06.656379 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-w6ltt"] Jul 27 18:36:06.660190 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:06.660065 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-w6ltt" Jul 27 18:36:06.663673 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:06.663641 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-qsr4x\"" Jul 27 18:36:06.674522 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:06.674499 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-w6ltt"] Jul 27 18:36:06.691078 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:06.691011 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:36:06.824280 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:06.824196 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xth7m\" (UniqueName: \"kubernetes.io/projected/2f32486f-5cb4-48a9-96e0-6ee08e76933a-kube-api-access-xth7m\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-w6ltt\" (UID: \"2f32486f-5cb4-48a9-96e0-6ee08e76933a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-w6ltt" Jul 27 18:36:06.824455 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:06.824287 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f32486f-5cb4-48a9-96e0-6ee08e76933a-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-w6ltt\" (UID: \"2f32486f-5cb4-48a9-96e0-6ee08e76933a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-w6ltt" Jul 27 18:36:06.824455 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:06.824319 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f32486f-5cb4-48a9-96e0-6ee08e76933a-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-w6ltt\" (UID: \"2f32486f-5cb4-48a9-96e0-6ee08e76933a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-w6ltt" Jul 27 18:36:06.824455 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:06.824357 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f32486f-5cb4-48a9-96e0-6ee08e76933a-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-w6ltt\" (UID: \"2f32486f-5cb4-48a9-96e0-6ee08e76933a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-w6ltt" Jul 27 18:36:06.824598 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:06.824460 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f32486f-5cb4-48a9-96e0-6ee08e76933a-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-w6ltt\" (UID: \"2f32486f-5cb4-48a9-96e0-6ee08e76933a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-w6ltt" Jul 27 18:36:06.824598 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:06.824488 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f32486f-5cb4-48a9-96e0-6ee08e76933a-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-w6ltt\" (UID: \"2f32486f-5cb4-48a9-96e0-6ee08e76933a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-w6ltt" Jul 27 18:36:06.824598 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:06.824583 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f32486f-5cb4-48a9-96e0-6ee08e76933a-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-w6ltt\" (UID: \"2f32486f-5cb4-48a9-96e0-6ee08e76933a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-w6ltt" Jul 27 18:36:06.824726 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:06.824618 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f32486f-5cb4-48a9-96e0-6ee08e76933a-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-w6ltt\" (UID: \"2f32486f-5cb4-48a9-96e0-6ee08e76933a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-w6ltt" Jul 27 18:36:06.824726 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:06.824651 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f32486f-5cb4-48a9-96e0-6ee08e76933a-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-w6ltt\" (UID: \"2f32486f-5cb4-48a9-96e0-6ee08e76933a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-w6ltt" Jul 27 18:36:06.925173 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:06.925073 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f32486f-5cb4-48a9-96e0-6ee08e76933a-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-w6ltt\" (UID: \"2f32486f-5cb4-48a9-96e0-6ee08e76933a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-w6ltt" Jul 27 18:36:06.925173 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:06.925128 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f32486f-5cb4-48a9-96e0-6ee08e76933a-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-w6ltt\" (UID: \"2f32486f-5cb4-48a9-96e0-6ee08e76933a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-w6ltt" Jul 27 18:36:06.925432 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:06.925182 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f32486f-5cb4-48a9-96e0-6ee08e76933a-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-w6ltt\" (UID: \"2f32486f-5cb4-48a9-96e0-6ee08e76933a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-w6ltt" Jul 27 18:36:06.925432 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:06.925206 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f32486f-5cb4-48a9-96e0-6ee08e76933a-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-w6ltt\" (UID: \"2f32486f-5cb4-48a9-96e0-6ee08e76933a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-w6ltt" Jul 27 18:36:06.925432 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:06.925254 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f32486f-5cb4-48a9-96e0-6ee08e76933a-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-w6ltt\" (UID: \"2f32486f-5cb4-48a9-96e0-6ee08e76933a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-w6ltt" Jul 27 18:36:06.925432 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:06.925288 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xth7m\" (UniqueName: \"kubernetes.io/projected/2f32486f-5cb4-48a9-96e0-6ee08e76933a-kube-api-access-xth7m\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-w6ltt\" (UID: \"2f32486f-5cb4-48a9-96e0-6ee08e76933a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-w6ltt" Jul 27 18:36:06.925432 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:06.925326 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f32486f-5cb4-48a9-96e0-6ee08e76933a-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-w6ltt\" (UID: \"2f32486f-5cb4-48a9-96e0-6ee08e76933a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-w6ltt" Jul 27 18:36:06.925432 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:06.925349 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f32486f-5cb4-48a9-96e0-6ee08e76933a-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-w6ltt\" (UID: \"2f32486f-5cb4-48a9-96e0-6ee08e76933a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-w6ltt" Jul 27 18:36:06.925432 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:06.925386 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f32486f-5cb4-48a9-96e0-6ee08e76933a-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-w6ltt\" (UID: \"2f32486f-5cb4-48a9-96e0-6ee08e76933a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-w6ltt" Jul 27 18:36:06.925790 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:06.925532 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f32486f-5cb4-48a9-96e0-6ee08e76933a-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-w6ltt\" (UID: \"2f32486f-5cb4-48a9-96e0-6ee08e76933a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-w6ltt" Jul 27 18:36:06.925846 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:06.925787 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f32486f-5cb4-48a9-96e0-6ee08e76933a-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-w6ltt\" (UID: \"2f32486f-5cb4-48a9-96e0-6ee08e76933a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-w6ltt" Jul 27 18:36:06.925921 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:06.925899 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f32486f-5cb4-48a9-96e0-6ee08e76933a-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-w6ltt\" (UID: \"2f32486f-5cb4-48a9-96e0-6ee08e76933a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-w6ltt" Jul 27 18:36:06.926055 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:06.926034 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f32486f-5cb4-48a9-96e0-6ee08e76933a-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-w6ltt\" (UID: \"2f32486f-5cb4-48a9-96e0-6ee08e76933a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-w6ltt" Jul 27 18:36:06.926663 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:06.926629 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f32486f-5cb4-48a9-96e0-6ee08e76933a-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-w6ltt\" (UID: \"2f32486f-5cb4-48a9-96e0-6ee08e76933a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-w6ltt" Jul 27 18:36:06.929997 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:06.929468 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f32486f-5cb4-48a9-96e0-6ee08e76933a-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-w6ltt\" (UID: \"2f32486f-5cb4-48a9-96e0-6ee08e76933a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-w6ltt" Jul 27 18:36:06.929997 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:06.929760 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f32486f-5cb4-48a9-96e0-6ee08e76933a-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-w6ltt\" (UID: \"2f32486f-5cb4-48a9-96e0-6ee08e76933a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-w6ltt" Jul 27 18:36:06.939445 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:06.939367 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:36:06.942979 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:06.942953 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xth7m\" (UniqueName: \"kubernetes.io/projected/2f32486f-5cb4-48a9-96e0-6ee08e76933a-kube-api-access-xth7m\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-w6ltt\" (UID: \"2f32486f-5cb4-48a9-96e0-6ee08e76933a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-w6ltt" Jul 27 18:36:06.948528 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:06.948479 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f32486f-5cb4-48a9-96e0-6ee08e76933a-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-w6ltt\" (UID: \"2f32486f-5cb4-48a9-96e0-6ee08e76933a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-w6ltt" Jul 27 18:36:06.974547 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:06.974501 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-w6ltt" Jul 27 18:36:07.038054 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:07.037995 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:36:07.416108 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:07.416079 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vhv7p" Jul 27 18:36:07.654482 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:07.654387 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-w6ltt"] Jul 27 18:36:07.657909 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:36:07.657880 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f32486f_5cb4_48a9_96e0_6ee08e76933a.slice/crio-ed7c5d2ff662ebc90a235b16dd92467862bf21f8450aa36f0f0cb376e72584a2 WatchSource:0}: Error finding container ed7c5d2ff662ebc90a235b16dd92467862bf21f8450aa36f0f0cb376e72584a2: Status 404 returned error can't find the container with id ed7c5d2ff662ebc90a235b16dd92467862bf21f8450aa36f0f0cb376e72584a2 Jul 27 18:36:08.420140 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:08.420097 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-w6ltt" event={"ID":"2f32486f-5cb4-48a9-96e0-6ee08e76933a","Type":"ContainerStarted","Data":"ed7c5d2ff662ebc90a235b16dd92467862bf21f8450aa36f0f0cb376e72584a2"} Jul 27 18:36:09.074447 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:09.074415 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vhv7p"] Jul 27 18:36:09.426894 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:09.426809 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-w9d4l" event={"ID":"52b6da00-246b-4954-adcc-5b482d1c41bc","Type":"ContainerStarted","Data":"23e6e786609d3f889ec82e3abdf5af5616a04dd90d3d2d9d1a685ae1119b805e"} Jul 27 18:36:09.427319 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:09.427012 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-w9d4l" Jul 27 18:36:09.429339 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:09.429305 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-z4tz5" event={"ID":"ebf77b9d-9e5d-47cd-aa09-ea05ff63234a","Type":"ContainerStarted","Data":"61d2e60758b8a5f76274c3933b326420b863f4ff7ef821d2671c556eb55068c6"} Jul 27 18:36:09.429455 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:09.429378 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-z4tz5" Jul 27 18:36:09.429574 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:09.429546 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vhv7p" podUID="0979e81f-93d2-43c3-be01-8355c6c4ccc2" containerName="discovery" containerID="cri-o://5fb80ed55f9e2ed75884cdb87fa515ce109e1ef3bc17c457ed542dc568b2c8e5" gracePeriod=30 Jul 27 18:36:09.454318 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:09.453902 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-w9d4l" podStartSLOduration=2.329152645 podStartE2EDuration="6.453883011s" totalImagesPullingTime="4.124730366s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:36:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:36:09.453796953 +0000 UTC m=+444.515485999" watchObservedRunningTime="2026-07-27 18:36:09.453883011 +0000 UTC m=+444.515572042" Jul 27 18:36:09.483034 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:09.482988 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-z4tz5" podStartSLOduration=2.373041269 podStartE2EDuration="6.482972649s" totalImagesPullingTime="4.10993138s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:36:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:36:09.481806411 +0000 UTC m=+444.543495443" watchObservedRunningTime="2026-07-27 18:36:09.482972649 +0000 UTC m=+444.544661658" Jul 27 18:36:10.372374 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:10.372339 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-sgg44" Jul 27 18:36:10.415157 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:10.415079 2579 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-vhv7p container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.24:8080/ready\": dial tcp 10.132.0.24:8080: connect: connection refused" start-of-body= Jul 27 18:36:10.415337 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:10.415156 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vhv7p" podUID="0979e81f-93d2-43c3-be01-8355c6c4ccc2" containerName="discovery" probeResult="failure" output="Get \"http://10.132.0.24:8080/ready\": dial tcp 10.132.0.24:8080: connect: connection refused" Jul 27 18:36:10.439495 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:10.439462 2579 generic.go:356] "Generic (PLEG): container finished" podID="0979e81f-93d2-43c3-be01-8355c6c4ccc2" containerID="5fb80ed55f9e2ed75884cdb87fa515ce109e1ef3bc17c457ed542dc568b2c8e5" exitCode=0 Jul 27 18:36:10.439942 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:10.439571 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vhv7p" event={"ID":"0979e81f-93d2-43c3-be01-8355c6c4ccc2","Type":"ContainerDied","Data":"5fb80ed55f9e2ed75884cdb87fa515ce109e1ef3bc17c457ed542dc568b2c8e5"} Jul 27 18:36:10.584497 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:10.584459 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 27 18:36:10.584571 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:10.584536 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 27 18:36:10.584609 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:10.584572 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 27 18:36:10.661414 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:10.661388 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vhv7p" Jul 27 18:36:10.766289 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:10.766256 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0979e81f-93d2-43c3-be01-8355c6c4ccc2-istio-csr-ca-configmap\") pod \"0979e81f-93d2-43c3-be01-8355c6c4ccc2\" (UID: \"0979e81f-93d2-43c3-be01-8355c6c4ccc2\") " Jul 27 18:36:10.766486 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:10.766339 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0979e81f-93d2-43c3-be01-8355c6c4ccc2-istio-token\") pod \"0979e81f-93d2-43c3-be01-8355c6c4ccc2\" (UID: \"0979e81f-93d2-43c3-be01-8355c6c4ccc2\") " Jul 27 18:36:10.766486 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:10.766373 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0979e81f-93d2-43c3-be01-8355c6c4ccc2-istio-csr-dns-cert\") pod \"0979e81f-93d2-43c3-be01-8355c6c4ccc2\" (UID: \"0979e81f-93d2-43c3-be01-8355c6c4ccc2\") " Jul 27 18:36:10.766486 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:10.766420 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0979e81f-93d2-43c3-be01-8355c6c4ccc2-cacerts\") pod \"0979e81f-93d2-43c3-be01-8355c6c4ccc2\" (UID: \"0979e81f-93d2-43c3-be01-8355c6c4ccc2\") " Jul 27 18:36:10.766486 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:10.766448 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0979e81f-93d2-43c3-be01-8355c6c4ccc2-local-certs\") pod \"0979e81f-93d2-43c3-be01-8355c6c4ccc2\" (UID: \"0979e81f-93d2-43c3-be01-8355c6c4ccc2\") " Jul 27 18:36:10.766690 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:10.766489 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0979e81f-93d2-43c3-be01-8355c6c4ccc2-istio-kubeconfig\") pod \"0979e81f-93d2-43c3-be01-8355c6c4ccc2\" (UID: \"0979e81f-93d2-43c3-be01-8355c6c4ccc2\") " Jul 27 18:36:10.766690 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:10.766531 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6g6fg\" (UniqueName: \"kubernetes.io/projected/0979e81f-93d2-43c3-be01-8355c6c4ccc2-kube-api-access-6g6fg\") pod \"0979e81f-93d2-43c3-be01-8355c6c4ccc2\" (UID: \"0979e81f-93d2-43c3-be01-8355c6c4ccc2\") " Jul 27 18:36:10.766810 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:10.766780 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0979e81f-93d2-43c3-be01-8355c6c4ccc2-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "0979e81f-93d2-43c3-be01-8355c6c4ccc2" (UID: "0979e81f-93d2-43c3-be01-8355c6c4ccc2"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 18:36:10.769652 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:10.769621 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0979e81f-93d2-43c3-be01-8355c6c4ccc2-local-certs" (OuterVolumeSpecName: "local-certs") pod "0979e81f-93d2-43c3-be01-8355c6c4ccc2" (UID: "0979e81f-93d2-43c3-be01-8355c6c4ccc2"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 18:36:10.769775 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:10.769671 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0979e81f-93d2-43c3-be01-8355c6c4ccc2-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "0979e81f-93d2-43c3-be01-8355c6c4ccc2" (UID: "0979e81f-93d2-43c3-be01-8355c6c4ccc2"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 18:36:10.769951 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:10.769925 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0979e81f-93d2-43c3-be01-8355c6c4ccc2-cacerts" (OuterVolumeSpecName: "cacerts") pod "0979e81f-93d2-43c3-be01-8355c6c4ccc2" (UID: "0979e81f-93d2-43c3-be01-8355c6c4ccc2"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 18:36:10.770025 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:10.769988 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0979e81f-93d2-43c3-be01-8355c6c4ccc2-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "0979e81f-93d2-43c3-be01-8355c6c4ccc2" (UID: "0979e81f-93d2-43c3-be01-8355c6c4ccc2"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 18:36:10.770079 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:10.770042 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0979e81f-93d2-43c3-be01-8355c6c4ccc2-kube-api-access-6g6fg" (OuterVolumeSpecName: "kube-api-access-6g6fg") pod "0979e81f-93d2-43c3-be01-8355c6c4ccc2" (UID: "0979e81f-93d2-43c3-be01-8355c6c4ccc2"). InnerVolumeSpecName "kube-api-access-6g6fg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 18:36:10.770397 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:10.770377 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0979e81f-93d2-43c3-be01-8355c6c4ccc2-istio-token" (OuterVolumeSpecName: "istio-token") pod "0979e81f-93d2-43c3-be01-8355c6c4ccc2" (UID: "0979e81f-93d2-43c3-be01-8355c6c4ccc2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 18:36:10.867907 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:10.867873 2579 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0979e81f-93d2-43c3-be01-8355c6c4ccc2-istio-csr-ca-configmap\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 18:36:10.867907 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:10.867902 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0979e81f-93d2-43c3-be01-8355c6c4ccc2-istio-token\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 18:36:10.867907 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:10.867911 2579 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0979e81f-93d2-43c3-be01-8355c6c4ccc2-istio-csr-dns-cert\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 18:36:10.868112 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:10.867920 2579 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0979e81f-93d2-43c3-be01-8355c6c4ccc2-cacerts\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 18:36:10.868112 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:10.867929 2579 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0979e81f-93d2-43c3-be01-8355c6c4ccc2-local-certs\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 18:36:10.868112 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:10.867937 2579 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0979e81f-93d2-43c3-be01-8355c6c4ccc2-istio-kubeconfig\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 18:36:10.868112 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:10.867944 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6g6fg\" (UniqueName: \"kubernetes.io/projected/0979e81f-93d2-43c3-be01-8355c6c4ccc2-kube-api-access-6g6fg\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 18:36:11.451057 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:11.451024 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vhv7p" Jul 27 18:36:11.451573 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:11.451019 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vhv7p" event={"ID":"0979e81f-93d2-43c3-be01-8355c6c4ccc2","Type":"ContainerDied","Data":"3b573709a6545648eaac4f3fdc4b4c8738094d0ffd2d848c33a68c629c5842ab"} Jul 27 18:36:11.451573 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:11.451222 2579 scope.go:117] "RemoveContainer" containerID="5fb80ed55f9e2ed75884cdb87fa515ce109e1ef3bc17c457ed542dc568b2c8e5" Jul 27 18:36:11.453004 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:11.452960 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-w6ltt" event={"ID":"2f32486f-5cb4-48a9-96e0-6ee08e76933a","Type":"ContainerStarted","Data":"6de9236cb6bbb7c88aa6412d37ef7496b7c3df709de0236c5c1e586070fff1c3"} Jul 27 18:36:11.481340 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:11.481199 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-w6ltt" podStartSLOduration=2.557669208 podStartE2EDuration="5.481185726s" totalImagesPullingTime="2.923516518s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:36:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:36:11.48022475 +0000 UTC m=+446.541913782" watchObservedRunningTime="2026-07-27 18:36:11.481185726 +0000 UTC m=+446.542874757" Jul 27 18:36:11.514188 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:11.514149 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vhv7p"] Jul 27 18:36:11.526161 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:11.526134 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vhv7p"] Jul 27 18:36:11.975667 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:11.975631 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-w6ltt" Jul 27 18:36:11.980531 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:11.980508 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-w6ltt" Jul 27 18:36:12.458223 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:12.458188 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-w6ltt" Jul 27 18:36:12.459417 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:12.459396 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-w6ltt" Jul 27 18:36:13.491988 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:13.491958 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0979e81f-93d2-43c3-be01-8355c6c4ccc2" path="/var/lib/kubelet/pods/0979e81f-93d2-43c3-be01-8355c6c4ccc2/volumes" Jul 27 18:36:31.603755 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:31.603720 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:36:31.664151 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:31.664114 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:36:40.441865 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:40.441834 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-w9d4l" Jul 27 18:36:40.445276 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:40.445256 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-z4tz5" Jul 27 18:36:54.426107 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:54.426070 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:36:54.517622 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:54.517587 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:36:54.535584 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:54.535549 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-fn9md"] Jul 27 18:36:54.535898 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:54.535886 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0979e81f-93d2-43c3-be01-8355c6c4ccc2" containerName="discovery" Jul 27 18:36:54.535984 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:54.535899 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="0979e81f-93d2-43c3-be01-8355c6c4ccc2" containerName="discovery" Jul 27 18:36:54.535984 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:54.535951 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="0979e81f-93d2-43c3-be01-8355c6c4ccc2" containerName="discovery" Jul 27 18:36:54.539874 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:54.539853 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-fn9md" Jul 27 18:36:54.542756 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:54.542734 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 27 18:36:54.544022 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:54.543998 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-8kmxg\"" Jul 27 18:36:54.544171 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:54.544044 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 27 18:36:54.551365 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:54.551340 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-fn9md"] Jul 27 18:36:54.620553 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:54.620512 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrkfk\" (UniqueName: \"kubernetes.io/projected/f9d30d0c-b61a-49b3-a4b2-3f75fe5f7fb4-kube-api-access-wrkfk\") pod \"kuadrant-operator-catalog-fn9md\" (UID: \"f9d30d0c-b61a-49b3-a4b2-3f75fe5f7fb4\") " pod="kuadrant-system/kuadrant-operator-catalog-fn9md" Jul 27 18:36:54.639063 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:54.639021 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:36:54.721264 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:54.721149 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrkfk\" (UniqueName: \"kubernetes.io/projected/f9d30d0c-b61a-49b3-a4b2-3f75fe5f7fb4-kube-api-access-wrkfk\") pod \"kuadrant-operator-catalog-fn9md\" (UID: \"f9d30d0c-b61a-49b3-a4b2-3f75fe5f7fb4\") " pod="kuadrant-system/kuadrant-operator-catalog-fn9md" Jul 27 18:36:54.730216 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:54.730186 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrkfk\" (UniqueName: \"kubernetes.io/projected/f9d30d0c-b61a-49b3-a4b2-3f75fe5f7fb4-kube-api-access-wrkfk\") pod \"kuadrant-operator-catalog-fn9md\" (UID: \"f9d30d0c-b61a-49b3-a4b2-3f75fe5f7fb4\") " pod="kuadrant-system/kuadrant-operator-catalog-fn9md" Jul 27 18:36:54.851042 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:54.851007 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-fn9md" Jul 27 18:36:54.903609 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:54.903577 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-fn9md"] Jul 27 18:36:54.983300 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:54.983276 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-fn9md"] Jul 27 18:36:54.986042 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:36:54.986013 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9d30d0c_b61a_49b3_a4b2_3f75fe5f7fb4.slice/crio-0e47a36b9afde1cfb0e99dd75c13b02f1003d553407a0cda218a520bfc37f9a4 WatchSource:0}: Error finding container 0e47a36b9afde1cfb0e99dd75c13b02f1003d553407a0cda218a520bfc37f9a4: Status 404 returned error can't find the container with id 0e47a36b9afde1cfb0e99dd75c13b02f1003d553407a0cda218a520bfc37f9a4 Jul 27 18:36:55.112888 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:55.112854 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-mkzvs"] Jul 27 18:36:55.115726 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:55.115707 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-mkzvs" Jul 27 18:36:55.125021 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:55.124999 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-mkzvs"] Jul 27 18:36:55.224441 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:55.224403 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcbqn\" (UniqueName: \"kubernetes.io/projected/04dbb35a-cea6-4fb7-aab6-29150d13090d-kube-api-access-jcbqn\") pod \"kuadrant-operator-catalog-mkzvs\" (UID: \"04dbb35a-cea6-4fb7-aab6-29150d13090d\") " pod="kuadrant-system/kuadrant-operator-catalog-mkzvs" Jul 27 18:36:55.325123 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:55.325087 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcbqn\" (UniqueName: \"kubernetes.io/projected/04dbb35a-cea6-4fb7-aab6-29150d13090d-kube-api-access-jcbqn\") pod \"kuadrant-operator-catalog-mkzvs\" (UID: \"04dbb35a-cea6-4fb7-aab6-29150d13090d\") " pod="kuadrant-system/kuadrant-operator-catalog-mkzvs" Jul 27 18:36:55.334348 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:55.334318 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcbqn\" (UniqueName: \"kubernetes.io/projected/04dbb35a-cea6-4fb7-aab6-29150d13090d-kube-api-access-jcbqn\") pod \"kuadrant-operator-catalog-mkzvs\" (UID: \"04dbb35a-cea6-4fb7-aab6-29150d13090d\") " pod="kuadrant-system/kuadrant-operator-catalog-mkzvs" Jul 27 18:36:55.338138 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:55.338113 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:36:55.425069 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:55.425035 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-mkzvs" Jul 27 18:36:55.543263 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:55.543144 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:36:55.564360 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:55.564334 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-mkzvs"] Jul 27 18:36:55.566534 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:36:55.566506 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04dbb35a_cea6_4fb7_aab6_29150d13090d.slice/crio-328526b26332f7aa0638ecb5d0f9772eaaee0c5ad785fcb103e8f494d7dbfb93 WatchSource:0}: Error finding container 328526b26332f7aa0638ecb5d0f9772eaaee0c5ad785fcb103e8f494d7dbfb93: Status 404 returned error can't find the container with id 328526b26332f7aa0638ecb5d0f9772eaaee0c5ad785fcb103e8f494d7dbfb93 Jul 27 18:36:55.639364 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:55.639325 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-mkzvs" event={"ID":"04dbb35a-cea6-4fb7-aab6-29150d13090d","Type":"ContainerStarted","Data":"328526b26332f7aa0638ecb5d0f9772eaaee0c5ad785fcb103e8f494d7dbfb93"} Jul 27 18:36:55.640557 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:55.640533 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-fn9md" event={"ID":"f9d30d0c-b61a-49b3-a4b2-3f75fe5f7fb4","Type":"ContainerStarted","Data":"0e47a36b9afde1cfb0e99dd75c13b02f1003d553407a0cda218a520bfc37f9a4"} Jul 27 18:36:57.650374 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:57.650343 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-mkzvs" event={"ID":"04dbb35a-cea6-4fb7-aab6-29150d13090d","Type":"ContainerStarted","Data":"a7845ed1a01560ff5f742c2b2b5962c99b2287b513ff0ac9cecb18586039380b"} Jul 27 18:36:57.654583 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:57.654554 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-fn9md" event={"ID":"f9d30d0c-b61a-49b3-a4b2-3f75fe5f7fb4","Type":"ContainerStarted","Data":"b98e7a153ef3eabd51b1a8306a9bb8ccc8063f8bf5fd58bf407745695c9bbf9d"} Jul 27 18:36:57.654738 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:57.654660 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-fn9md" podUID="f9d30d0c-b61a-49b3-a4b2-3f75fe5f7fb4" containerName="registry-server" containerID="cri-o://b98e7a153ef3eabd51b1a8306a9bb8ccc8063f8bf5fd58bf407745695c9bbf9d" gracePeriod=2 Jul 27 18:36:57.668187 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:57.668105 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-mkzvs" podStartSLOduration=1.113680316 podStartE2EDuration="2.668091625s" totalImagesPullingTime="1.554411309s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:36:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:36:57.66697371 +0000 UTC m=+492.728662742" watchObservedRunningTime="2026-07-27 18:36:57.668091625 +0000 UTC m=+492.729780655" Jul 27 18:36:57.686344 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:57.686301 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-fn9md" podStartSLOduration=1.552341341 podStartE2EDuration="3.686286742s" totalImagesPullingTime="2.133945401s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:36:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:36:57.684917415 +0000 UTC m=+492.746606446" watchObservedRunningTime="2026-07-27 18:36:57.686286742 +0000 UTC m=+492.747975773" Jul 27 18:36:57.890299 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:57.890276 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-fn9md" Jul 27 18:36:58.053374 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:58.053347 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wrkfk\" (UniqueName: \"kubernetes.io/projected/f9d30d0c-b61a-49b3-a4b2-3f75fe5f7fb4-kube-api-access-wrkfk\") pod \"f9d30d0c-b61a-49b3-a4b2-3f75fe5f7fb4\" (UID: \"f9d30d0c-b61a-49b3-a4b2-3f75fe5f7fb4\") " Jul 27 18:36:58.055630 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:58.055599 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9d30d0c-b61a-49b3-a4b2-3f75fe5f7fb4-kube-api-access-wrkfk" (OuterVolumeSpecName: "kube-api-access-wrkfk") pod "f9d30d0c-b61a-49b3-a4b2-3f75fe5f7fb4" (UID: "f9d30d0c-b61a-49b3-a4b2-3f75fe5f7fb4"). InnerVolumeSpecName "kube-api-access-wrkfk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 18:36:58.154215 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:58.154187 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrkfk\" (UniqueName: \"kubernetes.io/projected/f9d30d0c-b61a-49b3-a4b2-3f75fe5f7fb4-kube-api-access-wrkfk\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 18:36:58.660379 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:58.660347 2579 generic.go:356] "Generic (PLEG): container finished" podID="f9d30d0c-b61a-49b3-a4b2-3f75fe5f7fb4" containerID="b98e7a153ef3eabd51b1a8306a9bb8ccc8063f8bf5fd58bf407745695c9bbf9d" exitCode=0 Jul 27 18:36:58.660845 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:58.660392 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-fn9md" Jul 27 18:36:58.660845 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:58.660428 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-fn9md" event={"ID":"f9d30d0c-b61a-49b3-a4b2-3f75fe5f7fb4","Type":"ContainerDied","Data":"b98e7a153ef3eabd51b1a8306a9bb8ccc8063f8bf5fd58bf407745695c9bbf9d"} Jul 27 18:36:58.660845 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:58.660475 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-fn9md" event={"ID":"f9d30d0c-b61a-49b3-a4b2-3f75fe5f7fb4","Type":"ContainerDied","Data":"0e47a36b9afde1cfb0e99dd75c13b02f1003d553407a0cda218a520bfc37f9a4"} Jul 27 18:36:58.660845 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:58.660497 2579 scope.go:117] "RemoveContainer" containerID="b98e7a153ef3eabd51b1a8306a9bb8ccc8063f8bf5fd58bf407745695c9bbf9d" Jul 27 18:36:58.669913 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:58.669893 2579 scope.go:117] "RemoveContainer" containerID="b98e7a153ef3eabd51b1a8306a9bb8ccc8063f8bf5fd58bf407745695c9bbf9d" Jul 27 18:36:58.670177 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:36:58.670159 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b98e7a153ef3eabd51b1a8306a9bb8ccc8063f8bf5fd58bf407745695c9bbf9d\": container with ID starting with b98e7a153ef3eabd51b1a8306a9bb8ccc8063f8bf5fd58bf407745695c9bbf9d not found: ID does not exist" containerID="b98e7a153ef3eabd51b1a8306a9bb8ccc8063f8bf5fd58bf407745695c9bbf9d" Jul 27 18:36:58.670267 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:58.670184 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b98e7a153ef3eabd51b1a8306a9bb8ccc8063f8bf5fd58bf407745695c9bbf9d"} err="failed to get container status \"b98e7a153ef3eabd51b1a8306a9bb8ccc8063f8bf5fd58bf407745695c9bbf9d\": rpc error: code = NotFound desc = could not find container \"b98e7a153ef3eabd51b1a8306a9bb8ccc8063f8bf5fd58bf407745695c9bbf9d\": container with ID starting with b98e7a153ef3eabd51b1a8306a9bb8ccc8063f8bf5fd58bf407745695c9bbf9d not found: ID does not exist" Jul 27 18:36:58.681941 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:58.681916 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-fn9md"] Jul 27 18:36:58.688266 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:58.688223 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-fn9md"] Jul 27 18:36:59.493029 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:36:59.492995 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f9d30d0c-b61a-49b3-a4b2-3f75fe5f7fb4" path="/var/lib/kubelet/pods/f9d30d0c-b61a-49b3-a4b2-3f75fe5f7fb4/volumes" Jul 27 18:37:02.363124 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:02.363088 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:37:02.424092 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:02.424056 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:37:02.489209 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:02.489174 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:37:02.651961 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:02.651870 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:37:03.311155 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:03.311125 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:37:03.371414 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:03.371382 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:37:05.425377 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:05.425341 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-mkzvs" Jul 27 18:37:05.425781 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:05.425401 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-mkzvs" Jul 27 18:37:05.448061 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:05.448032 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-mkzvs" Jul 27 18:37:05.712388 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:05.712319 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-mkzvs" Jul 27 18:37:13.663334 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:13.663283 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:37:13.764894 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:13.764857 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:37:13.865861 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:13.865826 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:37:24.040418 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:24.040382 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:37:24.399067 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:24.398980 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:37:24.494975 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:24.494942 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:37:26.342246 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:26.342200 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:37:26.674127 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:26.674041 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-5bjl6"] Jul 27 18:37:26.674485 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:26.674468 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9d30d0c-b61a-49b3-a4b2-3f75fe5f7fb4" containerName="registry-server" Jul 27 18:37:26.674575 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:26.674489 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9d30d0c-b61a-49b3-a4b2-3f75fe5f7fb4" containerName="registry-server" Jul 27 18:37:26.674575 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:26.674565 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9d30d0c-b61a-49b3-a4b2-3f75fe5f7fb4" containerName="registry-server" Jul 27 18:37:26.681069 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:26.681047 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-5bjl6" Jul 27 18:37:26.683873 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:26.683852 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 27 18:37:26.684095 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:26.684073 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-xvqxn\"" Jul 27 18:37:26.688200 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:26.687848 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-5bjl6"] Jul 27 18:37:26.784257 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:26.784203 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckq5l\" (UniqueName: \"kubernetes.io/projected/e2b018f7-aa0d-40d9-8d51-cb3fcaaca11e-kube-api-access-ckq5l\") pod \"dns-operator-controller-manager-864dd78c4c-5bjl6\" (UID: \"e2b018f7-aa0d-40d9-8d51-cb3fcaaca11e\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-5bjl6" Jul 27 18:37:26.884986 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:26.884947 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckq5l\" (UniqueName: \"kubernetes.io/projected/e2b018f7-aa0d-40d9-8d51-cb3fcaaca11e-kube-api-access-ckq5l\") pod \"dns-operator-controller-manager-864dd78c4c-5bjl6\" (UID: \"e2b018f7-aa0d-40d9-8d51-cb3fcaaca11e\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-5bjl6" Jul 27 18:37:26.897682 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:26.897643 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckq5l\" (UniqueName: \"kubernetes.io/projected/e2b018f7-aa0d-40d9-8d51-cb3fcaaca11e-kube-api-access-ckq5l\") pod \"dns-operator-controller-manager-864dd78c4c-5bjl6\" (UID: \"e2b018f7-aa0d-40d9-8d51-cb3fcaaca11e\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-5bjl6" Jul 27 18:37:26.991909 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:26.991833 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-5bjl6" Jul 27 18:37:27.131751 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:27.131726 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-5bjl6"] Jul 27 18:37:27.134414 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:37:27.134389 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2b018f7_aa0d_40d9_8d51_cb3fcaaca11e.slice/crio-01cfe8a106ec9c4cbfaa7f59feccef00eaf4f838a80a0c4e1e77b2f5ec4680c2 WatchSource:0}: Error finding container 01cfe8a106ec9c4cbfaa7f59feccef00eaf4f838a80a0c4e1e77b2f5ec4680c2: Status 404 returned error can't find the container with id 01cfe8a106ec9c4cbfaa7f59feccef00eaf4f838a80a0c4e1e77b2f5ec4680c2 Jul 27 18:37:27.790936 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:27.790902 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-5bjl6" event={"ID":"e2b018f7-aa0d-40d9-8d51-cb3fcaaca11e","Type":"ContainerStarted","Data":"01cfe8a106ec9c4cbfaa7f59feccef00eaf4f838a80a0c4e1e77b2f5ec4680c2"} Jul 27 18:37:28.251876 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:28.251782 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:37:29.801203 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:29.801121 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-5bjl6" event={"ID":"e2b018f7-aa0d-40d9-8d51-cb3fcaaca11e","Type":"ContainerStarted","Data":"998bad292a1b2087d73aa3bf5886da3778fca882280f9398f24a518f6522a4ed"} Jul 27 18:37:29.801601 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:29.801276 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-5bjl6" Jul 27 18:37:29.819878 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:29.819831 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-5bjl6" podStartSLOduration=1.4305815769999999 podStartE2EDuration="3.819817258s" totalImagesPullingTime="2.389235681s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:37:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:37:29.818289611 +0000 UTC m=+524.879978642" watchObservedRunningTime="2026-07-27 18:37:29.819817258 +0000 UTC m=+524.881506288" Jul 27 18:37:30.150618 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:30.150547 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5nmgk"] Jul 27 18:37:30.153937 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:30.153918 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5nmgk" Jul 27 18:37:30.156766 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:30.156744 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-2n2bn\"" Jul 27 18:37:30.169660 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:30.169632 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5nmgk"] Jul 27 18:37:30.213550 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:30.213516 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrqcp\" (UniqueName: \"kubernetes.io/projected/a1c25cbc-3555-43ee-a5d7-745881024718-kube-api-access-vrqcp\") pod \"limitador-operator-controller-manager-859d7b78bd-5nmgk\" (UID: \"a1c25cbc-3555-43ee-a5d7-745881024718\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5nmgk" Jul 27 18:37:30.314070 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:30.314032 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrqcp\" (UniqueName: \"kubernetes.io/projected/a1c25cbc-3555-43ee-a5d7-745881024718-kube-api-access-vrqcp\") pod \"limitador-operator-controller-manager-859d7b78bd-5nmgk\" (UID: \"a1c25cbc-3555-43ee-a5d7-745881024718\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5nmgk" Jul 27 18:37:30.325910 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:30.325883 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrqcp\" (UniqueName: \"kubernetes.io/projected/a1c25cbc-3555-43ee-a5d7-745881024718-kube-api-access-vrqcp\") pod \"limitador-operator-controller-manager-859d7b78bd-5nmgk\" (UID: \"a1c25cbc-3555-43ee-a5d7-745881024718\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5nmgk" Jul 27 18:37:30.464076 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:30.463966 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5nmgk" Jul 27 18:37:30.620205 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:30.620183 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5nmgk"] Jul 27 18:37:30.622207 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:37:30.622178 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1c25cbc_3555_43ee_a5d7_745881024718.slice/crio-14a0a13da703bd4568c77178c39558bb6f936f3d012c8515ee02bfe908339bd8 WatchSource:0}: Error finding container 14a0a13da703bd4568c77178c39558bb6f936f3d012c8515ee02bfe908339bd8: Status 404 returned error can't find the container with id 14a0a13da703bd4568c77178c39558bb6f936f3d012c8515ee02bfe908339bd8 Jul 27 18:37:30.808895 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:30.808841 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5nmgk" event={"ID":"a1c25cbc-3555-43ee-a5d7-745881024718","Type":"ContainerStarted","Data":"14a0a13da703bd4568c77178c39558bb6f936f3d012c8515ee02bfe908339bd8"} Jul 27 18:37:30.850219 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:30.850173 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:37:31.016338 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:31.016302 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:37:32.069909 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:32.069871 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:37:32.820378 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:32.820295 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5nmgk" event={"ID":"a1c25cbc-3555-43ee-a5d7-745881024718","Type":"ContainerStarted","Data":"4b1d836683671b76faaebed2adff3377b7c87648f8ac5f09abfafedd0597d987"} Jul 27 18:37:32.820515 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:32.820419 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5nmgk" Jul 27 18:37:32.838492 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:32.838449 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5nmgk" podStartSLOduration=0.918596006 podStartE2EDuration="2.83843394s" totalImagesPullingTime="1.919837934s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:37:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:37:32.837830171 +0000 UTC m=+527.899519202" watchObservedRunningTime="2026-07-27 18:37:32.83843394 +0000 UTC m=+527.900122971" Jul 27 18:37:35.166591 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:35.166553 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qf772"] Jul 27 18:37:35.169887 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:35.169871 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qf772" Jul 27 18:37:35.172911 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:35.172887 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-q4btw\"" Jul 27 18:37:35.187346 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:35.187323 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qf772"] Jul 27 18:37:35.260905 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:35.260865 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a8bd966a-43df-41bc-a9bf-84059d281598-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-qf772\" (UID: \"a8bd966a-43df-41bc-a9bf-84059d281598\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qf772" Jul 27 18:37:35.261067 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:35.260921 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qqxx\" (UniqueName: \"kubernetes.io/projected/a8bd966a-43df-41bc-a9bf-84059d281598-kube-api-access-7qqxx\") pod \"kuadrant-operator-controller-manager-8684c9ff49-qf772\" (UID: \"a8bd966a-43df-41bc-a9bf-84059d281598\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qf772" Jul 27 18:37:35.361809 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:35.361774 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a8bd966a-43df-41bc-a9bf-84059d281598-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-qf772\" (UID: \"a8bd966a-43df-41bc-a9bf-84059d281598\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qf772" Jul 27 18:37:35.361967 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:35.361821 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qqxx\" (UniqueName: \"kubernetes.io/projected/a8bd966a-43df-41bc-a9bf-84059d281598-kube-api-access-7qqxx\") pod \"kuadrant-operator-controller-manager-8684c9ff49-qf772\" (UID: \"a8bd966a-43df-41bc-a9bf-84059d281598\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qf772" Jul 27 18:37:35.362144 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:35.362126 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a8bd966a-43df-41bc-a9bf-84059d281598-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-qf772\" (UID: \"a8bd966a-43df-41bc-a9bf-84059d281598\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qf772" Jul 27 18:37:35.374595 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:35.374572 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qqxx\" (UniqueName: \"kubernetes.io/projected/a8bd966a-43df-41bc-a9bf-84059d281598-kube-api-access-7qqxx\") pod \"kuadrant-operator-controller-manager-8684c9ff49-qf772\" (UID: \"a8bd966a-43df-41bc-a9bf-84059d281598\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qf772" Jul 27 18:37:35.396201 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:35.396161 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:37:35.480082 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:35.479972 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qf772" Jul 27 18:37:35.622745 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:35.622717 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qf772"] Jul 27 18:37:35.626022 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:37:35.625993 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8bd966a_43df_41bc_a9bf_84059d281598.slice/crio-3d713f14de65a5c1b4ed4e274ad91868b3613fffa987445b579c506a8f787c6c WatchSource:0}: Error finding container 3d713f14de65a5c1b4ed4e274ad91868b3613fffa987445b579c506a8f787c6c: Status 404 returned error can't find the container with id 3d713f14de65a5c1b4ed4e274ad91868b3613fffa987445b579c506a8f787c6c Jul 27 18:37:35.644634 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:35.644596 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:37:35.839634 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:35.839594 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qf772" event={"ID":"a8bd966a-43df-41bc-a9bf-84059d281598","Type":"ContainerStarted","Data":"3d713f14de65a5c1b4ed4e274ad91868b3613fffa987445b579c506a8f787c6c"} Jul 27 18:37:35.999601 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:35.999551 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:37:36.292422 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:36.292376 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:37:38.374532 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:38.374496 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-7q5jd"] Jul 27 18:37:38.377943 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:38.377919 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-7q5jd" Jul 27 18:37:38.381634 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:38.381614 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-2v6bt\"" Jul 27 18:37:38.394018 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:38.393995 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-7q5jd"] Jul 27 18:37:38.486306 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:38.486273 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfrfv\" (UniqueName: \"kubernetes.io/projected/76b982a1-38c1-4626-8e90-042e910fdffc-kube-api-access-mfrfv\") pod \"authorino-operator-75cc8d5cb7-7q5jd\" (UID: \"76b982a1-38c1-4626-8e90-042e910fdffc\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-7q5jd" Jul 27 18:37:38.587565 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:38.587516 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfrfv\" (UniqueName: \"kubernetes.io/projected/76b982a1-38c1-4626-8e90-042e910fdffc-kube-api-access-mfrfv\") pod \"authorino-operator-75cc8d5cb7-7q5jd\" (UID: \"76b982a1-38c1-4626-8e90-042e910fdffc\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-7q5jd" Jul 27 18:37:38.603447 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:38.603416 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfrfv\" (UniqueName: \"kubernetes.io/projected/76b982a1-38c1-4626-8e90-042e910fdffc-kube-api-access-mfrfv\") pod \"authorino-operator-75cc8d5cb7-7q5jd\" (UID: \"76b982a1-38c1-4626-8e90-042e910fdffc\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-7q5jd" Jul 27 18:37:38.687954 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:38.687871 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-7q5jd" Jul 27 18:37:38.842704 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:38.842588 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:37:38.843921 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:38.843888 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-7q5jd"] Jul 27 18:37:38.846371 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:37:38.846342 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76b982a1_38c1_4626_8e90_042e910fdffc.slice/crio-76a6469ccebf726b989195ec7d209a7160c363e06bbba053d1f8a4689ea6e6dc WatchSource:0}: Error finding container 76a6469ccebf726b989195ec7d209a7160c363e06bbba053d1f8a4689ea6e6dc: Status 404 returned error can't find the container with id 76a6469ccebf726b989195ec7d209a7160c363e06bbba053d1f8a4689ea6e6dc Jul 27 18:37:38.856724 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:38.856693 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-7q5jd" event={"ID":"76b982a1-38c1-4626-8e90-042e910fdffc","Type":"ContainerStarted","Data":"76a6469ccebf726b989195ec7d209a7160c363e06bbba053d1f8a4689ea6e6dc"} Jul 27 18:37:40.255937 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:40.255896 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:37:40.358964 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:40.358909 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:37:40.801250 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:40.801180 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:37:40.811865 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:40.811836 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-5bjl6" Jul 27 18:37:43.827107 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:43.827077 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5nmgk" Jul 27 18:37:43.885150 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:43.885119 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qf772" event={"ID":"a8bd966a-43df-41bc-a9bf-84059d281598","Type":"ContainerStarted","Data":"cbf3f10314dd66054f8f178e1f2307133beebfa05515ff9ddd6821390dc051ee"} Jul 27 18:37:43.885333 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:43.885268 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qf772" Jul 27 18:37:43.886885 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:43.886858 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-7q5jd" event={"ID":"76b982a1-38c1-4626-8e90-042e910fdffc","Type":"ContainerStarted","Data":"c209210241b3b27d5381f78f1cf7a9be871db87a4382476d8e784631f49c04fd"} Jul 27 18:37:43.887014 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:43.886914 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-7q5jd" Jul 27 18:37:43.904474 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:43.904428 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qf772" podStartSLOduration=1.640146753 podStartE2EDuration="8.904411173s" totalImagesPullingTime="7.26426442s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:37:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:37:43.903339624 +0000 UTC m=+538.965028654" watchObservedRunningTime="2026-07-27 18:37:43.904411173 +0000 UTC m=+538.966100206" Jul 27 18:37:43.919594 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:43.919553 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-7q5jd" podStartSLOduration=1.8858568980000001 podStartE2EDuration="5.919538957s" totalImagesPullingTime="4.033682059s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:37:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:37:43.918788608 +0000 UTC m=+538.980477638" watchObservedRunningTime="2026-07-27 18:37:43.919538957 +0000 UTC m=+538.981227987" Jul 27 18:37:44.254538 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:44.254447 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:37:44.357994 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:44.357958 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:37:45.626446 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:45.626415 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-lgnxn"] Jul 27 18:37:45.631438 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:45.631417 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-lgnxn" Jul 27 18:37:45.634245 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:45.634207 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 27 18:37:45.634415 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:45.634397 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 27 18:37:45.634487 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:45.634431 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-jv8qx\"" Jul 27 18:37:45.638383 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:45.638352 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:37:45.639130 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:45.639111 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-lgnxn"] Jul 27 18:37:45.751155 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:45.751052 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/d4cbf21a-5dad-42c5-93ef-f2fa08e5b320-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-lgnxn\" (UID: \"d4cbf21a-5dad-42c5-93ef-f2fa08e5b320\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-lgnxn" Jul 27 18:37:45.751155 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:45.751098 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d4cbf21a-5dad-42c5-93ef-f2fa08e5b320-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-lgnxn\" (UID: \"d4cbf21a-5dad-42c5-93ef-f2fa08e5b320\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-lgnxn" Jul 27 18:37:45.751155 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:45.751135 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zt8jf\" (UniqueName: \"kubernetes.io/projected/d4cbf21a-5dad-42c5-93ef-f2fa08e5b320-kube-api-access-zt8jf\") pod \"kuadrant-console-plugin-66b5774985-lgnxn\" (UID: \"d4cbf21a-5dad-42c5-93ef-f2fa08e5b320\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-lgnxn" Jul 27 18:37:45.851511 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:45.851483 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d4cbf21a-5dad-42c5-93ef-f2fa08e5b320-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-lgnxn\" (UID: \"d4cbf21a-5dad-42c5-93ef-f2fa08e5b320\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-lgnxn" Jul 27 18:37:45.851697 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:45.851527 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zt8jf\" (UniqueName: \"kubernetes.io/projected/d4cbf21a-5dad-42c5-93ef-f2fa08e5b320-kube-api-access-zt8jf\") pod \"kuadrant-console-plugin-66b5774985-lgnxn\" (UID: \"d4cbf21a-5dad-42c5-93ef-f2fa08e5b320\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-lgnxn" Jul 27 18:37:45.851697 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:45.851668 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/d4cbf21a-5dad-42c5-93ef-f2fa08e5b320-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-lgnxn\" (UID: \"d4cbf21a-5dad-42c5-93ef-f2fa08e5b320\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-lgnxn" Jul 27 18:37:45.852167 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:45.852144 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d4cbf21a-5dad-42c5-93ef-f2fa08e5b320-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-lgnxn\" (UID: \"d4cbf21a-5dad-42c5-93ef-f2fa08e5b320\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-lgnxn" Jul 27 18:37:45.854116 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:45.854097 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/d4cbf21a-5dad-42c5-93ef-f2fa08e5b320-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-lgnxn\" (UID: \"d4cbf21a-5dad-42c5-93ef-f2fa08e5b320\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-lgnxn" Jul 27 18:37:45.867464 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:45.867435 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zt8jf\" (UniqueName: \"kubernetes.io/projected/d4cbf21a-5dad-42c5-93ef-f2fa08e5b320-kube-api-access-zt8jf\") pod \"kuadrant-console-plugin-66b5774985-lgnxn\" (UID: \"d4cbf21a-5dad-42c5-93ef-f2fa08e5b320\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-lgnxn" Jul 27 18:37:45.906375 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:45.906333 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:37:45.942016 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:45.941983 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-lgnxn" Jul 27 18:37:46.080608 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:46.080580 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-lgnxn"] Jul 27 18:37:46.082619 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:37:46.082592 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd4cbf21a_5dad_42c5_93ef_f2fa08e5b320.slice/crio-8e7e21fd71c6d0bc57fd9ddc420662503bcd48710f9659c0227790a8d2e37ca7 WatchSource:0}: Error finding container 8e7e21fd71c6d0bc57fd9ddc420662503bcd48710f9659c0227790a8d2e37ca7: Status 404 returned error can't find the container with id 8e7e21fd71c6d0bc57fd9ddc420662503bcd48710f9659c0227790a8d2e37ca7 Jul 27 18:37:46.901096 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:46.901053 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-lgnxn" event={"ID":"d4cbf21a-5dad-42c5-93ef-f2fa08e5b320","Type":"ContainerStarted","Data":"8e7e21fd71c6d0bc57fd9ddc420662503bcd48710f9659c0227790a8d2e37ca7"} Jul 27 18:37:48.442203 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:48.442168 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:37:48.500918 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:48.500869 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:37:54.893063 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:54.893012 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qf772" Jul 27 18:37:54.893559 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:54.893352 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-7q5jd" Jul 27 18:37:56.037703 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:56.037668 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-bd7dv"] Jul 27 18:37:56.047303 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:56.047279 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-bd7dv" Jul 27 18:37:56.066448 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:56.066185 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-bd7dv"] Jul 27 18:37:56.145185 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:56.145120 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4fedfed5-fc9b-4549-b685-6ad4eb8fda9a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-bd7dv\" (UID: \"4fedfed5-fc9b-4549-b685-6ad4eb8fda9a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-bd7dv" Jul 27 18:37:56.145373 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:56.145316 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g462z\" (UniqueName: \"kubernetes.io/projected/4fedfed5-fc9b-4549-b685-6ad4eb8fda9a-kube-api-access-g462z\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-bd7dv\" (UID: \"4fedfed5-fc9b-4549-b685-6ad4eb8fda9a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-bd7dv" Jul 27 18:37:56.246609 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:56.246568 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g462z\" (UniqueName: \"kubernetes.io/projected/4fedfed5-fc9b-4549-b685-6ad4eb8fda9a-kube-api-access-g462z\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-bd7dv\" (UID: \"4fedfed5-fc9b-4549-b685-6ad4eb8fda9a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-bd7dv" Jul 27 18:37:56.246799 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:56.246641 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4fedfed5-fc9b-4549-b685-6ad4eb8fda9a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-bd7dv\" (UID: \"4fedfed5-fc9b-4549-b685-6ad4eb8fda9a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-bd7dv" Jul 27 18:37:56.247038 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:56.247011 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4fedfed5-fc9b-4549-b685-6ad4eb8fda9a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-bd7dv\" (UID: \"4fedfed5-fc9b-4549-b685-6ad4eb8fda9a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-bd7dv" Jul 27 18:37:56.260121 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:56.260067 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g462z\" (UniqueName: \"kubernetes.io/projected/4fedfed5-fc9b-4549-b685-6ad4eb8fda9a-kube-api-access-g462z\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-bd7dv\" (UID: \"4fedfed5-fc9b-4549-b685-6ad4eb8fda9a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-bd7dv" Jul 27 18:37:56.360656 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:56.360563 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-bd7dv" Jul 27 18:37:56.773692 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:56.773661 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-bd7dv"] Jul 27 18:37:56.776660 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:56.776633 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-bd7dv"] Jul 27 18:37:56.787961 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:56.787922 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qf772"] Jul 27 18:37:56.788165 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:56.788130 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qf772" podUID="a8bd966a-43df-41bc-a9bf-84059d281598" containerName="manager" containerID="cri-o://cbf3f10314dd66054f8f178e1f2307133beebfa05515ff9ddd6821390dc051ee" gracePeriod=2 Jul 27 18:37:56.799915 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:56.799888 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qf772"] Jul 27 18:37:56.802379 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:37:56.802319 2579 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"a8bd966a-43df-41bc-a9bf-84059d281598\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-8684c9ff49-qf772\": pods \"kuadrant-operator-controller-manager-8684c9ff49-qf772\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qf772" Jul 27 18:37:56.803769 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:56.803746 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-jwdzm"] Jul 27 18:37:56.804300 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:56.804280 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8bd966a-43df-41bc-a9bf-84059d281598" containerName="manager" Jul 27 18:37:56.804300 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:56.804300 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8bd966a-43df-41bc-a9bf-84059d281598" containerName="manager" Jul 27 18:37:56.804454 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:56.804397 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8bd966a-43df-41bc-a9bf-84059d281598" containerName="manager" Jul 27 18:37:56.811063 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:56.811033 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-jwdzm" Jul 27 18:37:56.813853 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:37:56.813818 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-qf772\" is forbidden: User \"system:node:ip-10-0-128-32.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-32.ec2.internal' and this object" podUID="a8bd966a-43df-41bc-a9bf-84059d281598" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qf772" Jul 27 18:37:56.813977 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:56.813942 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5nmgk"] Jul 27 18:37:56.814204 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:56.814163 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5nmgk" podUID="a1c25cbc-3555-43ee-a5d7-745881024718" containerName="manager" containerID="cri-o://4b1d836683671b76faaebed2adff3377b7c87648f8ac5f09abfafedd0597d987" gracePeriod=2 Jul 27 18:37:56.816308 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:56.816283 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-pbv7d"] Jul 27 18:37:56.825698 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:56.825326 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-pbv7d" Jul 27 18:37:56.833820 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:56.833791 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5nmgk"] Jul 27 18:37:56.836737 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:56.836494 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-jwdzm"] Jul 27 18:37:56.838619 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:56.838591 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-pbv7d"] Jul 27 18:37:56.847285 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:37:56.847247 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-qf772\" is forbidden: User \"system:node:ip-10-0-128-32.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-32.ec2.internal' and this object" podUID="a8bd966a-43df-41bc-a9bf-84059d281598" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qf772" Jul 27 18:37:56.874535 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:37:56.874490 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-5nmgk\" is forbidden: User \"system:node:ip-10-0-128-32.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-32.ec2.internal' and this object" podUID="a1c25cbc-3555-43ee-a5d7-745881024718" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5nmgk" Jul 27 18:37:56.952584 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:56.952542 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9jbn\" (UniqueName: \"kubernetes.io/projected/af04ced4-4b4a-4661-b63a-f0fcc3cc66f1-kube-api-access-w9jbn\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-jwdzm\" (UID: \"af04ced4-4b4a-4661-b63a-f0fcc3cc66f1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-jwdzm" Jul 27 18:37:56.952754 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:56.952604 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/150d56bd-a044-490c-a8d3-611f0948a955-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-pbv7d\" (UID: \"150d56bd-a044-490c-a8d3-611f0948a955\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-pbv7d" Jul 27 18:37:56.952754 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:56.952679 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bqj2\" (UniqueName: \"kubernetes.io/projected/150d56bd-a044-490c-a8d3-611f0948a955-kube-api-access-6bqj2\") pod \"kuadrant-operator-controller-manager-8684c9ff49-pbv7d\" (UID: \"150d56bd-a044-490c-a8d3-611f0948a955\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-pbv7d" Jul 27 18:37:56.952754 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:56.952715 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/af04ced4-4b4a-4661-b63a-f0fcc3cc66f1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-jwdzm\" (UID: \"af04ced4-4b4a-4661-b63a-f0fcc3cc66f1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-jwdzm" Jul 27 18:37:57.053593 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:57.053504 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/150d56bd-a044-490c-a8d3-611f0948a955-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-pbv7d\" (UID: \"150d56bd-a044-490c-a8d3-611f0948a955\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-pbv7d" Jul 27 18:37:57.053593 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:57.053563 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bqj2\" (UniqueName: \"kubernetes.io/projected/150d56bd-a044-490c-a8d3-611f0948a955-kube-api-access-6bqj2\") pod \"kuadrant-operator-controller-manager-8684c9ff49-pbv7d\" (UID: \"150d56bd-a044-490c-a8d3-611f0948a955\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-pbv7d" Jul 27 18:37:57.053593 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:57.053589 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/af04ced4-4b4a-4661-b63a-f0fcc3cc66f1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-jwdzm\" (UID: \"af04ced4-4b4a-4661-b63a-f0fcc3cc66f1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-jwdzm" Jul 27 18:37:57.054125 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:57.053694 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9jbn\" (UniqueName: \"kubernetes.io/projected/af04ced4-4b4a-4661-b63a-f0fcc3cc66f1-kube-api-access-w9jbn\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-jwdzm\" (UID: \"af04ced4-4b4a-4661-b63a-f0fcc3cc66f1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-jwdzm" Jul 27 18:37:57.054125 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:57.053932 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/150d56bd-a044-490c-a8d3-611f0948a955-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-pbv7d\" (UID: \"150d56bd-a044-490c-a8d3-611f0948a955\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-pbv7d" Jul 27 18:37:57.054125 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:57.054056 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/af04ced4-4b4a-4661-b63a-f0fcc3cc66f1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-jwdzm\" (UID: \"af04ced4-4b4a-4661-b63a-f0fcc3cc66f1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-jwdzm" Jul 27 18:37:57.071373 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:57.071340 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9jbn\" (UniqueName: \"kubernetes.io/projected/af04ced4-4b4a-4661-b63a-f0fcc3cc66f1-kube-api-access-w9jbn\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-jwdzm\" (UID: \"af04ced4-4b4a-4661-b63a-f0fcc3cc66f1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-jwdzm" Jul 27 18:37:57.071506 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:57.071342 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bqj2\" (UniqueName: \"kubernetes.io/projected/150d56bd-a044-490c-a8d3-611f0948a955-kube-api-access-6bqj2\") pod \"kuadrant-operator-controller-manager-8684c9ff49-pbv7d\" (UID: \"150d56bd-a044-490c-a8d3-611f0948a955\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-pbv7d" Jul 27 18:37:57.208206 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:57.208165 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-jwdzm" Jul 27 18:37:57.215354 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:57.215286 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-pbv7d" Jul 27 18:37:58.521889 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:58.521833 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:37:58.672324 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:58.672213 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:37:59.476752 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:59.476719 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:37:59.589021 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:37:59.588987 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:38:02.374447 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:02.374407 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:38:02.478116 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:02.478077 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:38:03.853459 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:03.853417 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:38:04.008536 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:04.008502 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:38:07.456184 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:07.456152 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:38:07.552998 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:07.552962 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:38:09.022065 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:09.021774 2579 generic.go:356] "Generic (PLEG): container finished" podID="a8bd966a-43df-41bc-a9bf-84059d281598" containerID="cbf3f10314dd66054f8f178e1f2307133beebfa05515ff9ddd6821390dc051ee" exitCode=0 Jul 27 18:38:09.024758 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:09.024586 2579 generic.go:356] "Generic (PLEG): container finished" podID="a1c25cbc-3555-43ee-a5d7-745881024718" containerID="4b1d836683671b76faaebed2adff3377b7c87648f8ac5f09abfafedd0597d987" exitCode=0 Jul 27 18:38:09.137963 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:09.137940 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5nmgk" Jul 27 18:38:09.141035 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:38:09.141005 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-5nmgk\" is forbidden: User \"system:node:ip-10-0-128-32.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-32.ec2.internal' and this object" podUID="a1c25cbc-3555-43ee-a5d7-745881024718" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5nmgk" Jul 27 18:38:09.141881 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:09.141862 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qf772" Jul 27 18:38:09.145459 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:38:09.145417 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-qf772\" is forbidden: User \"system:node:ip-10-0-128-32.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-32.ec2.internal' and this object" podUID="a8bd966a-43df-41bc-a9bf-84059d281598" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qf772" Jul 27 18:38:09.148538 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:38:09.148505 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-5nmgk\" is forbidden: User \"system:node:ip-10-0-128-32.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-32.ec2.internal' and this object" podUID="a1c25cbc-3555-43ee-a5d7-745881024718" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5nmgk" Jul 27 18:38:09.150395 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:09.150370 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-pbv7d"] Jul 27 18:38:09.151965 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:38:09.151944 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod150d56bd_a044_490c_a8d3_611f0948a955.slice/crio-1b0b290b677925e26b457a226558ff21ddc2d62fe138d8dab527ce4c2006bafd WatchSource:0}: Error finding container 1b0b290b677925e26b457a226558ff21ddc2d62fe138d8dab527ce4c2006bafd: Status 404 returned error can't find the container with id 1b0b290b677925e26b457a226558ff21ddc2d62fe138d8dab527ce4c2006bafd Jul 27 18:38:09.169681 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:09.169658 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a8bd966a-43df-41bc-a9bf-84059d281598-extensions-socket-volume\") pod \"a8bd966a-43df-41bc-a9bf-84059d281598\" (UID: \"a8bd966a-43df-41bc-a9bf-84059d281598\") " Jul 27 18:38:09.169823 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:09.169763 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7qqxx\" (UniqueName: \"kubernetes.io/projected/a8bd966a-43df-41bc-a9bf-84059d281598-kube-api-access-7qqxx\") pod \"a8bd966a-43df-41bc-a9bf-84059d281598\" (UID: \"a8bd966a-43df-41bc-a9bf-84059d281598\") " Jul 27 18:38:09.169823 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:09.169807 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vrqcp\" (UniqueName: \"kubernetes.io/projected/a1c25cbc-3555-43ee-a5d7-745881024718-kube-api-access-vrqcp\") pod \"a1c25cbc-3555-43ee-a5d7-745881024718\" (UID: \"a1c25cbc-3555-43ee-a5d7-745881024718\") " Jul 27 18:38:09.171289 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:09.170405 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8bd966a-43df-41bc-a9bf-84059d281598-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "a8bd966a-43df-41bc-a9bf-84059d281598" (UID: "a8bd966a-43df-41bc-a9bf-84059d281598"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 18:38:09.172880 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:09.172853 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8bd966a-43df-41bc-a9bf-84059d281598-kube-api-access-7qqxx" (OuterVolumeSpecName: "kube-api-access-7qqxx") pod "a8bd966a-43df-41bc-a9bf-84059d281598" (UID: "a8bd966a-43df-41bc-a9bf-84059d281598"). InnerVolumeSpecName "kube-api-access-7qqxx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 18:38:09.172958 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:09.172940 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1c25cbc-3555-43ee-a5d7-745881024718-kube-api-access-vrqcp" (OuterVolumeSpecName: "kube-api-access-vrqcp") pod "a1c25cbc-3555-43ee-a5d7-745881024718" (UID: "a1c25cbc-3555-43ee-a5d7-745881024718"). InnerVolumeSpecName "kube-api-access-vrqcp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 18:38:09.173073 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:09.173053 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-jwdzm"] Jul 27 18:38:09.178245 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:38:09.178200 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf04ced4_4b4a_4661_b63a_f0fcc3cc66f1.slice/crio-9ecc5e91ae38be051d19c2365c1ae66d60d8d8e0b3c9558d7dc91fb9fcb21d2f WatchSource:0}: Error finding container 9ecc5e91ae38be051d19c2365c1ae66d60d8d8e0b3c9558d7dc91fb9fcb21d2f: Status 404 returned error can't find the container with id 9ecc5e91ae38be051d19c2365c1ae66d60d8d8e0b3c9558d7dc91fb9fcb21d2f Jul 27 18:38:09.270350 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:09.270322 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qqxx\" (UniqueName: \"kubernetes.io/projected/a8bd966a-43df-41bc-a9bf-84059d281598-kube-api-access-7qqxx\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 18:38:09.270350 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:09.270347 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrqcp\" (UniqueName: \"kubernetes.io/projected/a1c25cbc-3555-43ee-a5d7-745881024718-kube-api-access-vrqcp\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 18:38:09.270350 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:09.270355 2579 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a8bd966a-43df-41bc-a9bf-84059d281598-extensions-socket-volume\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 18:38:09.493108 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:09.493029 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a1c25cbc-3555-43ee-a5d7-745881024718" path="/var/lib/kubelet/pods/a1c25cbc-3555-43ee-a5d7-745881024718/volumes" Jul 27 18:38:09.493389 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:09.493375 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a8bd966a-43df-41bc-a9bf-84059d281598" path="/var/lib/kubelet/pods/a8bd966a-43df-41bc-a9bf-84059d281598/volumes" Jul 27 18:38:10.031972 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:10.031939 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qf772" Jul 27 18:38:10.031972 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:10.031965 2579 scope.go:117] "RemoveContainer" containerID="cbf3f10314dd66054f8f178e1f2307133beebfa05515ff9ddd6821390dc051ee" Jul 27 18:38:10.033465 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:10.033434 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5nmgk" Jul 27 18:38:10.035584 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:10.035555 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-pbv7d" event={"ID":"150d56bd-a044-490c-a8d3-611f0948a955","Type":"ContainerStarted","Data":"086ec366913fb06a3ba7705e5b674d4d3d3674d5beb8fb3c17e16c3d6bb645a0"} Jul 27 18:38:10.035744 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:10.035715 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-pbv7d" event={"ID":"150d56bd-a044-490c-a8d3-611f0948a955","Type":"ContainerStarted","Data":"1b0b290b677925e26b457a226558ff21ddc2d62fe138d8dab527ce4c2006bafd"} Jul 27 18:38:10.035822 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:10.035752 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-pbv7d" Jul 27 18:38:10.035822 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:38:10.035626 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-qf772\" is forbidden: User \"system:node:ip-10-0-128-32.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-32.ec2.internal' and this object" podUID="a8bd966a-43df-41bc-a9bf-84059d281598" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qf772" Jul 27 18:38:10.037818 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:10.037791 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-jwdzm" event={"ID":"af04ced4-4b4a-4661-b63a-f0fcc3cc66f1","Type":"ContainerStarted","Data":"020a91124a6c560aee95239c4ba6061fc3781cb439637c3c2c719eb852e1a22f"} Jul 27 18:38:10.037965 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:10.037846 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-jwdzm" event={"ID":"af04ced4-4b4a-4661-b63a-f0fcc3cc66f1","Type":"ContainerStarted","Data":"9ecc5e91ae38be051d19c2365c1ae66d60d8d8e0b3c9558d7dc91fb9fcb21d2f"} Jul 27 18:38:10.037965 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:10.037894 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-jwdzm" Jul 27 18:38:10.039062 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:38:10.039033 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-qf772\" is forbidden: User \"system:node:ip-10-0-128-32.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-32.ec2.internal' and this object" podUID="a8bd966a-43df-41bc-a9bf-84059d281598" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qf772" Jul 27 18:38:10.042095 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:10.041699 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-lgnxn" event={"ID":"d4cbf21a-5dad-42c5-93ef-f2fa08e5b320","Type":"ContainerStarted","Data":"14eec546037c518c5e23ad1fc42a84d9f85fb7b219878a33b59f6cc02e3dd093"} Jul 27 18:38:10.043944 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:10.043921 2579 scope.go:117] "RemoveContainer" containerID="4b1d836683671b76faaebed2adff3377b7c87648f8ac5f09abfafedd0597d987" Jul 27 18:38:10.065456 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:10.065411 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-pbv7d" podStartSLOduration=14.06539471 podStartE2EDuration="14.06539471s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:37:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:38:10.064965383 +0000 UTC m=+565.126654415" watchObservedRunningTime="2026-07-27 18:38:10.06539471 +0000 UTC m=+565.127083740" Jul 27 18:38:10.067159 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:38:10.067135 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-5nmgk\" is forbidden: User \"system:node:ip-10-0-128-32.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-32.ec2.internal' and this object" podUID="a1c25cbc-3555-43ee-a5d7-745881024718" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5nmgk" Jul 27 18:38:10.071594 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:38:10.071570 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-qf772\" is forbidden: User \"system:node:ip-10-0-128-32.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-32.ec2.internal' and this object" podUID="a8bd966a-43df-41bc-a9bf-84059d281598" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qf772" Jul 27 18:38:10.073860 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:38:10.073834 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-5nmgk\" is forbidden: User \"system:node:ip-10-0-128-32.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-32.ec2.internal' and this object" podUID="a1c25cbc-3555-43ee-a5d7-745881024718" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5nmgk" Jul 27 18:38:10.106126 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:10.106083 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-jwdzm" podStartSLOduration=14.106069324 podStartE2EDuration="14.106069324s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:37:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:38:10.105916126 +0000 UTC m=+565.167605159" watchObservedRunningTime="2026-07-27 18:38:10.106069324 +0000 UTC m=+565.167758355" Jul 27 18:38:10.286687 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:10.286661 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:38:10.392729 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:10.392695 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:38:11.500863 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:38:11.500826 2579 log.go:32] "RunPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-6fb7879bc9-bd7dv_kuadrant-system_4fedfed5-fc9b-4549-b685-6ad4eb8fda9a_0(4dbda0082267e5234a3315a88d50e7d195208cf7caf319659017f163f1b65e62): error adding pod kuadrant-system_kuadrant-operator-controller-manager-6fb7879bc9-bd7dv to CNI network \"multus-cni-network\": plugin type=\"multus-shim\" name=\"multus-cni-network\" failed (add): CmdAdd (shim): failed to send CNI request: Post \"http://dummy/cni\": EOF: StdinData: {\"auxiliaryCNIChainName\":\"vendor-cni-chain\",\"binDir\":\"/var/lib/cni/bin\",\"clusterNetwork\":\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\",\"cniVersion\":\"0.3.1\",\"daemonSocketDir\":\"/run/multus/socket\",\"globalNamespaces\":\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\",\"logLevel\":\"verbose\",\"logToStderr\":true,\"name\":\"multus-cni-network\",\"namespaceIsolation\":true,\"type\":\"multus-shim\"}" Jul 27 18:38:11.501123 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:38:11.500884 2579 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err="rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-6fb7879bc9-bd7dv_kuadrant-system_4fedfed5-fc9b-4549-b685-6ad4eb8fda9a_0(4dbda0082267e5234a3315a88d50e7d195208cf7caf319659017f163f1b65e62): error adding pod kuadrant-system_kuadrant-operator-controller-manager-6fb7879bc9-bd7dv to CNI network \"multus-cni-network\": plugin type=\"multus-shim\" name=\"multus-cni-network\" failed (add): CmdAdd (shim): failed to send CNI request: Post \"http://dummy/cni\": EOF: StdinData: {\"auxiliaryCNIChainName\":\"vendor-cni-chain\",\"binDir\":\"/var/lib/cni/bin\",\"clusterNetwork\":\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\",\"cniVersion\":\"0.3.1\",\"daemonSocketDir\":\"/run/multus/socket\",\"globalNamespaces\":\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\",\"logLevel\":\"verbose\",\"logToStderr\":true,\"name\":\"multus-cni-network\",\"namespaceIsolation\":true,\"type\":\"multus-shim\"}" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-bd7dv" Jul 27 18:38:12.052645 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:12.052614 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-bd7dv" Jul 27 18:38:12.055429 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:38:12.055402 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-bd7dv\" is forbidden: User \"system:node:ip-10-0-128-32.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-32.ec2.internal' and this object" podUID="4fedfed5-fc9b-4549-b685-6ad4eb8fda9a" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-bd7dv" Jul 27 18:38:12.059195 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:12.059175 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-bd7dv" Jul 27 18:38:12.061671 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:38:12.061651 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-bd7dv\" is forbidden: User \"system:node:ip-10-0-128-32.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-32.ec2.internal' and this object" podUID="4fedfed5-fc9b-4549-b685-6ad4eb8fda9a" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-bd7dv" Jul 27 18:38:12.088004 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:12.087985 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4fedfed5-fc9b-4549-b685-6ad4eb8fda9a-extensions-socket-volume\") pod \"4fedfed5-fc9b-4549-b685-6ad4eb8fda9a\" (UID: \"4fedfed5-fc9b-4549-b685-6ad4eb8fda9a\") " Jul 27 18:38:12.088099 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:12.088085 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g462z\" (UniqueName: \"kubernetes.io/projected/4fedfed5-fc9b-4549-b685-6ad4eb8fda9a-kube-api-access-g462z\") pod \"4fedfed5-fc9b-4549-b685-6ad4eb8fda9a\" (UID: \"4fedfed5-fc9b-4549-b685-6ad4eb8fda9a\") " Jul 27 18:38:12.088276 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:12.088224 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fedfed5-fc9b-4549-b685-6ad4eb8fda9a-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "4fedfed5-fc9b-4549-b685-6ad4eb8fda9a" (UID: "4fedfed5-fc9b-4549-b685-6ad4eb8fda9a"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 18:38:12.090292 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:12.090274 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fedfed5-fc9b-4549-b685-6ad4eb8fda9a-kube-api-access-g462z" (OuterVolumeSpecName: "kube-api-access-g462z") pod "4fedfed5-fc9b-4549-b685-6ad4eb8fda9a" (UID: "4fedfed5-fc9b-4549-b685-6ad4eb8fda9a"). InnerVolumeSpecName "kube-api-access-g462z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 18:38:12.188744 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:12.188704 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g462z\" (UniqueName: \"kubernetes.io/projected/4fedfed5-fc9b-4549-b685-6ad4eb8fda9a-kube-api-access-g462z\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 18:38:12.188744 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:12.188743 2579 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4fedfed5-fc9b-4549-b685-6ad4eb8fda9a-extensions-socket-volume\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 18:38:13.057845 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:13.057813 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-bd7dv" Jul 27 18:38:13.060414 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:38:13.060385 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-bd7dv\" is forbidden: User \"system:node:ip-10-0-128-32.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-32.ec2.internal' and this object" podUID="4fedfed5-fc9b-4549-b685-6ad4eb8fda9a" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-bd7dv" Jul 27 18:38:13.065907 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:38:13.065881 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-bd7dv\" is forbidden: User \"system:node:ip-10-0-128-32.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-32.ec2.internal' and this object" podUID="4fedfed5-fc9b-4549-b685-6ad4eb8fda9a" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-bd7dv" Jul 27 18:38:13.496525 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:13.496451 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4fedfed5-fc9b-4549-b685-6ad4eb8fda9a" path="/var/lib/kubelet/pods/4fedfed5-fc9b-4549-b685-6ad4eb8fda9a/volumes" Jul 27 18:38:19.417628 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:19.417593 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:38:19.532118 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:19.532079 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:38:21.050451 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:21.050418 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-jwdzm" Jul 27 18:38:21.050869 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:21.050634 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-pbv7d" Jul 27 18:38:21.073088 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:21.073035 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-lgnxn" podStartSLOduration=13.144003293 podStartE2EDuration="36.073016741s" totalImagesPullingTime="22.929013448s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:37:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:38:10.14040211 +0000 UTC m=+565.202091141" watchObservedRunningTime="2026-07-27 18:38:21.073016741 +0000 UTC m=+576.134705773" Jul 27 18:38:21.128794 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:21.128762 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-pbv7d"] Jul 27 18:38:21.128977 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:21.128955 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-pbv7d" podUID="150d56bd-a044-490c-a8d3-611f0948a955" containerName="manager" containerID="cri-o://086ec366913fb06a3ba7705e5b674d4d3d3674d5beb8fb3c17e16c3d6bb645a0" gracePeriod=10 Jul 27 18:38:21.366686 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:21.366655 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mzvxm"] Jul 27 18:38:21.367194 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:21.367177 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1c25cbc-3555-43ee-a5d7-745881024718" containerName="manager" Jul 27 18:38:21.367287 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:21.367198 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1c25cbc-3555-43ee-a5d7-745881024718" containerName="manager" Jul 27 18:38:21.367362 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:21.367350 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1c25cbc-3555-43ee-a5d7-745881024718" containerName="manager" Jul 27 18:38:21.376982 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:21.376962 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mzvxm" Jul 27 18:38:21.378781 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:21.378762 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-pbv7d" Jul 27 18:38:21.387639 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:21.387611 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mzvxm"] Jul 27 18:38:21.460836 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:21.460808 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/150d56bd-a044-490c-a8d3-611f0948a955-extensions-socket-volume\") pod \"150d56bd-a044-490c-a8d3-611f0948a955\" (UID: \"150d56bd-a044-490c-a8d3-611f0948a955\") " Jul 27 18:38:21.460987 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:21.460916 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6bqj2\" (UniqueName: \"kubernetes.io/projected/150d56bd-a044-490c-a8d3-611f0948a955-kube-api-access-6bqj2\") pod \"150d56bd-a044-490c-a8d3-611f0948a955\" (UID: \"150d56bd-a044-490c-a8d3-611f0948a955\") " Jul 27 18:38:21.461114 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:21.461084 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s76w2\" (UniqueName: \"kubernetes.io/projected/ab81bc00-e76a-43be-882c-3ef4a8cae1d5-kube-api-access-s76w2\") pod \"kuadrant-operator-controller-manager-86878c769f-mzvxm\" (UID: \"ab81bc00-e76a-43be-882c-3ef4a8cae1d5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mzvxm" Jul 27 18:38:21.461254 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:21.461207 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/150d56bd-a044-490c-a8d3-611f0948a955-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "150d56bd-a044-490c-a8d3-611f0948a955" (UID: "150d56bd-a044-490c-a8d3-611f0948a955"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 18:38:21.461332 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:21.461308 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ab81bc00-e76a-43be-882c-3ef4a8cae1d5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-mzvxm\" (UID: \"ab81bc00-e76a-43be-882c-3ef4a8cae1d5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mzvxm" Jul 27 18:38:21.461512 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:21.461496 2579 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/150d56bd-a044-490c-a8d3-611f0948a955-extensions-socket-volume\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 18:38:21.463044 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:21.463017 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/150d56bd-a044-490c-a8d3-611f0948a955-kube-api-access-6bqj2" (OuterVolumeSpecName: "kube-api-access-6bqj2") pod "150d56bd-a044-490c-a8d3-611f0948a955" (UID: "150d56bd-a044-490c-a8d3-611f0948a955"). InnerVolumeSpecName "kube-api-access-6bqj2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 18:38:21.562125 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:21.562093 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ab81bc00-e76a-43be-882c-3ef4a8cae1d5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-mzvxm\" (UID: \"ab81bc00-e76a-43be-882c-3ef4a8cae1d5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mzvxm" Jul 27 18:38:21.562297 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:21.562180 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s76w2\" (UniqueName: \"kubernetes.io/projected/ab81bc00-e76a-43be-882c-3ef4a8cae1d5-kube-api-access-s76w2\") pod \"kuadrant-operator-controller-manager-86878c769f-mzvxm\" (UID: \"ab81bc00-e76a-43be-882c-3ef4a8cae1d5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mzvxm" Jul 27 18:38:21.562297 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:21.562276 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bqj2\" (UniqueName: \"kubernetes.io/projected/150d56bd-a044-490c-a8d3-611f0948a955-kube-api-access-6bqj2\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 18:38:21.562598 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:21.562576 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ab81bc00-e76a-43be-882c-3ef4a8cae1d5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-mzvxm\" (UID: \"ab81bc00-e76a-43be-882c-3ef4a8cae1d5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mzvxm" Jul 27 18:38:21.571774 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:21.571749 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s76w2\" (UniqueName: \"kubernetes.io/projected/ab81bc00-e76a-43be-882c-3ef4a8cae1d5-kube-api-access-s76w2\") pod \"kuadrant-operator-controller-manager-86878c769f-mzvxm\" (UID: \"ab81bc00-e76a-43be-882c-3ef4a8cae1d5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mzvxm" Jul 27 18:38:21.684657 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:21.684574 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:38:21.687425 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:21.687406 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mzvxm" Jul 27 18:38:21.786419 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:21.786392 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:38:21.840280 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:21.840256 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mzvxm"] Jul 27 18:38:21.842215 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:38:21.842189 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab81bc00_e76a_43be_882c_3ef4a8cae1d5.slice/crio-d11e63bf7ff3072d2e13fc7b10e91ff86bc4adefe387939f95341f7cb06efda2 WatchSource:0}: Error finding container d11e63bf7ff3072d2e13fc7b10e91ff86bc4adefe387939f95341f7cb06efda2: Status 404 returned error can't find the container with id d11e63bf7ff3072d2e13fc7b10e91ff86bc4adefe387939f95341f7cb06efda2 Jul 27 18:38:22.103532 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:22.103499 2579 generic.go:356] "Generic (PLEG): container finished" podID="150d56bd-a044-490c-a8d3-611f0948a955" containerID="086ec366913fb06a3ba7705e5b674d4d3d3674d5beb8fb3c17e16c3d6bb645a0" exitCode=0 Jul 27 18:38:22.103947 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:22.103539 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-pbv7d" Jul 27 18:38:22.103947 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:22.103593 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-pbv7d" event={"ID":"150d56bd-a044-490c-a8d3-611f0948a955","Type":"ContainerDied","Data":"086ec366913fb06a3ba7705e5b674d4d3d3674d5beb8fb3c17e16c3d6bb645a0"} Jul 27 18:38:22.103947 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:22.103647 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-pbv7d" event={"ID":"150d56bd-a044-490c-a8d3-611f0948a955","Type":"ContainerDied","Data":"1b0b290b677925e26b457a226558ff21ddc2d62fe138d8dab527ce4c2006bafd"} Jul 27 18:38:22.103947 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:22.103693 2579 scope.go:117] "RemoveContainer" containerID="086ec366913fb06a3ba7705e5b674d4d3d3674d5beb8fb3c17e16c3d6bb645a0" Jul 27 18:38:22.105394 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:22.105365 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mzvxm" event={"ID":"ab81bc00-e76a-43be-882c-3ef4a8cae1d5","Type":"ContainerStarted","Data":"c3a51c7612e7754c82339943a50445a10e045240ebb51eab6fb97325842b48db"} Jul 27 18:38:22.105527 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:22.105416 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mzvxm" event={"ID":"ab81bc00-e76a-43be-882c-3ef4a8cae1d5","Type":"ContainerStarted","Data":"d11e63bf7ff3072d2e13fc7b10e91ff86bc4adefe387939f95341f7cb06efda2"} Jul 27 18:38:22.105527 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:22.105462 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mzvxm" Jul 27 18:38:22.113052 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:22.113035 2579 scope.go:117] "RemoveContainer" containerID="086ec366913fb06a3ba7705e5b674d4d3d3674d5beb8fb3c17e16c3d6bb645a0" Jul 27 18:38:22.113347 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:38:22.113331 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"086ec366913fb06a3ba7705e5b674d4d3d3674d5beb8fb3c17e16c3d6bb645a0\": container with ID starting with 086ec366913fb06a3ba7705e5b674d4d3d3674d5beb8fb3c17e16c3d6bb645a0 not found: ID does not exist" containerID="086ec366913fb06a3ba7705e5b674d4d3d3674d5beb8fb3c17e16c3d6bb645a0" Jul 27 18:38:22.113417 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:22.113360 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"086ec366913fb06a3ba7705e5b674d4d3d3674d5beb8fb3c17e16c3d6bb645a0"} err="failed to get container status \"086ec366913fb06a3ba7705e5b674d4d3d3674d5beb8fb3c17e16c3d6bb645a0\": rpc error: code = NotFound desc = could not find container \"086ec366913fb06a3ba7705e5b674d4d3d3674d5beb8fb3c17e16c3d6bb645a0\": container with ID starting with 086ec366913fb06a3ba7705e5b674d4d3d3674d5beb8fb3c17e16c3d6bb645a0 not found: ID does not exist" Jul 27 18:38:22.120847 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:22.120794 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-pbv7d"] Jul 27 18:38:22.123224 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:22.123194 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-pbv7d"] Jul 27 18:38:22.141224 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:22.141172 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mzvxm" podStartSLOduration=1.14115796 podStartE2EDuration="1.14115796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:38:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:38:22.139527054 +0000 UTC m=+577.201216087" watchObservedRunningTime="2026-07-27 18:38:22.14115796 +0000 UTC m=+577.202846988" Jul 27 18:38:23.492949 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:23.492916 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="150d56bd-a044-490c-a8d3-611f0948a955" path="/var/lib/kubelet/pods/150d56bd-a044-490c-a8d3-611f0948a955/volumes" Jul 27 18:38:24.868192 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:24.868152 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:38:24.968122 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:24.968089 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:38:28.761306 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:28.761271 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:38:28.862476 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:28.862427 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:38:33.113487 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:33.113451 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mzvxm" Jul 27 18:38:33.169818 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:33.169781 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-jwdzm"] Jul 27 18:38:33.170053 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:33.170008 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-jwdzm" podUID="af04ced4-4b4a-4661-b63a-f0fcc3cc66f1" containerName="manager" containerID="cri-o://020a91124a6c560aee95239c4ba6061fc3781cb439637c3c2c719eb852e1a22f" gracePeriod=10 Jul 27 18:38:33.718589 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:33.718507 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-jwdzm" Jul 27 18:38:33.762760 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:33.762733 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/af04ced4-4b4a-4661-b63a-f0fcc3cc66f1-extensions-socket-volume\") pod \"af04ced4-4b4a-4661-b63a-f0fcc3cc66f1\" (UID: \"af04ced4-4b4a-4661-b63a-f0fcc3cc66f1\") " Jul 27 18:38:33.762760 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:33.762766 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w9jbn\" (UniqueName: \"kubernetes.io/projected/af04ced4-4b4a-4661-b63a-f0fcc3cc66f1-kube-api-access-w9jbn\") pod \"af04ced4-4b4a-4661-b63a-f0fcc3cc66f1\" (UID: \"af04ced4-4b4a-4661-b63a-f0fcc3cc66f1\") " Jul 27 18:38:33.763126 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:33.763099 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af04ced4-4b4a-4661-b63a-f0fcc3cc66f1-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "af04ced4-4b4a-4661-b63a-f0fcc3cc66f1" (UID: "af04ced4-4b4a-4661-b63a-f0fcc3cc66f1"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 18:38:33.764910 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:33.764880 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af04ced4-4b4a-4661-b63a-f0fcc3cc66f1-kube-api-access-w9jbn" (OuterVolumeSpecName: "kube-api-access-w9jbn") pod "af04ced4-4b4a-4661-b63a-f0fcc3cc66f1" (UID: "af04ced4-4b4a-4661-b63a-f0fcc3cc66f1"). InnerVolumeSpecName "kube-api-access-w9jbn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 18:38:33.847983 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:33.847943 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:38:33.864005 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:33.863980 2579 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/af04ced4-4b4a-4661-b63a-f0fcc3cc66f1-extensions-socket-volume\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 18:38:33.864005 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:33.864003 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9jbn\" (UniqueName: \"kubernetes.io/projected/af04ced4-4b4a-4661-b63a-f0fcc3cc66f1-kube-api-access-w9jbn\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 18:38:34.169301 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:34.169251 2579 generic.go:356] "Generic (PLEG): container finished" podID="af04ced4-4b4a-4661-b63a-f0fcc3cc66f1" containerID="020a91124a6c560aee95239c4ba6061fc3781cb439637c3c2c719eb852e1a22f" exitCode=0 Jul 27 18:38:34.169301 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:34.169291 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-jwdzm" Jul 27 18:38:34.169828 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:34.169317 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-jwdzm" event={"ID":"af04ced4-4b4a-4661-b63a-f0fcc3cc66f1","Type":"ContainerDied","Data":"020a91124a6c560aee95239c4ba6061fc3781cb439637c3c2c719eb852e1a22f"} Jul 27 18:38:34.169828 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:34.169364 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-jwdzm" event={"ID":"af04ced4-4b4a-4661-b63a-f0fcc3cc66f1","Type":"ContainerDied","Data":"9ecc5e91ae38be051d19c2365c1ae66d60d8d8e0b3c9558d7dc91fb9fcb21d2f"} Jul 27 18:38:34.169828 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:34.169387 2579 scope.go:117] "RemoveContainer" containerID="020a91124a6c560aee95239c4ba6061fc3781cb439637c3c2c719eb852e1a22f" Jul 27 18:38:34.179192 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:34.178947 2579 scope.go:117] "RemoveContainer" containerID="020a91124a6c560aee95239c4ba6061fc3781cb439637c3c2c719eb852e1a22f" Jul 27 18:38:34.179347 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:38:34.179278 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"020a91124a6c560aee95239c4ba6061fc3781cb439637c3c2c719eb852e1a22f\": container with ID starting with 020a91124a6c560aee95239c4ba6061fc3781cb439637c3c2c719eb852e1a22f not found: ID does not exist" containerID="020a91124a6c560aee95239c4ba6061fc3781cb439637c3c2c719eb852e1a22f" Jul 27 18:38:34.179347 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:34.179305 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"020a91124a6c560aee95239c4ba6061fc3781cb439637c3c2c719eb852e1a22f"} err="failed to get container status \"020a91124a6c560aee95239c4ba6061fc3781cb439637c3c2c719eb852e1a22f\": rpc error: code = NotFound desc = could not find container \"020a91124a6c560aee95239c4ba6061fc3781cb439637c3c2c719eb852e1a22f\": container with ID starting with 020a91124a6c560aee95239c4ba6061fc3781cb439637c3c2c719eb852e1a22f not found: ID does not exist" Jul 27 18:38:34.206386 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:34.206362 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-jwdzm"] Jul 27 18:38:34.210973 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:34.210949 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-jwdzm"] Jul 27 18:38:35.492824 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:35.492791 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="af04ced4-4b4a-4661-b63a-f0fcc3cc66f1" path="/var/lib/kubelet/pods/af04ced4-4b4a-4661-b63a-f0fcc3cc66f1/volumes" Jul 27 18:38:37.576333 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:37.576298 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-j2vmg"] Jul 27 18:38:37.576684 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:37.576663 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af04ced4-4b4a-4661-b63a-f0fcc3cc66f1" containerName="manager" Jul 27 18:38:37.576684 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:37.576677 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="af04ced4-4b4a-4661-b63a-f0fcc3cc66f1" containerName="manager" Jul 27 18:38:37.576754 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:37.576696 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="150d56bd-a044-490c-a8d3-611f0948a955" containerName="manager" Jul 27 18:38:37.576754 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:37.576703 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="150d56bd-a044-490c-a8d3-611f0948a955" containerName="manager" Jul 27 18:38:37.576817 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:37.576757 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="af04ced4-4b4a-4661-b63a-f0fcc3cc66f1" containerName="manager" Jul 27 18:38:37.576817 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:37.576765 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="150d56bd-a044-490c-a8d3-611f0948a955" containerName="manager" Jul 27 18:38:37.605472 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:37.605433 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-j2vmg"] Jul 27 18:38:37.605631 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:37.605541 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-j2vmg" Jul 27 18:38:37.608087 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:37.608058 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:38:37.608300 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:37.608282 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-n97zp\"" Jul 27 18:38:37.673006 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:37.672971 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:38:37.692455 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:37.692426 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/429930eb-3d30-450f-8e99-75ab3f80ba28-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-j2vmg\" (UID: \"429930eb-3d30-450f-8e99-75ab3f80ba28\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-j2vmg" Jul 27 18:38:37.692605 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:37.692467 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/429930eb-3d30-450f-8e99-75ab3f80ba28-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-j2vmg\" (UID: \"429930eb-3d30-450f-8e99-75ab3f80ba28\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-j2vmg" Jul 27 18:38:37.692605 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:37.692515 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/429930eb-3d30-450f-8e99-75ab3f80ba28-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-j2vmg\" (UID: \"429930eb-3d30-450f-8e99-75ab3f80ba28\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-j2vmg" Jul 27 18:38:37.692605 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:37.692542 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/429930eb-3d30-450f-8e99-75ab3f80ba28-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-j2vmg\" (UID: \"429930eb-3d30-450f-8e99-75ab3f80ba28\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-j2vmg" Jul 27 18:38:37.692605 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:37.692592 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/429930eb-3d30-450f-8e99-75ab3f80ba28-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-j2vmg\" (UID: \"429930eb-3d30-450f-8e99-75ab3f80ba28\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-j2vmg" Jul 27 18:38:37.692738 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:37.692610 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/429930eb-3d30-450f-8e99-75ab3f80ba28-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-j2vmg\" (UID: \"429930eb-3d30-450f-8e99-75ab3f80ba28\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-j2vmg" Jul 27 18:38:37.692738 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:37.692657 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kq4g8\" (UniqueName: \"kubernetes.io/projected/429930eb-3d30-450f-8e99-75ab3f80ba28-kube-api-access-kq4g8\") pod \"maas-default-gateway-openshift-default-78bb78ff78-j2vmg\" (UID: \"429930eb-3d30-450f-8e99-75ab3f80ba28\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-j2vmg" Jul 27 18:38:37.692738 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:37.692694 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/429930eb-3d30-450f-8e99-75ab3f80ba28-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-j2vmg\" (UID: \"429930eb-3d30-450f-8e99-75ab3f80ba28\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-j2vmg" Jul 27 18:38:37.692738 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:37.692713 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/429930eb-3d30-450f-8e99-75ab3f80ba28-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-j2vmg\" (UID: \"429930eb-3d30-450f-8e99-75ab3f80ba28\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-j2vmg" Jul 27 18:38:37.793393 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:37.793357 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/429930eb-3d30-450f-8e99-75ab3f80ba28-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-j2vmg\" (UID: \"429930eb-3d30-450f-8e99-75ab3f80ba28\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-j2vmg" Jul 27 18:38:37.793552 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:37.793406 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/429930eb-3d30-450f-8e99-75ab3f80ba28-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-j2vmg\" (UID: \"429930eb-3d30-450f-8e99-75ab3f80ba28\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-j2vmg" Jul 27 18:38:37.793552 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:37.793467 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/429930eb-3d30-450f-8e99-75ab3f80ba28-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-j2vmg\" (UID: \"429930eb-3d30-450f-8e99-75ab3f80ba28\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-j2vmg" Jul 27 18:38:37.793552 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:37.793492 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/429930eb-3d30-450f-8e99-75ab3f80ba28-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-j2vmg\" (UID: \"429930eb-3d30-450f-8e99-75ab3f80ba28\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-j2vmg" Jul 27 18:38:37.793704 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:37.793619 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kq4g8\" (UniqueName: \"kubernetes.io/projected/429930eb-3d30-450f-8e99-75ab3f80ba28-kube-api-access-kq4g8\") pod \"maas-default-gateway-openshift-default-78bb78ff78-j2vmg\" (UID: \"429930eb-3d30-450f-8e99-75ab3f80ba28\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-j2vmg" Jul 27 18:38:37.793704 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:37.793658 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/429930eb-3d30-450f-8e99-75ab3f80ba28-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-j2vmg\" (UID: \"429930eb-3d30-450f-8e99-75ab3f80ba28\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-j2vmg" Jul 27 18:38:37.793704 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:37.793680 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/429930eb-3d30-450f-8e99-75ab3f80ba28-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-j2vmg\" (UID: \"429930eb-3d30-450f-8e99-75ab3f80ba28\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-j2vmg" Jul 27 18:38:37.794264 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:37.794171 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/429930eb-3d30-450f-8e99-75ab3f80ba28-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-j2vmg\" (UID: \"429930eb-3d30-450f-8e99-75ab3f80ba28\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-j2vmg" Jul 27 18:38:37.794264 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:37.794218 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/429930eb-3d30-450f-8e99-75ab3f80ba28-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-j2vmg\" (UID: \"429930eb-3d30-450f-8e99-75ab3f80ba28\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-j2vmg" Jul 27 18:38:37.794470 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:37.794280 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/429930eb-3d30-450f-8e99-75ab3f80ba28-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-j2vmg\" (UID: \"429930eb-3d30-450f-8e99-75ab3f80ba28\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-j2vmg" Jul 27 18:38:37.794470 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:37.794391 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/429930eb-3d30-450f-8e99-75ab3f80ba28-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-j2vmg\" (UID: \"429930eb-3d30-450f-8e99-75ab3f80ba28\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-j2vmg" Jul 27 18:38:37.794470 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:37.794448 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/429930eb-3d30-450f-8e99-75ab3f80ba28-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-j2vmg\" (UID: \"429930eb-3d30-450f-8e99-75ab3f80ba28\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-j2vmg" Jul 27 18:38:37.794621 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:37.794542 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/429930eb-3d30-450f-8e99-75ab3f80ba28-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-j2vmg\" (UID: \"429930eb-3d30-450f-8e99-75ab3f80ba28\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-j2vmg" Jul 27 18:38:37.799253 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:37.794715 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/429930eb-3d30-450f-8e99-75ab3f80ba28-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-j2vmg\" (UID: \"429930eb-3d30-450f-8e99-75ab3f80ba28\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-j2vmg" Jul 27 18:38:37.799253 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:37.797662 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/429930eb-3d30-450f-8e99-75ab3f80ba28-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-j2vmg\" (UID: \"429930eb-3d30-450f-8e99-75ab3f80ba28\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-j2vmg" Jul 27 18:38:37.800271 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:37.799997 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/429930eb-3d30-450f-8e99-75ab3f80ba28-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-j2vmg\" (UID: \"429930eb-3d30-450f-8e99-75ab3f80ba28\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-j2vmg" Jul 27 18:38:37.806812 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:37.806787 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/429930eb-3d30-450f-8e99-75ab3f80ba28-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-j2vmg\" (UID: \"429930eb-3d30-450f-8e99-75ab3f80ba28\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-j2vmg" Jul 27 18:38:37.806942 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:37.806816 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kq4g8\" (UniqueName: \"kubernetes.io/projected/429930eb-3d30-450f-8e99-75ab3f80ba28-kube-api-access-kq4g8\") pod \"maas-default-gateway-openshift-default-78bb78ff78-j2vmg\" (UID: \"429930eb-3d30-450f-8e99-75ab3f80ba28\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-j2vmg" Jul 27 18:38:37.918683 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:37.918601 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-j2vmg" Jul 27 18:38:38.149453 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:38.149431 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-j2vmg"] Jul 27 18:38:38.151820 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:38:38.151785 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod429930eb_3d30_450f_8e99_75ab3f80ba28.slice/crio-196aa1bc195471b632f73ede0122280a5006681879695d3857bc7671d711742b WatchSource:0}: Error finding container 196aa1bc195471b632f73ede0122280a5006681879695d3857bc7671d711742b: Status 404 returned error can't find the container with id 196aa1bc195471b632f73ede0122280a5006681879695d3857bc7671d711742b Jul 27 18:38:38.153964 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:38.153929 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 27 18:38:38.154039 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:38.154003 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 27 18:38:38.154082 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:38.154035 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 27 18:38:38.192869 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:38.192838 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-j2vmg" event={"ID":"429930eb-3d30-450f-8e99-75ab3f80ba28","Type":"ContainerStarted","Data":"196aa1bc195471b632f73ede0122280a5006681879695d3857bc7671d711742b"} Jul 27 18:38:38.465998 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:38.465887 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:38:38.528087 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:38.528056 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:38:39.075365 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:39.075320 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:38:39.199429 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:39.199392 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-j2vmg" event={"ID":"429930eb-3d30-450f-8e99-75ab3f80ba28","Type":"ContainerStarted","Data":"04505a7b08b666acf97a43ab1d913b874815b7a753a2faaeee5cbe54c10f4441"} Jul 27 18:38:39.225031 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:39.224975 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-j2vmg" podStartSLOduration=2.22495701 podStartE2EDuration="2.22495701s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:38:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:38:39.222743133 +0000 UTC m=+594.284432164" watchObservedRunningTime="2026-07-27 18:38:39.22495701 +0000 UTC m=+594.286646041" Jul 27 18:38:39.432450 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:39.432339 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:38:39.919562 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:39.919528 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-j2vmg" Jul 27 18:38:39.924301 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:39.924275 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-j2vmg" Jul 27 18:38:40.204336 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:40.204253 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-j2vmg" Jul 27 18:38:40.205283 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:40.205260 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-j2vmg" Jul 27 18:38:45.385562 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:45.385533 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-mv4hb_715b2084-d79e-4cbf-a057-f2595b4e6b9b/console-operator/1.log" Jul 27 18:38:45.389462 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:45.389443 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-mv4hb_715b2084-d79e-4cbf-a057-f2595b4e6b9b/console-operator/1.log" Jul 27 18:38:51.475187 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:51.475148 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:38:52.953451 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:52.953414 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:38:52.963053 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:52.963029 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-tf7lk"] Jul 27 18:38:52.966276 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:52.966256 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-tf7lk" Jul 27 18:38:52.969363 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:52.969342 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 27 18:38:52.974301 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:52.974279 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-tf7lk"] Jul 27 18:38:53.057718 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:53.057679 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-tf7lk"] Jul 27 18:38:53.071658 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:53.071623 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:38:53.132832 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:53.132797 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2wqm\" (UniqueName: \"kubernetes.io/projected/a1df17f8-2947-4e18-ab1d-7ddec4838584-kube-api-access-q2wqm\") pod \"limitador-limitador-64c94fd64f-tf7lk\" (UID: \"a1df17f8-2947-4e18-ab1d-7ddec4838584\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-tf7lk" Jul 27 18:38:53.133006 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:53.132846 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/a1df17f8-2947-4e18-ab1d-7ddec4838584-config-file\") pod \"limitador-limitador-64c94fd64f-tf7lk\" (UID: \"a1df17f8-2947-4e18-ab1d-7ddec4838584\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-tf7lk" Jul 27 18:38:53.134125 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:53.134101 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:38:53.234330 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:53.234218 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2wqm\" (UniqueName: \"kubernetes.io/projected/a1df17f8-2947-4e18-ab1d-7ddec4838584-kube-api-access-q2wqm\") pod \"limitador-limitador-64c94fd64f-tf7lk\" (UID: \"a1df17f8-2947-4e18-ab1d-7ddec4838584\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-tf7lk" Jul 27 18:38:53.234330 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:53.234301 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/a1df17f8-2947-4e18-ab1d-7ddec4838584-config-file\") pod \"limitador-limitador-64c94fd64f-tf7lk\" (UID: \"a1df17f8-2947-4e18-ab1d-7ddec4838584\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-tf7lk" Jul 27 18:38:53.234971 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:53.234952 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/a1df17f8-2947-4e18-ab1d-7ddec4838584-config-file\") pod \"limitador-limitador-64c94fd64f-tf7lk\" (UID: \"a1df17f8-2947-4e18-ab1d-7ddec4838584\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-tf7lk" Jul 27 18:38:53.244120 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:53.244076 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2wqm\" (UniqueName: \"kubernetes.io/projected/a1df17f8-2947-4e18-ab1d-7ddec4838584-kube-api-access-q2wqm\") pod \"limitador-limitador-64c94fd64f-tf7lk\" (UID: \"a1df17f8-2947-4e18-ab1d-7ddec4838584\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-tf7lk" Jul 27 18:38:53.277096 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:53.277066 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-tf7lk" Jul 27 18:38:53.417534 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:53.417451 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-tf7lk"] Jul 27 18:38:53.420149 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:38:53.420114 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1df17f8_2947_4e18_ab1d_7ddec4838584.slice/crio-731d1de008e121476008c664bdcf152487d63050f833c4ff0c59146986aabebf WatchSource:0}: Error finding container 731d1de008e121476008c664bdcf152487d63050f833c4ff0c59146986aabebf: Status 404 returned error can't find the container with id 731d1de008e121476008c664bdcf152487d63050f833c4ff0c59146986aabebf Jul 27 18:38:53.743820 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:53.743761 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-r849t"] Jul 27 18:38:53.748682 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:53.748659 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-r849t" Jul 27 18:38:53.751446 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:53.751423 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-d825p\"" Jul 27 18:38:53.759803 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:53.759777 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-r849t"] Jul 27 18:38:53.778366 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:53.778340 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:38:53.838422 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:53.838389 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:38:53.840524 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:53.840502 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rlf6\" (UniqueName: \"kubernetes.io/projected/af2904cc-5c49-4690-b6c9-dfc651fefaf1-kube-api-access-5rlf6\") pod \"authorino-58cd955675-r849t\" (UID: \"af2904cc-5c49-4690-b6c9-dfc651fefaf1\") " pod="kuadrant-system/authorino-58cd955675-r849t" Jul 27 18:38:53.941948 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:53.941918 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rlf6\" (UniqueName: \"kubernetes.io/projected/af2904cc-5c49-4690-b6c9-dfc651fefaf1-kube-api-access-5rlf6\") pod \"authorino-58cd955675-r849t\" (UID: \"af2904cc-5c49-4690-b6c9-dfc651fefaf1\") " pod="kuadrant-system/authorino-58cd955675-r849t" Jul 27 18:38:53.950992 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:53.950959 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rlf6\" (UniqueName: \"kubernetes.io/projected/af2904cc-5c49-4690-b6c9-dfc651fefaf1-kube-api-access-5rlf6\") pod \"authorino-58cd955675-r849t\" (UID: \"af2904cc-5c49-4690-b6c9-dfc651fefaf1\") " pod="kuadrant-system/authorino-58cd955675-r849t" Jul 27 18:38:54.058781 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:54.058754 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-r849t" Jul 27 18:38:54.205041 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:54.205006 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-r849t"] Jul 27 18:38:54.209160 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:38:54.209116 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaf2904cc_5c49_4690_b6c9_dfc651fefaf1.slice/crio-6a9f1ecb4156bbceb44dfac289e07e45c86d2d2091c9f159dc932911cf7b3aff WatchSource:0}: Error finding container 6a9f1ecb4156bbceb44dfac289e07e45c86d2d2091c9f159dc932911cf7b3aff: Status 404 returned error can't find the container with id 6a9f1ecb4156bbceb44dfac289e07e45c86d2d2091c9f159dc932911cf7b3aff Jul 27 18:38:54.278999 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:54.278925 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-r849t" event={"ID":"af2904cc-5c49-4690-b6c9-dfc651fefaf1","Type":"ContainerStarted","Data":"6a9f1ecb4156bbceb44dfac289e07e45c86d2d2091c9f159dc932911cf7b3aff"} Jul 27 18:38:54.281043 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:54.281007 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-tf7lk" event={"ID":"a1df17f8-2947-4e18-ab1d-7ddec4838584","Type":"ContainerStarted","Data":"731d1de008e121476008c664bdcf152487d63050f833c4ff0c59146986aabebf"} Jul 27 18:38:57.300509 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:57.300475 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-tf7lk" event={"ID":"a1df17f8-2947-4e18-ab1d-7ddec4838584","Type":"ContainerStarted","Data":"dec90461cc16e9feeecd6bab558993ecf1a1e980545226d77abc2261e955be4a"} Jul 27 18:38:57.300985 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:57.300587 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-tf7lk" Jul 27 18:38:57.320578 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:57.320511 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-tf7lk" podStartSLOduration=2.377147788 podStartE2EDuration="5.320489772s" totalImagesPullingTime="2.943341984s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:38:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:38:57.318876595 +0000 UTC m=+612.380565627" watchObservedRunningTime="2026-07-27 18:38:57.320489772 +0000 UTC m=+612.382178804" Jul 27 18:38:59.035054 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:59.035016 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-r849t"] Jul 27 18:38:59.312536 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:59.312441 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-r849t" event={"ID":"af2904cc-5c49-4690-b6c9-dfc651fefaf1","Type":"ContainerStarted","Data":"1f11c75bb31cc76eb647891f3f4e70609f3f3a7947c1f04127337eaefb0dfc13"} Jul 27 18:38:59.329316 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:38:59.329266 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-r849t" podStartSLOduration=2.091531674 podStartE2EDuration="6.329251774s" totalImagesPullingTime="4.2377201s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:38:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:38:59.327833986 +0000 UTC m=+614.389523016" watchObservedRunningTime="2026-07-27 18:38:59.329251774 +0000 UTC m=+614.390940806" Jul 27 18:39:00.316818 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:00.316775 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-r849t" podUID="af2904cc-5c49-4690-b6c9-dfc651fefaf1" containerName="authorino" containerID="cri-o://1f11c75bb31cc76eb647891f3f4e70609f3f3a7947c1f04127337eaefb0dfc13" gracePeriod=30 Jul 27 18:39:00.566800 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:00.566772 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-r849t" Jul 27 18:39:00.706498 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:00.706410 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5rlf6\" (UniqueName: \"kubernetes.io/projected/af2904cc-5c49-4690-b6c9-dfc651fefaf1-kube-api-access-5rlf6\") pod \"af2904cc-5c49-4690-b6c9-dfc651fefaf1\" (UID: \"af2904cc-5c49-4690-b6c9-dfc651fefaf1\") " Jul 27 18:39:00.708632 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:00.708604 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af2904cc-5c49-4690-b6c9-dfc651fefaf1-kube-api-access-5rlf6" (OuterVolumeSpecName: "kube-api-access-5rlf6") pod "af2904cc-5c49-4690-b6c9-dfc651fefaf1" (UID: "af2904cc-5c49-4690-b6c9-dfc651fefaf1"). InnerVolumeSpecName "kube-api-access-5rlf6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 18:39:00.807285 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:00.807253 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rlf6\" (UniqueName: \"kubernetes.io/projected/af2904cc-5c49-4690-b6c9-dfc651fefaf1-kube-api-access-5rlf6\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 18:39:01.323601 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:01.323564 2579 generic.go:356] "Generic (PLEG): container finished" podID="af2904cc-5c49-4690-b6c9-dfc651fefaf1" containerID="1f11c75bb31cc76eb647891f3f4e70609f3f3a7947c1f04127337eaefb0dfc13" exitCode=0 Jul 27 18:39:01.324057 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:01.323608 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-r849t" Jul 27 18:39:01.324057 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:01.323627 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-r849t" event={"ID":"af2904cc-5c49-4690-b6c9-dfc651fefaf1","Type":"ContainerDied","Data":"1f11c75bb31cc76eb647891f3f4e70609f3f3a7947c1f04127337eaefb0dfc13"} Jul 27 18:39:01.324057 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:01.323667 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-r849t" event={"ID":"af2904cc-5c49-4690-b6c9-dfc651fefaf1","Type":"ContainerDied","Data":"6a9f1ecb4156bbceb44dfac289e07e45c86d2d2091c9f159dc932911cf7b3aff"} Jul 27 18:39:01.324057 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:01.323689 2579 scope.go:117] "RemoveContainer" containerID="1f11c75bb31cc76eb647891f3f4e70609f3f3a7947c1f04127337eaefb0dfc13" Jul 27 18:39:01.332701 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:01.332684 2579 scope.go:117] "RemoveContainer" containerID="1f11c75bb31cc76eb647891f3f4e70609f3f3a7947c1f04127337eaefb0dfc13" Jul 27 18:39:01.332978 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:39:01.332960 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f11c75bb31cc76eb647891f3f4e70609f3f3a7947c1f04127337eaefb0dfc13\": container with ID starting with 1f11c75bb31cc76eb647891f3f4e70609f3f3a7947c1f04127337eaefb0dfc13 not found: ID does not exist" containerID="1f11c75bb31cc76eb647891f3f4e70609f3f3a7947c1f04127337eaefb0dfc13" Jul 27 18:39:01.333049 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:01.332992 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f11c75bb31cc76eb647891f3f4e70609f3f3a7947c1f04127337eaefb0dfc13"} err="failed to get container status \"1f11c75bb31cc76eb647891f3f4e70609f3f3a7947c1f04127337eaefb0dfc13\": rpc error: code = NotFound desc = could not find container \"1f11c75bb31cc76eb647891f3f4e70609f3f3a7947c1f04127337eaefb0dfc13\": container with ID starting with 1f11c75bb31cc76eb647891f3f4e70609f3f3a7947c1f04127337eaefb0dfc13 not found: ID does not exist" Jul 27 18:39:01.346556 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:01.346527 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-r849t"] Jul 27 18:39:01.348108 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:01.348085 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-r849t"] Jul 27 18:39:01.493091 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:01.493054 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="af2904cc-5c49-4690-b6c9-dfc651fefaf1" path="/var/lib/kubelet/pods/af2904cc-5c49-4690-b6c9-dfc651fefaf1/volumes" Jul 27 18:39:07.770062 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:07.769972 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-tf7lk"] Jul 27 18:39:07.770788 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:07.770255 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-tf7lk" podUID="a1df17f8-2947-4e18-ab1d-7ddec4838584" containerName="limitador" containerID="cri-o://dec90461cc16e9feeecd6bab558993ecf1a1e980545226d77abc2261e955be4a" gracePeriod=30 Jul 27 18:39:07.771150 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:07.771116 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-tf7lk" Jul 27 18:39:08.328543 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:08.328518 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-tf7lk" Jul 27 18:39:08.358883 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:08.358798 2579 generic.go:356] "Generic (PLEG): container finished" podID="a1df17f8-2947-4e18-ab1d-7ddec4838584" containerID="dec90461cc16e9feeecd6bab558993ecf1a1e980545226d77abc2261e955be4a" exitCode=0 Jul 27 18:39:08.358883 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:08.358823 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-tf7lk" event={"ID":"a1df17f8-2947-4e18-ab1d-7ddec4838584","Type":"ContainerDied","Data":"dec90461cc16e9feeecd6bab558993ecf1a1e980545226d77abc2261e955be4a"} Jul 27 18:39:08.358883 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:08.358839 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-tf7lk" Jul 27 18:39:08.359193 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:08.358888 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-tf7lk" event={"ID":"a1df17f8-2947-4e18-ab1d-7ddec4838584","Type":"ContainerDied","Data":"731d1de008e121476008c664bdcf152487d63050f833c4ff0c59146986aabebf"} Jul 27 18:39:08.359193 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:08.358919 2579 scope.go:117] "RemoveContainer" containerID="dec90461cc16e9feeecd6bab558993ecf1a1e980545226d77abc2261e955be4a" Jul 27 18:39:08.364314 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:08.364289 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/a1df17f8-2947-4e18-ab1d-7ddec4838584-config-file\") pod \"a1df17f8-2947-4e18-ab1d-7ddec4838584\" (UID: \"a1df17f8-2947-4e18-ab1d-7ddec4838584\") " Jul 27 18:39:08.364451 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:08.364376 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q2wqm\" (UniqueName: \"kubernetes.io/projected/a1df17f8-2947-4e18-ab1d-7ddec4838584-kube-api-access-q2wqm\") pod \"a1df17f8-2947-4e18-ab1d-7ddec4838584\" (UID: \"a1df17f8-2947-4e18-ab1d-7ddec4838584\") " Jul 27 18:39:08.364702 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:08.364679 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1df17f8-2947-4e18-ab1d-7ddec4838584-config-file" (OuterVolumeSpecName: "config-file") pod "a1df17f8-2947-4e18-ab1d-7ddec4838584" (UID: "a1df17f8-2947-4e18-ab1d-7ddec4838584"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 18:39:08.366652 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:08.366628 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1df17f8-2947-4e18-ab1d-7ddec4838584-kube-api-access-q2wqm" (OuterVolumeSpecName: "kube-api-access-q2wqm") pod "a1df17f8-2947-4e18-ab1d-7ddec4838584" (UID: "a1df17f8-2947-4e18-ab1d-7ddec4838584"). InnerVolumeSpecName "kube-api-access-q2wqm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 18:39:08.368020 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:08.368003 2579 scope.go:117] "RemoveContainer" containerID="dec90461cc16e9feeecd6bab558993ecf1a1e980545226d77abc2261e955be4a" Jul 27 18:39:08.368315 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:39:08.368298 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dec90461cc16e9feeecd6bab558993ecf1a1e980545226d77abc2261e955be4a\": container with ID starting with dec90461cc16e9feeecd6bab558993ecf1a1e980545226d77abc2261e955be4a not found: ID does not exist" containerID="dec90461cc16e9feeecd6bab558993ecf1a1e980545226d77abc2261e955be4a" Jul 27 18:39:08.368370 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:08.368324 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dec90461cc16e9feeecd6bab558993ecf1a1e980545226d77abc2261e955be4a"} err="failed to get container status \"dec90461cc16e9feeecd6bab558993ecf1a1e980545226d77abc2261e955be4a\": rpc error: code = NotFound desc = could not find container \"dec90461cc16e9feeecd6bab558993ecf1a1e980545226d77abc2261e955be4a\": container with ID starting with dec90461cc16e9feeecd6bab558993ecf1a1e980545226d77abc2261e955be4a not found: ID does not exist" Jul 27 18:39:08.465205 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:08.465167 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2wqm\" (UniqueName: \"kubernetes.io/projected/a1df17f8-2947-4e18-ab1d-7ddec4838584-kube-api-access-q2wqm\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 18:39:08.465205 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:08.465204 2579 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/a1df17f8-2947-4e18-ab1d-7ddec4838584-config-file\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 18:39:08.681571 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:08.681539 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-tf7lk"] Jul 27 18:39:08.683044 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:08.683024 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-tf7lk"] Jul 27 18:39:09.492862 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:09.492829 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a1df17f8-2947-4e18-ab1d-7ddec4838584" path="/var/lib/kubelet/pods/a1df17f8-2947-4e18-ab1d-7ddec4838584/volumes" Jul 27 18:39:11.893776 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:11.893742 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:39:12.369671 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:12.369586 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-66qk2"] Jul 27 18:39:12.370162 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:12.370141 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1df17f8-2947-4e18-ab1d-7ddec4838584" containerName="limitador" Jul 27 18:39:12.370296 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:12.370165 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1df17f8-2947-4e18-ab1d-7ddec4838584" containerName="limitador" Jul 27 18:39:12.370296 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:12.370195 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af2904cc-5c49-4690-b6c9-dfc651fefaf1" containerName="authorino" Jul 27 18:39:12.370296 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:12.370204 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="af2904cc-5c49-4690-b6c9-dfc651fefaf1" containerName="authorino" Jul 27 18:39:12.370426 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:12.370333 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1df17f8-2947-4e18-ab1d-7ddec4838584" containerName="limitador" Jul 27 18:39:12.370426 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:12.370347 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="af2904cc-5c49-4690-b6c9-dfc651fefaf1" containerName="authorino" Jul 27 18:39:12.374657 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:12.374634 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-66qk2" Jul 27 18:39:12.377531 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:12.377505 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-9j445\"" Jul 27 18:39:12.377654 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:12.377505 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Jul 27 18:39:12.378777 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:12.378758 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 27 18:39:12.378855 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:12.378829 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 27 18:39:12.382732 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:12.382708 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-66qk2"] Jul 27 18:39:12.396510 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:12.396483 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b66ps\" (UniqueName: \"kubernetes.io/projected/31d0c692-c97b-472b-b909-9cccd4109f7f-kube-api-access-b66ps\") pod \"postgres-69fb4594cf-66qk2\" (UID: \"31d0c692-c97b-472b-b909-9cccd4109f7f\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-66qk2" Jul 27 18:39:12.396662 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:12.396641 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/31d0c692-c97b-472b-b909-9cccd4109f7f-data\") pod \"postgres-69fb4594cf-66qk2\" (UID: \"31d0c692-c97b-472b-b909-9cccd4109f7f\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-66qk2" Jul 27 18:39:12.497629 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:12.497596 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/31d0c692-c97b-472b-b909-9cccd4109f7f-data\") pod \"postgres-69fb4594cf-66qk2\" (UID: \"31d0c692-c97b-472b-b909-9cccd4109f7f\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-66qk2" Jul 27 18:39:12.497816 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:12.497640 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b66ps\" (UniqueName: \"kubernetes.io/projected/31d0c692-c97b-472b-b909-9cccd4109f7f-kube-api-access-b66ps\") pod \"postgres-69fb4594cf-66qk2\" (UID: \"31d0c692-c97b-472b-b909-9cccd4109f7f\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-66qk2" Jul 27 18:39:12.497987 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:12.497967 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/31d0c692-c97b-472b-b909-9cccd4109f7f-data\") pod \"postgres-69fb4594cf-66qk2\" (UID: \"31d0c692-c97b-472b-b909-9cccd4109f7f\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-66qk2" Jul 27 18:39:12.507478 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:12.507450 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b66ps\" (UniqueName: \"kubernetes.io/projected/31d0c692-c97b-472b-b909-9cccd4109f7f-kube-api-access-b66ps\") pod \"postgres-69fb4594cf-66qk2\" (UID: \"31d0c692-c97b-472b-b909-9cccd4109f7f\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-66qk2" Jul 27 18:39:12.688940 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:12.688848 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-66qk2" Jul 27 18:39:12.820589 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:12.820566 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-66qk2"] Jul 27 18:39:12.822634 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:39:12.822607 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31d0c692_c97b_472b_b909_9cccd4109f7f.slice/crio-757b5f0556f7e61cf804ba7854047869e5bc371d9947b7391f0f351400c23fcb WatchSource:0}: Error finding container 757b5f0556f7e61cf804ba7854047869e5bc371d9947b7391f0f351400c23fcb: Status 404 returned error can't find the container with id 757b5f0556f7e61cf804ba7854047869e5bc371d9947b7391f0f351400c23fcb Jul 27 18:39:12.823871 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:12.823842 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 27 18:39:13.384362 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:13.384330 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-66qk2" event={"ID":"31d0c692-c97b-472b-b909-9cccd4109f7f","Type":"ContainerStarted","Data":"757b5f0556f7e61cf804ba7854047869e5bc371d9947b7391f0f351400c23fcb"} Jul 27 18:39:14.621609 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:14.621566 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:39:18.411033 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:18.410989 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-66qk2" event={"ID":"31d0c692-c97b-472b-b909-9cccd4109f7f","Type":"ContainerStarted","Data":"1d9e2ceefe086badb9ebf2ae2c2431712287988355787fda83df167a30f48400"} Jul 27 18:39:18.411518 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:18.411072 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-66qk2" Jul 27 18:39:18.428317 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:18.428269 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-66qk2" podStartSLOduration=1.322284148 podStartE2EDuration="6.428252765s" totalImagesPullingTime="5.105968617s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:39:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:39:18.427872908 +0000 UTC m=+633.489561940" watchObservedRunningTime="2026-07-27 18:39:18.428252765 +0000 UTC m=+633.489941790" Jul 27 18:39:24.446180 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:24.446151 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-66qk2" Jul 27 18:39:24.763785 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:24.763691 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:39:24.965584 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:24.965550 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:39:24.988592 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:24.988563 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-dt7hq"] Jul 27 18:39:24.994340 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:24.994321 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-dt7hq" Jul 27 18:39:24.996832 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:24.996812 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-d825p\"" Jul 27 18:39:25.001845 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:25.001829 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-dt7hq"] Jul 27 18:39:25.109374 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:25.109339 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4c8x6\" (UniqueName: \"kubernetes.io/projected/f64c9643-5b83-4d1b-af13-9d107619ed45-kube-api-access-4c8x6\") pod \"authorino-5d4b67c446-dt7hq\" (UID: \"f64c9643-5b83-4d1b-af13-9d107619ed45\") " pod="kuadrant-system/authorino-5d4b67c446-dt7hq" Jul 27 18:39:25.178859 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:25.178818 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-dt7hq"] Jul 27 18:39:25.179118 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:39:25.179093 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-4c8x6], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-5d4b67c446-dt7hq" podUID="f64c9643-5b83-4d1b-af13-9d107619ed45" Jul 27 18:39:25.209593 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:25.209565 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-76877994-v9g8l"] Jul 27 18:39:25.209862 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:25.209845 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4c8x6\" (UniqueName: \"kubernetes.io/projected/f64c9643-5b83-4d1b-af13-9d107619ed45-kube-api-access-4c8x6\") pod \"authorino-5d4b67c446-dt7hq\" (UID: \"f64c9643-5b83-4d1b-af13-9d107619ed45\") " pod="kuadrant-system/authorino-5d4b67c446-dt7hq" Jul 27 18:39:25.213574 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:25.213553 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-v9g8l" Jul 27 18:39:25.216563 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:25.216544 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 27 18:39:25.225275 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:25.225223 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-76877994-v9g8l"] Jul 27 18:39:25.227165 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:25.227145 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4c8x6\" (UniqueName: \"kubernetes.io/projected/f64c9643-5b83-4d1b-af13-9d107619ed45-kube-api-access-4c8x6\") pod \"authorino-5d4b67c446-dt7hq\" (UID: \"f64c9643-5b83-4d1b-af13-9d107619ed45\") " pod="kuadrant-system/authorino-5d4b67c446-dt7hq" Jul 27 18:39:25.262464 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:25.262430 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-76877994-v9g8l"] Jul 27 18:39:25.262742 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:39:25.262718 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-llq7z tls-cert], unattached volumes=[], failed to process volumes=[kube-api-access-llq7z tls-cert]: context canceled" pod="kuadrant-system/authorino-76877994-v9g8l" podUID="7d0bdbd9-0daf-487d-996e-26420563a444" Jul 27 18:39:25.296780 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:25.296748 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-d649b6669-84m9q"] Jul 27 18:39:25.300684 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:25.300662 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-d649b6669-84m9q" Jul 27 18:39:25.309764 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:25.309739 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-d649b6669-84m9q"] Jul 27 18:39:25.310652 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:25.310630 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7d0bdbd9-0daf-487d-996e-26420563a444-tls-cert\") pod \"authorino-76877994-v9g8l\" (UID: \"7d0bdbd9-0daf-487d-996e-26420563a444\") " pod="kuadrant-system/authorino-76877994-v9g8l" Jul 27 18:39:25.310821 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:25.310744 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llq7z\" (UniqueName: \"kubernetes.io/projected/7d0bdbd9-0daf-487d-996e-26420563a444-kube-api-access-llq7z\") pod \"authorino-76877994-v9g8l\" (UID: \"7d0bdbd9-0daf-487d-996e-26420563a444\") " pod="kuadrant-system/authorino-76877994-v9g8l" Jul 27 18:39:25.412136 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:25.412056 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/55a32b36-c97c-438c-9f8e-fd47dc903775-tls-cert\") pod \"authorino-d649b6669-84m9q\" (UID: \"55a32b36-c97c-438c-9f8e-fd47dc903775\") " pod="kuadrant-system/authorino-d649b6669-84m9q" Jul 27 18:39:25.412136 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:25.412095 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llq7z\" (UniqueName: \"kubernetes.io/projected/7d0bdbd9-0daf-487d-996e-26420563a444-kube-api-access-llq7z\") pod \"authorino-76877994-v9g8l\" (UID: \"7d0bdbd9-0daf-487d-996e-26420563a444\") " pod="kuadrant-system/authorino-76877994-v9g8l" Jul 27 18:39:25.412343 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:25.412173 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7d0bdbd9-0daf-487d-996e-26420563a444-tls-cert\") pod \"authorino-76877994-v9g8l\" (UID: \"7d0bdbd9-0daf-487d-996e-26420563a444\") " pod="kuadrant-system/authorino-76877994-v9g8l" Jul 27 18:39:25.412343 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:25.412219 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbp6z\" (UniqueName: \"kubernetes.io/projected/55a32b36-c97c-438c-9f8e-fd47dc903775-kube-api-access-gbp6z\") pod \"authorino-d649b6669-84m9q\" (UID: \"55a32b36-c97c-438c-9f8e-fd47dc903775\") " pod="kuadrant-system/authorino-d649b6669-84m9q" Jul 27 18:39:25.414837 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:25.414814 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7d0bdbd9-0daf-487d-996e-26420563a444-tls-cert\") pod \"authorino-76877994-v9g8l\" (UID: \"7d0bdbd9-0daf-487d-996e-26420563a444\") " pod="kuadrant-system/authorino-76877994-v9g8l" Jul 27 18:39:25.420803 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:25.420778 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-llq7z\" (UniqueName: \"kubernetes.io/projected/7d0bdbd9-0daf-487d-996e-26420563a444-kube-api-access-llq7z\") pod \"authorino-76877994-v9g8l\" (UID: \"7d0bdbd9-0daf-487d-996e-26420563a444\") " pod="kuadrant-system/authorino-76877994-v9g8l" Jul 27 18:39:25.449433 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:25.449408 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-dt7hq" Jul 27 18:39:25.449828 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:25.449406 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-v9g8l" Jul 27 18:39:25.457395 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:25.457370 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-dt7hq" Jul 27 18:39:25.460330 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:25.460313 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-v9g8l" Jul 27 18:39:25.512993 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:25.512967 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-llq7z\" (UniqueName: \"kubernetes.io/projected/7d0bdbd9-0daf-487d-996e-26420563a444-kube-api-access-llq7z\") pod \"7d0bdbd9-0daf-487d-996e-26420563a444\" (UID: \"7d0bdbd9-0daf-487d-996e-26420563a444\") " Jul 27 18:39:25.513157 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:25.513011 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4c8x6\" (UniqueName: \"kubernetes.io/projected/f64c9643-5b83-4d1b-af13-9d107619ed45-kube-api-access-4c8x6\") pod \"f64c9643-5b83-4d1b-af13-9d107619ed45\" (UID: \"f64c9643-5b83-4d1b-af13-9d107619ed45\") " Jul 27 18:39:25.513157 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:25.513032 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7d0bdbd9-0daf-487d-996e-26420563a444-tls-cert\") pod \"7d0bdbd9-0daf-487d-996e-26420563a444\" (UID: \"7d0bdbd9-0daf-487d-996e-26420563a444\") " Jul 27 18:39:25.513157 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:25.513145 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbp6z\" (UniqueName: \"kubernetes.io/projected/55a32b36-c97c-438c-9f8e-fd47dc903775-kube-api-access-gbp6z\") pod \"authorino-d649b6669-84m9q\" (UID: \"55a32b36-c97c-438c-9f8e-fd47dc903775\") " pod="kuadrant-system/authorino-d649b6669-84m9q" Jul 27 18:39:25.513364 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:25.513189 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/55a32b36-c97c-438c-9f8e-fd47dc903775-tls-cert\") pod \"authorino-d649b6669-84m9q\" (UID: \"55a32b36-c97c-438c-9f8e-fd47dc903775\") " pod="kuadrant-system/authorino-d649b6669-84m9q" Jul 27 18:39:25.515385 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:25.515352 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d0bdbd9-0daf-487d-996e-26420563a444-kube-api-access-llq7z" (OuterVolumeSpecName: "kube-api-access-llq7z") pod "7d0bdbd9-0daf-487d-996e-26420563a444" (UID: "7d0bdbd9-0daf-487d-996e-26420563a444"). InnerVolumeSpecName "kube-api-access-llq7z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 18:39:25.515512 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:25.515384 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d0bdbd9-0daf-487d-996e-26420563a444-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "7d0bdbd9-0daf-487d-996e-26420563a444" (UID: "7d0bdbd9-0daf-487d-996e-26420563a444"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 18:39:25.515512 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:25.515451 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f64c9643-5b83-4d1b-af13-9d107619ed45-kube-api-access-4c8x6" (OuterVolumeSpecName: "kube-api-access-4c8x6") pod "f64c9643-5b83-4d1b-af13-9d107619ed45" (UID: "f64c9643-5b83-4d1b-af13-9d107619ed45"). InnerVolumeSpecName "kube-api-access-4c8x6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 18:39:25.515957 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:25.515937 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/55a32b36-c97c-438c-9f8e-fd47dc903775-tls-cert\") pod \"authorino-d649b6669-84m9q\" (UID: \"55a32b36-c97c-438c-9f8e-fd47dc903775\") " pod="kuadrant-system/authorino-d649b6669-84m9q" Jul 27 18:39:25.521772 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:25.521749 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbp6z\" (UniqueName: \"kubernetes.io/projected/55a32b36-c97c-438c-9f8e-fd47dc903775-kube-api-access-gbp6z\") pod \"authorino-d649b6669-84m9q\" (UID: \"55a32b36-c97c-438c-9f8e-fd47dc903775\") " pod="kuadrant-system/authorino-d649b6669-84m9q" Jul 27 18:39:25.611436 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:25.611404 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-d649b6669-84m9q" Jul 27 18:39:25.613809 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:25.613790 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-llq7z\" (UniqueName: \"kubernetes.io/projected/7d0bdbd9-0daf-487d-996e-26420563a444-kube-api-access-llq7z\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 18:39:25.613809 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:25.613810 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4c8x6\" (UniqueName: \"kubernetes.io/projected/f64c9643-5b83-4d1b-af13-9d107619ed45-kube-api-access-4c8x6\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 18:39:25.613930 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:25.613820 2579 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7d0bdbd9-0daf-487d-996e-26420563a444-tls-cert\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 18:39:25.764005 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:25.763984 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-d649b6669-84m9q"] Jul 27 18:39:25.766417 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:39:25.766382 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod55a32b36_c97c_438c_9f8e_fd47dc903775.slice/crio-95d878993f8bb129ca5d09c14a50902a4d1cee3de6f04375c9ef60b5adcf49e0 WatchSource:0}: Error finding container 95d878993f8bb129ca5d09c14a50902a4d1cee3de6f04375c9ef60b5adcf49e0: Status 404 returned error can't find the container with id 95d878993f8bb129ca5d09c14a50902a4d1cee3de6f04375c9ef60b5adcf49e0 Jul 27 18:39:26.456357 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:26.456259 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-d649b6669-84m9q" event={"ID":"55a32b36-c97c-438c-9f8e-fd47dc903775","Type":"ContainerStarted","Data":"f958feca9ce84a01e307abdfb364458f6686f9b02517add5a4495bb24df49bd7"} Jul 27 18:39:26.456357 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:26.456309 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-dt7hq" Jul 27 18:39:26.456357 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:26.456311 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-d649b6669-84m9q" event={"ID":"55a32b36-c97c-438c-9f8e-fd47dc903775","Type":"ContainerStarted","Data":"95d878993f8bb129ca5d09c14a50902a4d1cee3de6f04375c9ef60b5adcf49e0"} Jul 27 18:39:26.456357 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:26.456329 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-v9g8l" Jul 27 18:39:26.474576 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:26.474532 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-d649b6669-84m9q" podStartSLOduration=1.119701413 podStartE2EDuration="1.474516812s" totalImagesPullingTime="354.815399ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:39:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:39:26.473671911 +0000 UTC m=+641.535360942" watchObservedRunningTime="2026-07-27 18:39:26.474516812 +0000 UTC m=+641.536205843" Jul 27 18:39:26.506435 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:26.506405 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-dt7hq"] Jul 27 18:39:26.516858 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:26.516829 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-dt7hq"] Jul 27 18:39:26.542699 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:26.542672 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-76877994-v9g8l"] Jul 27 18:39:26.546021 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:26.545992 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-76877994-v9g8l"] Jul 27 18:39:27.493320 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:27.493289 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7d0bdbd9-0daf-487d-996e-26420563a444" path="/var/lib/kubelet/pods/7d0bdbd9-0daf-487d-996e-26420563a444/volumes" Jul 27 18:39:27.493757 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:27.493531 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f64c9643-5b83-4d1b-af13-9d107619ed45" path="/var/lib/kubelet/pods/f64c9643-5b83-4d1b-af13-9d107619ed45/volumes" Jul 27 18:39:31.604699 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:31.604664 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:39:31.623178 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:31.623150 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-74dd665f98-9bqkk"] Jul 27 18:39:31.704055 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:31.704020 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:39:31.709917 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:31.709898 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-74dd665f98-9bqkk"] Jul 27 18:39:31.710037 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:31.709984 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-74dd665f98-9bqkk" Jul 27 18:39:31.712939 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:31.712910 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 27 18:39:31.712939 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:31.712931 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-chdqz\"" Jul 27 18:39:31.713113 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:31.712916 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 27 18:39:31.759310 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:31.759268 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:39:31.767411 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:31.767389 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqvlg\" (UniqueName: \"kubernetes.io/projected/b463c930-d9dc-4788-a38b-ecab7543646a-kube-api-access-xqvlg\") pod \"maas-controller-74dd665f98-9bqkk\" (UID: \"b463c930-d9dc-4788-a38b-ecab7543646a\") " pod="opendatahub/maas-controller-74dd665f98-9bqkk" Jul 27 18:39:31.767513 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:31.767498 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b463c930-d9dc-4788-a38b-ecab7543646a-webhook-cert\") pod \"maas-controller-74dd665f98-9bqkk\" (UID: \"b463c930-d9dc-4788-a38b-ecab7543646a\") " pod="opendatahub/maas-controller-74dd665f98-9bqkk" Jul 27 18:39:31.868132 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:31.868053 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b463c930-d9dc-4788-a38b-ecab7543646a-webhook-cert\") pod \"maas-controller-74dd665f98-9bqkk\" (UID: \"b463c930-d9dc-4788-a38b-ecab7543646a\") " pod="opendatahub/maas-controller-74dd665f98-9bqkk" Jul 27 18:39:31.868132 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:31.868092 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqvlg\" (UniqueName: \"kubernetes.io/projected/b463c930-d9dc-4788-a38b-ecab7543646a-kube-api-access-xqvlg\") pod \"maas-controller-74dd665f98-9bqkk\" (UID: \"b463c930-d9dc-4788-a38b-ecab7543646a\") " pod="opendatahub/maas-controller-74dd665f98-9bqkk" Jul 27 18:39:31.868358 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:39:31.868203 2579 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found Jul 27 18:39:31.868358 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:39:31.868307 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b463c930-d9dc-4788-a38b-ecab7543646a-webhook-cert podName:b463c930-d9dc-4788-a38b-ecab7543646a nodeName:}" failed. No retries permitted until 2026-07-27 18:39:32.368291444 +0000 UTC m=+647.429980453 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/b463c930-d9dc-4788-a38b-ecab7543646a-webhook-cert") pod "maas-controller-74dd665f98-9bqkk" (UID: "b463c930-d9dc-4788-a38b-ecab7543646a") : secret "maas-controller-webhook-cert" not found Jul 27 18:39:31.879003 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:31.878973 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqvlg\" (UniqueName: \"kubernetes.io/projected/b463c930-d9dc-4788-a38b-ecab7543646a-kube-api-access-xqvlg\") pod \"maas-controller-74dd665f98-9bqkk\" (UID: \"b463c930-d9dc-4788-a38b-ecab7543646a\") " pod="opendatahub/maas-controller-74dd665f98-9bqkk" Jul 27 18:39:31.939712 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:31.939680 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:39:32.373434 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:32.373399 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b463c930-d9dc-4788-a38b-ecab7543646a-webhook-cert\") pod \"maas-controller-74dd665f98-9bqkk\" (UID: \"b463c930-d9dc-4788-a38b-ecab7543646a\") " pod="opendatahub/maas-controller-74dd665f98-9bqkk" Jul 27 18:39:32.376058 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:32.376036 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b463c930-d9dc-4788-a38b-ecab7543646a-webhook-cert\") pod \"maas-controller-74dd665f98-9bqkk\" (UID: \"b463c930-d9dc-4788-a38b-ecab7543646a\") " pod="opendatahub/maas-controller-74dd665f98-9bqkk" Jul 27 18:39:32.620124 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:32.620088 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-74dd665f98-9bqkk" Jul 27 18:39:32.859621 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:32.859589 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-74dd665f98-9bqkk"] Jul 27 18:39:32.861684 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:39:32.861657 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb463c930_d9dc_4788_a38b_ecab7543646a.slice/crio-f8a2427e85b2d295e1e5c26fa46759c257a5a105bb5977d4412aa80926712146 WatchSource:0}: Error finding container f8a2427e85b2d295e1e5c26fa46759c257a5a105bb5977d4412aa80926712146: Status 404 returned error can't find the container with id f8a2427e85b2d295e1e5c26fa46759c257a5a105bb5977d4412aa80926712146 Jul 27 18:39:33.495181 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:33.495113 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-74dd665f98-9bqkk" event={"ID":"b463c930-d9dc-4788-a38b-ecab7543646a","Type":"ContainerStarted","Data":"f8a2427e85b2d295e1e5c26fa46759c257a5a105bb5977d4412aa80926712146"} Jul 27 18:39:35.187431 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:35.187373 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:39:35.364870 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:35.364819 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:39:36.507336 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:36.507298 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-74dd665f98-9bqkk" event={"ID":"b463c930-d9dc-4788-a38b-ecab7543646a","Type":"ContainerStarted","Data":"7e458ee5db7484d5d4abd5c262f623d5449a4bc2c62d77eb75aa32bfa4d9d8fb"} Jul 27 18:39:36.507721 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:36.507503 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-74dd665f98-9bqkk" Jul 27 18:39:36.529146 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:36.529086 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-74dd665f98-9bqkk" podStartSLOduration=2.783974337 podStartE2EDuration="5.529071698s" totalImagesPullingTime="2.745097361s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:39:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:39:36.526679649 +0000 UTC m=+651.588368681" watchObservedRunningTime="2026-07-27 18:39:36.529071698 +0000 UTC m=+651.590760729" Jul 27 18:39:46.242409 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:46.242375 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-74dd665f98-9bqkk"] Jul 27 18:39:46.242793 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:46.242660 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-74dd665f98-9bqkk" podUID="b463c930-d9dc-4788-a38b-ecab7543646a" containerName="manager" containerID="cri-o://7e458ee5db7484d5d4abd5c262f623d5449a4bc2c62d77eb75aa32bfa4d9d8fb" gracePeriod=30 Jul 27 18:39:46.245653 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:46.245623 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-74dd665f98-9bqkk" Jul 27 18:39:46.497658 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:46.497604 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-74dd665f98-9bqkk" Jul 27 18:39:46.554563 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:46.554532 2579 generic.go:356] "Generic (PLEG): container finished" podID="b463c930-d9dc-4788-a38b-ecab7543646a" containerID="7e458ee5db7484d5d4abd5c262f623d5449a4bc2c62d77eb75aa32bfa4d9d8fb" exitCode=0 Jul 27 18:39:46.554725 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:46.554577 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-74dd665f98-9bqkk" event={"ID":"b463c930-d9dc-4788-a38b-ecab7543646a","Type":"ContainerDied","Data":"7e458ee5db7484d5d4abd5c262f623d5449a4bc2c62d77eb75aa32bfa4d9d8fb"} Jul 27 18:39:46.554725 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:46.554614 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-74dd665f98-9bqkk" event={"ID":"b463c930-d9dc-4788-a38b-ecab7543646a","Type":"ContainerDied","Data":"f8a2427e85b2d295e1e5c26fa46759c257a5a105bb5977d4412aa80926712146"} Jul 27 18:39:46.554725 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:46.554635 2579 scope.go:117] "RemoveContainer" containerID="7e458ee5db7484d5d4abd5c262f623d5449a4bc2c62d77eb75aa32bfa4d9d8fb" Jul 27 18:39:46.554725 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:46.554580 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-74dd665f98-9bqkk" Jul 27 18:39:46.563273 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:46.563248 2579 scope.go:117] "RemoveContainer" containerID="7e458ee5db7484d5d4abd5c262f623d5449a4bc2c62d77eb75aa32bfa4d9d8fb" Jul 27 18:39:46.563516 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:39:46.563494 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e458ee5db7484d5d4abd5c262f623d5449a4bc2c62d77eb75aa32bfa4d9d8fb\": container with ID starting with 7e458ee5db7484d5d4abd5c262f623d5449a4bc2c62d77eb75aa32bfa4d9d8fb not found: ID does not exist" containerID="7e458ee5db7484d5d4abd5c262f623d5449a4bc2c62d77eb75aa32bfa4d9d8fb" Jul 27 18:39:46.563568 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:46.563525 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e458ee5db7484d5d4abd5c262f623d5449a4bc2c62d77eb75aa32bfa4d9d8fb"} err="failed to get container status \"7e458ee5db7484d5d4abd5c262f623d5449a4bc2c62d77eb75aa32bfa4d9d8fb\": rpc error: code = NotFound desc = could not find container \"7e458ee5db7484d5d4abd5c262f623d5449a4bc2c62d77eb75aa32bfa4d9d8fb\": container with ID starting with 7e458ee5db7484d5d4abd5c262f623d5449a4bc2c62d77eb75aa32bfa4d9d8fb not found: ID does not exist" Jul 27 18:39:46.594444 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:46.594415 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xqvlg\" (UniqueName: \"kubernetes.io/projected/b463c930-d9dc-4788-a38b-ecab7543646a-kube-api-access-xqvlg\") pod \"b463c930-d9dc-4788-a38b-ecab7543646a\" (UID: \"b463c930-d9dc-4788-a38b-ecab7543646a\") " Jul 27 18:39:46.594577 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:46.594468 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b463c930-d9dc-4788-a38b-ecab7543646a-webhook-cert\") pod \"b463c930-d9dc-4788-a38b-ecab7543646a\" (UID: \"b463c930-d9dc-4788-a38b-ecab7543646a\") " Jul 27 18:39:46.596712 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:46.596681 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b463c930-d9dc-4788-a38b-ecab7543646a-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "b463c930-d9dc-4788-a38b-ecab7543646a" (UID: "b463c930-d9dc-4788-a38b-ecab7543646a"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 18:39:46.596814 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:46.596727 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b463c930-d9dc-4788-a38b-ecab7543646a-kube-api-access-xqvlg" (OuterVolumeSpecName: "kube-api-access-xqvlg") pod "b463c930-d9dc-4788-a38b-ecab7543646a" (UID: "b463c930-d9dc-4788-a38b-ecab7543646a"). InnerVolumeSpecName "kube-api-access-xqvlg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 18:39:46.695608 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:46.695563 2579 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b463c930-d9dc-4788-a38b-ecab7543646a-webhook-cert\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 18:39:46.695608 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:46.695602 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xqvlg\" (UniqueName: \"kubernetes.io/projected/b463c930-d9dc-4788-a38b-ecab7543646a-kube-api-access-xqvlg\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 18:39:46.876905 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:46.876878 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-74dd665f98-9bqkk"] Jul 27 18:39:46.883149 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:46.883123 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-74dd665f98-9bqkk"] Jul 27 18:39:47.492917 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:39:47.492888 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b463c930-d9dc-4788-a38b-ecab7543646a" path="/var/lib/kubelet/pods/b463c930-d9dc-4788-a38b-ecab7543646a/volumes" Jul 27 18:40:06.481729 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:40:06.481691 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:40:06.587612 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:40:06.587578 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:40:07.012941 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:40:07.012898 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:40:07.242712 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:40:07.242678 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:40:07.813508 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:40:07.813475 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:40:07.914485 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:40:07.914455 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:40:07.957730 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:40:07.957694 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-8rfpl"] Jul 27 18:40:07.958099 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:40:07.958085 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b463c930-d9dc-4788-a38b-ecab7543646a" containerName="manager" Jul 27 18:40:07.958149 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:40:07.958104 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="b463c930-d9dc-4788-a38b-ecab7543646a" containerName="manager" Jul 27 18:40:07.958194 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:40:07.958182 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="b463c930-d9dc-4788-a38b-ecab7543646a" containerName="manager" Jul 27 18:40:07.961255 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:40:07.961223 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-8rfpl" Jul 27 18:40:07.968420 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:40:07.968398 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 27 18:40:07.983650 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:40:07.983629 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-8rfpl"] Jul 27 18:40:08.078971 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:40:08.078881 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2w42\" (UniqueName: \"kubernetes.io/projected/77a26a98-9824-423f-be2e-c316fc57d11b-kube-api-access-h2w42\") pod \"payload-processing-5cc6cd444b-8rfpl\" (UID: \"77a26a98-9824-423f-be2e-c316fc57d11b\") " pod="openshift-ingress/payload-processing-5cc6cd444b-8rfpl" Jul 27 18:40:08.078971 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:40:08.078930 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77a26a98-9824-423f-be2e-c316fc57d11b-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-8rfpl\" (UID: \"77a26a98-9824-423f-be2e-c316fc57d11b\") " pod="openshift-ingress/payload-processing-5cc6cd444b-8rfpl" Jul 27 18:40:08.180271 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:40:08.180219 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2w42\" (UniqueName: \"kubernetes.io/projected/77a26a98-9824-423f-be2e-c316fc57d11b-kube-api-access-h2w42\") pod \"payload-processing-5cc6cd444b-8rfpl\" (UID: \"77a26a98-9824-423f-be2e-c316fc57d11b\") " pod="openshift-ingress/payload-processing-5cc6cd444b-8rfpl" Jul 27 18:40:08.180271 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:40:08.180276 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77a26a98-9824-423f-be2e-c316fc57d11b-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-8rfpl\" (UID: \"77a26a98-9824-423f-be2e-c316fc57d11b\") " pod="openshift-ingress/payload-processing-5cc6cd444b-8rfpl" Jul 27 18:40:08.181046 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:40:08.181023 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77a26a98-9824-423f-be2e-c316fc57d11b-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-8rfpl\" (UID: \"77a26a98-9824-423f-be2e-c316fc57d11b\") " pod="openshift-ingress/payload-processing-5cc6cd444b-8rfpl" Jul 27 18:40:08.197111 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:40:08.197083 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2w42\" (UniqueName: \"kubernetes.io/projected/77a26a98-9824-423f-be2e-c316fc57d11b-kube-api-access-h2w42\") pod \"payload-processing-5cc6cd444b-8rfpl\" (UID: \"77a26a98-9824-423f-be2e-c316fc57d11b\") " pod="openshift-ingress/payload-processing-5cc6cd444b-8rfpl" Jul 27 18:40:08.271507 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:40:08.271473 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-8rfpl" Jul 27 18:40:08.419703 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:40:08.419678 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-8rfpl"] Jul 27 18:40:08.422205 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:40:08.422172 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77a26a98_9824_423f_be2e_c316fc57d11b.slice/crio-af37bbb5f4cdd4991fc973b8947883b0ac2314d0fb3e075203b050f99ee85f1b WatchSource:0}: Error finding container af37bbb5f4cdd4991fc973b8947883b0ac2314d0fb3e075203b050f99ee85f1b: Status 404 returned error can't find the container with id af37bbb5f4cdd4991fc973b8947883b0ac2314d0fb3e075203b050f99ee85f1b Jul 27 18:40:08.659113 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:40:08.659013 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-8rfpl" event={"ID":"77a26a98-9824-423f-be2e-c316fc57d11b","Type":"ContainerStarted","Data":"af37bbb5f4cdd4991fc973b8947883b0ac2314d0fb3e075203b050f99ee85f1b"} Jul 27 18:40:08.703622 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:40:08.703585 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:40:08.805390 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:40:08.805352 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:40:11.683425 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:40:11.683390 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-8rfpl" event={"ID":"77a26a98-9824-423f-be2e-c316fc57d11b","Type":"ContainerStarted","Data":"f365de9adcf5eb4da5d8769520b2ada36b909cc70829ba693dabe73daea2649e"} Jul 27 18:40:11.683986 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:40:11.683530 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5cc6cd444b-8rfpl" Jul 27 18:40:11.707103 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:40:11.707046 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5cc6cd444b-8rfpl" podStartSLOduration=1.950381038 podStartE2EDuration="4.707030178s" totalImagesPullingTime="2.75664914s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:40:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:40:11.705722973 +0000 UTC m=+686.767411995" watchObservedRunningTime="2026-07-27 18:40:11.707030178 +0000 UTC m=+686.768719186" Jul 27 18:40:15.705306 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:40:15.705179 2579 generic.go:356] "Generic (PLEG): container finished" podID="429930eb-3d30-450f-8e99-75ab3f80ba28" containerID="04505a7b08b666acf97a43ab1d913b874815b7a753a2faaeee5cbe54c10f4441" exitCode=0 Jul 27 18:40:15.705306 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:40:15.705261 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-j2vmg" event={"ID":"429930eb-3d30-450f-8e99-75ab3f80ba28","Type":"ContainerDied","Data":"04505a7b08b666acf97a43ab1d913b874815b7a753a2faaeee5cbe54c10f4441"} Jul 27 18:40:15.705735 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:40:15.705669 2579 scope.go:117] "RemoveContainer" containerID="04505a7b08b666acf97a43ab1d913b874815b7a753a2faaeee5cbe54c10f4441" Jul 27 18:40:15.706675 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:40:15.706648 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 27 18:40:15.706788 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:40:15.706728 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 27 18:40:15.706788 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:40:15.706776 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 27 18:40:15.882017 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:40:15.881977 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:40:16.711695 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:40:16.711658 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-j2vmg" event={"ID":"429930eb-3d30-450f-8e99-75ab3f80ba28","Type":"ContainerStarted","Data":"3afc3f1ea0d7c2604461fa7dd7a319aca408fd68ea69ed7f9e2a8b4836ecb48d"} Jul 27 18:40:16.918948 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:40:16.918913 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-j2vmg" Jul 27 18:40:16.919156 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:40:16.919101 2579 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-j2vmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body= Jul 27 18:40:16.919259 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:40:16.919158 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-j2vmg" podUID="429930eb-3d30-450f-8e99-75ab3f80ba28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" Jul 27 18:40:17.919449 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:40:17.919411 2579 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-j2vmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body= Jul 27 18:40:17.919882 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:40:17.919483 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-j2vmg" podUID="429930eb-3d30-450f-8e99-75ab3f80ba28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" Jul 27 18:40:18.085961 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:40:18.085918 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:40:18.186847 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:40:18.186754 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:40:18.919959 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:40:18.919921 2579 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-j2vmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body= Jul 27 18:40:18.920404 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:40:18.919986 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-j2vmg" podUID="429930eb-3d30-450f-8e99-75ab3f80ba28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" Jul 27 18:40:19.919281 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:40:19.919218 2579 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-j2vmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body= Jul 27 18:40:19.919473 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:40:19.919321 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-j2vmg" podUID="429930eb-3d30-450f-8e99-75ab3f80ba28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" Jul 27 18:40:20.923735 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:40:20.923700 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-j2vmg" Jul 27 18:40:20.924207 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:40:20.923991 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-j2vmg" Jul 27 18:40:20.924869 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:40:20.924855 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-j2vmg" Jul 27 18:40:22.691833 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:40:22.691804 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5cc6cd444b-8rfpl" Jul 27 18:40:26.494589 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:40:26.494557 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9jcc5"] Jul 27 18:40:26.497188 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:40:26.497168 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9jcc5" Jul 27 18:40:26.500067 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:40:26.500045 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 27 18:40:26.501452 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:40:26.501429 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-6w4tt\"" Jul 27 18:40:26.501452 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:40:26.501453 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 27 18:40:26.501625 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:40:26.501486 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Jul 27 18:40:26.507417 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:40:26.507395 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9jcc5"] Jul 27 18:40:26.538385 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:40:26.538350 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ebf412c-8d28-4361-9384-574b538c5c8c-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9jcc5\" (UID: \"3ebf412c-8d28-4361-9384-574b538c5c8c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9jcc5" Jul 27 18:40:26.538528 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:40:26.538402 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ebf412c-8d28-4361-9384-574b538c5c8c-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9jcc5\" (UID: \"3ebf412c-8d28-4361-9384-574b538c5c8c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9jcc5" Jul 27 18:40:26.538528 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:40:26.538455 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ebf412c-8d28-4361-9384-574b538c5c8c-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9jcc5\" (UID: \"3ebf412c-8d28-4361-9384-574b538c5c8c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9jcc5" Jul 27 18:40:26.538528 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:40:26.538481 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ebf412c-8d28-4361-9384-574b538c5c8c-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9jcc5\" (UID: \"3ebf412c-8d28-4361-9384-574b538c5c8c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9jcc5" Jul 27 18:40:26.538662 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:40:26.538599 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82blq\" (UniqueName: \"kubernetes.io/projected/3ebf412c-8d28-4361-9384-574b538c5c8c-kube-api-access-82blq\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9jcc5\" (UID: \"3ebf412c-8d28-4361-9384-574b538c5c8c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9jcc5" Jul 27 18:40:26.538712 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:40:26.538656 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ebf412c-8d28-4361-9384-574b538c5c8c-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9jcc5\" (UID: \"3ebf412c-8d28-4361-9384-574b538c5c8c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9jcc5" Jul 27 18:40:26.574024 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:40:26.573991 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:40:26.639719 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:40:26.639690 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ebf412c-8d28-4361-9384-574b538c5c8c-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9jcc5\" (UID: \"3ebf412c-8d28-4361-9384-574b538c5c8c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9jcc5" Jul 27 18:40:26.639719 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:40:26.639723 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ebf412c-8d28-4361-9384-574b538c5c8c-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9jcc5\" (UID: \"3ebf412c-8d28-4361-9384-574b538c5c8c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9jcc5" Jul 27 18:40:26.639905 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:40:26.639754 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82blq\" (UniqueName: \"kubernetes.io/projected/3ebf412c-8d28-4361-9384-574b538c5c8c-kube-api-access-82blq\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9jcc5\" (UID: \"3ebf412c-8d28-4361-9384-574b538c5c8c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9jcc5" Jul 27 18:40:26.639905 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:40:26.639795 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ebf412c-8d28-4361-9384-574b538c5c8c-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9jcc5\" (UID: \"3ebf412c-8d28-4361-9384-574b538c5c8c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9jcc5" Jul 27 18:40:26.639905 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:40:26.639868 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ebf412c-8d28-4361-9384-574b538c5c8c-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9jcc5\" (UID: \"3ebf412c-8d28-4361-9384-574b538c5c8c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9jcc5" Jul 27 18:40:26.640061 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:40:26.639909 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ebf412c-8d28-4361-9384-574b538c5c8c-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9jcc5\" (UID: \"3ebf412c-8d28-4361-9384-574b538c5c8c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9jcc5" Jul 27 18:40:26.640195 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:40:26.640175 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ebf412c-8d28-4361-9384-574b538c5c8c-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9jcc5\" (UID: \"3ebf412c-8d28-4361-9384-574b538c5c8c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9jcc5" Jul 27 18:40:26.640291 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:40:26.640204 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ebf412c-8d28-4361-9384-574b538c5c8c-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9jcc5\" (UID: \"3ebf412c-8d28-4361-9384-574b538c5c8c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9jcc5" Jul 27 18:40:26.640291 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:40:26.640264 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ebf412c-8d28-4361-9384-574b538c5c8c-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9jcc5\" (UID: \"3ebf412c-8d28-4361-9384-574b538c5c8c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9jcc5" Jul 27 18:40:26.642805 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:40:26.642778 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ebf412c-8d28-4361-9384-574b538c5c8c-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9jcc5\" (UID: \"3ebf412c-8d28-4361-9384-574b538c5c8c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9jcc5" Jul 27 18:40:26.643159 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:40:26.643139 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ebf412c-8d28-4361-9384-574b538c5c8c-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9jcc5\" (UID: \"3ebf412c-8d28-4361-9384-574b538c5c8c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9jcc5" Jul 27 18:40:26.649605 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:40:26.649575 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-82blq\" (UniqueName: \"kubernetes.io/projected/3ebf412c-8d28-4361-9384-574b538c5c8c-kube-api-access-82blq\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9jcc5\" (UID: \"3ebf412c-8d28-4361-9384-574b538c5c8c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9jcc5" Jul 27 18:40:26.808396 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:40:26.808365 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9jcc5" Jul 27 18:40:26.953180 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:40:26.953153 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9jcc5"] Jul 27 18:40:26.954862 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:40:26.954837 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ebf412c_8d28_4361_9384_574b538c5c8c.slice/crio-6b38f86771effa2b4ad9c0362fdf87a6a42ca0db619ba2a0fb3f75fc029b926a WatchSource:0}: Error finding container 6b38f86771effa2b4ad9c0362fdf87a6a42ca0db619ba2a0fb3f75fc029b926a: Status 404 returned error can't find the container with id 6b38f86771effa2b4ad9c0362fdf87a6a42ca0db619ba2a0fb3f75fc029b926a Jul 27 18:40:27.779361 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:40:27.779321 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9jcc5" event={"ID":"3ebf412c-8d28-4361-9384-574b538c5c8c","Type":"ContainerStarted","Data":"6b38f86771effa2b4ad9c0362fdf87a6a42ca0db619ba2a0fb3f75fc029b926a"} Jul 27 18:40:28.785815 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:40:28.785781 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9jcc5" event={"ID":"3ebf412c-8d28-4361-9384-574b538c5c8c","Type":"ContainerStarted","Data":"faa1e901f9f1fb6c40bed5089bed7cc5fdd263c72ee76b54ef91fd1ee5a52932"} Jul 27 18:40:28.807970 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:40:28.807918 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9jcc5" podStartSLOduration=1.613773331 podStartE2EDuration="2.807903795s" totalImagesPullingTime="1.194130464s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:40:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:40:28.805946313 +0000 UTC m=+703.867635345" watchObservedRunningTime="2026-07-27 18:40:28.807903795 +0000 UTC m=+703.869592825" Jul 27 18:40:36.809349 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:40:36.809265 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9jcc5" Jul 27 18:40:36.809349 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:40:36.809329 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9jcc5" Jul 27 18:40:36.821693 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:40:36.821663 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9jcc5" Jul 27 18:40:36.837528 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:40:36.837500 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9jcc5" Jul 27 18:40:39.281832 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:40:39.281798 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-rdx2x"] Jul 27 18:40:39.285083 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:40:39.285065 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rdx2x" Jul 27 18:40:39.287963 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:40:39.287945 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Jul 27 18:40:39.297280 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:40:39.297258 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-rdx2x"] Jul 27 18:40:39.315052 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:40:39.315009 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:40:39.353577 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:40:39.353533 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a43bc0e3-cb23-49fa-afec-a2b93eea86d7-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-rdx2x\" (UID: \"a43bc0e3-cb23-49fa-afec-a2b93eea86d7\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rdx2x" Jul 27 18:40:39.353577 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:40:39.353576 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a43bc0e3-cb23-49fa-afec-a2b93eea86d7-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-rdx2x\" (UID: \"a43bc0e3-cb23-49fa-afec-a2b93eea86d7\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rdx2x" Jul 27 18:40:39.353761 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:40:39.353710 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r69qf\" (UniqueName: \"kubernetes.io/projected/a43bc0e3-cb23-49fa-afec-a2b93eea86d7-kube-api-access-r69qf\") pod \"e2e-distinct-simulated-kserve-59975d69f9-rdx2x\" (UID: \"a43bc0e3-cb23-49fa-afec-a2b93eea86d7\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rdx2x" Jul 27 18:40:39.353761 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:40:39.353732 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a43bc0e3-cb23-49fa-afec-a2b93eea86d7-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-rdx2x\" (UID: \"a43bc0e3-cb23-49fa-afec-a2b93eea86d7\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rdx2x" Jul 27 18:40:39.353834 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:40:39.353789 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a43bc0e3-cb23-49fa-afec-a2b93eea86d7-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-rdx2x\" (UID: \"a43bc0e3-cb23-49fa-afec-a2b93eea86d7\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rdx2x" Jul 27 18:40:39.353876 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:40:39.353839 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a43bc0e3-cb23-49fa-afec-a2b93eea86d7-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-rdx2x\" (UID: \"a43bc0e3-cb23-49fa-afec-a2b93eea86d7\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rdx2x" Jul 27 18:40:39.454719 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:40:39.454684 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r69qf\" (UniqueName: \"kubernetes.io/projected/a43bc0e3-cb23-49fa-afec-a2b93eea86d7-kube-api-access-r69qf\") pod \"e2e-distinct-simulated-kserve-59975d69f9-rdx2x\" (UID: \"a43bc0e3-cb23-49fa-afec-a2b93eea86d7\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rdx2x" Jul 27 18:40:39.454904 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:40:39.454726 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a43bc0e3-cb23-49fa-afec-a2b93eea86d7-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-rdx2x\" (UID: \"a43bc0e3-cb23-49fa-afec-a2b93eea86d7\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rdx2x" Jul 27 18:40:39.454904 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:40:39.454759 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a43bc0e3-cb23-49fa-afec-a2b93eea86d7-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-rdx2x\" (UID: \"a43bc0e3-cb23-49fa-afec-a2b93eea86d7\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rdx2x" Jul 27 18:40:39.454904 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:40:39.454799 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a43bc0e3-cb23-49fa-afec-a2b93eea86d7-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-rdx2x\" (UID: \"a43bc0e3-cb23-49fa-afec-a2b93eea86d7\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rdx2x" Jul 27 18:40:39.454904 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:40:39.454847 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a43bc0e3-cb23-49fa-afec-a2b93eea86d7-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-rdx2x\" (UID: \"a43bc0e3-cb23-49fa-afec-a2b93eea86d7\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rdx2x" Jul 27 18:40:39.454904 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:40:39.454872 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a43bc0e3-cb23-49fa-afec-a2b93eea86d7-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-rdx2x\" (UID: \"a43bc0e3-cb23-49fa-afec-a2b93eea86d7\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rdx2x" Jul 27 18:40:39.455334 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:40:39.455304 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a43bc0e3-cb23-49fa-afec-a2b93eea86d7-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-rdx2x\" (UID: \"a43bc0e3-cb23-49fa-afec-a2b93eea86d7\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rdx2x" Jul 27 18:40:39.455473 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:40:39.455352 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a43bc0e3-cb23-49fa-afec-a2b93eea86d7-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-rdx2x\" (UID: \"a43bc0e3-cb23-49fa-afec-a2b93eea86d7\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rdx2x" Jul 27 18:40:39.455473 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:40:39.455407 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a43bc0e3-cb23-49fa-afec-a2b93eea86d7-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-rdx2x\" (UID: \"a43bc0e3-cb23-49fa-afec-a2b93eea86d7\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rdx2x" Jul 27 18:40:39.457423 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:40:39.457394 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a43bc0e3-cb23-49fa-afec-a2b93eea86d7-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-rdx2x\" (UID: \"a43bc0e3-cb23-49fa-afec-a2b93eea86d7\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rdx2x" Jul 27 18:40:39.457423 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:40:39.457418 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a43bc0e3-cb23-49fa-afec-a2b93eea86d7-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-rdx2x\" (UID: \"a43bc0e3-cb23-49fa-afec-a2b93eea86d7\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rdx2x" Jul 27 18:40:39.463447 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:40:39.463425 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r69qf\" (UniqueName: \"kubernetes.io/projected/a43bc0e3-cb23-49fa-afec-a2b93eea86d7-kube-api-access-r69qf\") pod \"e2e-distinct-simulated-kserve-59975d69f9-rdx2x\" (UID: \"a43bc0e3-cb23-49fa-afec-a2b93eea86d7\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rdx2x" Jul 27 18:40:39.596148 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:40:39.596052 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rdx2x" Jul 27 18:40:39.761161 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:40:39.761131 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-rdx2x"] Jul 27 18:40:39.763002 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:40:39.762979 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda43bc0e3_cb23_49fa_afec_a2b93eea86d7.slice/crio-05a892bf8edb3c0c5cd7e5ef427cafe63f2a316bab29027e83790f1f60211052 WatchSource:0}: Error finding container 05a892bf8edb3c0c5cd7e5ef427cafe63f2a316bab29027e83790f1f60211052: Status 404 returned error can't find the container with id 05a892bf8edb3c0c5cd7e5ef427cafe63f2a316bab29027e83790f1f60211052 Jul 27 18:40:39.843007 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:40:39.842972 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rdx2x" event={"ID":"a43bc0e3-cb23-49fa-afec-a2b93eea86d7","Type":"ContainerStarted","Data":"05a892bf8edb3c0c5cd7e5ef427cafe63f2a316bab29027e83790f1f60211052"} Jul 27 18:40:40.849162 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:40:40.849124 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rdx2x" event={"ID":"a43bc0e3-cb23-49fa-afec-a2b93eea86d7","Type":"ContainerStarted","Data":"49aea6c2ba62c8b0d6c54a0d4ff3ff680092c31a5eb34a78edaf4e542efa25a2"} Jul 27 18:40:40.871770 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:40:40.871715 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rdx2x" podStartSLOduration=1.6122555589999998 podStartE2EDuration="1.871694677s" totalImagesPullingTime="259.439118ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:40:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:40:40.871108705 +0000 UTC m=+715.932797737" watchObservedRunningTime="2026-07-27 18:40:40.871694677 +0000 UTC m=+715.933383709" Jul 27 18:40:49.596383 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:40:49.596345 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rdx2x" Jul 27 18:40:49.596798 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:40:49.596668 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rdx2x" Jul 27 18:40:49.608825 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:40:49.608800 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rdx2x" Jul 27 18:40:49.902214 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:40:49.902127 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rdx2x" Jul 27 18:41:07.853355 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:41:07.853310 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:41:17.303863 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:41:17.303828 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:41:19.423022 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:41:19.422989 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:41:42.294714 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:41:42.294684 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-gmxtx"] Jul 27 18:41:42.298106 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:41:42.298087 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gmxtx" Jul 27 18:41:42.302317 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:41:42.302295 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Jul 27 18:41:42.310382 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:41:42.310359 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-gmxtx"] Jul 27 18:41:42.311506 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:41:42.311476 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2370a328-d5e9-4e99-b0b6-b35739ee7b66-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gmxtx\" (UID: \"2370a328-d5e9-4e99-b0b6-b35739ee7b66\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gmxtx" Jul 27 18:41:42.311630 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:41:42.311579 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2370a328-d5e9-4e99-b0b6-b35739ee7b66-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gmxtx\" (UID: \"2370a328-d5e9-4e99-b0b6-b35739ee7b66\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gmxtx" Jul 27 18:41:42.311703 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:41:42.311637 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2370a328-d5e9-4e99-b0b6-b35739ee7b66-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gmxtx\" (UID: \"2370a328-d5e9-4e99-b0b6-b35739ee7b66\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gmxtx" Jul 27 18:41:42.311703 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:41:42.311669 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpxtk\" (UniqueName: \"kubernetes.io/projected/2370a328-d5e9-4e99-b0b6-b35739ee7b66-kube-api-access-gpxtk\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gmxtx\" (UID: \"2370a328-d5e9-4e99-b0b6-b35739ee7b66\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gmxtx" Jul 27 18:41:42.311795 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:41:42.311741 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2370a328-d5e9-4e99-b0b6-b35739ee7b66-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gmxtx\" (UID: \"2370a328-d5e9-4e99-b0b6-b35739ee7b66\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gmxtx" Jul 27 18:41:42.311795 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:41:42.311774 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2370a328-d5e9-4e99-b0b6-b35739ee7b66-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gmxtx\" (UID: \"2370a328-d5e9-4e99-b0b6-b35739ee7b66\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gmxtx" Jul 27 18:41:42.363706 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:41:42.363662 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:41:42.413109 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:41:42.413079 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2370a328-d5e9-4e99-b0b6-b35739ee7b66-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gmxtx\" (UID: \"2370a328-d5e9-4e99-b0b6-b35739ee7b66\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gmxtx" Jul 27 18:41:42.413301 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:41:42.413139 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2370a328-d5e9-4e99-b0b6-b35739ee7b66-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gmxtx\" (UID: \"2370a328-d5e9-4e99-b0b6-b35739ee7b66\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gmxtx" Jul 27 18:41:42.413301 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:41:42.413183 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2370a328-d5e9-4e99-b0b6-b35739ee7b66-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gmxtx\" (UID: \"2370a328-d5e9-4e99-b0b6-b35739ee7b66\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gmxtx" Jul 27 18:41:42.413301 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:41:42.413211 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpxtk\" (UniqueName: \"kubernetes.io/projected/2370a328-d5e9-4e99-b0b6-b35739ee7b66-kube-api-access-gpxtk\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gmxtx\" (UID: \"2370a328-d5e9-4e99-b0b6-b35739ee7b66\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gmxtx" Jul 27 18:41:42.413301 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:41:42.413279 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2370a328-d5e9-4e99-b0b6-b35739ee7b66-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gmxtx\" (UID: \"2370a328-d5e9-4e99-b0b6-b35739ee7b66\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gmxtx" Jul 27 18:41:42.413512 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:41:42.413303 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2370a328-d5e9-4e99-b0b6-b35739ee7b66-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gmxtx\" (UID: \"2370a328-d5e9-4e99-b0b6-b35739ee7b66\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gmxtx" Jul 27 18:41:42.413591 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:41:42.413570 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2370a328-d5e9-4e99-b0b6-b35739ee7b66-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gmxtx\" (UID: \"2370a328-d5e9-4e99-b0b6-b35739ee7b66\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gmxtx" Jul 27 18:41:42.413891 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:41:42.413866 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2370a328-d5e9-4e99-b0b6-b35739ee7b66-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gmxtx\" (UID: \"2370a328-d5e9-4e99-b0b6-b35739ee7b66\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gmxtx" Jul 27 18:41:42.413891 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:41:42.413874 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2370a328-d5e9-4e99-b0b6-b35739ee7b66-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gmxtx\" (UID: \"2370a328-d5e9-4e99-b0b6-b35739ee7b66\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gmxtx" Jul 27 18:41:42.416211 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:41:42.416183 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2370a328-d5e9-4e99-b0b6-b35739ee7b66-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gmxtx\" (UID: \"2370a328-d5e9-4e99-b0b6-b35739ee7b66\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gmxtx" Jul 27 18:41:42.416447 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:41:42.416431 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2370a328-d5e9-4e99-b0b6-b35739ee7b66-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gmxtx\" (UID: \"2370a328-d5e9-4e99-b0b6-b35739ee7b66\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gmxtx" Jul 27 18:41:42.422578 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:41:42.422552 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpxtk\" (UniqueName: \"kubernetes.io/projected/2370a328-d5e9-4e99-b0b6-b35739ee7b66-kube-api-access-gpxtk\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gmxtx\" (UID: \"2370a328-d5e9-4e99-b0b6-b35739ee7b66\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gmxtx" Jul 27 18:41:42.608420 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:41:42.608335 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gmxtx" Jul 27 18:41:42.750412 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:41:42.750381 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-gmxtx"] Jul 27 18:41:42.751840 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:41:42.751811 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2370a328_d5e9_4e99_b0b6_b35739ee7b66.slice/crio-26b52c0b832bc24efecaf94e0aa493d230628a78a539be884534c3fdf74964c4 WatchSource:0}: Error finding container 26b52c0b832bc24efecaf94e0aa493d230628a78a539be884534c3fdf74964c4: Status 404 returned error can't find the container with id 26b52c0b832bc24efecaf94e0aa493d230628a78a539be884534c3fdf74964c4 Jul 27 18:41:43.154596 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:41:43.154558 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gmxtx" event={"ID":"2370a328-d5e9-4e99-b0b6-b35739ee7b66","Type":"ContainerStarted","Data":"fa2fe1af1d3aa80029f465fdc17a1342dd3cb29ff87455dc258817cd3b51a33a"} Jul 27 18:41:43.154794 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:41:43.154613 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gmxtx" event={"ID":"2370a328-d5e9-4e99-b0b6-b35739ee7b66","Type":"ContainerStarted","Data":"26b52c0b832bc24efecaf94e0aa493d230628a78a539be884534c3fdf74964c4"} Jul 27 18:41:43.177554 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:41:43.177493 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gmxtx" podStartSLOduration=0.945197791 podStartE2EDuration="1.177475577s" totalImagesPullingTime="232.277786ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:41:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:41:43.175025349 +0000 UTC m=+778.236714380" watchObservedRunningTime="2026-07-27 18:41:43.177475577 +0000 UTC m=+778.239164624" Jul 27 18:41:49.827059 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:41:49.827012 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:41:52.609119 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:41:52.609068 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gmxtx" Jul 27 18:41:52.609590 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:41:52.609145 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gmxtx" Jul 27 18:41:52.625220 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:41:52.625192 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gmxtx" Jul 27 18:41:53.220094 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:41:53.220059 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gmxtx" Jul 27 18:42:49.902411 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:42:49.902379 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:42:49.990275 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:42:49.990203 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:42:51.079197 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:42:51.079160 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:42:51.203851 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:42:51.203812 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:42:51.339770 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:42:51.339689 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-d649b6669-84m9q"] Jul 27 18:42:51.339919 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:42:51.339869 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-d649b6669-84m9q" podUID="55a32b36-c97c-438c-9f8e-fd47dc903775" containerName="authorino" containerID="cri-o://f958feca9ce84a01e307abdfb364458f6686f9b02517add5a4495bb24df49bd7" gracePeriod=30 Jul 27 18:42:51.499260 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:42:51.499194 2579 generic.go:356] "Generic (PLEG): container finished" podID="55a32b36-c97c-438c-9f8e-fd47dc903775" containerID="f958feca9ce84a01e307abdfb364458f6686f9b02517add5a4495bb24df49bd7" exitCode=0 Jul 27 18:42:51.499444 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:42:51.499270 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-d649b6669-84m9q" event={"ID":"55a32b36-c97c-438c-9f8e-fd47dc903775","Type":"ContainerDied","Data":"f958feca9ce84a01e307abdfb364458f6686f9b02517add5a4495bb24df49bd7"} Jul 27 18:42:51.590404 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:42:51.590340 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-d649b6669-84m9q" Jul 27 18:42:51.741472 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:42:51.741438 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gbp6z\" (UniqueName: \"kubernetes.io/projected/55a32b36-c97c-438c-9f8e-fd47dc903775-kube-api-access-gbp6z\") pod \"55a32b36-c97c-438c-9f8e-fd47dc903775\" (UID: \"55a32b36-c97c-438c-9f8e-fd47dc903775\") " Jul 27 18:42:51.741670 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:42:51.741490 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/55a32b36-c97c-438c-9f8e-fd47dc903775-tls-cert\") pod \"55a32b36-c97c-438c-9f8e-fd47dc903775\" (UID: \"55a32b36-c97c-438c-9f8e-fd47dc903775\") " Jul 27 18:42:51.743723 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:42:51.743698 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55a32b36-c97c-438c-9f8e-fd47dc903775-kube-api-access-gbp6z" (OuterVolumeSpecName: "kube-api-access-gbp6z") pod "55a32b36-c97c-438c-9f8e-fd47dc903775" (UID: "55a32b36-c97c-438c-9f8e-fd47dc903775"). InnerVolumeSpecName "kube-api-access-gbp6z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 18:42:51.752256 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:42:51.752198 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55a32b36-c97c-438c-9f8e-fd47dc903775-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "55a32b36-c97c-438c-9f8e-fd47dc903775" (UID: "55a32b36-c97c-438c-9f8e-fd47dc903775"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 18:42:51.842487 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:42:51.842415 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbp6z\" (UniqueName: \"kubernetes.io/projected/55a32b36-c97c-438c-9f8e-fd47dc903775-kube-api-access-gbp6z\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 18:42:51.842487 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:42:51.842441 2579 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/55a32b36-c97c-438c-9f8e-fd47dc903775-tls-cert\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 18:42:52.505608 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:42:52.505566 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-d649b6669-84m9q" event={"ID":"55a32b36-c97c-438c-9f8e-fd47dc903775","Type":"ContainerDied","Data":"95d878993f8bb129ca5d09c14a50902a4d1cee3de6f04375c9ef60b5adcf49e0"} Jul 27 18:42:52.506098 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:42:52.505619 2579 scope.go:117] "RemoveContainer" containerID="f958feca9ce84a01e307abdfb364458f6686f9b02517add5a4495bb24df49bd7" Jul 27 18:42:52.506098 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:42:52.505582 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-d649b6669-84m9q" Jul 27 18:42:52.533410 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:42:52.533386 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-d649b6669-84m9q"] Jul 27 18:42:52.537211 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:42:52.537189 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-d649b6669-84m9q"] Jul 27 18:42:53.499660 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:42:53.499617 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="55a32b36-c97c-438c-9f8e-fd47dc903775" path="/var/lib/kubelet/pods/55a32b36-c97c-438c-9f8e-fd47dc903775/volumes" Jul 27 18:43:06.832182 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:43:06.832149 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:43:34.638707 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:43:34.638668 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:43:45.415758 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:43:45.415731 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-mv4hb_715b2084-d79e-4cbf-a057-f2595b4e6b9b/console-operator/1.log" Jul 27 18:43:45.416480 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:43:45.416454 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-mv4hb_715b2084-d79e-4cbf-a057-f2595b4e6b9b/console-operator/1.log" Jul 27 18:44:31.350338 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:44:31.350278 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:44:39.360984 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:44:39.360951 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-77ff6dccdd-mfx9d"] Jul 27 18:44:39.361400 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:44:39.361385 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55a32b36-c97c-438c-9f8e-fd47dc903775" containerName="authorino" Jul 27 18:44:39.361445 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:44:39.361401 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="55a32b36-c97c-438c-9f8e-fd47dc903775" containerName="authorino" Jul 27 18:44:39.361480 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:44:39.361468 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="55a32b36-c97c-438c-9f8e-fd47dc903775" containerName="authorino" Jul 27 18:44:39.364391 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:44:39.364374 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-77ff6dccdd-mfx9d" Jul 27 18:44:39.369196 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:44:39.369174 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 27 18:44:39.369196 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:44:39.369193 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-chdqz\"" Jul 27 18:44:39.369407 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:44:39.369194 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 27 18:44:39.376579 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:44:39.376557 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-77ff6dccdd-mfx9d"] Jul 27 18:44:39.508852 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:44:39.508822 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e2f8816b-5ec7-432a-a4ad-cfc5bcc52879-webhook-cert\") pod \"maas-controller-77ff6dccdd-mfx9d\" (UID: \"e2f8816b-5ec7-432a-a4ad-cfc5bcc52879\") " pod="opendatahub/maas-controller-77ff6dccdd-mfx9d" Jul 27 18:44:39.509025 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:44:39.508864 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzqhz\" (UniqueName: \"kubernetes.io/projected/e2f8816b-5ec7-432a-a4ad-cfc5bcc52879-kube-api-access-rzqhz\") pod \"maas-controller-77ff6dccdd-mfx9d\" (UID: \"e2f8816b-5ec7-432a-a4ad-cfc5bcc52879\") " pod="opendatahub/maas-controller-77ff6dccdd-mfx9d" Jul 27 18:44:39.609739 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:44:39.609704 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzqhz\" (UniqueName: \"kubernetes.io/projected/e2f8816b-5ec7-432a-a4ad-cfc5bcc52879-kube-api-access-rzqhz\") pod \"maas-controller-77ff6dccdd-mfx9d\" (UID: \"e2f8816b-5ec7-432a-a4ad-cfc5bcc52879\") " pod="opendatahub/maas-controller-77ff6dccdd-mfx9d" Jul 27 18:44:39.609923 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:44:39.609803 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e2f8816b-5ec7-432a-a4ad-cfc5bcc52879-webhook-cert\") pod \"maas-controller-77ff6dccdd-mfx9d\" (UID: \"e2f8816b-5ec7-432a-a4ad-cfc5bcc52879\") " pod="opendatahub/maas-controller-77ff6dccdd-mfx9d" Jul 27 18:44:39.612483 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:44:39.612428 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e2f8816b-5ec7-432a-a4ad-cfc5bcc52879-webhook-cert\") pod \"maas-controller-77ff6dccdd-mfx9d\" (UID: \"e2f8816b-5ec7-432a-a4ad-cfc5bcc52879\") " pod="opendatahub/maas-controller-77ff6dccdd-mfx9d" Jul 27 18:44:39.619200 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:44:39.619024 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzqhz\" (UniqueName: \"kubernetes.io/projected/e2f8816b-5ec7-432a-a4ad-cfc5bcc52879-kube-api-access-rzqhz\") pod \"maas-controller-77ff6dccdd-mfx9d\" (UID: \"e2f8816b-5ec7-432a-a4ad-cfc5bcc52879\") " pod="opendatahub/maas-controller-77ff6dccdd-mfx9d" Jul 27 18:44:39.675355 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:44:39.675320 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-77ff6dccdd-mfx9d" Jul 27 18:44:39.828784 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:44:39.828758 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-77ff6dccdd-mfx9d"] Jul 27 18:44:39.831129 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:44:39.831096 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2f8816b_5ec7_432a_a4ad_cfc5bcc52879.slice/crio-1352463074f84ce8fd7217358a656555d82c7c807b2aed7725da26b86605b361 WatchSource:0}: Error finding container 1352463074f84ce8fd7217358a656555d82c7c807b2aed7725da26b86605b361: Status 404 returned error can't find the container with id 1352463074f84ce8fd7217358a656555d82c7c807b2aed7725da26b86605b361 Jul 27 18:44:39.832728 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:44:39.832705 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 27 18:44:40.032642 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:44:40.032610 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-77ff6dccdd-mfx9d" event={"ID":"e2f8816b-5ec7-432a-a4ad-cfc5bcc52879","Type":"ContainerStarted","Data":"1352463074f84ce8fd7217358a656555d82c7c807b2aed7725da26b86605b361"} Jul 27 18:44:41.040278 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:44:41.040213 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-77ff6dccdd-mfx9d" event={"ID":"e2f8816b-5ec7-432a-a4ad-cfc5bcc52879","Type":"ContainerStarted","Data":"c600441f634be836dc9bee28383199786e9f3e6fb1317248d7fcb4f0ed401f79"} Jul 27 18:44:41.040653 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:44:41.040336 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-77ff6dccdd-mfx9d" Jul 27 18:44:41.064170 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:44:41.064117 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-77ff6dccdd-mfx9d" podStartSLOduration=1.542597426 podStartE2EDuration="2.064102111s" totalImagesPullingTime="521.504685ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:44:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:44:41.06339995 +0000 UTC m=+956.125088982" watchObservedRunningTime="2026-07-27 18:44:41.064102111 +0000 UTC m=+956.125791141" Jul 27 18:44:52.047798 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:44:52.047759 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-77ff6dccdd-mfx9d" Jul 27 18:44:56.690972 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:44:56.690930 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:44:56.806141 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:44:56.806107 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-77ff6dccdd-mfx9d"] Jul 27 18:44:56.806344 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:44:56.806317 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-77ff6dccdd-mfx9d" podUID="e2f8816b-5ec7-432a-a4ad-cfc5bcc52879" containerName="manager" containerID="cri-o://c600441f634be836dc9bee28383199786e9f3e6fb1317248d7fcb4f0ed401f79" gracePeriod=30 Jul 27 18:44:57.120713 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:44:57.120683 2579 generic.go:356] "Generic (PLEG): container finished" podID="e2f8816b-5ec7-432a-a4ad-cfc5bcc52879" containerID="c600441f634be836dc9bee28383199786e9f3e6fb1317248d7fcb4f0ed401f79" exitCode=0 Jul 27 18:44:57.120863 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:44:57.120760 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-77ff6dccdd-mfx9d" event={"ID":"e2f8816b-5ec7-432a-a4ad-cfc5bcc52879","Type":"ContainerDied","Data":"c600441f634be836dc9bee28383199786e9f3e6fb1317248d7fcb4f0ed401f79"} Jul 27 18:44:57.163705 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:44:57.163681 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-77ff6dccdd-mfx9d" Jul 27 18:44:57.259571 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:44:57.259541 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e2f8816b-5ec7-432a-a4ad-cfc5bcc52879-webhook-cert\") pod \"e2f8816b-5ec7-432a-a4ad-cfc5bcc52879\" (UID: \"e2f8816b-5ec7-432a-a4ad-cfc5bcc52879\") " Jul 27 18:44:57.259779 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:44:57.259642 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rzqhz\" (UniqueName: \"kubernetes.io/projected/e2f8816b-5ec7-432a-a4ad-cfc5bcc52879-kube-api-access-rzqhz\") pod \"e2f8816b-5ec7-432a-a4ad-cfc5bcc52879\" (UID: \"e2f8816b-5ec7-432a-a4ad-cfc5bcc52879\") " Jul 27 18:44:57.261793 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:44:57.261767 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2f8816b-5ec7-432a-a4ad-cfc5bcc52879-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "e2f8816b-5ec7-432a-a4ad-cfc5bcc52879" (UID: "e2f8816b-5ec7-432a-a4ad-cfc5bcc52879"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 18:44:57.261793 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:44:57.261773 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2f8816b-5ec7-432a-a4ad-cfc5bcc52879-kube-api-access-rzqhz" (OuterVolumeSpecName: "kube-api-access-rzqhz") pod "e2f8816b-5ec7-432a-a4ad-cfc5bcc52879" (UID: "e2f8816b-5ec7-432a-a4ad-cfc5bcc52879"). InnerVolumeSpecName "kube-api-access-rzqhz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 18:44:57.361190 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:44:57.361104 2579 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e2f8816b-5ec7-432a-a4ad-cfc5bcc52879-webhook-cert\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 18:44:57.361190 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:44:57.361132 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzqhz\" (UniqueName: \"kubernetes.io/projected/e2f8816b-5ec7-432a-a4ad-cfc5bcc52879-kube-api-access-rzqhz\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 18:44:58.127015 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:44:58.126984 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-77ff6dccdd-mfx9d" Jul 27 18:44:58.127462 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:44:58.126985 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-77ff6dccdd-mfx9d" event={"ID":"e2f8816b-5ec7-432a-a4ad-cfc5bcc52879","Type":"ContainerDied","Data":"1352463074f84ce8fd7217358a656555d82c7c807b2aed7725da26b86605b361"} Jul 27 18:44:58.127462 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:44:58.127141 2579 scope.go:117] "RemoveContainer" containerID="c600441f634be836dc9bee28383199786e9f3e6fb1317248d7fcb4f0ed401f79" Jul 27 18:44:58.148345 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:44:58.148317 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-77ff6dccdd-mfx9d"] Jul 27 18:44:58.156096 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:44:58.156072 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-77ff6dccdd-mfx9d"] Jul 27 18:44:58.914454 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:44:58.914421 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:44:59.020011 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:44:59.019979 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-77ff6dccdd-sdkdx"] Jul 27 18:44:59.020423 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:44:59.020409 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2f8816b-5ec7-432a-a4ad-cfc5bcc52879" containerName="manager" Jul 27 18:44:59.020475 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:44:59.020425 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2f8816b-5ec7-432a-a4ad-cfc5bcc52879" containerName="manager" Jul 27 18:44:59.020516 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:44:59.020501 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2f8816b-5ec7-432a-a4ad-cfc5bcc52879" containerName="manager" Jul 27 18:44:59.024896 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:44:59.024875 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-77ff6dccdd-sdkdx" Jul 27 18:44:59.029120 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:44:59.029100 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 27 18:44:59.029267 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:44:59.029161 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-chdqz\"" Jul 27 18:44:59.029267 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:44:59.029209 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 27 18:44:59.032972 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:44:59.032949 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-77ff6dccdd-sdkdx"] Jul 27 18:44:59.078054 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:44:59.078022 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/281fa175-bbef-4309-bbaa-a3de9bf7e6a4-webhook-cert\") pod \"maas-controller-77ff6dccdd-sdkdx\" (UID: \"281fa175-bbef-4309-bbaa-a3de9bf7e6a4\") " pod="opendatahub/maas-controller-77ff6dccdd-sdkdx" Jul 27 18:44:59.078198 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:44:59.078091 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkbvw\" (UniqueName: \"kubernetes.io/projected/281fa175-bbef-4309-bbaa-a3de9bf7e6a4-kube-api-access-lkbvw\") pod \"maas-controller-77ff6dccdd-sdkdx\" (UID: \"281fa175-bbef-4309-bbaa-a3de9bf7e6a4\") " pod="opendatahub/maas-controller-77ff6dccdd-sdkdx" Jul 27 18:44:59.178957 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:44:59.178864 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkbvw\" (UniqueName: \"kubernetes.io/projected/281fa175-bbef-4309-bbaa-a3de9bf7e6a4-kube-api-access-lkbvw\") pod \"maas-controller-77ff6dccdd-sdkdx\" (UID: \"281fa175-bbef-4309-bbaa-a3de9bf7e6a4\") " pod="opendatahub/maas-controller-77ff6dccdd-sdkdx" Jul 27 18:44:59.178957 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:44:59.178927 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/281fa175-bbef-4309-bbaa-a3de9bf7e6a4-webhook-cert\") pod \"maas-controller-77ff6dccdd-sdkdx\" (UID: \"281fa175-bbef-4309-bbaa-a3de9bf7e6a4\") " pod="opendatahub/maas-controller-77ff6dccdd-sdkdx" Jul 27 18:44:59.181609 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:44:59.181586 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/281fa175-bbef-4309-bbaa-a3de9bf7e6a4-webhook-cert\") pod \"maas-controller-77ff6dccdd-sdkdx\" (UID: \"281fa175-bbef-4309-bbaa-a3de9bf7e6a4\") " pod="opendatahub/maas-controller-77ff6dccdd-sdkdx" Jul 27 18:44:59.188899 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:44:59.188855 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkbvw\" (UniqueName: \"kubernetes.io/projected/281fa175-bbef-4309-bbaa-a3de9bf7e6a4-kube-api-access-lkbvw\") pod \"maas-controller-77ff6dccdd-sdkdx\" (UID: \"281fa175-bbef-4309-bbaa-a3de9bf7e6a4\") " pod="opendatahub/maas-controller-77ff6dccdd-sdkdx" Jul 27 18:44:59.336355 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:44:59.336316 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-77ff6dccdd-sdkdx" Jul 27 18:44:59.482545 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:44:59.482516 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-77ff6dccdd-sdkdx"] Jul 27 18:44:59.484120 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:44:59.484098 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod281fa175_bbef_4309_bbaa_a3de9bf7e6a4.slice/crio-c575ad41f017afcae9e52622cfaf8c3779833382a4d369dcb7ff92a644e5f047 WatchSource:0}: Error finding container c575ad41f017afcae9e52622cfaf8c3779833382a4d369dcb7ff92a644e5f047: Status 404 returned error can't find the container with id c575ad41f017afcae9e52622cfaf8c3779833382a4d369dcb7ff92a644e5f047 Jul 27 18:44:59.494720 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:44:59.494694 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e2f8816b-5ec7-432a-a4ad-cfc5bcc52879" path="/var/lib/kubelet/pods/e2f8816b-5ec7-432a-a4ad-cfc5bcc52879/volumes" Jul 27 18:45:00.140683 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:45:00.140598 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-77ff6dccdd-sdkdx" event={"ID":"281fa175-bbef-4309-bbaa-a3de9bf7e6a4","Type":"ContainerStarted","Data":"e85e1b56e0495ba1e2cdc56b3be8a717d182c88e4fc21467e1cdc80342f47461"} Jul 27 18:45:00.140683 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:45:00.140658 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-77ff6dccdd-sdkdx" event={"ID":"281fa175-bbef-4309-bbaa-a3de9bf7e6a4","Type":"ContainerStarted","Data":"c575ad41f017afcae9e52622cfaf8c3779833382a4d369dcb7ff92a644e5f047"} Jul 27 18:45:00.140878 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:45:00.140740 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-77ff6dccdd-sdkdx" Jul 27 18:45:11.148288 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:45:11.148258 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-77ff6dccdd-sdkdx" Jul 27 18:45:11.177898 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:45:11.177837 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-77ff6dccdd-sdkdx" podStartSLOduration=12.825105893 podStartE2EDuration="13.177815965s" totalImagesPullingTime="352.710072ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:44:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:45:00.165356891 +0000 UTC m=+975.227045925" watchObservedRunningTime="2026-07-27 18:45:11.177815965 +0000 UTC m=+986.239504996" Jul 27 18:45:39.918119 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:45:39.918078 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:45:40.017274 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:45:40.017220 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:45:40.165372 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:45:40.165334 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:45:46.949857 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:45:46.949814 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:45:47.107858 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:45:47.107823 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:45:47.255552 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:45:47.255429 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:45:47.648690 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:45:47.648653 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:45:53.972719 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:45:53.972679 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:45:54.080791 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:45:54.080761 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:46:39.884259 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:46:39.884143 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:48:45.465691 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:48:45.465664 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-mv4hb_715b2084-d79e-4cbf-a057-f2595b4e6b9b/console-operator/1.log" Jul 27 18:48:45.467671 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:48:45.467650 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-mv4hb_715b2084-d79e-4cbf-a057-f2595b4e6b9b/console-operator/1.log" Jul 27 18:49:55.033648 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:49:55.033613 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mzvxm"] Jul 27 18:49:55.034209 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:49:55.033823 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mzvxm" podUID="ab81bc00-e76a-43be-882c-3ef4a8cae1d5" containerName="manager" containerID="cri-o://c3a51c7612e7754c82339943a50445a10e045240ebb51eab6fb97325842b48db" gracePeriod=10 Jul 27 18:49:57.590798 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:49:57.590763 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mzvxm" Jul 27 18:49:57.619353 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:49:57.619322 2579 generic.go:356] "Generic (PLEG): container finished" podID="ab81bc00-e76a-43be-882c-3ef4a8cae1d5" containerID="c3a51c7612e7754c82339943a50445a10e045240ebb51eab6fb97325842b48db" exitCode=0 Jul 27 18:49:57.619523 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:49:57.619371 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mzvxm" Jul 27 18:49:57.619523 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:49:57.619397 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mzvxm" event={"ID":"ab81bc00-e76a-43be-882c-3ef4a8cae1d5","Type":"ContainerDied","Data":"c3a51c7612e7754c82339943a50445a10e045240ebb51eab6fb97325842b48db"} Jul 27 18:49:57.619523 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:49:57.619442 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mzvxm" event={"ID":"ab81bc00-e76a-43be-882c-3ef4a8cae1d5","Type":"ContainerDied","Data":"d11e63bf7ff3072d2e13fc7b10e91ff86bc4adefe387939f95341f7cb06efda2"} Jul 27 18:49:57.619523 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:49:57.619475 2579 scope.go:117] "RemoveContainer" containerID="c3a51c7612e7754c82339943a50445a10e045240ebb51eab6fb97325842b48db" Jul 27 18:49:57.632113 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:49:57.632082 2579 scope.go:117] "RemoveContainer" containerID="c3a51c7612e7754c82339943a50445a10e045240ebb51eab6fb97325842b48db" Jul 27 18:49:57.632473 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:49:57.632449 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3a51c7612e7754c82339943a50445a10e045240ebb51eab6fb97325842b48db\": container with ID starting with c3a51c7612e7754c82339943a50445a10e045240ebb51eab6fb97325842b48db not found: ID does not exist" containerID="c3a51c7612e7754c82339943a50445a10e045240ebb51eab6fb97325842b48db" Jul 27 18:49:57.632560 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:49:57.632484 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3a51c7612e7754c82339943a50445a10e045240ebb51eab6fb97325842b48db"} err="failed to get container status \"c3a51c7612e7754c82339943a50445a10e045240ebb51eab6fb97325842b48db\": rpc error: code = NotFound desc = could not find container \"c3a51c7612e7754c82339943a50445a10e045240ebb51eab6fb97325842b48db\": container with ID starting with c3a51c7612e7754c82339943a50445a10e045240ebb51eab6fb97325842b48db not found: ID does not exist" Jul 27 18:49:57.727155 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:49:57.727050 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s76w2\" (UniqueName: \"kubernetes.io/projected/ab81bc00-e76a-43be-882c-3ef4a8cae1d5-kube-api-access-s76w2\") pod \"ab81bc00-e76a-43be-882c-3ef4a8cae1d5\" (UID: \"ab81bc00-e76a-43be-882c-3ef4a8cae1d5\") " Jul 27 18:49:57.727155 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:49:57.727117 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ab81bc00-e76a-43be-882c-3ef4a8cae1d5-extensions-socket-volume\") pod \"ab81bc00-e76a-43be-882c-3ef4a8cae1d5\" (UID: \"ab81bc00-e76a-43be-882c-3ef4a8cae1d5\") " Jul 27 18:49:57.727520 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:49:57.727497 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab81bc00-e76a-43be-882c-3ef4a8cae1d5-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "ab81bc00-e76a-43be-882c-3ef4a8cae1d5" (UID: "ab81bc00-e76a-43be-882c-3ef4a8cae1d5"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 18:49:57.729483 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:49:57.729453 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab81bc00-e76a-43be-882c-3ef4a8cae1d5-kube-api-access-s76w2" (OuterVolumeSpecName: "kube-api-access-s76w2") pod "ab81bc00-e76a-43be-882c-3ef4a8cae1d5" (UID: "ab81bc00-e76a-43be-882c-3ef4a8cae1d5"). InnerVolumeSpecName "kube-api-access-s76w2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 18:49:57.827996 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:49:57.827967 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s76w2\" (UniqueName: \"kubernetes.io/projected/ab81bc00-e76a-43be-882c-3ef4a8cae1d5-kube-api-access-s76w2\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 18:49:57.827996 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:49:57.827993 2579 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ab81bc00-e76a-43be-882c-3ef4a8cae1d5-extensions-socket-volume\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 18:49:57.950966 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:49:57.950937 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mzvxm"] Jul 27 18:49:57.954816 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:49:57.954792 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mzvxm"] Jul 27 18:49:59.492995 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:49:59.492959 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ab81bc00-e76a-43be-882c-3ef4a8cae1d5" path="/var/lib/kubelet/pods/ab81bc00-e76a-43be-882c-3ef4a8cae1d5/volumes" Jul 27 18:50:46.902165 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:50:46.902125 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:50:56.289425 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:50:56.289390 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tgppf"] Jul 27 18:50:56.289888 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:50:56.289760 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab81bc00-e76a-43be-882c-3ef4a8cae1d5" containerName="manager" Jul 27 18:50:56.289888 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:50:56.289770 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab81bc00-e76a-43be-882c-3ef4a8cae1d5" containerName="manager" Jul 27 18:50:56.289888 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:50:56.289847 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab81bc00-e76a-43be-882c-3ef4a8cae1d5" containerName="manager" Jul 27 18:50:56.294042 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:50:56.294011 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tgppf" Jul 27 18:50:56.298163 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:50:56.298139 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-q4btw\"" Jul 27 18:50:56.316004 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:50:56.315976 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tgppf"] Jul 27 18:50:56.432357 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:50:56.432318 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d2145611-dcfa-4590-8f47-e742fe5beca7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-tgppf\" (UID: \"d2145611-dcfa-4590-8f47-e742fe5beca7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tgppf" Jul 27 18:50:56.432519 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:50:56.432387 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrfrx\" (UniqueName: \"kubernetes.io/projected/d2145611-dcfa-4590-8f47-e742fe5beca7-kube-api-access-hrfrx\") pod \"kuadrant-operator-controller-manager-86878c769f-tgppf\" (UID: \"d2145611-dcfa-4590-8f47-e742fe5beca7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tgppf" Jul 27 18:50:56.533452 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:50:56.533418 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrfrx\" (UniqueName: \"kubernetes.io/projected/d2145611-dcfa-4590-8f47-e742fe5beca7-kube-api-access-hrfrx\") pod \"kuadrant-operator-controller-manager-86878c769f-tgppf\" (UID: \"d2145611-dcfa-4590-8f47-e742fe5beca7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tgppf" Jul 27 18:50:56.533605 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:50:56.533492 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d2145611-dcfa-4590-8f47-e742fe5beca7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-tgppf\" (UID: \"d2145611-dcfa-4590-8f47-e742fe5beca7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tgppf" Jul 27 18:50:56.533818 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:50:56.533802 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d2145611-dcfa-4590-8f47-e742fe5beca7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-tgppf\" (UID: \"d2145611-dcfa-4590-8f47-e742fe5beca7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tgppf" Jul 27 18:50:56.544344 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:50:56.544276 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrfrx\" (UniqueName: \"kubernetes.io/projected/d2145611-dcfa-4590-8f47-e742fe5beca7-kube-api-access-hrfrx\") pod \"kuadrant-operator-controller-manager-86878c769f-tgppf\" (UID: \"d2145611-dcfa-4590-8f47-e742fe5beca7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tgppf" Jul 27 18:50:56.604831 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:50:56.604802 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tgppf" Jul 27 18:50:56.763910 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:50:56.763723 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tgppf"] Jul 27 18:50:56.767142 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:50:56.767114 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2145611_dcfa_4590_8f47_e742fe5beca7.slice/crio-d19312108fd773b9e3606985d70b02154df02a1518ff005e9587735b9ecc9a15 WatchSource:0}: Error finding container d19312108fd773b9e3606985d70b02154df02a1518ff005e9587735b9ecc9a15: Status 404 returned error can't find the container with id d19312108fd773b9e3606985d70b02154df02a1518ff005e9587735b9ecc9a15 Jul 27 18:50:56.917704 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:50:56.917670 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tgppf" event={"ID":"d2145611-dcfa-4590-8f47-e742fe5beca7","Type":"ContainerStarted","Data":"020fc3dacb0c08de2c0c793de6e1449ce55d705adc1f066b5abe5ead7d066fed"} Jul 27 18:50:56.917866 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:50:56.917727 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tgppf" event={"ID":"d2145611-dcfa-4590-8f47-e742fe5beca7","Type":"ContainerStarted","Data":"d19312108fd773b9e3606985d70b02154df02a1518ff005e9587735b9ecc9a15"} Jul 27 18:50:56.917935 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:50:56.917867 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tgppf" Jul 27 18:50:56.952881 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:50:56.952831 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tgppf" podStartSLOduration=0.952817254 podStartE2EDuration="952.817254ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:50:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:50:56.951331856 +0000 UTC m=+1332.013020888" watchObservedRunningTime="2026-07-27 18:50:56.952817254 +0000 UTC m=+1332.014506285" Jul 27 18:51:07.924986 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:51:07.924955 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tgppf" Jul 27 18:52:12.931247 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:52:12.931188 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:52:13.716385 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:52:13.716348 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:52:13.792261 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:52:13.792211 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:53:37.489031 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:53:37.488992 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:53:45.496518 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:53:45.496477 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-mv4hb_715b2084-d79e-4cbf-a057-f2595b4e6b9b/console-operator/1.log" Jul 27 18:53:45.499845 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:53:45.499826 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-mv4hb_715b2084-d79e-4cbf-a057-f2595b4e6b9b/console-operator/1.log" Jul 27 18:53:57.945332 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:53:57.945299 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:53:58.045206 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:53:58.045171 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:54:02.500455 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:54:02.500420 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-a92e6c29-kserve-67654cf847-qr7nr"] Jul 27 18:54:02.503957 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:54:02.503933 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-a92e6c29-kserve-67654cf847-qr7nr" Jul 27 18:54:02.507476 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:54:02.507453 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-a92e6c29-kserve-self-signed-certs\"" Jul 27 18:54:02.523868 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:54:02.523839 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-a92e6c29-kserve-67654cf847-qr7nr"] Jul 27 18:54:02.560944 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:54:02.560912 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:54:02.634374 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:54:02.634335 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfbgf\" (UniqueName: \"kubernetes.io/projected/dc5033f7-15cf-41f1-957e-e33695f22f23-kube-api-access-jfbgf\") pod \"e2e-same-id-b-a92e6c29-kserve-67654cf847-qr7nr\" (UID: \"dc5033f7-15cf-41f1-957e-e33695f22f23\") " pod="llm/e2e-same-id-b-a92e6c29-kserve-67654cf847-qr7nr" Jul 27 18:54:02.634553 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:54:02.634421 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc5033f7-15cf-41f1-957e-e33695f22f23-model-cache\") pod \"e2e-same-id-b-a92e6c29-kserve-67654cf847-qr7nr\" (UID: \"dc5033f7-15cf-41f1-957e-e33695f22f23\") " pod="llm/e2e-same-id-b-a92e6c29-kserve-67654cf847-qr7nr" Jul 27 18:54:02.634553 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:54:02.634443 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc5033f7-15cf-41f1-957e-e33695f22f23-dshm\") pod \"e2e-same-id-b-a92e6c29-kserve-67654cf847-qr7nr\" (UID: \"dc5033f7-15cf-41f1-957e-e33695f22f23\") " pod="llm/e2e-same-id-b-a92e6c29-kserve-67654cf847-qr7nr" Jul 27 18:54:02.634553 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:54:02.634462 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc5033f7-15cf-41f1-957e-e33695f22f23-home\") pod \"e2e-same-id-b-a92e6c29-kserve-67654cf847-qr7nr\" (UID: \"dc5033f7-15cf-41f1-957e-e33695f22f23\") " pod="llm/e2e-same-id-b-a92e6c29-kserve-67654cf847-qr7nr" Jul 27 18:54:02.634694 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:54:02.634573 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc5033f7-15cf-41f1-957e-e33695f22f23-tmp-dir\") pod \"e2e-same-id-b-a92e6c29-kserve-67654cf847-qr7nr\" (UID: \"dc5033f7-15cf-41f1-957e-e33695f22f23\") " pod="llm/e2e-same-id-b-a92e6c29-kserve-67654cf847-qr7nr" Jul 27 18:54:02.634694 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:54:02.634613 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc5033f7-15cf-41f1-957e-e33695f22f23-tls-certs\") pod \"e2e-same-id-b-a92e6c29-kserve-67654cf847-qr7nr\" (UID: \"dc5033f7-15cf-41f1-957e-e33695f22f23\") " pod="llm/e2e-same-id-b-a92e6c29-kserve-67654cf847-qr7nr" Jul 27 18:54:02.736187 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:54:02.736153 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc5033f7-15cf-41f1-957e-e33695f22f23-model-cache\") pod \"e2e-same-id-b-a92e6c29-kserve-67654cf847-qr7nr\" (UID: \"dc5033f7-15cf-41f1-957e-e33695f22f23\") " pod="llm/e2e-same-id-b-a92e6c29-kserve-67654cf847-qr7nr" Jul 27 18:54:02.736403 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:54:02.736203 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc5033f7-15cf-41f1-957e-e33695f22f23-dshm\") pod \"e2e-same-id-b-a92e6c29-kserve-67654cf847-qr7nr\" (UID: \"dc5033f7-15cf-41f1-957e-e33695f22f23\") " pod="llm/e2e-same-id-b-a92e6c29-kserve-67654cf847-qr7nr" Jul 27 18:54:02.736403 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:54:02.736253 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc5033f7-15cf-41f1-957e-e33695f22f23-home\") pod \"e2e-same-id-b-a92e6c29-kserve-67654cf847-qr7nr\" (UID: \"dc5033f7-15cf-41f1-957e-e33695f22f23\") " pod="llm/e2e-same-id-b-a92e6c29-kserve-67654cf847-qr7nr" Jul 27 18:54:02.736403 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:54:02.736301 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc5033f7-15cf-41f1-957e-e33695f22f23-tmp-dir\") pod \"e2e-same-id-b-a92e6c29-kserve-67654cf847-qr7nr\" (UID: \"dc5033f7-15cf-41f1-957e-e33695f22f23\") " pod="llm/e2e-same-id-b-a92e6c29-kserve-67654cf847-qr7nr" Jul 27 18:54:02.736403 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:54:02.736327 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc5033f7-15cf-41f1-957e-e33695f22f23-tls-certs\") pod \"e2e-same-id-b-a92e6c29-kserve-67654cf847-qr7nr\" (UID: \"dc5033f7-15cf-41f1-957e-e33695f22f23\") " pod="llm/e2e-same-id-b-a92e6c29-kserve-67654cf847-qr7nr" Jul 27 18:54:02.736403 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:54:02.736392 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfbgf\" (UniqueName: \"kubernetes.io/projected/dc5033f7-15cf-41f1-957e-e33695f22f23-kube-api-access-jfbgf\") pod \"e2e-same-id-b-a92e6c29-kserve-67654cf847-qr7nr\" (UID: \"dc5033f7-15cf-41f1-957e-e33695f22f23\") " pod="llm/e2e-same-id-b-a92e6c29-kserve-67654cf847-qr7nr" Jul 27 18:54:02.736695 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:54:02.736632 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc5033f7-15cf-41f1-957e-e33695f22f23-model-cache\") pod \"e2e-same-id-b-a92e6c29-kserve-67654cf847-qr7nr\" (UID: \"dc5033f7-15cf-41f1-957e-e33695f22f23\") " pod="llm/e2e-same-id-b-a92e6c29-kserve-67654cf847-qr7nr" Jul 27 18:54:02.736786 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:54:02.736743 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc5033f7-15cf-41f1-957e-e33695f22f23-tmp-dir\") pod \"e2e-same-id-b-a92e6c29-kserve-67654cf847-qr7nr\" (UID: \"dc5033f7-15cf-41f1-957e-e33695f22f23\") " pod="llm/e2e-same-id-b-a92e6c29-kserve-67654cf847-qr7nr" Jul 27 18:54:02.736923 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:54:02.736746 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc5033f7-15cf-41f1-957e-e33695f22f23-home\") pod \"e2e-same-id-b-a92e6c29-kserve-67654cf847-qr7nr\" (UID: \"dc5033f7-15cf-41f1-957e-e33695f22f23\") " pod="llm/e2e-same-id-b-a92e6c29-kserve-67654cf847-qr7nr" Jul 27 18:54:02.739067 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:54:02.739042 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc5033f7-15cf-41f1-957e-e33695f22f23-dshm\") pod \"e2e-same-id-b-a92e6c29-kserve-67654cf847-qr7nr\" (UID: \"dc5033f7-15cf-41f1-957e-e33695f22f23\") " pod="llm/e2e-same-id-b-a92e6c29-kserve-67654cf847-qr7nr" Jul 27 18:54:02.739281 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:54:02.739257 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc5033f7-15cf-41f1-957e-e33695f22f23-tls-certs\") pod \"e2e-same-id-b-a92e6c29-kserve-67654cf847-qr7nr\" (UID: \"dc5033f7-15cf-41f1-957e-e33695f22f23\") " pod="llm/e2e-same-id-b-a92e6c29-kserve-67654cf847-qr7nr" Jul 27 18:54:02.745938 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:54:02.745916 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfbgf\" (UniqueName: \"kubernetes.io/projected/dc5033f7-15cf-41f1-957e-e33695f22f23-kube-api-access-jfbgf\") pod \"e2e-same-id-b-a92e6c29-kserve-67654cf847-qr7nr\" (UID: \"dc5033f7-15cf-41f1-957e-e33695f22f23\") " pod="llm/e2e-same-id-b-a92e6c29-kserve-67654cf847-qr7nr" Jul 27 18:54:02.813914 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:54:02.813880 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-a92e6c29-kserve-67654cf847-qr7nr" Jul 27 18:54:02.959793 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:54:02.959765 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-a92e6c29-kserve-67654cf847-qr7nr"] Jul 27 18:54:02.961507 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:54:02.961470 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddc5033f7_15cf_41f1_957e_e33695f22f23.slice/crio-86e7fb64de35e077eaf73f466d3e0411de907d1480ca10919125548bf949beb9 WatchSource:0}: Error finding container 86e7fb64de35e077eaf73f466d3e0411de907d1480ca10919125548bf949beb9: Status 404 returned error can't find the container with id 86e7fb64de35e077eaf73f466d3e0411de907d1480ca10919125548bf949beb9 Jul 27 18:54:03.870745 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:54:03.870703 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-a92e6c29-kserve-67654cf847-qr7nr" event={"ID":"dc5033f7-15cf-41f1-957e-e33695f22f23","Type":"ContainerStarted","Data":"3f0b28d8aafd773ddf43b451e9e3af3136010a056c8f4fb55d6bb524eee4e663"} Jul 27 18:54:03.871166 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:54:03.870753 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-a92e6c29-kserve-67654cf847-qr7nr" event={"ID":"dc5033f7-15cf-41f1-957e-e33695f22f23","Type":"ContainerStarted","Data":"86e7fb64de35e077eaf73f466d3e0411de907d1480ca10919125548bf949beb9"} Jul 27 18:54:03.893080 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:54:03.893030 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-a92e6c29-kserve-67654cf847-qr7nr" podStartSLOduration=1.8930164 podStartE2EDuration="1.8930164s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:54:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:54:03.891063714 +0000 UTC m=+1518.952752756" watchObservedRunningTime="2026-07-27 18:54:03.8930164 +0000 UTC m=+1518.954705431" Jul 27 18:54:12.814419 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:54:12.814306 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-a92e6c29-kserve-67654cf847-qr7nr" Jul 27 18:54:12.814419 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:54:12.814373 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-a92e6c29-kserve-67654cf847-qr7nr" Jul 27 18:54:12.830441 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:54:12.830410 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-a92e6c29-kserve-67654cf847-qr7nr" Jul 27 18:54:12.927317 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:54:12.927267 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-a92e6c29-kserve-67654cf847-qr7nr" Jul 27 18:54:31.194809 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:54:31.194772 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:54:32.812350 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:54:32.812317 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-a92e6c29-kserve-67654cf847-qr7nr"] Jul 27 18:54:32.836465 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:54:32.836429 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:54:47.833098 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:54:47.833026 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-a92e6c29-kserve-67654cf847-qr7nr" podUID="dc5033f7-15cf-41f1-957e-e33695f22f23" containerName="main" containerID="cri-o://3f0b28d8aafd773ddf43b451e9e3af3136010a056c8f4fb55d6bb524eee4e663" gracePeriod=45 Jul 27 18:54:48.091221 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:54:48.091139 2579 generic.go:356] "Generic (PLEG): container finished" podID="dc5033f7-15cf-41f1-957e-e33695f22f23" containerID="3f0b28d8aafd773ddf43b451e9e3af3136010a056c8f4fb55d6bb524eee4e663" exitCode=0 Jul 27 18:54:48.091371 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:54:48.091213 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-a92e6c29-kserve-67654cf847-qr7nr" event={"ID":"dc5033f7-15cf-41f1-957e-e33695f22f23","Type":"ContainerDied","Data":"3f0b28d8aafd773ddf43b451e9e3af3136010a056c8f4fb55d6bb524eee4e663"} Jul 27 18:54:48.091371 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:54:48.091285 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-a92e6c29-kserve-67654cf847-qr7nr" event={"ID":"dc5033f7-15cf-41f1-957e-e33695f22f23","Type":"ContainerDied","Data":"86e7fb64de35e077eaf73f466d3e0411de907d1480ca10919125548bf949beb9"} Jul 27 18:54:48.091371 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:54:48.091297 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="86e7fb64de35e077eaf73f466d3e0411de907d1480ca10919125548bf949beb9" Jul 27 18:54:48.098565 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:54:48.098545 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-a92e6c29-kserve-67654cf847-qr7nr" Jul 27 18:54:48.161154 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:54:48.161126 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc5033f7-15cf-41f1-957e-e33695f22f23-dshm\") pod \"dc5033f7-15cf-41f1-957e-e33695f22f23\" (UID: \"dc5033f7-15cf-41f1-957e-e33695f22f23\") " Jul 27 18:54:48.161154 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:54:48.161157 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc5033f7-15cf-41f1-957e-e33695f22f23-tmp-dir\") pod \"dc5033f7-15cf-41f1-957e-e33695f22f23\" (UID: \"dc5033f7-15cf-41f1-957e-e33695f22f23\") " Jul 27 18:54:48.161393 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:54:48.161242 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jfbgf\" (UniqueName: \"kubernetes.io/projected/dc5033f7-15cf-41f1-957e-e33695f22f23-kube-api-access-jfbgf\") pod \"dc5033f7-15cf-41f1-957e-e33695f22f23\" (UID: \"dc5033f7-15cf-41f1-957e-e33695f22f23\") " Jul 27 18:54:48.161393 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:54:48.161267 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc5033f7-15cf-41f1-957e-e33695f22f23-home\") pod \"dc5033f7-15cf-41f1-957e-e33695f22f23\" (UID: \"dc5033f7-15cf-41f1-957e-e33695f22f23\") " Jul 27 18:54:48.161393 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:54:48.161329 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc5033f7-15cf-41f1-957e-e33695f22f23-tls-certs\") pod \"dc5033f7-15cf-41f1-957e-e33695f22f23\" (UID: \"dc5033f7-15cf-41f1-957e-e33695f22f23\") " Jul 27 18:54:48.161393 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:54:48.161351 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc5033f7-15cf-41f1-957e-e33695f22f23-model-cache\") pod \"dc5033f7-15cf-41f1-957e-e33695f22f23\" (UID: \"dc5033f7-15cf-41f1-957e-e33695f22f23\") " Jul 27 18:54:48.161611 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:54:48.161499 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc5033f7-15cf-41f1-957e-e33695f22f23-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "dc5033f7-15cf-41f1-957e-e33695f22f23" (UID: "dc5033f7-15cf-41f1-957e-e33695f22f23"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 18:54:48.161611 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:54:48.161595 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc5033f7-15cf-41f1-957e-e33695f22f23-home" (OuterVolumeSpecName: "home") pod "dc5033f7-15cf-41f1-957e-e33695f22f23" (UID: "dc5033f7-15cf-41f1-957e-e33695f22f23"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 18:54:48.161731 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:54:48.161608 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc5033f7-15cf-41f1-957e-e33695f22f23-model-cache" (OuterVolumeSpecName: "model-cache") pod "dc5033f7-15cf-41f1-957e-e33695f22f23" (UID: "dc5033f7-15cf-41f1-957e-e33695f22f23"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 18:54:48.161731 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:54:48.161728 2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc5033f7-15cf-41f1-957e-e33695f22f23-home\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 18:54:48.161858 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:54:48.161739 2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc5033f7-15cf-41f1-957e-e33695f22f23-model-cache\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 18:54:48.161858 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:54:48.161768 2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc5033f7-15cf-41f1-957e-e33695f22f23-tmp-dir\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 18:54:48.163621 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:54:48.163593 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc5033f7-15cf-41f1-957e-e33695f22f23-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "dc5033f7-15cf-41f1-957e-e33695f22f23" (UID: "dc5033f7-15cf-41f1-957e-e33695f22f23"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 18:54:48.163621 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:54:48.163601 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc5033f7-15cf-41f1-957e-e33695f22f23-dshm" (OuterVolumeSpecName: "dshm") pod "dc5033f7-15cf-41f1-957e-e33695f22f23" (UID: "dc5033f7-15cf-41f1-957e-e33695f22f23"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 18:54:48.163621 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:54:48.163619 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc5033f7-15cf-41f1-957e-e33695f22f23-kube-api-access-jfbgf" (OuterVolumeSpecName: "kube-api-access-jfbgf") pod "dc5033f7-15cf-41f1-957e-e33695f22f23" (UID: "dc5033f7-15cf-41f1-957e-e33695f22f23"). InnerVolumeSpecName "kube-api-access-jfbgf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 18:54:48.263344 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:54:48.263307 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfbgf\" (UniqueName: \"kubernetes.io/projected/dc5033f7-15cf-41f1-957e-e33695f22f23-kube-api-access-jfbgf\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 18:54:48.263344 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:54:48.263337 2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc5033f7-15cf-41f1-957e-e33695f22f23-tls-certs\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 18:54:48.263344 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:54:48.263347 2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc5033f7-15cf-41f1-957e-e33695f22f23-dshm\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 18:54:49.096533 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:54:49.096504 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-a92e6c29-kserve-67654cf847-qr7nr" Jul 27 18:54:49.119981 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:54:49.119957 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-a92e6c29-kserve-67654cf847-qr7nr"] Jul 27 18:54:49.127676 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:54:49.127649 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-a92e6c29-kserve-67654cf847-qr7nr"] Jul 27 18:54:49.493682 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:54:49.493608 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dc5033f7-15cf-41f1-957e-e33695f22f23" path="/var/lib/kubelet/pods/dc5033f7-15cf-41f1-957e-e33695f22f23/volumes" Jul 27 18:57:39.697536 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:57:39.697459 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:58:01.516979 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:01.516940 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:58:04.967243 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:04.967191 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:58:05.079527 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:05.079485 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:58:05.457794 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:05.457765 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:58:11.761642 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:11.761609 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:58:11.863321 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:11.863291 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:58:12.028146 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:12.028119 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:58:12.481386 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:12.481310 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-001f5ecd-openshift-default-bbcdf955b-mgmbz"] Jul 27 18:58:12.481745 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:12.481733 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc5033f7-15cf-41f1-957e-e33695f22f23" containerName="main" Jul 27 18:58:12.481797 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:12.481747 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc5033f7-15cf-41f1-957e-e33695f22f23" containerName="main" Jul 27 18:58:12.481835 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:12.481807 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc5033f7-15cf-41f1-957e-e33695f22f23" containerName="main" Jul 27 18:58:12.484465 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:12.484450 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-001f5ecd-openshift-default-bbcdf955b-mgmbz" Jul 27 18:58:12.494504 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:12.494480 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-001f5ecd-openshift-default-dockercfg-h2p7k\"" Jul 27 18:58:12.528446 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:12.528410 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-001f5ecd-openshift-default-bbcdf955b-mgmbz"] Jul 27 18:58:12.534777 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:12.534751 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:58:12.615492 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:12.615455 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22f63302-7966-40e7-8cfb-59201c5b99ae-istio-podinfo\") pod \"e2e-ait-001f5ecd-openshift-default-bbcdf955b-mgmbz\" (UID: \"22f63302-7966-40e7-8cfb-59201c5b99ae\") " pod="openshift-ingress/e2e-ait-001f5ecd-openshift-default-bbcdf955b-mgmbz" Jul 27 18:58:12.615685 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:12.615500 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22f63302-7966-40e7-8cfb-59201c5b99ae-istio-token\") pod \"e2e-ait-001f5ecd-openshift-default-bbcdf955b-mgmbz\" (UID: \"22f63302-7966-40e7-8cfb-59201c5b99ae\") " pod="openshift-ingress/e2e-ait-001f5ecd-openshift-default-bbcdf955b-mgmbz" Jul 27 18:58:12.615685 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:12.615578 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22f63302-7966-40e7-8cfb-59201c5b99ae-istiod-ca-cert\") pod \"e2e-ait-001f5ecd-openshift-default-bbcdf955b-mgmbz\" (UID: \"22f63302-7966-40e7-8cfb-59201c5b99ae\") " pod="openshift-ingress/e2e-ait-001f5ecd-openshift-default-bbcdf955b-mgmbz" Jul 27 18:58:12.615685 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:12.615621 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22f63302-7966-40e7-8cfb-59201c5b99ae-workload-certs\") pod \"e2e-ait-001f5ecd-openshift-default-bbcdf955b-mgmbz\" (UID: \"22f63302-7966-40e7-8cfb-59201c5b99ae\") " pod="openshift-ingress/e2e-ait-001f5ecd-openshift-default-bbcdf955b-mgmbz" Jul 27 18:58:12.615876 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:12.615702 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22f63302-7966-40e7-8cfb-59201c5b99ae-workload-socket\") pod \"e2e-ait-001f5ecd-openshift-default-bbcdf955b-mgmbz\" (UID: \"22f63302-7966-40e7-8cfb-59201c5b99ae\") " pod="openshift-ingress/e2e-ait-001f5ecd-openshift-default-bbcdf955b-mgmbz" Jul 27 18:58:12.615876 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:12.615734 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22f63302-7966-40e7-8cfb-59201c5b99ae-istio-envoy\") pod \"e2e-ait-001f5ecd-openshift-default-bbcdf955b-mgmbz\" (UID: \"22f63302-7966-40e7-8cfb-59201c5b99ae\") " pod="openshift-ingress/e2e-ait-001f5ecd-openshift-default-bbcdf955b-mgmbz" Jul 27 18:58:12.615876 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:12.615766 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22f63302-7966-40e7-8cfb-59201c5b99ae-credential-socket\") pod \"e2e-ait-001f5ecd-openshift-default-bbcdf955b-mgmbz\" (UID: \"22f63302-7966-40e7-8cfb-59201c5b99ae\") " pod="openshift-ingress/e2e-ait-001f5ecd-openshift-default-bbcdf955b-mgmbz" Jul 27 18:58:12.615876 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:12.615824 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22f63302-7966-40e7-8cfb-59201c5b99ae-istio-data\") pod \"e2e-ait-001f5ecd-openshift-default-bbcdf955b-mgmbz\" (UID: \"22f63302-7966-40e7-8cfb-59201c5b99ae\") " pod="openshift-ingress/e2e-ait-001f5ecd-openshift-default-bbcdf955b-mgmbz" Jul 27 18:58:12.616014 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:12.615956 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ppx6\" (UniqueName: \"kubernetes.io/projected/22f63302-7966-40e7-8cfb-59201c5b99ae-kube-api-access-2ppx6\") pod \"e2e-ait-001f5ecd-openshift-default-bbcdf955b-mgmbz\" (UID: \"22f63302-7966-40e7-8cfb-59201c5b99ae\") " pod="openshift-ingress/e2e-ait-001f5ecd-openshift-default-bbcdf955b-mgmbz" Jul 27 18:58:12.716569 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:12.716534 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22f63302-7966-40e7-8cfb-59201c5b99ae-istiod-ca-cert\") pod \"e2e-ait-001f5ecd-openshift-default-bbcdf955b-mgmbz\" (UID: \"22f63302-7966-40e7-8cfb-59201c5b99ae\") " pod="openshift-ingress/e2e-ait-001f5ecd-openshift-default-bbcdf955b-mgmbz" Jul 27 18:58:12.716569 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:12.716576 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22f63302-7966-40e7-8cfb-59201c5b99ae-workload-certs\") pod \"e2e-ait-001f5ecd-openshift-default-bbcdf955b-mgmbz\" (UID: \"22f63302-7966-40e7-8cfb-59201c5b99ae\") " pod="openshift-ingress/e2e-ait-001f5ecd-openshift-default-bbcdf955b-mgmbz" Jul 27 18:58:12.716813 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:12.716626 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22f63302-7966-40e7-8cfb-59201c5b99ae-workload-socket\") pod \"e2e-ait-001f5ecd-openshift-default-bbcdf955b-mgmbz\" (UID: \"22f63302-7966-40e7-8cfb-59201c5b99ae\") " pod="openshift-ingress/e2e-ait-001f5ecd-openshift-default-bbcdf955b-mgmbz" Jul 27 18:58:12.716813 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:12.716660 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22f63302-7966-40e7-8cfb-59201c5b99ae-istio-envoy\") pod \"e2e-ait-001f5ecd-openshift-default-bbcdf955b-mgmbz\" (UID: \"22f63302-7966-40e7-8cfb-59201c5b99ae\") " pod="openshift-ingress/e2e-ait-001f5ecd-openshift-default-bbcdf955b-mgmbz" Jul 27 18:58:12.716813 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:12.716789 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22f63302-7966-40e7-8cfb-59201c5b99ae-credential-socket\") pod \"e2e-ait-001f5ecd-openshift-default-bbcdf955b-mgmbz\" (UID: \"22f63302-7966-40e7-8cfb-59201c5b99ae\") " pod="openshift-ingress/e2e-ait-001f5ecd-openshift-default-bbcdf955b-mgmbz" Jul 27 18:58:12.716983 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:12.716830 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22f63302-7966-40e7-8cfb-59201c5b99ae-istio-data\") pod \"e2e-ait-001f5ecd-openshift-default-bbcdf955b-mgmbz\" (UID: \"22f63302-7966-40e7-8cfb-59201c5b99ae\") " pod="openshift-ingress/e2e-ait-001f5ecd-openshift-default-bbcdf955b-mgmbz" Jul 27 18:58:12.716983 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:12.716883 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ppx6\" (UniqueName: \"kubernetes.io/projected/22f63302-7966-40e7-8cfb-59201c5b99ae-kube-api-access-2ppx6\") pod \"e2e-ait-001f5ecd-openshift-default-bbcdf955b-mgmbz\" (UID: \"22f63302-7966-40e7-8cfb-59201c5b99ae\") " pod="openshift-ingress/e2e-ait-001f5ecd-openshift-default-bbcdf955b-mgmbz" Jul 27 18:58:12.716983 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:12.716946 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22f63302-7966-40e7-8cfb-59201c5b99ae-istio-podinfo\") pod \"e2e-ait-001f5ecd-openshift-default-bbcdf955b-mgmbz\" (UID: \"22f63302-7966-40e7-8cfb-59201c5b99ae\") " pod="openshift-ingress/e2e-ait-001f5ecd-openshift-default-bbcdf955b-mgmbz" Jul 27 18:58:12.717135 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:12.716984 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22f63302-7966-40e7-8cfb-59201c5b99ae-istio-token\") pod \"e2e-ait-001f5ecd-openshift-default-bbcdf955b-mgmbz\" (UID: \"22f63302-7966-40e7-8cfb-59201c5b99ae\") " pod="openshift-ingress/e2e-ait-001f5ecd-openshift-default-bbcdf955b-mgmbz" Jul 27 18:58:12.717135 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:12.717026 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22f63302-7966-40e7-8cfb-59201c5b99ae-workload-socket\") pod \"e2e-ait-001f5ecd-openshift-default-bbcdf955b-mgmbz\" (UID: \"22f63302-7966-40e7-8cfb-59201c5b99ae\") " pod="openshift-ingress/e2e-ait-001f5ecd-openshift-default-bbcdf955b-mgmbz" Jul 27 18:58:12.717272 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:12.717140 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22f63302-7966-40e7-8cfb-59201c5b99ae-credential-socket\") pod \"e2e-ait-001f5ecd-openshift-default-bbcdf955b-mgmbz\" (UID: \"22f63302-7966-40e7-8cfb-59201c5b99ae\") " pod="openshift-ingress/e2e-ait-001f5ecd-openshift-default-bbcdf955b-mgmbz" Jul 27 18:58:12.717338 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:12.717311 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22f63302-7966-40e7-8cfb-59201c5b99ae-istio-data\") pod \"e2e-ait-001f5ecd-openshift-default-bbcdf955b-mgmbz\" (UID: \"22f63302-7966-40e7-8cfb-59201c5b99ae\") " pod="openshift-ingress/e2e-ait-001f5ecd-openshift-default-bbcdf955b-mgmbz" Jul 27 18:58:12.717387 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:12.717358 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22f63302-7966-40e7-8cfb-59201c5b99ae-istiod-ca-cert\") pod \"e2e-ait-001f5ecd-openshift-default-bbcdf955b-mgmbz\" (UID: \"22f63302-7966-40e7-8cfb-59201c5b99ae\") " pod="openshift-ingress/e2e-ait-001f5ecd-openshift-default-bbcdf955b-mgmbz" Jul 27 18:58:12.717548 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:12.717524 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22f63302-7966-40e7-8cfb-59201c5b99ae-workload-certs\") pod \"e2e-ait-001f5ecd-openshift-default-bbcdf955b-mgmbz\" (UID: \"22f63302-7966-40e7-8cfb-59201c5b99ae\") " pod="openshift-ingress/e2e-ait-001f5ecd-openshift-default-bbcdf955b-mgmbz" Jul 27 18:58:12.719211 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:12.719192 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22f63302-7966-40e7-8cfb-59201c5b99ae-istio-envoy\") pod \"e2e-ait-001f5ecd-openshift-default-bbcdf955b-mgmbz\" (UID: \"22f63302-7966-40e7-8cfb-59201c5b99ae\") " pod="openshift-ingress/e2e-ait-001f5ecd-openshift-default-bbcdf955b-mgmbz" Jul 27 18:58:12.719574 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:12.719556 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22f63302-7966-40e7-8cfb-59201c5b99ae-istio-podinfo\") pod \"e2e-ait-001f5ecd-openshift-default-bbcdf955b-mgmbz\" (UID: \"22f63302-7966-40e7-8cfb-59201c5b99ae\") " pod="openshift-ingress/e2e-ait-001f5ecd-openshift-default-bbcdf955b-mgmbz" Jul 27 18:58:12.727167 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:12.727145 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22f63302-7966-40e7-8cfb-59201c5b99ae-istio-token\") pod \"e2e-ait-001f5ecd-openshift-default-bbcdf955b-mgmbz\" (UID: \"22f63302-7966-40e7-8cfb-59201c5b99ae\") " pod="openshift-ingress/e2e-ait-001f5ecd-openshift-default-bbcdf955b-mgmbz" Jul 27 18:58:12.727456 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:12.727424 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ppx6\" (UniqueName: \"kubernetes.io/projected/22f63302-7966-40e7-8cfb-59201c5b99ae-kube-api-access-2ppx6\") pod \"e2e-ait-001f5ecd-openshift-default-bbcdf955b-mgmbz\" (UID: \"22f63302-7966-40e7-8cfb-59201c5b99ae\") " pod="openshift-ingress/e2e-ait-001f5ecd-openshift-default-bbcdf955b-mgmbz" Jul 27 18:58:12.796043 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:12.796012 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-001f5ecd-openshift-default-bbcdf955b-mgmbz" Jul 27 18:58:12.941795 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:12.941772 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-001f5ecd-openshift-default-bbcdf955b-mgmbz"] Jul 27 18:58:12.943464 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:58:12.943434 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22f63302_7966_40e7_8cfb_59201c5b99ae.slice/crio-89db7302d608e7b8c257da6c0f985758390a11071ff470a596fda85109c18dcd WatchSource:0}: Error finding container 89db7302d608e7b8c257da6c0f985758390a11071ff470a596fda85109c18dcd: Status 404 returned error can't find the container with id 89db7302d608e7b8c257da6c0f985758390a11071ff470a596fda85109c18dcd Jul 27 18:58:12.945717 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:12.945684 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 27 18:58:12.945841 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:12.945753 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 27 18:58:12.945841 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:12.945781 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 27 18:58:12.961499 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:12.961473 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:58:13.144999 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:13.144893 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-001f5ecd-openshift-default-bbcdf955b-mgmbz" event={"ID":"22f63302-7966-40e7-8cfb-59201c5b99ae","Type":"ContainerStarted","Data":"6ab3537f3b5048b8b84e2b048bb16c20a1e7c621b020d1d5a01309bc64b13efa"} Jul 27 18:58:13.144999 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:13.144960 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-001f5ecd-openshift-default-bbcdf955b-mgmbz" event={"ID":"22f63302-7966-40e7-8cfb-59201c5b99ae","Type":"ContainerStarted","Data":"89db7302d608e7b8c257da6c0f985758390a11071ff470a596fda85109c18dcd"} Jul 27 18:58:13.168991 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:13.168925 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-001f5ecd-openshift-default-bbcdf955b-mgmbz" podStartSLOduration=1.168901822 podStartE2EDuration="1.168901822s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:58:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:58:13.168554982 +0000 UTC m=+1768.230244014" watchObservedRunningTime="2026-07-27 18:58:13.168901822 +0000 UTC m=+1768.230590853" Jul 27 18:58:13.797139 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:13.797105 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-001f5ecd-openshift-default-bbcdf955b-mgmbz" Jul 27 18:58:14.632633 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:14.632596 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:58:14.732028 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:14.731994 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:58:14.802292 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:14.802260 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-001f5ecd-openshift-default-bbcdf955b-mgmbz" Jul 27 18:58:14.850712 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:14.850673 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:58:15.155763 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:15.155732 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-001f5ecd-openshift-default-bbcdf955b-mgmbz" Jul 27 18:58:15.156630 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:15.156612 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-001f5ecd-openshift-default-bbcdf955b-mgmbz" Jul 27 18:58:15.164202 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:15.164180 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:58:15.468625 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:15.468521 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:58:15.565997 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:15.565959 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:58:15.591655 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:15.591616 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-001f5ecd-78fd99558d-2pgdq"] Jul 27 18:58:15.595187 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:15.595167 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-001f5ecd-78fd99558d-2pgdq" Jul 27 18:58:15.598346 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:15.598325 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-001f5ecd\"" Jul 27 18:58:15.606776 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:15.606754 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-001f5ecd-78fd99558d-2pgdq"] Jul 27 18:58:15.650086 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:15.650039 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-001f5ecd-6948f7748d-67m67"] Jul 27 18:58:15.653631 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:15.653612 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-001f5ecd-6948f7748d-67m67" Jul 27 18:58:15.656612 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:15.656594 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-001f5ecd-dockercfg-qdk6g\"" Jul 27 18:58:15.665382 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:15.665357 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-001f5ecd-6948f7748d-67m67"] Jul 27 18:58:15.746630 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:15.746532 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09dd3ba6-e46a-41ff-93ee-fe7700b11640-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-001f5ecd-6948f7748d-67m67\" (UID: \"09dd3ba6-e46a-41ff-93ee-fe7700b11640\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-001f5ecd-6948f7748d-67m67" Jul 27 18:58:15.746802 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:15.746640 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnfkz\" (UniqueName: \"kubernetes.io/projected/b34cf2e5-ef5f-44a7-b081-e383e168e28c-kube-api-access-pnfkz\") pod \"payload-processing-e2e-ait-001f5ecd-78fd99558d-2pgdq\" (UID: \"b34cf2e5-ef5f-44a7-b081-e383e168e28c\") " pod="openshift-ingress/payload-processing-e2e-ait-001f5ecd-78fd99558d-2pgdq" Jul 27 18:58:15.746802 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:15.746669 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2ntw\" (UniqueName: \"kubernetes.io/projected/09dd3ba6-e46a-41ff-93ee-fe7700b11640-kube-api-access-r2ntw\") pod \"payload-pre-processing-e2e-ait-001f5ecd-6948f7748d-67m67\" (UID: \"09dd3ba6-e46a-41ff-93ee-fe7700b11640\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-001f5ecd-6948f7748d-67m67" Jul 27 18:58:15.746802 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:15.746705 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b34cf2e5-ef5f-44a7-b081-e383e168e28c-plugins-config-volume\") pod \"payload-processing-e2e-ait-001f5ecd-78fd99558d-2pgdq\" (UID: \"b34cf2e5-ef5f-44a7-b081-e383e168e28c\") " pod="openshift-ingress/payload-processing-e2e-ait-001f5ecd-78fd99558d-2pgdq" Jul 27 18:58:15.848161 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:15.848122 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnfkz\" (UniqueName: \"kubernetes.io/projected/b34cf2e5-ef5f-44a7-b081-e383e168e28c-kube-api-access-pnfkz\") pod \"payload-processing-e2e-ait-001f5ecd-78fd99558d-2pgdq\" (UID: \"b34cf2e5-ef5f-44a7-b081-e383e168e28c\") " pod="openshift-ingress/payload-processing-e2e-ait-001f5ecd-78fd99558d-2pgdq" Jul 27 18:58:15.848161 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:15.848163 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2ntw\" (UniqueName: \"kubernetes.io/projected/09dd3ba6-e46a-41ff-93ee-fe7700b11640-kube-api-access-r2ntw\") pod \"payload-pre-processing-e2e-ait-001f5ecd-6948f7748d-67m67\" (UID: \"09dd3ba6-e46a-41ff-93ee-fe7700b11640\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-001f5ecd-6948f7748d-67m67" Jul 27 18:58:15.848746 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:15.848203 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b34cf2e5-ef5f-44a7-b081-e383e168e28c-plugins-config-volume\") pod \"payload-processing-e2e-ait-001f5ecd-78fd99558d-2pgdq\" (UID: \"b34cf2e5-ef5f-44a7-b081-e383e168e28c\") " pod="openshift-ingress/payload-processing-e2e-ait-001f5ecd-78fd99558d-2pgdq" Jul 27 18:58:15.848746 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:15.848307 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09dd3ba6-e46a-41ff-93ee-fe7700b11640-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-001f5ecd-6948f7748d-67m67\" (UID: \"09dd3ba6-e46a-41ff-93ee-fe7700b11640\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-001f5ecd-6948f7748d-67m67" Jul 27 18:58:15.849009 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:15.848986 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b34cf2e5-ef5f-44a7-b081-e383e168e28c-plugins-config-volume\") pod \"payload-processing-e2e-ait-001f5ecd-78fd99558d-2pgdq\" (UID: \"b34cf2e5-ef5f-44a7-b081-e383e168e28c\") " pod="openshift-ingress/payload-processing-e2e-ait-001f5ecd-78fd99558d-2pgdq" Jul 27 18:58:15.849160 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:15.849139 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09dd3ba6-e46a-41ff-93ee-fe7700b11640-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-001f5ecd-6948f7748d-67m67\" (UID: \"09dd3ba6-e46a-41ff-93ee-fe7700b11640\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-001f5ecd-6948f7748d-67m67" Jul 27 18:58:15.857886 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:15.857852 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2ntw\" (UniqueName: \"kubernetes.io/projected/09dd3ba6-e46a-41ff-93ee-fe7700b11640-kube-api-access-r2ntw\") pod \"payload-pre-processing-e2e-ait-001f5ecd-6948f7748d-67m67\" (UID: \"09dd3ba6-e46a-41ff-93ee-fe7700b11640\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-001f5ecd-6948f7748d-67m67" Jul 27 18:58:15.857989 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:15.857981 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnfkz\" (UniqueName: \"kubernetes.io/projected/b34cf2e5-ef5f-44a7-b081-e383e168e28c-kube-api-access-pnfkz\") pod \"payload-processing-e2e-ait-001f5ecd-78fd99558d-2pgdq\" (UID: \"b34cf2e5-ef5f-44a7-b081-e383e168e28c\") " pod="openshift-ingress/payload-processing-e2e-ait-001f5ecd-78fd99558d-2pgdq" Jul 27 18:58:15.907895 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:15.907872 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-001f5ecd-78fd99558d-2pgdq" Jul 27 18:58:15.964532 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:15.964503 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-001f5ecd-6948f7748d-67m67" Jul 27 18:58:16.073360 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:16.073141 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-001f5ecd-78fd99558d-2pgdq"] Jul 27 18:58:16.077752 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:58:16.077723 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb34cf2e5_ef5f_44a7_b081_e383e168e28c.slice/crio-ade915b2dd6ea0c10691852f3581a70018edc1dd931aa6f9d517ee063bd35733 WatchSource:0}: Error finding container ade915b2dd6ea0c10691852f3581a70018edc1dd931aa6f9d517ee063bd35733: Status 404 returned error can't find the container with id ade915b2dd6ea0c10691852f3581a70018edc1dd931aa6f9d517ee063bd35733 Jul 27 18:58:16.134022 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:16.133987 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-001f5ecd-6948f7748d-67m67"] Jul 27 18:58:16.134758 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:58:16.134732 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09dd3ba6_e46a_41ff_93ee_fe7700b11640.slice/crio-0555f38916f85a315678141dd68d096e7e747cd98a4998389b694e3bc608c5c0 WatchSource:0}: Error finding container 0555f38916f85a315678141dd68d096e7e747cd98a4998389b694e3bc608c5c0: Status 404 returned error can't find the container with id 0555f38916f85a315678141dd68d096e7e747cd98a4998389b694e3bc608c5c0 Jul 27 18:58:16.161771 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:16.161743 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-001f5ecd-6948f7748d-67m67" event={"ID":"09dd3ba6-e46a-41ff-93ee-fe7700b11640","Type":"ContainerStarted","Data":"0555f38916f85a315678141dd68d096e7e747cd98a4998389b694e3bc608c5c0"} Jul 27 18:58:16.163292 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:16.163264 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-001f5ecd-78fd99558d-2pgdq" event={"ID":"b34cf2e5-ef5f-44a7-b081-e383e168e28c","Type":"ContainerStarted","Data":"cd6fbc817daff4b5254beedbb6606956b8b60d6d3aab546a0a1f2046b3f326ca"} Jul 27 18:58:16.163399 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:16.163310 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-001f5ecd-78fd99558d-2pgdq" event={"ID":"b34cf2e5-ef5f-44a7-b081-e383e168e28c","Type":"ContainerStarted","Data":"ade915b2dd6ea0c10691852f3581a70018edc1dd931aa6f9d517ee063bd35733"} Jul 27 18:58:16.163484 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:16.163469 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-001f5ecd-78fd99558d-2pgdq" Jul 27 18:58:16.188686 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:16.188644 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-001f5ecd-78fd99558d-2pgdq" podStartSLOduration=1.188624858 podStartE2EDuration="1.188624858s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:58:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:58:16.18733035 +0000 UTC m=+1771.249019380" watchObservedRunningTime="2026-07-27 18:58:16.188624858 +0000 UTC m=+1771.250313890" Jul 27 18:58:17.170898 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:17.170859 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-001f5ecd-6948f7748d-67m67" event={"ID":"09dd3ba6-e46a-41ff-93ee-fe7700b11640","Type":"ContainerStarted","Data":"077389d89a2de13a66221d2428e63095da2f64a66fdb4b684c445b325b735f3c"} Jul 27 18:58:17.171354 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:17.170926 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-001f5ecd-6948f7748d-67m67" Jul 27 18:58:17.197417 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:17.197365 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-001f5ecd-6948f7748d-67m67" podStartSLOduration=2.197348656 podStartE2EDuration="2.197348656s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:58:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:58:17.196680904 +0000 UTC m=+1772.258369947" watchObservedRunningTime="2026-07-27 18:58:17.197348656 +0000 UTC m=+1772.259037748" Jul 27 18:58:27.173382 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:27.173344 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-001f5ecd-78fd99558d-2pgdq" Jul 27 18:58:28.178809 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:28.178725 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-001f5ecd-6948f7748d-67m67" Jul 27 18:58:30.667094 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:30.667058 2579 patch_prober.go:28] interesting pod/e2e-ait-001f5ecd-openshift-default-bbcdf955b-mgmbz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": read tcp 10.132.0.2:35322->10.132.0.55:15021: read: connection reset by peer" start-of-body= Jul 27 18:58:30.667521 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:30.667173 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-001f5ecd-openshift-default-bbcdf955b-mgmbz" podUID="22f63302-7966-40e7-8cfb-59201c5b99ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": read tcp 10.132.0.2:35322->10.132.0.55:15021: read: connection reset by peer" Jul 27 18:58:31.253686 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:31.253654 2579 generic.go:356] "Generic (PLEG): container finished" podID="22f63302-7966-40e7-8cfb-59201c5b99ae" containerID="6ab3537f3b5048b8b84e2b048bb16c20a1e7c621b020d1d5a01309bc64b13efa" exitCode=0 Jul 27 18:58:31.253867 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:31.253742 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-001f5ecd-openshift-default-bbcdf955b-mgmbz" event={"ID":"22f63302-7966-40e7-8cfb-59201c5b99ae","Type":"ContainerDied","Data":"6ab3537f3b5048b8b84e2b048bb16c20a1e7c621b020d1d5a01309bc64b13efa"} Jul 27 18:58:31.254159 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:31.254145 2579 scope.go:117] "RemoveContainer" containerID="6ab3537f3b5048b8b84e2b048bb16c20a1e7c621b020d1d5a01309bc64b13efa" Jul 27 18:58:31.255186 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:31.255145 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 27 18:58:31.255315 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:31.255266 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 27 18:58:31.255382 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:31.255328 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 27 18:58:32.260594 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:32.260558 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-001f5ecd-openshift-default-bbcdf955b-mgmbz" event={"ID":"22f63302-7966-40e7-8cfb-59201c5b99ae","Type":"ContainerStarted","Data":"c05a0216d7d2f322ce6a437ec1f9511b701c26172d18cb2dd012768a03c0909e"} Jul 27 18:58:32.796317 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:32.796276 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-001f5ecd-openshift-default-bbcdf955b-mgmbz" Jul 27 18:58:32.796640 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:32.796617 2579 patch_prober.go:28] interesting pod/e2e-ait-001f5ecd-openshift-default-bbcdf955b-mgmbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Jul 27 18:58:32.796687 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:32.796665 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-001f5ecd-openshift-default-bbcdf955b-mgmbz" podUID="22f63302-7966-40e7-8cfb-59201c5b99ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Jul 27 18:58:33.797288 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:33.797250 2579 patch_prober.go:28] interesting pod/e2e-ait-001f5ecd-openshift-default-bbcdf955b-mgmbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Jul 27 18:58:33.797693 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:33.797307 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-001f5ecd-openshift-default-bbcdf955b-mgmbz" podUID="22f63302-7966-40e7-8cfb-59201c5b99ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Jul 27 18:58:34.796966 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:34.796931 2579 patch_prober.go:28] interesting pod/e2e-ait-001f5ecd-openshift-default-bbcdf955b-mgmbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Jul 27 18:58:34.797146 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:34.796992 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-001f5ecd-openshift-default-bbcdf955b-mgmbz" podUID="22f63302-7966-40e7-8cfb-59201c5b99ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Jul 27 18:58:35.797430 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:35.797345 2579 patch_prober.go:28] interesting pod/e2e-ait-001f5ecd-openshift-default-bbcdf955b-mgmbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Jul 27 18:58:35.797430 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:35.797407 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-001f5ecd-openshift-default-bbcdf955b-mgmbz" podUID="22f63302-7966-40e7-8cfb-59201c5b99ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Jul 27 18:58:36.800018 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:36.799981 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-001f5ecd-openshift-default-bbcdf955b-mgmbz" Jul 27 18:58:36.800510 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:36.800325 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-001f5ecd-openshift-default-bbcdf955b-mgmbz" Jul 27 18:58:36.800980 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:36.800960 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-001f5ecd-openshift-default-bbcdf955b-mgmbz" Jul 27 18:58:45.527562 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:45.527527 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-mv4hb_715b2084-d79e-4cbf-a057-f2595b4e6b9b/console-operator/1.log" Jul 27 18:58:45.531797 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:45.531776 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-mv4hb_715b2084-d79e-4cbf-a057-f2595b4e6b9b/console-operator/1.log" Jul 27 18:58:50.855976 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:50.855944 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:58:50.897295 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:50.897258 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-001f5ecd-78fd99558d-2pgdq"] Jul 27 18:58:50.897569 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:50.897517 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-001f5ecd-78fd99558d-2pgdq" podUID="b34cf2e5-ef5f-44a7-b081-e383e168e28c" containerName="payload-processing" containerID="cri-o://cd6fbc817daff4b5254beedbb6606956b8b60d6d3aab546a0a1f2046b3f326ca" gracePeriod=30 Jul 27 18:58:50.911520 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:50.911494 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-001f5ecd-6948f7748d-67m67"] Jul 27 18:58:50.911716 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:50.911696 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-001f5ecd-6948f7748d-67m67" podUID="09dd3ba6-e46a-41ff-93ee-fe7700b11640" containerName="payload-pre-processing" containerID="cri-o://077389d89a2de13a66221d2428e63095da2f64a66fdb4b684c445b325b735f3c" gracePeriod=30 Jul 27 18:58:50.969322 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:50.969298 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:58:51.054757 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:51.054725 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:58:51.200483 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:51.200456 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-001f5ecd-78fd99558d-2pgdq" Jul 27 18:58:51.203213 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:51.203197 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-001f5ecd-6948f7748d-67m67" Jul 27 18:58:51.297684 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:51.297652 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pnfkz\" (UniqueName: \"kubernetes.io/projected/b34cf2e5-ef5f-44a7-b081-e383e168e28c-kube-api-access-pnfkz\") pod \"b34cf2e5-ef5f-44a7-b081-e383e168e28c\" (UID: \"b34cf2e5-ef5f-44a7-b081-e383e168e28c\") " Jul 27 18:58:51.297876 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:51.297698 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r2ntw\" (UniqueName: \"kubernetes.io/projected/09dd3ba6-e46a-41ff-93ee-fe7700b11640-kube-api-access-r2ntw\") pod \"09dd3ba6-e46a-41ff-93ee-fe7700b11640\" (UID: \"09dd3ba6-e46a-41ff-93ee-fe7700b11640\") " Jul 27 18:58:51.297876 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:51.297741 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b34cf2e5-ef5f-44a7-b081-e383e168e28c-plugins-config-volume\") pod \"b34cf2e5-ef5f-44a7-b081-e383e168e28c\" (UID: \"b34cf2e5-ef5f-44a7-b081-e383e168e28c\") " Jul 27 18:58:51.297876 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:51.297802 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09dd3ba6-e46a-41ff-93ee-fe7700b11640-plugins-config-volume\") pod \"09dd3ba6-e46a-41ff-93ee-fe7700b11640\" (UID: \"09dd3ba6-e46a-41ff-93ee-fe7700b11640\") " Jul 27 18:58:51.298135 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:51.298098 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b34cf2e5-ef5f-44a7-b081-e383e168e28c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b34cf2e5-ef5f-44a7-b081-e383e168e28c" (UID: "b34cf2e5-ef5f-44a7-b081-e383e168e28c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 18:58:51.298305 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:51.298191 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09dd3ba6-e46a-41ff-93ee-fe7700b11640-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "09dd3ba6-e46a-41ff-93ee-fe7700b11640" (UID: "09dd3ba6-e46a-41ff-93ee-fe7700b11640"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 18:58:51.299945 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:51.299922 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b34cf2e5-ef5f-44a7-b081-e383e168e28c-kube-api-access-pnfkz" (OuterVolumeSpecName: "kube-api-access-pnfkz") pod "b34cf2e5-ef5f-44a7-b081-e383e168e28c" (UID: "b34cf2e5-ef5f-44a7-b081-e383e168e28c"). InnerVolumeSpecName "kube-api-access-pnfkz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 18:58:51.300099 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:51.300084 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09dd3ba6-e46a-41ff-93ee-fe7700b11640-kube-api-access-r2ntw" (OuterVolumeSpecName: "kube-api-access-r2ntw") pod "09dd3ba6-e46a-41ff-93ee-fe7700b11640" (UID: "09dd3ba6-e46a-41ff-93ee-fe7700b11640"). InnerVolumeSpecName "kube-api-access-r2ntw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 18:58:51.358927 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:51.358896 2579 generic.go:356] "Generic (PLEG): container finished" podID="b34cf2e5-ef5f-44a7-b081-e383e168e28c" containerID="cd6fbc817daff4b5254beedbb6606956b8b60d6d3aab546a0a1f2046b3f326ca" exitCode=0 Jul 27 18:58:51.359071 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:51.358948 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-001f5ecd-78fd99558d-2pgdq" Jul 27 18:58:51.359071 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:51.358975 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-001f5ecd-78fd99558d-2pgdq" event={"ID":"b34cf2e5-ef5f-44a7-b081-e383e168e28c","Type":"ContainerDied","Data":"cd6fbc817daff4b5254beedbb6606956b8b60d6d3aab546a0a1f2046b3f326ca"} Jul 27 18:58:51.359071 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:51.359065 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-001f5ecd-78fd99558d-2pgdq" event={"ID":"b34cf2e5-ef5f-44a7-b081-e383e168e28c","Type":"ContainerDied","Data":"ade915b2dd6ea0c10691852f3581a70018edc1dd931aa6f9d517ee063bd35733"} Jul 27 18:58:51.359264 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:51.359092 2579 scope.go:117] "RemoveContainer" containerID="cd6fbc817daff4b5254beedbb6606956b8b60d6d3aab546a0a1f2046b3f326ca" Jul 27 18:58:51.360547 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:51.360516 2579 generic.go:356] "Generic (PLEG): container finished" podID="09dd3ba6-e46a-41ff-93ee-fe7700b11640" containerID="077389d89a2de13a66221d2428e63095da2f64a66fdb4b684c445b325b735f3c" exitCode=0 Jul 27 18:58:51.360643 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:51.360570 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-001f5ecd-6948f7748d-67m67" Jul 27 18:58:51.360712 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:51.360564 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-001f5ecd-6948f7748d-67m67" event={"ID":"09dd3ba6-e46a-41ff-93ee-fe7700b11640","Type":"ContainerDied","Data":"077389d89a2de13a66221d2428e63095da2f64a66fdb4b684c445b325b735f3c"} Jul 27 18:58:51.360773 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:51.360710 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-001f5ecd-6948f7748d-67m67" event={"ID":"09dd3ba6-e46a-41ff-93ee-fe7700b11640","Type":"ContainerDied","Data":"0555f38916f85a315678141dd68d096e7e747cd98a4998389b694e3bc608c5c0"} Jul 27 18:58:51.369454 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:51.369434 2579 scope.go:117] "RemoveContainer" containerID="cd6fbc817daff4b5254beedbb6606956b8b60d6d3aab546a0a1f2046b3f326ca" Jul 27 18:58:51.369817 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:58:51.369758 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd6fbc817daff4b5254beedbb6606956b8b60d6d3aab546a0a1f2046b3f326ca\": container with ID starting with cd6fbc817daff4b5254beedbb6606956b8b60d6d3aab546a0a1f2046b3f326ca not found: ID does not exist" containerID="cd6fbc817daff4b5254beedbb6606956b8b60d6d3aab546a0a1f2046b3f326ca" Jul 27 18:58:51.369901 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:51.369826 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd6fbc817daff4b5254beedbb6606956b8b60d6d3aab546a0a1f2046b3f326ca"} err="failed to get container status \"cd6fbc817daff4b5254beedbb6606956b8b60d6d3aab546a0a1f2046b3f326ca\": rpc error: code = NotFound desc = could not find container \"cd6fbc817daff4b5254beedbb6606956b8b60d6d3aab546a0a1f2046b3f326ca\": container with ID starting with cd6fbc817daff4b5254beedbb6606956b8b60d6d3aab546a0a1f2046b3f326ca not found: ID does not exist" Jul 27 18:58:51.369901 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:51.369843 2579 scope.go:117] "RemoveContainer" containerID="077389d89a2de13a66221d2428e63095da2f64a66fdb4b684c445b325b735f3c" Jul 27 18:58:51.378194 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:51.378176 2579 scope.go:117] "RemoveContainer" containerID="077389d89a2de13a66221d2428e63095da2f64a66fdb4b684c445b325b735f3c" Jul 27 18:58:51.378474 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:58:51.378457 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"077389d89a2de13a66221d2428e63095da2f64a66fdb4b684c445b325b735f3c\": container with ID starting with 077389d89a2de13a66221d2428e63095da2f64a66fdb4b684c445b325b735f3c not found: ID does not exist" containerID="077389d89a2de13a66221d2428e63095da2f64a66fdb4b684c445b325b735f3c" Jul 27 18:58:51.378531 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:51.378481 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"077389d89a2de13a66221d2428e63095da2f64a66fdb4b684c445b325b735f3c"} err="failed to get container status \"077389d89a2de13a66221d2428e63095da2f64a66fdb4b684c445b325b735f3c\": rpc error: code = NotFound desc = could not find container \"077389d89a2de13a66221d2428e63095da2f64a66fdb4b684c445b325b735f3c\": container with ID starting with 077389d89a2de13a66221d2428e63095da2f64a66fdb4b684c445b325b735f3c not found: ID does not exist" Jul 27 18:58:51.389023 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:51.389002 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-001f5ecd-6948f7748d-67m67"] Jul 27 18:58:51.390893 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:51.390874 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-001f5ecd-6948f7748d-67m67"] Jul 27 18:58:51.398814 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:51.398795 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09dd3ba6-e46a-41ff-93ee-fe7700b11640-plugins-config-volume\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 18:58:51.398814 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:51.398815 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pnfkz\" (UniqueName: \"kubernetes.io/projected/b34cf2e5-ef5f-44a7-b081-e383e168e28c-kube-api-access-pnfkz\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 18:58:51.398933 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:51.398825 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2ntw\" (UniqueName: \"kubernetes.io/projected/09dd3ba6-e46a-41ff-93ee-fe7700b11640-kube-api-access-r2ntw\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 18:58:51.398933 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:51.398834 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b34cf2e5-ef5f-44a7-b081-e383e168e28c-plugins-config-volume\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 18:58:51.408096 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:51.408076 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-001f5ecd-78fd99558d-2pgdq"] Jul 27 18:58:51.413642 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:51.413623 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-001f5ecd-78fd99558d-2pgdq"] Jul 27 18:58:51.493913 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:51.493874 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="09dd3ba6-e46a-41ff-93ee-fe7700b11640" path="/var/lib/kubelet/pods/09dd3ba6-e46a-41ff-93ee-fe7700b11640/volumes" Jul 27 18:58:51.494376 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:51.494355 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b34cf2e5-ef5f-44a7-b081-e383e168e28c" path="/var/lib/kubelet/pods/b34cf2e5-ef5f-44a7-b081-e383e168e28c/volumes" Jul 27 18:58:55.337797 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:55.337755 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:58:55.566364 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:55.566328 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-001f5ecd-openshift-default-bbcdf955b-mgmbz"] Jul 27 18:58:55.566570 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:55.566543 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-001f5ecd-openshift-default-bbcdf955b-mgmbz" podUID="22f63302-7966-40e7-8cfb-59201c5b99ae" containerName="istio-proxy" containerID="cri-o://c05a0216d7d2f322ce6a437ec1f9511b701c26172d18cb2dd012768a03c0909e" gracePeriod=30 Jul 27 18:58:55.577306 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:55.577278 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:58:55.782644 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:55.782609 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:58:56.079986 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:56.079881 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:58:56.195857 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:56.195810 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:58:56.310608 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:56.310576 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:58:56.596273 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:56.596222 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-09eed4cd-gw-openshift-default-7757b48cf6-5wdrs"] Jul 27 18:58:56.596749 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:56.596631 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09dd3ba6-e46a-41ff-93ee-fe7700b11640" containerName="payload-pre-processing" Jul 27 18:58:56.596749 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:56.596642 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="09dd3ba6-e46a-41ff-93ee-fe7700b11640" containerName="payload-pre-processing" Jul 27 18:58:56.596749 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:56.596658 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b34cf2e5-ef5f-44a7-b081-e383e168e28c" containerName="payload-processing" Jul 27 18:58:56.596749 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:56.596664 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="b34cf2e5-ef5f-44a7-b081-e383e168e28c" containerName="payload-processing" Jul 27 18:58:56.596749 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:56.596748 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="09dd3ba6-e46a-41ff-93ee-fe7700b11640" containerName="payload-pre-processing" Jul 27 18:58:56.597005 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:56.596759 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="b34cf2e5-ef5f-44a7-b081-e383e168e28c" containerName="payload-processing" Jul 27 18:58:56.601391 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:56.601371 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-09eed4cd-gw-openshift-default-7757b48cf6-5wdrs" Jul 27 18:58:56.604388 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:56.604367 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-09eed4cd-gw-openshift-default-dockercfg-49m6l\"" Jul 27 18:58:56.612775 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:56.612754 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-09eed4cd-gw-openshift-default-7757b48cf6-5wdrs"] Jul 27 18:58:56.631964 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:56.631913 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:58:56.722759 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:56.722722 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:58:56.746996 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:56.746962 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a4a4a51-fa4b-4527-a02a-240c945d17c3-istio-envoy\") pod \"e2e-migrate-09eed4cd-gw-openshift-default-7757b48cf6-5wdrs\" (UID: \"7a4a4a51-fa4b-4527-a02a-240c945d17c3\") " pod="openshift-ingress/e2e-migrate-09eed4cd-gw-openshift-default-7757b48cf6-5wdrs" Jul 27 18:58:56.746996 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:56.747001 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a4a4a51-fa4b-4527-a02a-240c945d17c3-workload-socket\") pod \"e2e-migrate-09eed4cd-gw-openshift-default-7757b48cf6-5wdrs\" (UID: \"7a4a4a51-fa4b-4527-a02a-240c945d17c3\") " pod="openshift-ingress/e2e-migrate-09eed4cd-gw-openshift-default-7757b48cf6-5wdrs" Jul 27 18:58:56.747313 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:56.747027 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a4a4a51-fa4b-4527-a02a-240c945d17c3-istio-token\") pod \"e2e-migrate-09eed4cd-gw-openshift-default-7757b48cf6-5wdrs\" (UID: \"7a4a4a51-fa4b-4527-a02a-240c945d17c3\") " pod="openshift-ingress/e2e-migrate-09eed4cd-gw-openshift-default-7757b48cf6-5wdrs" Jul 27 18:58:56.747313 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:56.747187 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4l789\" (UniqueName: \"kubernetes.io/projected/7a4a4a51-fa4b-4527-a02a-240c945d17c3-kube-api-access-4l789\") pod \"e2e-migrate-09eed4cd-gw-openshift-default-7757b48cf6-5wdrs\" (UID: \"7a4a4a51-fa4b-4527-a02a-240c945d17c3\") " pod="openshift-ingress/e2e-migrate-09eed4cd-gw-openshift-default-7757b48cf6-5wdrs" Jul 27 18:58:56.747446 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:56.747321 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a4a4a51-fa4b-4527-a02a-240c945d17c3-istiod-ca-cert\") pod \"e2e-migrate-09eed4cd-gw-openshift-default-7757b48cf6-5wdrs\" (UID: \"7a4a4a51-fa4b-4527-a02a-240c945d17c3\") " pod="openshift-ingress/e2e-migrate-09eed4cd-gw-openshift-default-7757b48cf6-5wdrs" Jul 27 18:58:56.747446 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:56.747383 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a4a4a51-fa4b-4527-a02a-240c945d17c3-istio-data\") pod \"e2e-migrate-09eed4cd-gw-openshift-default-7757b48cf6-5wdrs\" (UID: \"7a4a4a51-fa4b-4527-a02a-240c945d17c3\") " pod="openshift-ingress/e2e-migrate-09eed4cd-gw-openshift-default-7757b48cf6-5wdrs" Jul 27 18:58:56.747446 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:56.747423 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a4a4a51-fa4b-4527-a02a-240c945d17c3-istio-podinfo\") pod \"e2e-migrate-09eed4cd-gw-openshift-default-7757b48cf6-5wdrs\" (UID: \"7a4a4a51-fa4b-4527-a02a-240c945d17c3\") " pod="openshift-ingress/e2e-migrate-09eed4cd-gw-openshift-default-7757b48cf6-5wdrs" Jul 27 18:58:56.747595 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:56.747455 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a4a4a51-fa4b-4527-a02a-240c945d17c3-credential-socket\") pod \"e2e-migrate-09eed4cd-gw-openshift-default-7757b48cf6-5wdrs\" (UID: \"7a4a4a51-fa4b-4527-a02a-240c945d17c3\") " pod="openshift-ingress/e2e-migrate-09eed4cd-gw-openshift-default-7757b48cf6-5wdrs" Jul 27 18:58:56.747595 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:56.747517 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a4a4a51-fa4b-4527-a02a-240c945d17c3-workload-certs\") pod \"e2e-migrate-09eed4cd-gw-openshift-default-7757b48cf6-5wdrs\" (UID: \"7a4a4a51-fa4b-4527-a02a-240c945d17c3\") " pod="openshift-ingress/e2e-migrate-09eed4cd-gw-openshift-default-7757b48cf6-5wdrs" Jul 27 18:58:56.848744 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:56.848652 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4l789\" (UniqueName: \"kubernetes.io/projected/7a4a4a51-fa4b-4527-a02a-240c945d17c3-kube-api-access-4l789\") pod \"e2e-migrate-09eed4cd-gw-openshift-default-7757b48cf6-5wdrs\" (UID: \"7a4a4a51-fa4b-4527-a02a-240c945d17c3\") " pod="openshift-ingress/e2e-migrate-09eed4cd-gw-openshift-default-7757b48cf6-5wdrs" Jul 27 18:58:56.848744 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:56.848739 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a4a4a51-fa4b-4527-a02a-240c945d17c3-istiod-ca-cert\") pod \"e2e-migrate-09eed4cd-gw-openshift-default-7757b48cf6-5wdrs\" (UID: \"7a4a4a51-fa4b-4527-a02a-240c945d17c3\") " pod="openshift-ingress/e2e-migrate-09eed4cd-gw-openshift-default-7757b48cf6-5wdrs" Jul 27 18:58:56.848982 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:56.848767 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a4a4a51-fa4b-4527-a02a-240c945d17c3-istio-data\") pod \"e2e-migrate-09eed4cd-gw-openshift-default-7757b48cf6-5wdrs\" (UID: \"7a4a4a51-fa4b-4527-a02a-240c945d17c3\") " pod="openshift-ingress/e2e-migrate-09eed4cd-gw-openshift-default-7757b48cf6-5wdrs" Jul 27 18:58:56.848982 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:56.848791 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a4a4a51-fa4b-4527-a02a-240c945d17c3-istio-podinfo\") pod \"e2e-migrate-09eed4cd-gw-openshift-default-7757b48cf6-5wdrs\" (UID: \"7a4a4a51-fa4b-4527-a02a-240c945d17c3\") " pod="openshift-ingress/e2e-migrate-09eed4cd-gw-openshift-default-7757b48cf6-5wdrs" Jul 27 18:58:56.848982 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:56.848810 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a4a4a51-fa4b-4527-a02a-240c945d17c3-credential-socket\") pod \"e2e-migrate-09eed4cd-gw-openshift-default-7757b48cf6-5wdrs\" (UID: \"7a4a4a51-fa4b-4527-a02a-240c945d17c3\") " pod="openshift-ingress/e2e-migrate-09eed4cd-gw-openshift-default-7757b48cf6-5wdrs" Jul 27 18:58:56.848982 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:56.848947 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a4a4a51-fa4b-4527-a02a-240c945d17c3-workload-certs\") pod \"e2e-migrate-09eed4cd-gw-openshift-default-7757b48cf6-5wdrs\" (UID: \"7a4a4a51-fa4b-4527-a02a-240c945d17c3\") " pod="openshift-ingress/e2e-migrate-09eed4cd-gw-openshift-default-7757b48cf6-5wdrs" Jul 27 18:58:56.849196 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:56.849001 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a4a4a51-fa4b-4527-a02a-240c945d17c3-istio-envoy\") pod \"e2e-migrate-09eed4cd-gw-openshift-default-7757b48cf6-5wdrs\" (UID: \"7a4a4a51-fa4b-4527-a02a-240c945d17c3\") " pod="openshift-ingress/e2e-migrate-09eed4cd-gw-openshift-default-7757b48cf6-5wdrs" Jul 27 18:58:56.849196 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:56.849033 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a4a4a51-fa4b-4527-a02a-240c945d17c3-workload-socket\") pod \"e2e-migrate-09eed4cd-gw-openshift-default-7757b48cf6-5wdrs\" (UID: \"7a4a4a51-fa4b-4527-a02a-240c945d17c3\") " pod="openshift-ingress/e2e-migrate-09eed4cd-gw-openshift-default-7757b48cf6-5wdrs" Jul 27 18:58:56.849196 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:56.849063 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a4a4a51-fa4b-4527-a02a-240c945d17c3-istio-token\") pod \"e2e-migrate-09eed4cd-gw-openshift-default-7757b48cf6-5wdrs\" (UID: \"7a4a4a51-fa4b-4527-a02a-240c945d17c3\") " pod="openshift-ingress/e2e-migrate-09eed4cd-gw-openshift-default-7757b48cf6-5wdrs" Jul 27 18:58:56.849381 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:56.849364 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a4a4a51-fa4b-4527-a02a-240c945d17c3-istio-data\") pod \"e2e-migrate-09eed4cd-gw-openshift-default-7757b48cf6-5wdrs\" (UID: \"7a4a4a51-fa4b-4527-a02a-240c945d17c3\") " pod="openshift-ingress/e2e-migrate-09eed4cd-gw-openshift-default-7757b48cf6-5wdrs" Jul 27 18:58:56.849441 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:56.849364 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a4a4a51-fa4b-4527-a02a-240c945d17c3-credential-socket\") pod \"e2e-migrate-09eed4cd-gw-openshift-default-7757b48cf6-5wdrs\" (UID: \"7a4a4a51-fa4b-4527-a02a-240c945d17c3\") " pod="openshift-ingress/e2e-migrate-09eed4cd-gw-openshift-default-7757b48cf6-5wdrs" Jul 27 18:58:56.849441 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:56.849425 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a4a4a51-fa4b-4527-a02a-240c945d17c3-workload-socket\") pod \"e2e-migrate-09eed4cd-gw-openshift-default-7757b48cf6-5wdrs\" (UID: \"7a4a4a51-fa4b-4527-a02a-240c945d17c3\") " pod="openshift-ingress/e2e-migrate-09eed4cd-gw-openshift-default-7757b48cf6-5wdrs" Jul 27 18:58:56.849552 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:56.849494 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a4a4a51-fa4b-4527-a02a-240c945d17c3-workload-certs\") pod \"e2e-migrate-09eed4cd-gw-openshift-default-7757b48cf6-5wdrs\" (UID: \"7a4a4a51-fa4b-4527-a02a-240c945d17c3\") " pod="openshift-ingress/e2e-migrate-09eed4cd-gw-openshift-default-7757b48cf6-5wdrs" Jul 27 18:58:56.849615 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:56.849596 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a4a4a51-fa4b-4527-a02a-240c945d17c3-istiod-ca-cert\") pod \"e2e-migrate-09eed4cd-gw-openshift-default-7757b48cf6-5wdrs\" (UID: \"7a4a4a51-fa4b-4527-a02a-240c945d17c3\") " pod="openshift-ingress/e2e-migrate-09eed4cd-gw-openshift-default-7757b48cf6-5wdrs" Jul 27 18:58:56.851702 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:56.851680 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a4a4a51-fa4b-4527-a02a-240c945d17c3-istio-podinfo\") pod \"e2e-migrate-09eed4cd-gw-openshift-default-7757b48cf6-5wdrs\" (UID: \"7a4a4a51-fa4b-4527-a02a-240c945d17c3\") " pod="openshift-ingress/e2e-migrate-09eed4cd-gw-openshift-default-7757b48cf6-5wdrs" Jul 27 18:58:56.851895 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:56.851874 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a4a4a51-fa4b-4527-a02a-240c945d17c3-istio-envoy\") pod \"e2e-migrate-09eed4cd-gw-openshift-default-7757b48cf6-5wdrs\" (UID: \"7a4a4a51-fa4b-4527-a02a-240c945d17c3\") " pod="openshift-ingress/e2e-migrate-09eed4cd-gw-openshift-default-7757b48cf6-5wdrs" Jul 27 18:58:56.862543 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:56.862521 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a4a4a51-fa4b-4527-a02a-240c945d17c3-istio-token\") pod \"e2e-migrate-09eed4cd-gw-openshift-default-7757b48cf6-5wdrs\" (UID: \"7a4a4a51-fa4b-4527-a02a-240c945d17c3\") " pod="openshift-ingress/e2e-migrate-09eed4cd-gw-openshift-default-7757b48cf6-5wdrs" Jul 27 18:58:56.863255 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:56.863215 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4l789\" (UniqueName: \"kubernetes.io/projected/7a4a4a51-fa4b-4527-a02a-240c945d17c3-kube-api-access-4l789\") pod \"e2e-migrate-09eed4cd-gw-openshift-default-7757b48cf6-5wdrs\" (UID: \"7a4a4a51-fa4b-4527-a02a-240c945d17c3\") " pod="openshift-ingress/e2e-migrate-09eed4cd-gw-openshift-default-7757b48cf6-5wdrs" Jul 27 18:58:56.912357 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:56.912326 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-09eed4cd-gw-openshift-default-7757b48cf6-5wdrs" Jul 27 18:58:57.061505 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:57.061482 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-09eed4cd-gw-openshift-default-7757b48cf6-5wdrs"] Jul 27 18:58:57.063567 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:58:57.063541 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a4a4a51_fa4b_4527_a02a_240c945d17c3.slice/crio-0d0218ac1d739bd4561a50fc1751df2aced424c1b6ada68e95a528b5c28ea2ad WatchSource:0}: Error finding container 0d0218ac1d739bd4561a50fc1751df2aced424c1b6ada68e95a528b5c28ea2ad: Status 404 returned error can't find the container with id 0d0218ac1d739bd4561a50fc1751df2aced424c1b6ada68e95a528b5c28ea2ad Jul 27 18:58:57.065591 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:57.065565 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 27 18:58:57.065668 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:57.065623 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 27 18:58:57.065668 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:57.065651 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 27 18:58:57.396016 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:57.395923 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-09eed4cd-gw-openshift-default-7757b48cf6-5wdrs" event={"ID":"7a4a4a51-fa4b-4527-a02a-240c945d17c3","Type":"ContainerStarted","Data":"43ee28ee525910b3979adae300320fecbf4421f5806e3947834f6467e1513289"} Jul 27 18:58:57.396016 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:57.395979 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-09eed4cd-gw-openshift-default-7757b48cf6-5wdrs" event={"ID":"7a4a4a51-fa4b-4527-a02a-240c945d17c3","Type":"ContainerStarted","Data":"0d0218ac1d739bd4561a50fc1751df2aced424c1b6ada68e95a528b5c28ea2ad"} Jul 27 18:58:57.418508 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:57.418458 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-09eed4cd-gw-openshift-default-7757b48cf6-5wdrs" podStartSLOduration=1.4184419 podStartE2EDuration="1.4184419s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:58:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:58:57.416572312 +0000 UTC m=+1812.478261344" watchObservedRunningTime="2026-07-27 18:58:57.4184419 +0000 UTC m=+1812.480130931" Jul 27 18:58:57.913203 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:57.913168 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-09eed4cd-gw-openshift-default-7757b48cf6-5wdrs" Jul 27 18:58:58.918840 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:58.918800 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-09eed4cd-gw-openshift-default-7757b48cf6-5wdrs" Jul 27 18:58:58.922829 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:58.922787 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:58:59.127217 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:59.127181 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:58:59.334936 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:59.334891 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:58:59.406807 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:59.406775 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-09eed4cd-gw-openshift-default-7757b48cf6-5wdrs" Jul 27 18:58:59.407764 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:59.407744 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-09eed4cd-gw-openshift-default-7757b48cf6-5wdrs" Jul 27 18:58:59.572521 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:59.572483 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:58:59.666307 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:58:59.666189 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:59:00.845055 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:00.844993 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-001f5ecd-openshift-default-bbcdf955b-mgmbz" Jul 27 18:59:00.886853 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:00.886821 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22f63302-7966-40e7-8cfb-59201c5b99ae-istio-data\") pod \"22f63302-7966-40e7-8cfb-59201c5b99ae\" (UID: \"22f63302-7966-40e7-8cfb-59201c5b99ae\") " Jul 27 18:59:00.887041 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:00.886873 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22f63302-7966-40e7-8cfb-59201c5b99ae-istio-envoy\") pod \"22f63302-7966-40e7-8cfb-59201c5b99ae\" (UID: \"22f63302-7966-40e7-8cfb-59201c5b99ae\") " Jul 27 18:59:00.887041 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:00.886893 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2ppx6\" (UniqueName: \"kubernetes.io/projected/22f63302-7966-40e7-8cfb-59201c5b99ae-kube-api-access-2ppx6\") pod \"22f63302-7966-40e7-8cfb-59201c5b99ae\" (UID: \"22f63302-7966-40e7-8cfb-59201c5b99ae\") " Jul 27 18:59:00.887041 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:00.886939 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22f63302-7966-40e7-8cfb-59201c5b99ae-istio-token\") pod \"22f63302-7966-40e7-8cfb-59201c5b99ae\" (UID: \"22f63302-7966-40e7-8cfb-59201c5b99ae\") " Jul 27 18:59:00.887041 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:00.886974 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22f63302-7966-40e7-8cfb-59201c5b99ae-istio-podinfo\") pod \"22f63302-7966-40e7-8cfb-59201c5b99ae\" (UID: \"22f63302-7966-40e7-8cfb-59201c5b99ae\") " Jul 27 18:59:00.887041 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:00.887002 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22f63302-7966-40e7-8cfb-59201c5b99ae-credential-socket\") pod \"22f63302-7966-40e7-8cfb-59201c5b99ae\" (UID: \"22f63302-7966-40e7-8cfb-59201c5b99ae\") " Jul 27 18:59:00.887353 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:00.887047 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22f63302-7966-40e7-8cfb-59201c5b99ae-workload-certs\") pod \"22f63302-7966-40e7-8cfb-59201c5b99ae\" (UID: \"22f63302-7966-40e7-8cfb-59201c5b99ae\") " Jul 27 18:59:00.887353 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:00.887114 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22f63302-7966-40e7-8cfb-59201c5b99ae-workload-socket\") pod \"22f63302-7966-40e7-8cfb-59201c5b99ae\" (UID: \"22f63302-7966-40e7-8cfb-59201c5b99ae\") " Jul 27 18:59:00.887353 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:00.887165 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22f63302-7966-40e7-8cfb-59201c5b99ae-istiod-ca-cert\") pod \"22f63302-7966-40e7-8cfb-59201c5b99ae\" (UID: \"22f63302-7966-40e7-8cfb-59201c5b99ae\") " Jul 27 18:59:00.887742 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:00.887698 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22f63302-7966-40e7-8cfb-59201c5b99ae-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "22f63302-7966-40e7-8cfb-59201c5b99ae" (UID: "22f63302-7966-40e7-8cfb-59201c5b99ae"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 18:59:00.887873 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:00.887795 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22f63302-7966-40e7-8cfb-59201c5b99ae-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "22f63302-7966-40e7-8cfb-59201c5b99ae" (UID: "22f63302-7966-40e7-8cfb-59201c5b99ae"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 18:59:00.887939 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:00.887914 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22f63302-7966-40e7-8cfb-59201c5b99ae-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "22f63302-7966-40e7-8cfb-59201c5b99ae" (UID: "22f63302-7966-40e7-8cfb-59201c5b99ae"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 18:59:00.887995 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:00.887961 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22f63302-7966-40e7-8cfb-59201c5b99ae-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "22f63302-7966-40e7-8cfb-59201c5b99ae" (UID: "22f63302-7966-40e7-8cfb-59201c5b99ae"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 18:59:00.888100 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:00.888069 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22f63302-7966-40e7-8cfb-59201c5b99ae-istio-data" (OuterVolumeSpecName: "istio-data") pod "22f63302-7966-40e7-8cfb-59201c5b99ae" (UID: "22f63302-7966-40e7-8cfb-59201c5b99ae"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 18:59:00.890078 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:00.890026 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/22f63302-7966-40e7-8cfb-59201c5b99ae-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "22f63302-7966-40e7-8cfb-59201c5b99ae" (UID: "22f63302-7966-40e7-8cfb-59201c5b99ae"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 27 18:59:00.890274 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:00.890224 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22f63302-7966-40e7-8cfb-59201c5b99ae-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "22f63302-7966-40e7-8cfb-59201c5b99ae" (UID: "22f63302-7966-40e7-8cfb-59201c5b99ae"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 18:59:00.890471 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:00.890450 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22f63302-7966-40e7-8cfb-59201c5b99ae-istio-token" (OuterVolumeSpecName: "istio-token") pod "22f63302-7966-40e7-8cfb-59201c5b99ae" (UID: "22f63302-7966-40e7-8cfb-59201c5b99ae"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 18:59:00.890555 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:00.890532 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22f63302-7966-40e7-8cfb-59201c5b99ae-kube-api-access-2ppx6" (OuterVolumeSpecName: "kube-api-access-2ppx6") pod "22f63302-7966-40e7-8cfb-59201c5b99ae" (UID: "22f63302-7966-40e7-8cfb-59201c5b99ae"). InnerVolumeSpecName "kube-api-access-2ppx6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 18:59:00.988154 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:00.988123 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22f63302-7966-40e7-8cfb-59201c5b99ae-istio-token\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 18:59:00.988154 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:00.988151 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22f63302-7966-40e7-8cfb-59201c5b99ae-istio-podinfo\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 18:59:00.988154 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:00.988160 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22f63302-7966-40e7-8cfb-59201c5b99ae-credential-socket\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 18:59:00.988396 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:00.988171 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22f63302-7966-40e7-8cfb-59201c5b99ae-workload-certs\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 18:59:00.988396 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:00.988179 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22f63302-7966-40e7-8cfb-59201c5b99ae-workload-socket\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 18:59:00.988396 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:00.988187 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22f63302-7966-40e7-8cfb-59201c5b99ae-istiod-ca-cert\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 18:59:00.988396 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:00.988195 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22f63302-7966-40e7-8cfb-59201c5b99ae-istio-data\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 18:59:00.988396 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:00.988202 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22f63302-7966-40e7-8cfb-59201c5b99ae-istio-envoy\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 18:59:00.988396 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:00.988209 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ppx6\" (UniqueName: \"kubernetes.io/projected/22f63302-7966-40e7-8cfb-59201c5b99ae-kube-api-access-2ppx6\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 18:59:01.418379 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:01.418347 2579 generic.go:356] "Generic (PLEG): container finished" podID="22f63302-7966-40e7-8cfb-59201c5b99ae" containerID="c05a0216d7d2f322ce6a437ec1f9511b701c26172d18cb2dd012768a03c0909e" exitCode=0 Jul 27 18:59:01.418573 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:01.418401 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-001f5ecd-openshift-default-bbcdf955b-mgmbz" Jul 27 18:59:01.418573 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:01.418441 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-001f5ecd-openshift-default-bbcdf955b-mgmbz" event={"ID":"22f63302-7966-40e7-8cfb-59201c5b99ae","Type":"ContainerDied","Data":"c05a0216d7d2f322ce6a437ec1f9511b701c26172d18cb2dd012768a03c0909e"} Jul 27 18:59:01.418573 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:01.418500 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-001f5ecd-openshift-default-bbcdf955b-mgmbz" event={"ID":"22f63302-7966-40e7-8cfb-59201c5b99ae","Type":"ContainerDied","Data":"89db7302d608e7b8c257da6c0f985758390a11071ff470a596fda85109c18dcd"} Jul 27 18:59:01.418573 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:01.418524 2579 scope.go:117] "RemoveContainer" containerID="c05a0216d7d2f322ce6a437ec1f9511b701c26172d18cb2dd012768a03c0909e" Jul 27 18:59:01.428541 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:01.428521 2579 scope.go:117] "RemoveContainer" containerID="6ab3537f3b5048b8b84e2b048bb16c20a1e7c621b020d1d5a01309bc64b13efa" Jul 27 18:59:01.440662 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:01.440643 2579 scope.go:117] "RemoveContainer" containerID="c05a0216d7d2f322ce6a437ec1f9511b701c26172d18cb2dd012768a03c0909e" Jul 27 18:59:01.440953 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:59:01.440932 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c05a0216d7d2f322ce6a437ec1f9511b701c26172d18cb2dd012768a03c0909e\": container with ID starting with c05a0216d7d2f322ce6a437ec1f9511b701c26172d18cb2dd012768a03c0909e not found: ID does not exist" containerID="c05a0216d7d2f322ce6a437ec1f9511b701c26172d18cb2dd012768a03c0909e" Jul 27 18:59:01.441012 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:01.440962 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c05a0216d7d2f322ce6a437ec1f9511b701c26172d18cb2dd012768a03c0909e"} err="failed to get container status \"c05a0216d7d2f322ce6a437ec1f9511b701c26172d18cb2dd012768a03c0909e\": rpc error: code = NotFound desc = could not find container \"c05a0216d7d2f322ce6a437ec1f9511b701c26172d18cb2dd012768a03c0909e\": container with ID starting with c05a0216d7d2f322ce6a437ec1f9511b701c26172d18cb2dd012768a03c0909e not found: ID does not exist" Jul 27 18:59:01.441012 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:01.440980 2579 scope.go:117] "RemoveContainer" containerID="6ab3537f3b5048b8b84e2b048bb16c20a1e7c621b020d1d5a01309bc64b13efa" Jul 27 18:59:01.441298 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:59:01.441275 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ab3537f3b5048b8b84e2b048bb16c20a1e7c621b020d1d5a01309bc64b13efa\": container with ID starting with 6ab3537f3b5048b8b84e2b048bb16c20a1e7c621b020d1d5a01309bc64b13efa not found: ID does not exist" containerID="6ab3537f3b5048b8b84e2b048bb16c20a1e7c621b020d1d5a01309bc64b13efa" Jul 27 18:59:01.441437 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:01.441302 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ab3537f3b5048b8b84e2b048bb16c20a1e7c621b020d1d5a01309bc64b13efa"} err="failed to get container status \"6ab3537f3b5048b8b84e2b048bb16c20a1e7c621b020d1d5a01309bc64b13efa\": rpc error: code = NotFound desc = could not find container \"6ab3537f3b5048b8b84e2b048bb16c20a1e7c621b020d1d5a01309bc64b13efa\": container with ID starting with 6ab3537f3b5048b8b84e2b048bb16c20a1e7c621b020d1d5a01309bc64b13efa not found: ID does not exist" Jul 27 18:59:01.444160 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:01.444138 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-001f5ecd-openshift-default-bbcdf955b-mgmbz"] Jul 27 18:59:01.446390 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:01.446368 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-001f5ecd-openshift-default-bbcdf955b-mgmbz"] Jul 27 18:59:01.494440 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:01.494414 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="22f63302-7966-40e7-8cfb-59201c5b99ae" path="/var/lib/kubelet/pods/22f63302-7966-40e7-8cfb-59201c5b99ae/volumes" Jul 27 18:59:14.408633 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:14.408596 2579 patch_prober.go:28] interesting pod/e2e-migrate-09eed4cd-gw-openshift-default-7757b48cf6-5wdrs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Jul 27 18:59:14.409051 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:14.408651 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-09eed4cd-gw-openshift-default-7757b48cf6-5wdrs" podUID="7a4a4a51-fa4b-4527-a02a-240c945d17c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Jul 27 18:59:14.494088 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:14.494056 2579 generic.go:356] "Generic (PLEG): container finished" podID="7a4a4a51-fa4b-4527-a02a-240c945d17c3" containerID="43ee28ee525910b3979adae300320fecbf4421f5806e3947834f6467e1513289" exitCode=0 Jul 27 18:59:14.494293 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:14.494111 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-09eed4cd-gw-openshift-default-7757b48cf6-5wdrs" event={"ID":"7a4a4a51-fa4b-4527-a02a-240c945d17c3","Type":"ContainerDied","Data":"43ee28ee525910b3979adae300320fecbf4421f5806e3947834f6467e1513289"} Jul 27 18:59:14.494547 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:14.494527 2579 scope.go:117] "RemoveContainer" containerID="43ee28ee525910b3979adae300320fecbf4421f5806e3947834f6467e1513289" Jul 27 18:59:14.495536 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:14.495509 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 27 18:59:14.495596 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:14.495575 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 27 18:59:14.495638 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:14.495608 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 27 18:59:15.501049 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:15.501014 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-09eed4cd-gw-openshift-default-7757b48cf6-5wdrs" event={"ID":"7a4a4a51-fa4b-4527-a02a-240c945d17c3","Type":"ContainerStarted","Data":"8e94a7e210f10c0c123fa9366e16fc47c9dea27da87f856873c27f33047fceb3"} Jul 27 18:59:15.912687 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:15.912657 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-09eed4cd-gw-openshift-default-7757b48cf6-5wdrs" Jul 27 18:59:15.912969 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:15.912942 2579 patch_prober.go:28] interesting pod/e2e-migrate-09eed4cd-gw-openshift-default-7757b48cf6-5wdrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Jul 27 18:59:15.913087 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:15.912990 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-09eed4cd-gw-openshift-default-7757b48cf6-5wdrs" podUID="7a4a4a51-fa4b-4527-a02a-240c945d17c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Jul 27 18:59:16.912923 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:16.912889 2579 patch_prober.go:28] interesting pod/e2e-migrate-09eed4cd-gw-openshift-default-7757b48cf6-5wdrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Jul 27 18:59:16.913330 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:16.912949 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-09eed4cd-gw-openshift-default-7757b48cf6-5wdrs" podUID="7a4a4a51-fa4b-4527-a02a-240c945d17c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Jul 27 18:59:17.913638 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:17.913598 2579 patch_prober.go:28] interesting pod/e2e-migrate-09eed4cd-gw-openshift-default-7757b48cf6-5wdrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Jul 27 18:59:17.914029 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:17.913654 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-09eed4cd-gw-openshift-default-7757b48cf6-5wdrs" podUID="7a4a4a51-fa4b-4527-a02a-240c945d17c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Jul 27 18:59:18.912877 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:18.912841 2579 patch_prober.go:28] interesting pod/e2e-migrate-09eed4cd-gw-openshift-default-7757b48cf6-5wdrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Jul 27 18:59:18.913059 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:18.912904 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-09eed4cd-gw-openshift-default-7757b48cf6-5wdrs" podUID="7a4a4a51-fa4b-4527-a02a-240c945d17c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Jul 27 18:59:19.917510 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:19.917476 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-09eed4cd-gw-openshift-default-7757b48cf6-5wdrs" Jul 27 18:59:19.917965 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:19.917943 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-09eed4cd-gw-openshift-default-7757b48cf6-5wdrs" Jul 27 18:59:19.918582 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:19.918561 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-09eed4cd-gw-openshift-default-7757b48cf6-5wdrs" Jul 27 18:59:34.087471 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:34.087432 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:59:34.157206 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:34.157163 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:59:34.640845 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:34.640807 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:59:34.766557 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:34.766505 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:59:34.877196 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:34.877153 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:59:39.165380 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:39.165348 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:59:39.319482 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:39.319409 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-09eed4cd-gw-openshift-default-7757b48cf6-5wdrs"] Jul 27 18:59:39.319676 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:39.319650 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-09eed4cd-gw-openshift-default-7757b48cf6-5wdrs" podUID="7a4a4a51-fa4b-4527-a02a-240c945d17c3" containerName="istio-proxy" containerID="cri-o://8e94a7e210f10c0c123fa9366e16fc47c9dea27da87f856873c27f33047fceb3" gracePeriod=30 Jul 27 18:59:39.481222 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:39.481136 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:59:39.647621 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:39.647585 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:59:39.891690 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:39.891650 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:59:40.222092 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:40.222009 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-217fd0d6-openshift-default-6c5946694d-s4xj9"] Jul 27 18:59:40.222506 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:40.222491 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22f63302-7966-40e7-8cfb-59201c5b99ae" containerName="istio-proxy" Jul 27 18:59:40.222553 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:40.222509 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="22f63302-7966-40e7-8cfb-59201c5b99ae" containerName="istio-proxy" Jul 27 18:59:40.222553 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:40.222522 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22f63302-7966-40e7-8cfb-59201c5b99ae" containerName="istio-proxy" Jul 27 18:59:40.222553 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:40.222528 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="22f63302-7966-40e7-8cfb-59201c5b99ae" containerName="istio-proxy" Jul 27 18:59:40.222645 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:40.222602 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="22f63302-7966-40e7-8cfb-59201c5b99ae" containerName="istio-proxy" Jul 27 18:59:40.222645 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:40.222611 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="22f63302-7966-40e7-8cfb-59201c5b99ae" containerName="istio-proxy" Jul 27 18:59:40.226001 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:40.225975 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-217fd0d6-openshift-default-6c5946694d-s4xj9" Jul 27 18:59:40.229770 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:40.229749 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-217fd0d6-openshift-default-dockercfg-bc5px\"" Jul 27 18:59:40.240025 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:40.239997 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-217fd0d6-openshift-default-6c5946694d-s4xj9"] Jul 27 18:59:40.247927 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:40.247903 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:59:40.367773 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:40.367730 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b302badc-8586-4808-b8d6-3e77e5588803-istio-envoy\") pod \"e2e-ait-217fd0d6-openshift-default-6c5946694d-s4xj9\" (UID: \"b302badc-8586-4808-b8d6-3e77e5588803\") " pod="openshift-ingress/e2e-ait-217fd0d6-openshift-default-6c5946694d-s4xj9" Jul 27 18:59:40.367960 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:40.367794 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b302badc-8586-4808-b8d6-3e77e5588803-istiod-ca-cert\") pod \"e2e-ait-217fd0d6-openshift-default-6c5946694d-s4xj9\" (UID: \"b302badc-8586-4808-b8d6-3e77e5588803\") " pod="openshift-ingress/e2e-ait-217fd0d6-openshift-default-6c5946694d-s4xj9" Jul 27 18:59:40.367960 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:40.367833 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b302badc-8586-4808-b8d6-3e77e5588803-istio-data\") pod \"e2e-ait-217fd0d6-openshift-default-6c5946694d-s4xj9\" (UID: \"b302badc-8586-4808-b8d6-3e77e5588803\") " pod="openshift-ingress/e2e-ait-217fd0d6-openshift-default-6c5946694d-s4xj9" Jul 27 18:59:40.367960 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:40.367864 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b302badc-8586-4808-b8d6-3e77e5588803-workload-socket\") pod \"e2e-ait-217fd0d6-openshift-default-6c5946694d-s4xj9\" (UID: \"b302badc-8586-4808-b8d6-3e77e5588803\") " pod="openshift-ingress/e2e-ait-217fd0d6-openshift-default-6c5946694d-s4xj9" Jul 27 18:59:40.367960 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:40.367928 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b302badc-8586-4808-b8d6-3e77e5588803-workload-certs\") pod \"e2e-ait-217fd0d6-openshift-default-6c5946694d-s4xj9\" (UID: \"b302badc-8586-4808-b8d6-3e77e5588803\") " pod="openshift-ingress/e2e-ait-217fd0d6-openshift-default-6c5946694d-s4xj9" Jul 27 18:59:40.368125 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:40.367963 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b302badc-8586-4808-b8d6-3e77e5588803-istio-token\") pod \"e2e-ait-217fd0d6-openshift-default-6c5946694d-s4xj9\" (UID: \"b302badc-8586-4808-b8d6-3e77e5588803\") " pod="openshift-ingress/e2e-ait-217fd0d6-openshift-default-6c5946694d-s4xj9" Jul 27 18:59:40.368125 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:40.367993 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2gtm\" (UniqueName: \"kubernetes.io/projected/b302badc-8586-4808-b8d6-3e77e5588803-kube-api-access-w2gtm\") pod \"e2e-ait-217fd0d6-openshift-default-6c5946694d-s4xj9\" (UID: \"b302badc-8586-4808-b8d6-3e77e5588803\") " pod="openshift-ingress/e2e-ait-217fd0d6-openshift-default-6c5946694d-s4xj9" Jul 27 18:59:40.368125 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:40.368027 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b302badc-8586-4808-b8d6-3e77e5588803-istio-podinfo\") pod \"e2e-ait-217fd0d6-openshift-default-6c5946694d-s4xj9\" (UID: \"b302badc-8586-4808-b8d6-3e77e5588803\") " pod="openshift-ingress/e2e-ait-217fd0d6-openshift-default-6c5946694d-s4xj9" Jul 27 18:59:40.368125 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:40.368096 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b302badc-8586-4808-b8d6-3e77e5588803-credential-socket\") pod \"e2e-ait-217fd0d6-openshift-default-6c5946694d-s4xj9\" (UID: \"b302badc-8586-4808-b8d6-3e77e5588803\") " pod="openshift-ingress/e2e-ait-217fd0d6-openshift-default-6c5946694d-s4xj9" Jul 27 18:59:40.469521 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:40.469488 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b302badc-8586-4808-b8d6-3e77e5588803-istio-envoy\") pod \"e2e-ait-217fd0d6-openshift-default-6c5946694d-s4xj9\" (UID: \"b302badc-8586-4808-b8d6-3e77e5588803\") " pod="openshift-ingress/e2e-ait-217fd0d6-openshift-default-6c5946694d-s4xj9" Jul 27 18:59:40.469709 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:40.469535 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b302badc-8586-4808-b8d6-3e77e5588803-istiod-ca-cert\") pod \"e2e-ait-217fd0d6-openshift-default-6c5946694d-s4xj9\" (UID: \"b302badc-8586-4808-b8d6-3e77e5588803\") " pod="openshift-ingress/e2e-ait-217fd0d6-openshift-default-6c5946694d-s4xj9" Jul 27 18:59:40.469709 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:40.469566 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b302badc-8586-4808-b8d6-3e77e5588803-istio-data\") pod \"e2e-ait-217fd0d6-openshift-default-6c5946694d-s4xj9\" (UID: \"b302badc-8586-4808-b8d6-3e77e5588803\") " pod="openshift-ingress/e2e-ait-217fd0d6-openshift-default-6c5946694d-s4xj9" Jul 27 18:59:40.469709 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:40.469590 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b302badc-8586-4808-b8d6-3e77e5588803-workload-socket\") pod \"e2e-ait-217fd0d6-openshift-default-6c5946694d-s4xj9\" (UID: \"b302badc-8586-4808-b8d6-3e77e5588803\") " pod="openshift-ingress/e2e-ait-217fd0d6-openshift-default-6c5946694d-s4xj9" Jul 27 18:59:40.469709 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:40.469613 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b302badc-8586-4808-b8d6-3e77e5588803-workload-certs\") pod \"e2e-ait-217fd0d6-openshift-default-6c5946694d-s4xj9\" (UID: \"b302badc-8586-4808-b8d6-3e77e5588803\") " pod="openshift-ingress/e2e-ait-217fd0d6-openshift-default-6c5946694d-s4xj9" Jul 27 18:59:40.469709 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:40.469631 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b302badc-8586-4808-b8d6-3e77e5588803-istio-token\") pod \"e2e-ait-217fd0d6-openshift-default-6c5946694d-s4xj9\" (UID: \"b302badc-8586-4808-b8d6-3e77e5588803\") " pod="openshift-ingress/e2e-ait-217fd0d6-openshift-default-6c5946694d-s4xj9" Jul 27 18:59:40.469709 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:40.469655 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2gtm\" (UniqueName: \"kubernetes.io/projected/b302badc-8586-4808-b8d6-3e77e5588803-kube-api-access-w2gtm\") pod \"e2e-ait-217fd0d6-openshift-default-6c5946694d-s4xj9\" (UID: \"b302badc-8586-4808-b8d6-3e77e5588803\") " pod="openshift-ingress/e2e-ait-217fd0d6-openshift-default-6c5946694d-s4xj9" Jul 27 18:59:40.469709 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:40.469688 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b302badc-8586-4808-b8d6-3e77e5588803-istio-podinfo\") pod \"e2e-ait-217fd0d6-openshift-default-6c5946694d-s4xj9\" (UID: \"b302badc-8586-4808-b8d6-3e77e5588803\") " pod="openshift-ingress/e2e-ait-217fd0d6-openshift-default-6c5946694d-s4xj9" Jul 27 18:59:40.470082 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:40.469719 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b302badc-8586-4808-b8d6-3e77e5588803-credential-socket\") pod \"e2e-ait-217fd0d6-openshift-default-6c5946694d-s4xj9\" (UID: \"b302badc-8586-4808-b8d6-3e77e5588803\") " pod="openshift-ingress/e2e-ait-217fd0d6-openshift-default-6c5946694d-s4xj9" Jul 27 18:59:40.470082 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:40.470027 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b302badc-8586-4808-b8d6-3e77e5588803-istio-data\") pod \"e2e-ait-217fd0d6-openshift-default-6c5946694d-s4xj9\" (UID: \"b302badc-8586-4808-b8d6-3e77e5588803\") " pod="openshift-ingress/e2e-ait-217fd0d6-openshift-default-6c5946694d-s4xj9" Jul 27 18:59:40.470251 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:40.470202 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b302badc-8586-4808-b8d6-3e77e5588803-workload-certs\") pod \"e2e-ait-217fd0d6-openshift-default-6c5946694d-s4xj9\" (UID: \"b302badc-8586-4808-b8d6-3e77e5588803\") " pod="openshift-ingress/e2e-ait-217fd0d6-openshift-default-6c5946694d-s4xj9" Jul 27 18:59:40.470340 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:40.470322 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b302badc-8586-4808-b8d6-3e77e5588803-credential-socket\") pod \"e2e-ait-217fd0d6-openshift-default-6c5946694d-s4xj9\" (UID: \"b302badc-8586-4808-b8d6-3e77e5588803\") " pod="openshift-ingress/e2e-ait-217fd0d6-openshift-default-6c5946694d-s4xj9" Jul 27 18:59:40.470436 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:40.470411 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b302badc-8586-4808-b8d6-3e77e5588803-workload-socket\") pod \"e2e-ait-217fd0d6-openshift-default-6c5946694d-s4xj9\" (UID: \"b302badc-8586-4808-b8d6-3e77e5588803\") " pod="openshift-ingress/e2e-ait-217fd0d6-openshift-default-6c5946694d-s4xj9" Jul 27 18:59:40.470492 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:40.470439 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b302badc-8586-4808-b8d6-3e77e5588803-istiod-ca-cert\") pod \"e2e-ait-217fd0d6-openshift-default-6c5946694d-s4xj9\" (UID: \"b302badc-8586-4808-b8d6-3e77e5588803\") " pod="openshift-ingress/e2e-ait-217fd0d6-openshift-default-6c5946694d-s4xj9" Jul 27 18:59:40.471936 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:40.471917 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b302badc-8586-4808-b8d6-3e77e5588803-istio-envoy\") pod \"e2e-ait-217fd0d6-openshift-default-6c5946694d-s4xj9\" (UID: \"b302badc-8586-4808-b8d6-3e77e5588803\") " pod="openshift-ingress/e2e-ait-217fd0d6-openshift-default-6c5946694d-s4xj9" Jul 27 18:59:40.472307 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:40.472256 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b302badc-8586-4808-b8d6-3e77e5588803-istio-podinfo\") pod \"e2e-ait-217fd0d6-openshift-default-6c5946694d-s4xj9\" (UID: \"b302badc-8586-4808-b8d6-3e77e5588803\") " pod="openshift-ingress/e2e-ait-217fd0d6-openshift-default-6c5946694d-s4xj9" Jul 27 18:59:40.481558 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:40.481531 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b302badc-8586-4808-b8d6-3e77e5588803-istio-token\") pod \"e2e-ait-217fd0d6-openshift-default-6c5946694d-s4xj9\" (UID: \"b302badc-8586-4808-b8d6-3e77e5588803\") " pod="openshift-ingress/e2e-ait-217fd0d6-openshift-default-6c5946694d-s4xj9" Jul 27 18:59:40.482035 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:40.482019 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2gtm\" (UniqueName: \"kubernetes.io/projected/b302badc-8586-4808-b8d6-3e77e5588803-kube-api-access-w2gtm\") pod \"e2e-ait-217fd0d6-openshift-default-6c5946694d-s4xj9\" (UID: \"b302badc-8586-4808-b8d6-3e77e5588803\") " pod="openshift-ingress/e2e-ait-217fd0d6-openshift-default-6c5946694d-s4xj9" Jul 27 18:59:40.537907 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:40.537873 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-217fd0d6-openshift-default-6c5946694d-s4xj9" Jul 27 18:59:40.634082 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:40.633998 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:59:40.691744 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:40.691720 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-217fd0d6-openshift-default-6c5946694d-s4xj9"] Jul 27 18:59:40.693706 ip-10-0-128-32 kubenswrapper[2579]: W0727 18:59:40.693676 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb302badc_8586_4808_b8d6_3e77e5588803.slice/crio-c373b96d2917429462558b411f8587d9836ffc5372f91069d4efece02cfc9caf WatchSource:0}: Error finding container c373b96d2917429462558b411f8587d9836ffc5372f91069d4efece02cfc9caf: Status 404 returned error can't find the container with id c373b96d2917429462558b411f8587d9836ffc5372f91069d4efece02cfc9caf Jul 27 18:59:40.695799 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:40.695769 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 27 18:59:40.695868 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:40.695831 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 27 18:59:40.695868 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:40.695858 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 27 18:59:41.650668 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:41.650626 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-217fd0d6-openshift-default-6c5946694d-s4xj9" event={"ID":"b302badc-8586-4808-b8d6-3e77e5588803","Type":"ContainerStarted","Data":"29a0a48610d9da5373ccd4a57986bbff9618e87c41de3b80ef60b94c1df4da9d"} Jul 27 18:59:41.651092 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:41.650684 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-217fd0d6-openshift-default-6c5946694d-s4xj9" event={"ID":"b302badc-8586-4808-b8d6-3e77e5588803","Type":"ContainerStarted","Data":"c373b96d2917429462558b411f8587d9836ffc5372f91069d4efece02cfc9caf"} Jul 27 18:59:42.539078 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:42.539041 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-217fd0d6-openshift-default-6c5946694d-s4xj9" Jul 27 18:59:42.544275 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:42.544253 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-217fd0d6-openshift-default-6c5946694d-s4xj9" Jul 27 18:59:42.585038 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:42.584992 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-217fd0d6-openshift-default-6c5946694d-s4xj9" podStartSLOduration=2.584974721 podStartE2EDuration="2.584974721s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:59:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:59:41.686862671 +0000 UTC m=+1856.748551701" watchObservedRunningTime="2026-07-27 18:59:42.584974721 +0000 UTC m=+1857.646663753" Jul 27 18:59:42.656351 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:42.656328 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-217fd0d6-openshift-default-6c5946694d-s4xj9" Jul 27 18:59:42.657414 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:42.657395 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-217fd0d6-openshift-default-6c5946694d-s4xj9" Jul 27 18:59:43.236891 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:43.236844 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:59:43.388836 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:43.388800 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:59:43.651090 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:43.651055 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:59:43.836647 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:43.836612 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:59:44.363922 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:44.363896 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:59:44.495297 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:44.495210 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:59:44.584588 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:44.584525 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-09eed4cd-gw-openshift-default-7757b48cf6-5wdrs" Jul 27 18:59:44.606493 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:44.606464 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a4a4a51-fa4b-4527-a02a-240c945d17c3-workload-socket\") pod \"7a4a4a51-fa4b-4527-a02a-240c945d17c3\" (UID: \"7a4a4a51-fa4b-4527-a02a-240c945d17c3\") " Jul 27 18:59:44.606650 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:44.606512 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a4a4a51-fa4b-4527-a02a-240c945d17c3-workload-certs\") pod \"7a4a4a51-fa4b-4527-a02a-240c945d17c3\" (UID: \"7a4a4a51-fa4b-4527-a02a-240c945d17c3\") " Jul 27 18:59:44.606650 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:44.606534 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a4a4a51-fa4b-4527-a02a-240c945d17c3-credential-socket\") pod \"7a4a4a51-fa4b-4527-a02a-240c945d17c3\" (UID: \"7a4a4a51-fa4b-4527-a02a-240c945d17c3\") " Jul 27 18:59:44.606650 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:44.606578 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a4a4a51-fa4b-4527-a02a-240c945d17c3-istio-podinfo\") pod \"7a4a4a51-fa4b-4527-a02a-240c945d17c3\" (UID: \"7a4a4a51-fa4b-4527-a02a-240c945d17c3\") " Jul 27 18:59:44.606650 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:44.606598 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4l789\" (UniqueName: \"kubernetes.io/projected/7a4a4a51-fa4b-4527-a02a-240c945d17c3-kube-api-access-4l789\") pod \"7a4a4a51-fa4b-4527-a02a-240c945d17c3\" (UID: \"7a4a4a51-fa4b-4527-a02a-240c945d17c3\") " Jul 27 18:59:44.606650 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:44.606624 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a4a4a51-fa4b-4527-a02a-240c945d17c3-istio-data\") pod \"7a4a4a51-fa4b-4527-a02a-240c945d17c3\" (UID: \"7a4a4a51-fa4b-4527-a02a-240c945d17c3\") " Jul 27 18:59:44.606958 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:44.606750 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a4a4a51-fa4b-4527-a02a-240c945d17c3-istio-envoy\") pod \"7a4a4a51-fa4b-4527-a02a-240c945d17c3\" (UID: \"7a4a4a51-fa4b-4527-a02a-240c945d17c3\") " Jul 27 18:59:44.606958 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:44.606803 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a4a4a51-fa4b-4527-a02a-240c945d17c3-istio-token\") pod \"7a4a4a51-fa4b-4527-a02a-240c945d17c3\" (UID: \"7a4a4a51-fa4b-4527-a02a-240c945d17c3\") " Jul 27 18:59:44.606958 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:44.606803 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a4a4a51-fa4b-4527-a02a-240c945d17c3-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7a4a4a51-fa4b-4527-a02a-240c945d17c3" (UID: "7a4a4a51-fa4b-4527-a02a-240c945d17c3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 18:59:44.606958 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:44.606876 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a4a4a51-fa4b-4527-a02a-240c945d17c3-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7a4a4a51-fa4b-4527-a02a-240c945d17c3" (UID: "7a4a4a51-fa4b-4527-a02a-240c945d17c3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 18:59:44.606958 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:44.606886 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a4a4a51-fa4b-4527-a02a-240c945d17c3-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7a4a4a51-fa4b-4527-a02a-240c945d17c3" (UID: "7a4a4a51-fa4b-4527-a02a-240c945d17c3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 18:59:44.606958 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:44.606889 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a4a4a51-fa4b-4527-a02a-240c945d17c3-istiod-ca-cert\") pod \"7a4a4a51-fa4b-4527-a02a-240c945d17c3\" (UID: \"7a4a4a51-fa4b-4527-a02a-240c945d17c3\") " Jul 27 18:59:44.607377 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:44.607304 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a4a4a51-fa4b-4527-a02a-240c945d17c3-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7a4a4a51-fa4b-4527-a02a-240c945d17c3" (UID: "7a4a4a51-fa4b-4527-a02a-240c945d17c3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 18:59:44.607377 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:44.607376 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a4a4a51-fa4b-4527-a02a-240c945d17c3-workload-socket\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 18:59:44.607377 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:44.607398 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a4a4a51-fa4b-4527-a02a-240c945d17c3-workload-certs\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 18:59:44.607678 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:44.607413 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a4a4a51-fa4b-4527-a02a-240c945d17c3-credential-socket\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 18:59:44.607875 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:44.607851 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a4a4a51-fa4b-4527-a02a-240c945d17c3-istio-data" (OuterVolumeSpecName: "istio-data") pod "7a4a4a51-fa4b-4527-a02a-240c945d17c3" (UID: "7a4a4a51-fa4b-4527-a02a-240c945d17c3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 18:59:44.609168 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:44.609144 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a4a4a51-fa4b-4527-a02a-240c945d17c3-istio-token" (OuterVolumeSpecName: "istio-token") pod "7a4a4a51-fa4b-4527-a02a-240c945d17c3" (UID: "7a4a4a51-fa4b-4527-a02a-240c945d17c3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 18:59:44.609300 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:44.609163 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a4a4a51-fa4b-4527-a02a-240c945d17c3-kube-api-access-4l789" (OuterVolumeSpecName: "kube-api-access-4l789") pod "7a4a4a51-fa4b-4527-a02a-240c945d17c3" (UID: "7a4a4a51-fa4b-4527-a02a-240c945d17c3"). InnerVolumeSpecName "kube-api-access-4l789". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 18:59:44.609494 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:44.609462 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a4a4a51-fa4b-4527-a02a-240c945d17c3-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7a4a4a51-fa4b-4527-a02a-240c945d17c3" (UID: "7a4a4a51-fa4b-4527-a02a-240c945d17c3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 18:59:44.609582 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:44.609562 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7a4a4a51-fa4b-4527-a02a-240c945d17c3-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7a4a4a51-fa4b-4527-a02a-240c945d17c3" (UID: "7a4a4a51-fa4b-4527-a02a-240c945d17c3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 27 18:59:44.670209 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:44.670169 2579 generic.go:356] "Generic (PLEG): container finished" podID="7a4a4a51-fa4b-4527-a02a-240c945d17c3" containerID="8e94a7e210f10c0c123fa9366e16fc47c9dea27da87f856873c27f33047fceb3" exitCode=0 Jul 27 18:59:44.670427 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:44.670264 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-09eed4cd-gw-openshift-default-7757b48cf6-5wdrs" Jul 27 18:59:44.670427 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:44.670266 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-09eed4cd-gw-openshift-default-7757b48cf6-5wdrs" event={"ID":"7a4a4a51-fa4b-4527-a02a-240c945d17c3","Type":"ContainerDied","Data":"8e94a7e210f10c0c123fa9366e16fc47c9dea27da87f856873c27f33047fceb3"} Jul 27 18:59:44.670427 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:44.670381 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-09eed4cd-gw-openshift-default-7757b48cf6-5wdrs" event={"ID":"7a4a4a51-fa4b-4527-a02a-240c945d17c3","Type":"ContainerDied","Data":"0d0218ac1d739bd4561a50fc1751df2aced424c1b6ada68e95a528b5c28ea2ad"} Jul 27 18:59:44.670427 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:44.670425 2579 scope.go:117] "RemoveContainer" containerID="8e94a7e210f10c0c123fa9366e16fc47c9dea27da87f856873c27f33047fceb3" Jul 27 18:59:44.680751 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:44.680732 2579 scope.go:117] "RemoveContainer" containerID="43ee28ee525910b3979adae300320fecbf4421f5806e3947834f6467e1513289" Jul 27 18:59:44.690276 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:44.690221 2579 scope.go:117] "RemoveContainer" containerID="8e94a7e210f10c0c123fa9366e16fc47c9dea27da87f856873c27f33047fceb3" Jul 27 18:59:44.690575 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:59:44.690554 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e94a7e210f10c0c123fa9366e16fc47c9dea27da87f856873c27f33047fceb3\": container with ID starting with 8e94a7e210f10c0c123fa9366e16fc47c9dea27da87f856873c27f33047fceb3 not found: ID does not exist" containerID="8e94a7e210f10c0c123fa9366e16fc47c9dea27da87f856873c27f33047fceb3" Jul 27 18:59:44.690628 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:44.690584 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e94a7e210f10c0c123fa9366e16fc47c9dea27da87f856873c27f33047fceb3"} err="failed to get container status \"8e94a7e210f10c0c123fa9366e16fc47c9dea27da87f856873c27f33047fceb3\": rpc error: code = NotFound desc = could not find container \"8e94a7e210f10c0c123fa9366e16fc47c9dea27da87f856873c27f33047fceb3\": container with ID starting with 8e94a7e210f10c0c123fa9366e16fc47c9dea27da87f856873c27f33047fceb3 not found: ID does not exist" Jul 27 18:59:44.690628 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:44.690603 2579 scope.go:117] "RemoveContainer" containerID="43ee28ee525910b3979adae300320fecbf4421f5806e3947834f6467e1513289" Jul 27 18:59:44.690922 ip-10-0-128-32 kubenswrapper[2579]: E0727 18:59:44.690890 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43ee28ee525910b3979adae300320fecbf4421f5806e3947834f6467e1513289\": container with ID starting with 43ee28ee525910b3979adae300320fecbf4421f5806e3947834f6467e1513289 not found: ID does not exist" containerID="43ee28ee525910b3979adae300320fecbf4421f5806e3947834f6467e1513289" Jul 27 18:59:44.691021 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:44.690930 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43ee28ee525910b3979adae300320fecbf4421f5806e3947834f6467e1513289"} err="failed to get container status \"43ee28ee525910b3979adae300320fecbf4421f5806e3947834f6467e1513289\": rpc error: code = NotFound desc = could not find container \"43ee28ee525910b3979adae300320fecbf4421f5806e3947834f6467e1513289\": container with ID starting with 43ee28ee525910b3979adae300320fecbf4421f5806e3947834f6467e1513289 not found: ID does not exist" Jul 27 18:59:44.700300 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:44.700275 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-09eed4cd-gw-openshift-default-7757b48cf6-5wdrs"] Jul 27 18:59:44.708400 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:44.708377 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-09eed4cd-gw-openshift-default-7757b48cf6-5wdrs"] Jul 27 18:59:44.708731 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:44.708709 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a4a4a51-fa4b-4527-a02a-240c945d17c3-istiod-ca-cert\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 18:59:44.708731 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:44.708732 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a4a4a51-fa4b-4527-a02a-240c945d17c3-istio-podinfo\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 18:59:44.708883 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:44.708741 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4l789\" (UniqueName: \"kubernetes.io/projected/7a4a4a51-fa4b-4527-a02a-240c945d17c3-kube-api-access-4l789\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 18:59:44.708883 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:44.708752 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a4a4a51-fa4b-4527-a02a-240c945d17c3-istio-data\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 18:59:44.708883 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:44.708761 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a4a4a51-fa4b-4527-a02a-240c945d17c3-istio-envoy\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 18:59:44.708883 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:44.708768 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a4a4a51-fa4b-4527-a02a-240c945d17c3-istio-token\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 18:59:45.493097 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:45.493066 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7a4a4a51-fa4b-4527-a02a-240c945d17c3" path="/var/lib/kubelet/pods/7a4a4a51-fa4b-4527-a02a-240c945d17c3/volumes" Jul 27 18:59:55.109717 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:55.109676 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 18:59:57.657687 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:57.657650 2579 patch_prober.go:28] interesting pod/e2e-ait-217fd0d6-openshift-default-6c5946694d-s4xj9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Jul 27 18:59:57.658155 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:57.657715 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-217fd0d6-openshift-default-6c5946694d-s4xj9" podUID="b302badc-8586-4808-b8d6-3e77e5588803" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Jul 27 18:59:57.743375 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:57.743340 2579 generic.go:356] "Generic (PLEG): container finished" podID="b302badc-8586-4808-b8d6-3e77e5588803" containerID="29a0a48610d9da5373ccd4a57986bbff9618e87c41de3b80ef60b94c1df4da9d" exitCode=0 Jul 27 18:59:57.743543 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:57.743411 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-217fd0d6-openshift-default-6c5946694d-s4xj9" event={"ID":"b302badc-8586-4808-b8d6-3e77e5588803","Type":"ContainerDied","Data":"29a0a48610d9da5373ccd4a57986bbff9618e87c41de3b80ef60b94c1df4da9d"} Jul 27 18:59:57.743803 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:57.743789 2579 scope.go:117] "RemoveContainer" containerID="29a0a48610d9da5373ccd4a57986bbff9618e87c41de3b80ef60b94c1df4da9d" Jul 27 18:59:57.744938 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:57.744911 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 27 18:59:57.745013 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:57.744976 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 27 18:59:57.745013 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:57.745008 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 27 18:59:58.753121 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:58.753081 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-217fd0d6-openshift-default-6c5946694d-s4xj9" event={"ID":"b302badc-8586-4808-b8d6-3e77e5588803","Type":"ContainerStarted","Data":"4fbe988f8a26f9894565bed9ee3f52bc58f7ce07eb76db545e7c612461bbea5a"} Jul 27 18:59:59.538959 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:59.538923 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-217fd0d6-openshift-default-6c5946694d-s4xj9" Jul 27 18:59:59.539484 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:59.539453 2579 patch_prober.go:28] interesting pod/e2e-ait-217fd0d6-openshift-default-6c5946694d-s4xj9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Jul 27 18:59:59.539626 ip-10-0-128-32 kubenswrapper[2579]: I0727 18:59:59.539527 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-217fd0d6-openshift-default-6c5946694d-s4xj9" podUID="b302badc-8586-4808-b8d6-3e77e5588803" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Jul 27 19:00:00.538859 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:00:00.538803 2579 patch_prober.go:28] interesting pod/e2e-ait-217fd0d6-openshift-default-6c5946694d-s4xj9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Jul 27 19:00:00.539407 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:00:00.538883 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-217fd0d6-openshift-default-6c5946694d-s4xj9" podUID="b302badc-8586-4808-b8d6-3e77e5588803" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Jul 27 19:00:01.538710 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:00:01.538664 2579 patch_prober.go:28] interesting pod/e2e-ait-217fd0d6-openshift-default-6c5946694d-s4xj9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Jul 27 19:00:01.539041 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:00:01.539008 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-217fd0d6-openshift-default-6c5946694d-s4xj9" podUID="b302badc-8586-4808-b8d6-3e77e5588803" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Jul 27 19:00:02.547528 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:00:02.547494 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-217fd0d6-openshift-default-6c5946694d-s4xj9" Jul 27 19:00:02.547932 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:00:02.547867 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-217fd0d6-openshift-default-6c5946694d-s4xj9" Jul 27 19:00:02.548485 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:00:02.548466 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-217fd0d6-openshift-default-6c5946694d-s4xj9" Jul 27 19:00:19.595882 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:00:19.595798 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:00:19.690842 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:00:19.690806 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:00:19.764716 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:00:19.764678 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:00:24.260078 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:00:24.260038 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:00:25.368164 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:00:25.368128 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-217fd0d6-openshift-default-6c5946694d-s4xj9"] Jul 27 19:00:25.368619 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:00:25.368356 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-217fd0d6-openshift-default-6c5946694d-s4xj9" podUID="b302badc-8586-4808-b8d6-3e77e5588803" containerName="istio-proxy" containerID="cri-o://4fbe988f8a26f9894565bed9ee3f52bc58f7ce07eb76db545e7c612461bbea5a" gracePeriod=30 Jul 27 19:00:25.385719 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:00:25.385690 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:00:25.626501 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:00:25.626407 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:00:25.872945 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:00:25.872910 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:00:26.176577 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:00:26.176544 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-d042bdca-openshift-default-86c687b796-qkbwr"] Jul 27 19:00:26.176950 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:00:26.176938 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a4a4a51-fa4b-4527-a02a-240c945d17c3" containerName="istio-proxy" Jul 27 19:00:26.176997 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:00:26.176952 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a4a4a51-fa4b-4527-a02a-240c945d17c3" containerName="istio-proxy" Jul 27 19:00:26.176997 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:00:26.176965 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a4a4a51-fa4b-4527-a02a-240c945d17c3" containerName="istio-proxy" Jul 27 19:00:26.176997 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:00:26.176971 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a4a4a51-fa4b-4527-a02a-240c945d17c3" containerName="istio-proxy" Jul 27 19:00:26.177090 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:00:26.177066 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a4a4a51-fa4b-4527-a02a-240c945d17c3" containerName="istio-proxy" Jul 27 19:00:26.180140 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:00:26.180121 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-d042bdca-openshift-default-86c687b796-qkbwr" Jul 27 19:00:26.183658 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:00:26.183637 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-d042bdca-openshift-default-dockercfg-z4z2b\"" Jul 27 19:00:26.198522 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:00:26.198489 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e71cabb2-ca35-4a90-a203-538d4a3ec2b9-istio-podinfo\") pod \"e2e-derive-d042bdca-openshift-default-86c687b796-qkbwr\" (UID: \"e71cabb2-ca35-4a90-a203-538d4a3ec2b9\") " pod="openshift-ingress/e2e-derive-d042bdca-openshift-default-86c687b796-qkbwr" Jul 27 19:00:26.198635 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:00:26.198539 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e71cabb2-ca35-4a90-a203-538d4a3ec2b9-istio-token\") pod \"e2e-derive-d042bdca-openshift-default-86c687b796-qkbwr\" (UID: \"e71cabb2-ca35-4a90-a203-538d4a3ec2b9\") " pod="openshift-ingress/e2e-derive-d042bdca-openshift-default-86c687b796-qkbwr" Jul 27 19:00:26.198635 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:00:26.198572 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e71cabb2-ca35-4a90-a203-538d4a3ec2b9-istiod-ca-cert\") pod \"e2e-derive-d042bdca-openshift-default-86c687b796-qkbwr\" (UID: \"e71cabb2-ca35-4a90-a203-538d4a3ec2b9\") " pod="openshift-ingress/e2e-derive-d042bdca-openshift-default-86c687b796-qkbwr" Jul 27 19:00:26.198738 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:00:26.198625 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e71cabb2-ca35-4a90-a203-538d4a3ec2b9-istio-data\") pod \"e2e-derive-d042bdca-openshift-default-86c687b796-qkbwr\" (UID: \"e71cabb2-ca35-4a90-a203-538d4a3ec2b9\") " pod="openshift-ingress/e2e-derive-d042bdca-openshift-default-86c687b796-qkbwr" Jul 27 19:00:26.198738 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:00:26.198659 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e71cabb2-ca35-4a90-a203-538d4a3ec2b9-workload-certs\") pod \"e2e-derive-d042bdca-openshift-default-86c687b796-qkbwr\" (UID: \"e71cabb2-ca35-4a90-a203-538d4a3ec2b9\") " pod="openshift-ingress/e2e-derive-d042bdca-openshift-default-86c687b796-qkbwr" Jul 27 19:00:26.198738 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:00:26.198706 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e71cabb2-ca35-4a90-a203-538d4a3ec2b9-workload-socket\") pod \"e2e-derive-d042bdca-openshift-default-86c687b796-qkbwr\" (UID: \"e71cabb2-ca35-4a90-a203-538d4a3ec2b9\") " pod="openshift-ingress/e2e-derive-d042bdca-openshift-default-86c687b796-qkbwr" Jul 27 19:00:26.198893 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:00:26.198767 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e71cabb2-ca35-4a90-a203-538d4a3ec2b9-istio-envoy\") pod \"e2e-derive-d042bdca-openshift-default-86c687b796-qkbwr\" (UID: \"e71cabb2-ca35-4a90-a203-538d4a3ec2b9\") " pod="openshift-ingress/e2e-derive-d042bdca-openshift-default-86c687b796-qkbwr" Jul 27 19:00:26.198893 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:00:26.198810 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e71cabb2-ca35-4a90-a203-538d4a3ec2b9-credential-socket\") pod \"e2e-derive-d042bdca-openshift-default-86c687b796-qkbwr\" (UID: \"e71cabb2-ca35-4a90-a203-538d4a3ec2b9\") " pod="openshift-ingress/e2e-derive-d042bdca-openshift-default-86c687b796-qkbwr" Jul 27 19:00:26.199022 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:00:26.198931 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rq69\" (UniqueName: \"kubernetes.io/projected/e71cabb2-ca35-4a90-a203-538d4a3ec2b9-kube-api-access-2rq69\") pod \"e2e-derive-d042bdca-openshift-default-86c687b796-qkbwr\" (UID: \"e71cabb2-ca35-4a90-a203-538d4a3ec2b9\") " pod="openshift-ingress/e2e-derive-d042bdca-openshift-default-86c687b796-qkbwr" Jul 27 19:00:26.199509 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:00:26.199491 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-d042bdca-openshift-default-86c687b796-qkbwr"] Jul 27 19:00:26.228745 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:00:26.228712 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:00:26.300281 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:00:26.300221 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e71cabb2-ca35-4a90-a203-538d4a3ec2b9-workload-socket\") pod \"e2e-derive-d042bdca-openshift-default-86c687b796-qkbwr\" (UID: \"e71cabb2-ca35-4a90-a203-538d4a3ec2b9\") " pod="openshift-ingress/e2e-derive-d042bdca-openshift-default-86c687b796-qkbwr" Jul 27 19:00:26.300281 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:00:26.300285 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e71cabb2-ca35-4a90-a203-538d4a3ec2b9-istio-envoy\") pod \"e2e-derive-d042bdca-openshift-default-86c687b796-qkbwr\" (UID: \"e71cabb2-ca35-4a90-a203-538d4a3ec2b9\") " pod="openshift-ingress/e2e-derive-d042bdca-openshift-default-86c687b796-qkbwr" Jul 27 19:00:26.300667 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:00:26.300310 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e71cabb2-ca35-4a90-a203-538d4a3ec2b9-credential-socket\") pod \"e2e-derive-d042bdca-openshift-default-86c687b796-qkbwr\" (UID: \"e71cabb2-ca35-4a90-a203-538d4a3ec2b9\") " pod="openshift-ingress/e2e-derive-d042bdca-openshift-default-86c687b796-qkbwr" Jul 27 19:00:26.300667 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:00:26.300393 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rq69\" (UniqueName: \"kubernetes.io/projected/e71cabb2-ca35-4a90-a203-538d4a3ec2b9-kube-api-access-2rq69\") pod \"e2e-derive-d042bdca-openshift-default-86c687b796-qkbwr\" (UID: \"e71cabb2-ca35-4a90-a203-538d4a3ec2b9\") " pod="openshift-ingress/e2e-derive-d042bdca-openshift-default-86c687b796-qkbwr" Jul 27 19:00:26.300667 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:00:26.300433 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e71cabb2-ca35-4a90-a203-538d4a3ec2b9-istio-podinfo\") pod \"e2e-derive-d042bdca-openshift-default-86c687b796-qkbwr\" (UID: \"e71cabb2-ca35-4a90-a203-538d4a3ec2b9\") " pod="openshift-ingress/e2e-derive-d042bdca-openshift-default-86c687b796-qkbwr" Jul 27 19:00:26.300667 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:00:26.300469 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e71cabb2-ca35-4a90-a203-538d4a3ec2b9-istio-token\") pod \"e2e-derive-d042bdca-openshift-default-86c687b796-qkbwr\" (UID: \"e71cabb2-ca35-4a90-a203-538d4a3ec2b9\") " pod="openshift-ingress/e2e-derive-d042bdca-openshift-default-86c687b796-qkbwr" Jul 27 19:00:26.300667 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:00:26.300493 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e71cabb2-ca35-4a90-a203-538d4a3ec2b9-istiod-ca-cert\") pod \"e2e-derive-d042bdca-openshift-default-86c687b796-qkbwr\" (UID: \"e71cabb2-ca35-4a90-a203-538d4a3ec2b9\") " pod="openshift-ingress/e2e-derive-d042bdca-openshift-default-86c687b796-qkbwr" Jul 27 19:00:26.300667 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:00:26.300515 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e71cabb2-ca35-4a90-a203-538d4a3ec2b9-istio-data\") pod \"e2e-derive-d042bdca-openshift-default-86c687b796-qkbwr\" (UID: \"e71cabb2-ca35-4a90-a203-538d4a3ec2b9\") " pod="openshift-ingress/e2e-derive-d042bdca-openshift-default-86c687b796-qkbwr" Jul 27 19:00:26.300667 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:00:26.300537 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e71cabb2-ca35-4a90-a203-538d4a3ec2b9-workload-certs\") pod \"e2e-derive-d042bdca-openshift-default-86c687b796-qkbwr\" (UID: \"e71cabb2-ca35-4a90-a203-538d4a3ec2b9\") " pod="openshift-ingress/e2e-derive-d042bdca-openshift-default-86c687b796-qkbwr" Jul 27 19:00:26.300963 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:00:26.300686 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e71cabb2-ca35-4a90-a203-538d4a3ec2b9-workload-socket\") pod \"e2e-derive-d042bdca-openshift-default-86c687b796-qkbwr\" (UID: \"e71cabb2-ca35-4a90-a203-538d4a3ec2b9\") " pod="openshift-ingress/e2e-derive-d042bdca-openshift-default-86c687b796-qkbwr" Jul 27 19:00:26.300963 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:00:26.300792 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e71cabb2-ca35-4a90-a203-538d4a3ec2b9-credential-socket\") pod \"e2e-derive-d042bdca-openshift-default-86c687b796-qkbwr\" (UID: \"e71cabb2-ca35-4a90-a203-538d4a3ec2b9\") " pod="openshift-ingress/e2e-derive-d042bdca-openshift-default-86c687b796-qkbwr" Jul 27 19:00:26.300963 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:00:26.300869 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e71cabb2-ca35-4a90-a203-538d4a3ec2b9-workload-certs\") pod \"e2e-derive-d042bdca-openshift-default-86c687b796-qkbwr\" (UID: \"e71cabb2-ca35-4a90-a203-538d4a3ec2b9\") " pod="openshift-ingress/e2e-derive-d042bdca-openshift-default-86c687b796-qkbwr" Jul 27 19:00:26.300963 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:00:26.300926 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e71cabb2-ca35-4a90-a203-538d4a3ec2b9-istio-data\") pod \"e2e-derive-d042bdca-openshift-default-86c687b796-qkbwr\" (UID: \"e71cabb2-ca35-4a90-a203-538d4a3ec2b9\") " pod="openshift-ingress/e2e-derive-d042bdca-openshift-default-86c687b796-qkbwr" Jul 27 19:00:26.301461 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:00:26.301442 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e71cabb2-ca35-4a90-a203-538d4a3ec2b9-istiod-ca-cert\") pod \"e2e-derive-d042bdca-openshift-default-86c687b796-qkbwr\" (UID: \"e71cabb2-ca35-4a90-a203-538d4a3ec2b9\") " pod="openshift-ingress/e2e-derive-d042bdca-openshift-default-86c687b796-qkbwr" Jul 27 19:00:26.302737 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:00:26.302710 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e71cabb2-ca35-4a90-a203-538d4a3ec2b9-istio-envoy\") pod \"e2e-derive-d042bdca-openshift-default-86c687b796-qkbwr\" (UID: \"e71cabb2-ca35-4a90-a203-538d4a3ec2b9\") " pod="openshift-ingress/e2e-derive-d042bdca-openshift-default-86c687b796-qkbwr" Jul 27 19:00:26.302952 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:00:26.302935 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e71cabb2-ca35-4a90-a203-538d4a3ec2b9-istio-podinfo\") pod \"e2e-derive-d042bdca-openshift-default-86c687b796-qkbwr\" (UID: \"e71cabb2-ca35-4a90-a203-538d4a3ec2b9\") " pod="openshift-ingress/e2e-derive-d042bdca-openshift-default-86c687b796-qkbwr" Jul 27 19:00:26.311075 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:00:26.311054 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e71cabb2-ca35-4a90-a203-538d4a3ec2b9-istio-token\") pod \"e2e-derive-d042bdca-openshift-default-86c687b796-qkbwr\" (UID: \"e71cabb2-ca35-4a90-a203-538d4a3ec2b9\") " pod="openshift-ingress/e2e-derive-d042bdca-openshift-default-86c687b796-qkbwr" Jul 27 19:00:26.311622 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:00:26.311594 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rq69\" (UniqueName: \"kubernetes.io/projected/e71cabb2-ca35-4a90-a203-538d4a3ec2b9-kube-api-access-2rq69\") pod \"e2e-derive-d042bdca-openshift-default-86c687b796-qkbwr\" (UID: \"e71cabb2-ca35-4a90-a203-538d4a3ec2b9\") " pod="openshift-ingress/e2e-derive-d042bdca-openshift-default-86c687b796-qkbwr" Jul 27 19:00:26.490883 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:00:26.490796 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-d042bdca-openshift-default-86c687b796-qkbwr" Jul 27 19:00:26.645469 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:00:26.645441 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-d042bdca-openshift-default-86c687b796-qkbwr"] Jul 27 19:00:26.646878 ip-10-0-128-32 kubenswrapper[2579]: W0727 19:00:26.646852 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode71cabb2_ca35_4a90_a203_538d4a3ec2b9.slice/crio-f6107565d527b5b8576928fe109b98b882a277bff8c81f4472077551e6a06b0a WatchSource:0}: Error finding container f6107565d527b5b8576928fe109b98b882a277bff8c81f4472077551e6a06b0a: Status 404 returned error can't find the container with id f6107565d527b5b8576928fe109b98b882a277bff8c81f4472077551e6a06b0a Jul 27 19:00:26.649351 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:00:26.649320 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 27 19:00:26.649443 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:00:26.649381 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 27 19:00:26.649443 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:00:26.649412 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 27 19:00:26.814367 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:00:26.814331 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:00:26.901935 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:00:26.901894 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-d042bdca-openshift-default-86c687b796-qkbwr" event={"ID":"e71cabb2-ca35-4a90-a203-538d4a3ec2b9","Type":"ContainerStarted","Data":"67fe49dfc48662f9d9c317b565b1e960d5c63fc9fe945ddd9dbd6d0bc4a065a6"} Jul 27 19:00:26.902128 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:00:26.901960 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-d042bdca-openshift-default-86c687b796-qkbwr" event={"ID":"e71cabb2-ca35-4a90-a203-538d4a3ec2b9","Type":"ContainerStarted","Data":"f6107565d527b5b8576928fe109b98b882a277bff8c81f4472077551e6a06b0a"} Jul 27 19:00:26.923432 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:00:26.923369 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-d042bdca-openshift-default-86c687b796-qkbwr" podStartSLOduration=0.923350237 podStartE2EDuration="923.350237ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:00:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:00:26.920837226 +0000 UTC m=+1901.982526256" watchObservedRunningTime="2026-07-27 19:00:26.923350237 +0000 UTC m=+1901.985039274" Jul 27 19:00:27.492801 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:00:27.492767 2579 patch_prober.go:28] interesting pod/e2e-derive-d042bdca-openshift-default-86c687b796-qkbwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Jul 27 19:00:27.493199 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:00:27.492823 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-d042bdca-openshift-default-86c687b796-qkbwr" podUID="e71cabb2-ca35-4a90-a203-538d4a3ec2b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Jul 27 19:00:27.494315 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:00:27.494295 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-d042bdca-openshift-default-86c687b796-qkbwr" Jul 27 19:00:28.264142 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:00:28.264102 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:00:28.365425 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:00:28.365398 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:00:28.495544 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:00:28.495516 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-d042bdca-openshift-default-86c687b796-qkbwr" Jul 27 19:00:28.530033 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:00:28.529999 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:00:28.911668 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:00:28.911584 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-d042bdca-openshift-default-86c687b796-qkbwr" Jul 27 19:00:28.912693 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:00:28.912676 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-d042bdca-openshift-default-86c687b796-qkbwr" Jul 27 19:00:29.001774 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:00:29.001744 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:00:29.102968 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:00:29.102937 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:00:29.368412 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:00:29.368384 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:00:30.635126 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:00:30.635055 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-217fd0d6-openshift-default-6c5946694d-s4xj9" Jul 27 19:00:30.741555 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:00:30.741521 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b302badc-8586-4808-b8d6-3e77e5588803-istio-envoy\") pod \"b302badc-8586-4808-b8d6-3e77e5588803\" (UID: \"b302badc-8586-4808-b8d6-3e77e5588803\") " Jul 27 19:00:30.741753 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:00:30.741575 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w2gtm\" (UniqueName: \"kubernetes.io/projected/b302badc-8586-4808-b8d6-3e77e5588803-kube-api-access-w2gtm\") pod \"b302badc-8586-4808-b8d6-3e77e5588803\" (UID: \"b302badc-8586-4808-b8d6-3e77e5588803\") " Jul 27 19:00:30.741753 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:00:30.741642 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b302badc-8586-4808-b8d6-3e77e5588803-istio-podinfo\") pod \"b302badc-8586-4808-b8d6-3e77e5588803\" (UID: \"b302badc-8586-4808-b8d6-3e77e5588803\") " Jul 27 19:00:30.741753 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:00:30.741667 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b302badc-8586-4808-b8d6-3e77e5588803-credential-socket\") pod \"b302badc-8586-4808-b8d6-3e77e5588803\" (UID: \"b302badc-8586-4808-b8d6-3e77e5588803\") " Jul 27 19:00:30.741753 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:00:30.741685 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b302badc-8586-4808-b8d6-3e77e5588803-istio-data\") pod \"b302badc-8586-4808-b8d6-3e77e5588803\" (UID: \"b302badc-8586-4808-b8d6-3e77e5588803\") " Jul 27 19:00:30.741753 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:00:30.741714 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b302badc-8586-4808-b8d6-3e77e5588803-istio-token\") pod \"b302badc-8586-4808-b8d6-3e77e5588803\" (UID: \"b302badc-8586-4808-b8d6-3e77e5588803\") " Jul 27 19:00:30.741753 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:00:30.741733 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b302badc-8586-4808-b8d6-3e77e5588803-workload-certs\") pod \"b302badc-8586-4808-b8d6-3e77e5588803\" (UID: \"b302badc-8586-4808-b8d6-3e77e5588803\") " Jul 27 19:00:30.742078 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:00:30.741761 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b302badc-8586-4808-b8d6-3e77e5588803-workload-socket\") pod \"b302badc-8586-4808-b8d6-3e77e5588803\" (UID: \"b302badc-8586-4808-b8d6-3e77e5588803\") " Jul 27 19:00:30.742078 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:00:30.741834 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b302badc-8586-4808-b8d6-3e77e5588803-istiod-ca-cert\") pod \"b302badc-8586-4808-b8d6-3e77e5588803\" (UID: \"b302badc-8586-4808-b8d6-3e77e5588803\") " Jul 27 19:00:30.742078 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:00:30.741986 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b302badc-8586-4808-b8d6-3e77e5588803-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b302badc-8586-4808-b8d6-3e77e5588803" (UID: "b302badc-8586-4808-b8d6-3e77e5588803"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 19:00:30.742279 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:00:30.742162 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b302badc-8586-4808-b8d6-3e77e5588803-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b302badc-8586-4808-b8d6-3e77e5588803" (UID: "b302badc-8586-4808-b8d6-3e77e5588803"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 19:00:30.742279 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:00:30.742210 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b302badc-8586-4808-b8d6-3e77e5588803-credential-socket\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 19:00:30.742398 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:00:30.742300 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b302badc-8586-4808-b8d6-3e77e5588803-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b302badc-8586-4808-b8d6-3e77e5588803" (UID: "b302badc-8586-4808-b8d6-3e77e5588803"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 19:00:30.742691 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:00:30.742655 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b302badc-8586-4808-b8d6-3e77e5588803-istio-data" (OuterVolumeSpecName: "istio-data") pod "b302badc-8586-4808-b8d6-3e77e5588803" (UID: "b302badc-8586-4808-b8d6-3e77e5588803"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 19:00:30.742691 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:00:30.742680 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b302badc-8586-4808-b8d6-3e77e5588803-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b302badc-8586-4808-b8d6-3e77e5588803" (UID: "b302badc-8586-4808-b8d6-3e77e5588803"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 19:00:30.744159 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:00:30.744135 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b302badc-8586-4808-b8d6-3e77e5588803-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b302badc-8586-4808-b8d6-3e77e5588803" (UID: "b302badc-8586-4808-b8d6-3e77e5588803"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 27 19:00:30.744414 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:00:30.744385 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b302badc-8586-4808-b8d6-3e77e5588803-kube-api-access-w2gtm" (OuterVolumeSpecName: "kube-api-access-w2gtm") pod "b302badc-8586-4808-b8d6-3e77e5588803" (UID: "b302badc-8586-4808-b8d6-3e77e5588803"). InnerVolumeSpecName "kube-api-access-w2gtm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 19:00:30.744911 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:00:30.744887 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b302badc-8586-4808-b8d6-3e77e5588803-istio-token" (OuterVolumeSpecName: "istio-token") pod "b302badc-8586-4808-b8d6-3e77e5588803" (UID: "b302badc-8586-4808-b8d6-3e77e5588803"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 19:00:30.744988 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:00:30.744918 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b302badc-8586-4808-b8d6-3e77e5588803-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b302badc-8586-4808-b8d6-3e77e5588803" (UID: "b302badc-8586-4808-b8d6-3e77e5588803"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 19:00:30.842950 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:00:30.842918 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2gtm\" (UniqueName: \"kubernetes.io/projected/b302badc-8586-4808-b8d6-3e77e5588803-kube-api-access-w2gtm\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 19:00:30.842950 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:00:30.842944 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b302badc-8586-4808-b8d6-3e77e5588803-istio-podinfo\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 19:00:30.842950 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:00:30.842953 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b302badc-8586-4808-b8d6-3e77e5588803-istio-data\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 19:00:30.842950 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:00:30.842961 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b302badc-8586-4808-b8d6-3e77e5588803-istio-token\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 19:00:30.843287 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:00:30.842969 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b302badc-8586-4808-b8d6-3e77e5588803-workload-certs\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 19:00:30.843287 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:00:30.842977 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b302badc-8586-4808-b8d6-3e77e5588803-workload-socket\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 19:00:30.843287 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:00:30.842985 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b302badc-8586-4808-b8d6-3e77e5588803-istiod-ca-cert\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 19:00:30.843287 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:00:30.842994 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b302badc-8586-4808-b8d6-3e77e5588803-istio-envoy\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 19:00:30.923104 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:00:30.923005 2579 generic.go:356] "Generic (PLEG): container finished" podID="b302badc-8586-4808-b8d6-3e77e5588803" containerID="4fbe988f8a26f9894565bed9ee3f52bc58f7ce07eb76db545e7c612461bbea5a" exitCode=0 Jul 27 19:00:30.923104 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:00:30.923072 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-217fd0d6-openshift-default-6c5946694d-s4xj9" Jul 27 19:00:30.923104 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:00:30.923087 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-217fd0d6-openshift-default-6c5946694d-s4xj9" event={"ID":"b302badc-8586-4808-b8d6-3e77e5588803","Type":"ContainerDied","Data":"4fbe988f8a26f9894565bed9ee3f52bc58f7ce07eb76db545e7c612461bbea5a"} Jul 27 19:00:30.923381 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:00:30.923150 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-217fd0d6-openshift-default-6c5946694d-s4xj9" event={"ID":"b302badc-8586-4808-b8d6-3e77e5588803","Type":"ContainerDied","Data":"c373b96d2917429462558b411f8587d9836ffc5372f91069d4efece02cfc9caf"} Jul 27 19:00:30.923381 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:00:30.923173 2579 scope.go:117] "RemoveContainer" containerID="4fbe988f8a26f9894565bed9ee3f52bc58f7ce07eb76db545e7c612461bbea5a" Jul 27 19:00:30.933141 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:00:30.933122 2579 scope.go:117] "RemoveContainer" containerID="29a0a48610d9da5373ccd4a57986bbff9618e87c41de3b80ef60b94c1df4da9d" Jul 27 19:00:30.944218 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:00:30.944198 2579 scope.go:117] "RemoveContainer" containerID="4fbe988f8a26f9894565bed9ee3f52bc58f7ce07eb76db545e7c612461bbea5a" Jul 27 19:00:30.944606 ip-10-0-128-32 kubenswrapper[2579]: E0727 19:00:30.944577 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4fbe988f8a26f9894565bed9ee3f52bc58f7ce07eb76db545e7c612461bbea5a\": container with ID starting with 4fbe988f8a26f9894565bed9ee3f52bc58f7ce07eb76db545e7c612461bbea5a not found: ID does not exist" containerID="4fbe988f8a26f9894565bed9ee3f52bc58f7ce07eb76db545e7c612461bbea5a" Jul 27 19:00:30.944694 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:00:30.944619 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fbe988f8a26f9894565bed9ee3f52bc58f7ce07eb76db545e7c612461bbea5a"} err="failed to get container status \"4fbe988f8a26f9894565bed9ee3f52bc58f7ce07eb76db545e7c612461bbea5a\": rpc error: code = NotFound desc = could not find container \"4fbe988f8a26f9894565bed9ee3f52bc58f7ce07eb76db545e7c612461bbea5a\": container with ID starting with 4fbe988f8a26f9894565bed9ee3f52bc58f7ce07eb76db545e7c612461bbea5a not found: ID does not exist" Jul 27 19:00:30.944694 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:00:30.944646 2579 scope.go:117] "RemoveContainer" containerID="29a0a48610d9da5373ccd4a57986bbff9618e87c41de3b80ef60b94c1df4da9d" Jul 27 19:00:30.944898 ip-10-0-128-32 kubenswrapper[2579]: E0727 19:00:30.944882 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29a0a48610d9da5373ccd4a57986bbff9618e87c41de3b80ef60b94c1df4da9d\": container with ID starting with 29a0a48610d9da5373ccd4a57986bbff9618e87c41de3b80ef60b94c1df4da9d not found: ID does not exist" containerID="29a0a48610d9da5373ccd4a57986bbff9618e87c41de3b80ef60b94c1df4da9d" Jul 27 19:00:30.944938 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:00:30.944903 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29a0a48610d9da5373ccd4a57986bbff9618e87c41de3b80ef60b94c1df4da9d"} err="failed to get container status \"29a0a48610d9da5373ccd4a57986bbff9618e87c41de3b80ef60b94c1df4da9d\": rpc error: code = NotFound desc = could not find container \"29a0a48610d9da5373ccd4a57986bbff9618e87c41de3b80ef60b94c1df4da9d\": container with ID starting with 29a0a48610d9da5373ccd4a57986bbff9618e87c41de3b80ef60b94c1df4da9d not found: ID does not exist" Jul 27 19:00:30.948335 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:00:30.948310 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-217fd0d6-openshift-default-6c5946694d-s4xj9"] Jul 27 19:00:30.952250 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:00:30.952210 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-217fd0d6-openshift-default-6c5946694d-s4xj9"] Jul 27 19:00:31.493434 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:00:31.493398 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b302badc-8586-4808-b8d6-3e77e5588803" path="/var/lib/kubelet/pods/b302badc-8586-4808-b8d6-3e77e5588803/volumes" Jul 27 19:00:42.987922 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:00:42.987883 2579 generic.go:356] "Generic (PLEG): container finished" podID="e71cabb2-ca35-4a90-a203-538d4a3ec2b9" containerID="67fe49dfc48662f9d9c317b565b1e960d5c63fc9fe945ddd9dbd6d0bc4a065a6" exitCode=0 Jul 27 19:00:42.988353 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:00:42.987958 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-d042bdca-openshift-default-86c687b796-qkbwr" event={"ID":"e71cabb2-ca35-4a90-a203-538d4a3ec2b9","Type":"ContainerDied","Data":"67fe49dfc48662f9d9c317b565b1e960d5c63fc9fe945ddd9dbd6d0bc4a065a6"} Jul 27 19:00:42.988482 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:00:42.988462 2579 scope.go:117] "RemoveContainer" containerID="67fe49dfc48662f9d9c317b565b1e960d5c63fc9fe945ddd9dbd6d0bc4a065a6" Jul 27 19:00:42.989543 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:00:42.989514 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 27 19:00:42.989606 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:00:42.989582 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 27 19:00:42.989655 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:00:42.989623 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 27 19:00:43.911882 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:00:43.911851 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-d042bdca-openshift-default-86c687b796-qkbwr" Jul 27 19:00:43.994579 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:00:43.994541 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-d042bdca-openshift-default-86c687b796-qkbwr" event={"ID":"e71cabb2-ca35-4a90-a203-538d4a3ec2b9","Type":"ContainerStarted","Data":"22c03ddc67242e1d530547e62a18d4783c4e64af2f8727b57a95a81f1564fc22"} Jul 27 19:00:44.491421 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:00:44.491376 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-d042bdca-openshift-default-86c687b796-qkbwr" Jul 27 19:00:44.492586 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:00:44.492560 2579 patch_prober.go:28] interesting pod/e2e-derive-d042bdca-openshift-default-86c687b796-qkbwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Jul 27 19:00:44.492731 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:00:44.492609 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-d042bdca-openshift-default-86c687b796-qkbwr" podUID="e71cabb2-ca35-4a90-a203-538d4a3ec2b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Jul 27 19:00:45.491421 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:00:45.491385 2579 patch_prober.go:28] interesting pod/e2e-derive-d042bdca-openshift-default-86c687b796-qkbwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Jul 27 19:00:45.491897 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:00:45.491478 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-d042bdca-openshift-default-86c687b796-qkbwr" podUID="e71cabb2-ca35-4a90-a203-538d4a3ec2b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Jul 27 19:00:45.634296 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:00:45.634254 2579 scope.go:117] "RemoveContainer" containerID="3f0b28d8aafd773ddf43b451e9e3af3136010a056c8f4fb55d6bb524eee4e663" Jul 27 19:00:46.492065 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:00:46.492026 2579 patch_prober.go:28] interesting pod/e2e-derive-d042bdca-openshift-default-86c687b796-qkbwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Jul 27 19:00:46.492489 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:00:46.492090 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-d042bdca-openshift-default-86c687b796-qkbwr" podUID="e71cabb2-ca35-4a90-a203-538d4a3ec2b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Jul 27 19:00:47.491297 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:00:47.491263 2579 patch_prober.go:28] interesting pod/e2e-derive-d042bdca-openshift-default-86c687b796-qkbwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Jul 27 19:00:47.491477 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:00:47.491319 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-d042bdca-openshift-default-86c687b796-qkbwr" podUID="e71cabb2-ca35-4a90-a203-538d4a3ec2b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Jul 27 19:00:48.492054 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:00:48.492013 2579 patch_prober.go:28] interesting pod/e2e-derive-d042bdca-openshift-default-86c687b796-qkbwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Jul 27 19:00:48.492496 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:00:48.492072 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-d042bdca-openshift-default-86c687b796-qkbwr" podUID="e71cabb2-ca35-4a90-a203-538d4a3ec2b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Jul 27 19:00:49.495541 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:00:49.495512 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-d042bdca-openshift-default-86c687b796-qkbwr" Jul 27 19:00:49.495994 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:00:49.495892 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-d042bdca-openshift-default-86c687b796-qkbwr" Jul 27 19:00:49.496468 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:00:49.496453 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-d042bdca-openshift-default-86c687b796-qkbwr" Jul 27 19:01:03.711128 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:01:03.711085 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:01:03.799002 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:01:03.798922 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:01:03.861383 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:01:03.861348 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:01:08.249651 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:01:08.249609 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:01:08.397289 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:01:08.397258 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-d042bdca-openshift-default-86c687b796-qkbwr"] Jul 27 19:01:08.397562 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:01:08.397532 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-d042bdca-openshift-default-86c687b796-qkbwr" podUID="e71cabb2-ca35-4a90-a203-538d4a3ec2b9" containerName="istio-proxy" containerID="cri-o://22c03ddc67242e1d530547e62a18d4783c4e64af2f8727b57a95a81f1564fc22" gracePeriod=30 Jul 27 19:01:08.418900 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:01:08.418866 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:01:08.669593 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:01:08.669556 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:01:09.207917 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:01:09.207866 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:01:09.311344 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:01:09.311309 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:01:09.414244 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:01:09.414206 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:01:10.863361 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:01:10.863329 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:01:13.664750 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:01:13.664687 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-d042bdca-openshift-default-86c687b796-qkbwr" Jul 27 19:01:13.745612 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:01:13.745579 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2rq69\" (UniqueName: \"kubernetes.io/projected/e71cabb2-ca35-4a90-a203-538d4a3ec2b9-kube-api-access-2rq69\") pod \"e71cabb2-ca35-4a90-a203-538d4a3ec2b9\" (UID: \"e71cabb2-ca35-4a90-a203-538d4a3ec2b9\") " Jul 27 19:01:13.745770 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:01:13.745629 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e71cabb2-ca35-4a90-a203-538d4a3ec2b9-istio-data\") pod \"e71cabb2-ca35-4a90-a203-538d4a3ec2b9\" (UID: \"e71cabb2-ca35-4a90-a203-538d4a3ec2b9\") " Jul 27 19:01:13.745770 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:01:13.745658 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e71cabb2-ca35-4a90-a203-538d4a3ec2b9-workload-socket\") pod \"e71cabb2-ca35-4a90-a203-538d4a3ec2b9\" (UID: \"e71cabb2-ca35-4a90-a203-538d4a3ec2b9\") " Jul 27 19:01:13.745770 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:01:13.745688 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e71cabb2-ca35-4a90-a203-538d4a3ec2b9-istiod-ca-cert\") pod \"e71cabb2-ca35-4a90-a203-538d4a3ec2b9\" (UID: \"e71cabb2-ca35-4a90-a203-538d4a3ec2b9\") " Jul 27 19:01:13.745770 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:01:13.745752 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e71cabb2-ca35-4a90-a203-538d4a3ec2b9-istio-token\") pod \"e71cabb2-ca35-4a90-a203-538d4a3ec2b9\" (UID: \"e71cabb2-ca35-4a90-a203-538d4a3ec2b9\") " Jul 27 19:01:13.745982 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:01:13.745782 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e71cabb2-ca35-4a90-a203-538d4a3ec2b9-istio-podinfo\") pod \"e71cabb2-ca35-4a90-a203-538d4a3ec2b9\" (UID: \"e71cabb2-ca35-4a90-a203-538d4a3ec2b9\") " Jul 27 19:01:13.745982 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:01:13.745813 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e71cabb2-ca35-4a90-a203-538d4a3ec2b9-workload-certs\") pod \"e71cabb2-ca35-4a90-a203-538d4a3ec2b9\" (UID: \"e71cabb2-ca35-4a90-a203-538d4a3ec2b9\") " Jul 27 19:01:13.745982 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:01:13.745844 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e71cabb2-ca35-4a90-a203-538d4a3ec2b9-istio-envoy\") pod \"e71cabb2-ca35-4a90-a203-538d4a3ec2b9\" (UID: \"e71cabb2-ca35-4a90-a203-538d4a3ec2b9\") " Jul 27 19:01:13.745982 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:01:13.745870 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e71cabb2-ca35-4a90-a203-538d4a3ec2b9-credential-socket\") pod \"e71cabb2-ca35-4a90-a203-538d4a3ec2b9\" (UID: \"e71cabb2-ca35-4a90-a203-538d4a3ec2b9\") " Jul 27 19:01:13.746207 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:01:13.746011 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e71cabb2-ca35-4a90-a203-538d4a3ec2b9-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e71cabb2-ca35-4a90-a203-538d4a3ec2b9" (UID: "e71cabb2-ca35-4a90-a203-538d4a3ec2b9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 19:01:13.746207 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:01:13.746125 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e71cabb2-ca35-4a90-a203-538d4a3ec2b9-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e71cabb2-ca35-4a90-a203-538d4a3ec2b9" (UID: "e71cabb2-ca35-4a90-a203-538d4a3ec2b9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 19:01:13.746338 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:01:13.746206 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e71cabb2-ca35-4a90-a203-538d4a3ec2b9-workload-socket\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 19:01:13.746338 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:01:13.746211 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e71cabb2-ca35-4a90-a203-538d4a3ec2b9-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e71cabb2-ca35-4a90-a203-538d4a3ec2b9" (UID: "e71cabb2-ca35-4a90-a203-538d4a3ec2b9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 19:01:13.746338 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:01:13.746255 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e71cabb2-ca35-4a90-a203-538d4a3ec2b9-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e71cabb2-ca35-4a90-a203-538d4a3ec2b9" (UID: "e71cabb2-ca35-4a90-a203-538d4a3ec2b9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 19:01:13.746669 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:01:13.746587 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e71cabb2-ca35-4a90-a203-538d4a3ec2b9-istio-data" (OuterVolumeSpecName: "istio-data") pod "e71cabb2-ca35-4a90-a203-538d4a3ec2b9" (UID: "e71cabb2-ca35-4a90-a203-538d4a3ec2b9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 19:01:13.747867 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:01:13.747843 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e71cabb2-ca35-4a90-a203-538d4a3ec2b9-kube-api-access-2rq69" (OuterVolumeSpecName: "kube-api-access-2rq69") pod "e71cabb2-ca35-4a90-a203-538d4a3ec2b9" (UID: "e71cabb2-ca35-4a90-a203-538d4a3ec2b9"). InnerVolumeSpecName "kube-api-access-2rq69". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 19:01:13.748006 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:01:13.747894 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e71cabb2-ca35-4a90-a203-538d4a3ec2b9-istio-token" (OuterVolumeSpecName: "istio-token") pod "e71cabb2-ca35-4a90-a203-538d4a3ec2b9" (UID: "e71cabb2-ca35-4a90-a203-538d4a3ec2b9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 19:01:13.748160 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:01:13.748143 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e71cabb2-ca35-4a90-a203-538d4a3ec2b9-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e71cabb2-ca35-4a90-a203-538d4a3ec2b9" (UID: "e71cabb2-ca35-4a90-a203-538d4a3ec2b9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 27 19:01:13.748618 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:01:13.748594 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e71cabb2-ca35-4a90-a203-538d4a3ec2b9-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e71cabb2-ca35-4a90-a203-538d4a3ec2b9" (UID: "e71cabb2-ca35-4a90-a203-538d4a3ec2b9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 19:01:13.847276 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:01:13.847214 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e71cabb2-ca35-4a90-a203-538d4a3ec2b9-istiod-ca-cert\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 19:01:13.847276 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:01:13.847269 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e71cabb2-ca35-4a90-a203-538d4a3ec2b9-istio-token\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 19:01:13.847276 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:01:13.847283 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e71cabb2-ca35-4a90-a203-538d4a3ec2b9-istio-podinfo\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 19:01:13.847499 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:01:13.847294 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e71cabb2-ca35-4a90-a203-538d4a3ec2b9-workload-certs\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 19:01:13.847499 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:01:13.847304 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e71cabb2-ca35-4a90-a203-538d4a3ec2b9-istio-envoy\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 19:01:13.847499 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:01:13.847316 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e71cabb2-ca35-4a90-a203-538d4a3ec2b9-credential-socket\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 19:01:13.847499 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:01:13.847327 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rq69\" (UniqueName: \"kubernetes.io/projected/e71cabb2-ca35-4a90-a203-538d4a3ec2b9-kube-api-access-2rq69\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 19:01:13.847499 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:01:13.847339 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e71cabb2-ca35-4a90-a203-538d4a3ec2b9-istio-data\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 19:01:14.145474 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:01:14.145435 2579 generic.go:356] "Generic (PLEG): container finished" podID="e71cabb2-ca35-4a90-a203-538d4a3ec2b9" containerID="22c03ddc67242e1d530547e62a18d4783c4e64af2f8727b57a95a81f1564fc22" exitCode=0 Jul 27 19:01:14.145638 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:01:14.145485 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-d042bdca-openshift-default-86c687b796-qkbwr" Jul 27 19:01:14.145638 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:01:14.145505 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-d042bdca-openshift-default-86c687b796-qkbwr" event={"ID":"e71cabb2-ca35-4a90-a203-538d4a3ec2b9","Type":"ContainerDied","Data":"22c03ddc67242e1d530547e62a18d4783c4e64af2f8727b57a95a81f1564fc22"} Jul 27 19:01:14.145638 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:01:14.145562 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-d042bdca-openshift-default-86c687b796-qkbwr" event={"ID":"e71cabb2-ca35-4a90-a203-538d4a3ec2b9","Type":"ContainerDied","Data":"f6107565d527b5b8576928fe109b98b882a277bff8c81f4472077551e6a06b0a"} Jul 27 19:01:14.145638 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:01:14.145584 2579 scope.go:117] "RemoveContainer" containerID="22c03ddc67242e1d530547e62a18d4783c4e64af2f8727b57a95a81f1564fc22" Jul 27 19:01:14.161639 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:01:14.161618 2579 scope.go:117] "RemoveContainer" containerID="67fe49dfc48662f9d9c317b565b1e960d5c63fc9fe945ddd9dbd6d0bc4a065a6" Jul 27 19:01:14.170252 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:01:14.170183 2579 scope.go:117] "RemoveContainer" containerID="22c03ddc67242e1d530547e62a18d4783c4e64af2f8727b57a95a81f1564fc22" Jul 27 19:01:14.170795 ip-10-0-128-32 kubenswrapper[2579]: E0727 19:01:14.170533 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22c03ddc67242e1d530547e62a18d4783c4e64af2f8727b57a95a81f1564fc22\": container with ID starting with 22c03ddc67242e1d530547e62a18d4783c4e64af2f8727b57a95a81f1564fc22 not found: ID does not exist" containerID="22c03ddc67242e1d530547e62a18d4783c4e64af2f8727b57a95a81f1564fc22" Jul 27 19:01:14.170795 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:01:14.170574 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22c03ddc67242e1d530547e62a18d4783c4e64af2f8727b57a95a81f1564fc22"} err="failed to get container status \"22c03ddc67242e1d530547e62a18d4783c4e64af2f8727b57a95a81f1564fc22\": rpc error: code = NotFound desc = could not find container \"22c03ddc67242e1d530547e62a18d4783c4e64af2f8727b57a95a81f1564fc22\": container with ID starting with 22c03ddc67242e1d530547e62a18d4783c4e64af2f8727b57a95a81f1564fc22 not found: ID does not exist" Jul 27 19:01:14.170795 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:01:14.170597 2579 scope.go:117] "RemoveContainer" containerID="67fe49dfc48662f9d9c317b565b1e960d5c63fc9fe945ddd9dbd6d0bc4a065a6" Jul 27 19:01:14.171013 ip-10-0-128-32 kubenswrapper[2579]: E0727 19:01:14.170915 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67fe49dfc48662f9d9c317b565b1e960d5c63fc9fe945ddd9dbd6d0bc4a065a6\": container with ID starting with 67fe49dfc48662f9d9c317b565b1e960d5c63fc9fe945ddd9dbd6d0bc4a065a6 not found: ID does not exist" containerID="67fe49dfc48662f9d9c317b565b1e960d5c63fc9fe945ddd9dbd6d0bc4a065a6" Jul 27 19:01:14.171013 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:01:14.170938 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67fe49dfc48662f9d9c317b565b1e960d5c63fc9fe945ddd9dbd6d0bc4a065a6"} err="failed to get container status \"67fe49dfc48662f9d9c317b565b1e960d5c63fc9fe945ddd9dbd6d0bc4a065a6\": rpc error: code = NotFound desc = could not find container \"67fe49dfc48662f9d9c317b565b1e960d5c63fc9fe945ddd9dbd6d0bc4a065a6\": container with ID starting with 67fe49dfc48662f9d9c317b565b1e960d5c63fc9fe945ddd9dbd6d0bc4a065a6 not found: ID does not exist" Jul 27 19:01:14.173216 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:01:14.173197 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-d042bdca-openshift-default-86c687b796-qkbwr"] Jul 27 19:01:14.178639 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:01:14.178620 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-d042bdca-openshift-default-86c687b796-qkbwr"] Jul 27 19:01:15.493640 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:01:15.493606 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e71cabb2-ca35-4a90-a203-538d4a3ec2b9" path="/var/lib/kubelet/pods/e71cabb2-ca35-4a90-a203-538d4a3ec2b9/volumes" Jul 27 19:01:16.945201 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:01:16.945164 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:01:17.052703 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:01:17.052671 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:01:17.163106 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:01:17.163064 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:01:17.274118 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:01:17.274086 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:01:17.700865 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:01:17.700771 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:01:19.028408 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:01:19.028374 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:01:19.128249 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:01:19.128195 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:01:43.688291 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:01:43.688252 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:01:50.228052 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:01:50.228015 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:01:50.331349 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:01:50.331312 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:01:50.430343 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:01:50.430305 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:02:06.349281 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:02:06.349246 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:02:12.587821 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:02:12.587790 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:02:12.690109 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:02:12.690074 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:02:12.788082 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:02:12.788056 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:02:23.347595 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:02:23.347558 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:02:24.185918 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:02:24.185884 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:02:31.338195 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:02:31.338162 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:02:31.445595 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:02:31.445565 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:02:31.546193 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:02:31.546156 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:02:31.647708 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:02:31.647616 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:02:32.504098 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:02:32.504062 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:02:32.605796 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:02:32.605761 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:02:32.707714 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:02:32.707686 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:02:34.408430 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:02:34.408396 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:02:40.748908 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:02:40.748871 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:02:40.846486 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:02:40.846447 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:02:42.169831 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:02:42.169798 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:02:47.660546 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:02:47.660505 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:02:47.768873 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:02:47.768840 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:02:47.907561 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:02:47.907521 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:02:48.304762 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:02:48.304728 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-9aed4c03-openshift-default-5845bf9744-t4whw"] Jul 27 19:02:48.305115 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:02:48.305103 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e71cabb2-ca35-4a90-a203-538d4a3ec2b9" containerName="istio-proxy" Jul 27 19:02:48.305115 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:02:48.305116 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e71cabb2-ca35-4a90-a203-538d4a3ec2b9" containerName="istio-proxy" Jul 27 19:02:48.305202 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:02:48.305129 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b302badc-8586-4808-b8d6-3e77e5588803" containerName="istio-proxy" Jul 27 19:02:48.305202 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:02:48.305135 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="b302badc-8586-4808-b8d6-3e77e5588803" containerName="istio-proxy" Jul 27 19:02:48.305202 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:02:48.305143 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b302badc-8586-4808-b8d6-3e77e5588803" containerName="istio-proxy" Jul 27 19:02:48.305202 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:02:48.305149 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="b302badc-8586-4808-b8d6-3e77e5588803" containerName="istio-proxy" Jul 27 19:02:48.305202 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:02:48.305159 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e71cabb2-ca35-4a90-a203-538d4a3ec2b9" containerName="istio-proxy" Jul 27 19:02:48.305202 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:02:48.305164 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e71cabb2-ca35-4a90-a203-538d4a3ec2b9" containerName="istio-proxy" Jul 27 19:02:48.305405 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:02:48.305244 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="b302badc-8586-4808-b8d6-3e77e5588803" containerName="istio-proxy" Jul 27 19:02:48.305405 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:02:48.305257 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e71cabb2-ca35-4a90-a203-538d4a3ec2b9" containerName="istio-proxy" Jul 27 19:02:48.305405 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:02:48.305264 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e71cabb2-ca35-4a90-a203-538d4a3ec2b9" containerName="istio-proxy" Jul 27 19:02:48.305405 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:02:48.305270 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a4a4a51-fa4b-4527-a02a-240c945d17c3" containerName="istio-proxy" Jul 27 19:02:48.307820 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:02:48.307804 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-9aed4c03-openshift-default-5845bf9744-t4whw" Jul 27 19:02:48.311742 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:02:48.311717 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-9aed4c03-openshift-default-dockercfg-jnpcf\"" Jul 27 19:02:48.334729 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:02:48.334702 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-9aed4c03-openshift-default-5845bf9744-t4whw"] Jul 27 19:02:48.358452 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:02:48.358413 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:02:48.402738 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:02:48.402704 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf98d888-f70a-414c-a6b8-60c11d79d76b-istio-data\") pod \"e2e-mt-9aed4c03-openshift-default-5845bf9744-t4whw\" (UID: \"bf98d888-f70a-414c-a6b8-60c11d79d76b\") " pod="openshift-ingress/e2e-mt-9aed4c03-openshift-default-5845bf9744-t4whw" Jul 27 19:02:48.402934 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:02:48.402745 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf98d888-f70a-414c-a6b8-60c11d79d76b-istiod-ca-cert\") pod \"e2e-mt-9aed4c03-openshift-default-5845bf9744-t4whw\" (UID: \"bf98d888-f70a-414c-a6b8-60c11d79d76b\") " pod="openshift-ingress/e2e-mt-9aed4c03-openshift-default-5845bf9744-t4whw" Jul 27 19:02:48.402934 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:02:48.402886 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf98d888-f70a-414c-a6b8-60c11d79d76b-workload-certs\") pod \"e2e-mt-9aed4c03-openshift-default-5845bf9744-t4whw\" (UID: \"bf98d888-f70a-414c-a6b8-60c11d79d76b\") " pod="openshift-ingress/e2e-mt-9aed4c03-openshift-default-5845bf9744-t4whw" Jul 27 19:02:48.403041 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:02:48.402943 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf98d888-f70a-414c-a6b8-60c11d79d76b-istio-envoy\") pod \"e2e-mt-9aed4c03-openshift-default-5845bf9744-t4whw\" (UID: \"bf98d888-f70a-414c-a6b8-60c11d79d76b\") " pod="openshift-ingress/e2e-mt-9aed4c03-openshift-default-5845bf9744-t4whw" Jul 27 19:02:48.403041 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:02:48.402980 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf98d888-f70a-414c-a6b8-60c11d79d76b-istio-token\") pod \"e2e-mt-9aed4c03-openshift-default-5845bf9744-t4whw\" (UID: \"bf98d888-f70a-414c-a6b8-60c11d79d76b\") " pod="openshift-ingress/e2e-mt-9aed4c03-openshift-default-5845bf9744-t4whw" Jul 27 19:02:48.403041 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:02:48.403006 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zp49\" (UniqueName: \"kubernetes.io/projected/bf98d888-f70a-414c-a6b8-60c11d79d76b-kube-api-access-5zp49\") pod \"e2e-mt-9aed4c03-openshift-default-5845bf9744-t4whw\" (UID: \"bf98d888-f70a-414c-a6b8-60c11d79d76b\") " pod="openshift-ingress/e2e-mt-9aed4c03-openshift-default-5845bf9744-t4whw" Jul 27 19:02:48.403190 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:02:48.403041 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf98d888-f70a-414c-a6b8-60c11d79d76b-workload-socket\") pod \"e2e-mt-9aed4c03-openshift-default-5845bf9744-t4whw\" (UID: \"bf98d888-f70a-414c-a6b8-60c11d79d76b\") " pod="openshift-ingress/e2e-mt-9aed4c03-openshift-default-5845bf9744-t4whw" Jul 27 19:02:48.403190 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:02:48.403104 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf98d888-f70a-414c-a6b8-60c11d79d76b-credential-socket\") pod \"e2e-mt-9aed4c03-openshift-default-5845bf9744-t4whw\" (UID: \"bf98d888-f70a-414c-a6b8-60c11d79d76b\") " pod="openshift-ingress/e2e-mt-9aed4c03-openshift-default-5845bf9744-t4whw" Jul 27 19:02:48.403190 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:02:48.403178 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf98d888-f70a-414c-a6b8-60c11d79d76b-istio-podinfo\") pod \"e2e-mt-9aed4c03-openshift-default-5845bf9744-t4whw\" (UID: \"bf98d888-f70a-414c-a6b8-60c11d79d76b\") " pod="openshift-ingress/e2e-mt-9aed4c03-openshift-default-5845bf9744-t4whw" Jul 27 19:02:48.504649 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:02:48.504610 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf98d888-f70a-414c-a6b8-60c11d79d76b-workload-certs\") pod \"e2e-mt-9aed4c03-openshift-default-5845bf9744-t4whw\" (UID: \"bf98d888-f70a-414c-a6b8-60c11d79d76b\") " pod="openshift-ingress/e2e-mt-9aed4c03-openshift-default-5845bf9744-t4whw" Jul 27 19:02:48.504840 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:02:48.504658 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf98d888-f70a-414c-a6b8-60c11d79d76b-istio-envoy\") pod \"e2e-mt-9aed4c03-openshift-default-5845bf9744-t4whw\" (UID: \"bf98d888-f70a-414c-a6b8-60c11d79d76b\") " pod="openshift-ingress/e2e-mt-9aed4c03-openshift-default-5845bf9744-t4whw" Jul 27 19:02:48.504840 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:02:48.504747 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf98d888-f70a-414c-a6b8-60c11d79d76b-istio-token\") pod \"e2e-mt-9aed4c03-openshift-default-5845bf9744-t4whw\" (UID: \"bf98d888-f70a-414c-a6b8-60c11d79d76b\") " pod="openshift-ingress/e2e-mt-9aed4c03-openshift-default-5845bf9744-t4whw" Jul 27 19:02:48.504840 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:02:48.504777 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zp49\" (UniqueName: \"kubernetes.io/projected/bf98d888-f70a-414c-a6b8-60c11d79d76b-kube-api-access-5zp49\") pod \"e2e-mt-9aed4c03-openshift-default-5845bf9744-t4whw\" (UID: \"bf98d888-f70a-414c-a6b8-60c11d79d76b\") " pod="openshift-ingress/e2e-mt-9aed4c03-openshift-default-5845bf9744-t4whw" Jul 27 19:02:48.504840 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:02:48.504804 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf98d888-f70a-414c-a6b8-60c11d79d76b-workload-socket\") pod \"e2e-mt-9aed4c03-openshift-default-5845bf9744-t4whw\" (UID: \"bf98d888-f70a-414c-a6b8-60c11d79d76b\") " pod="openshift-ingress/e2e-mt-9aed4c03-openshift-default-5845bf9744-t4whw" Jul 27 19:02:48.505082 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:02:48.504953 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf98d888-f70a-414c-a6b8-60c11d79d76b-credential-socket\") pod \"e2e-mt-9aed4c03-openshift-default-5845bf9744-t4whw\" (UID: \"bf98d888-f70a-414c-a6b8-60c11d79d76b\") " pod="openshift-ingress/e2e-mt-9aed4c03-openshift-default-5845bf9744-t4whw" Jul 27 19:02:48.505082 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:02:48.505033 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf98d888-f70a-414c-a6b8-60c11d79d76b-istio-podinfo\") pod \"e2e-mt-9aed4c03-openshift-default-5845bf9744-t4whw\" (UID: \"bf98d888-f70a-414c-a6b8-60c11d79d76b\") " pod="openshift-ingress/e2e-mt-9aed4c03-openshift-default-5845bf9744-t4whw" Jul 27 19:02:48.505082 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:02:48.505047 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf98d888-f70a-414c-a6b8-60c11d79d76b-workload-certs\") pod \"e2e-mt-9aed4c03-openshift-default-5845bf9744-t4whw\" (UID: \"bf98d888-f70a-414c-a6b8-60c11d79d76b\") " pod="openshift-ingress/e2e-mt-9aed4c03-openshift-default-5845bf9744-t4whw" Jul 27 19:02:48.505257 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:02:48.505085 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf98d888-f70a-414c-a6b8-60c11d79d76b-istio-data\") pod \"e2e-mt-9aed4c03-openshift-default-5845bf9744-t4whw\" (UID: \"bf98d888-f70a-414c-a6b8-60c11d79d76b\") " pod="openshift-ingress/e2e-mt-9aed4c03-openshift-default-5845bf9744-t4whw" Jul 27 19:02:48.505257 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:02:48.505129 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf98d888-f70a-414c-a6b8-60c11d79d76b-istiod-ca-cert\") pod \"e2e-mt-9aed4c03-openshift-default-5845bf9744-t4whw\" (UID: \"bf98d888-f70a-414c-a6b8-60c11d79d76b\") " pod="openshift-ingress/e2e-mt-9aed4c03-openshift-default-5845bf9744-t4whw" Jul 27 19:02:48.505257 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:02:48.505152 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf98d888-f70a-414c-a6b8-60c11d79d76b-workload-socket\") pod \"e2e-mt-9aed4c03-openshift-default-5845bf9744-t4whw\" (UID: \"bf98d888-f70a-414c-a6b8-60c11d79d76b\") " pod="openshift-ingress/e2e-mt-9aed4c03-openshift-default-5845bf9744-t4whw" Jul 27 19:02:48.505429 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:02:48.505358 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf98d888-f70a-414c-a6b8-60c11d79d76b-credential-socket\") pod \"e2e-mt-9aed4c03-openshift-default-5845bf9744-t4whw\" (UID: \"bf98d888-f70a-414c-a6b8-60c11d79d76b\") " pod="openshift-ingress/e2e-mt-9aed4c03-openshift-default-5845bf9744-t4whw" Jul 27 19:02:48.505429 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:02:48.505399 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf98d888-f70a-414c-a6b8-60c11d79d76b-istio-data\") pod \"e2e-mt-9aed4c03-openshift-default-5845bf9744-t4whw\" (UID: \"bf98d888-f70a-414c-a6b8-60c11d79d76b\") " pod="openshift-ingress/e2e-mt-9aed4c03-openshift-default-5845bf9744-t4whw" Jul 27 19:02:48.505818 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:02:48.505797 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf98d888-f70a-414c-a6b8-60c11d79d76b-istiod-ca-cert\") pod \"e2e-mt-9aed4c03-openshift-default-5845bf9744-t4whw\" (UID: \"bf98d888-f70a-414c-a6b8-60c11d79d76b\") " pod="openshift-ingress/e2e-mt-9aed4c03-openshift-default-5845bf9744-t4whw" Jul 27 19:02:48.507160 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:02:48.507142 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf98d888-f70a-414c-a6b8-60c11d79d76b-istio-envoy\") pod \"e2e-mt-9aed4c03-openshift-default-5845bf9744-t4whw\" (UID: \"bf98d888-f70a-414c-a6b8-60c11d79d76b\") " pod="openshift-ingress/e2e-mt-9aed4c03-openshift-default-5845bf9744-t4whw" Jul 27 19:02:48.507525 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:02:48.507506 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf98d888-f70a-414c-a6b8-60c11d79d76b-istio-podinfo\") pod \"e2e-mt-9aed4c03-openshift-default-5845bf9744-t4whw\" (UID: \"bf98d888-f70a-414c-a6b8-60c11d79d76b\") " pod="openshift-ingress/e2e-mt-9aed4c03-openshift-default-5845bf9744-t4whw" Jul 27 19:02:48.513368 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:02:48.513342 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf98d888-f70a-414c-a6b8-60c11d79d76b-istio-token\") pod \"e2e-mt-9aed4c03-openshift-default-5845bf9744-t4whw\" (UID: \"bf98d888-f70a-414c-a6b8-60c11d79d76b\") " pod="openshift-ingress/e2e-mt-9aed4c03-openshift-default-5845bf9744-t4whw" Jul 27 19:02:48.513484 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:02:48.513447 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zp49\" (UniqueName: \"kubernetes.io/projected/bf98d888-f70a-414c-a6b8-60c11d79d76b-kube-api-access-5zp49\") pod \"e2e-mt-9aed4c03-openshift-default-5845bf9744-t4whw\" (UID: \"bf98d888-f70a-414c-a6b8-60c11d79d76b\") " pod="openshift-ingress/e2e-mt-9aed4c03-openshift-default-5845bf9744-t4whw" Jul 27 19:02:48.618466 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:02:48.618382 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-9aed4c03-openshift-default-5845bf9744-t4whw" Jul 27 19:02:48.773758 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:02:48.773731 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-9aed4c03-openshift-default-5845bf9744-t4whw"] Jul 27 19:02:48.775518 ip-10-0-128-32 kubenswrapper[2579]: W0727 19:02:48.775484 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf98d888_f70a_414c_a6b8_60c11d79d76b.slice/crio-60f5bdfc0bb2bbe668f12c64d4a51e3c52c4e4319e953b317ebc20d96282173f WatchSource:0}: Error finding container 60f5bdfc0bb2bbe668f12c64d4a51e3c52c4e4319e953b317ebc20d96282173f: Status 404 returned error can't find the container with id 60f5bdfc0bb2bbe668f12c64d4a51e3c52c4e4319e953b317ebc20d96282173f Jul 27 19:02:48.777944 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:02:48.777905 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 27 19:02:48.778061 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:02:48.777979 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 27 19:02:48.778061 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:02:48.778033 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 27 19:02:48.820702 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:02:48.820653 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:02:49.616175 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:02:49.616130 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9aed4c03-openshift-default-5845bf9744-t4whw" event={"ID":"bf98d888-f70a-414c-a6b8-60c11d79d76b","Type":"ContainerStarted","Data":"1bc92d1b59948f93add8776555233af8c3f86f1f86bb18f878f6564390e63d3d"} Jul 27 19:02:49.616383 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:02:49.616199 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9aed4c03-openshift-default-5845bf9744-t4whw" event={"ID":"bf98d888-f70a-414c-a6b8-60c11d79d76b","Type":"ContainerStarted","Data":"60f5bdfc0bb2bbe668f12c64d4a51e3c52c4e4319e953b317ebc20d96282173f"} Jul 27 19:02:49.619491 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:02:49.619469 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-9aed4c03-openshift-default-5845bf9744-t4whw" Jul 27 19:02:49.625269 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:02:49.625223 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-9aed4c03-openshift-default-5845bf9744-t4whw" Jul 27 19:02:49.642639 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:02:49.642587 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-9aed4c03-openshift-default-5845bf9744-t4whw" podStartSLOduration=1.6425739639999999 podStartE2EDuration="1.642573964s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:02:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:02:49.640610126 +0000 UTC m=+2044.702299195" watchObservedRunningTime="2026-07-27 19:02:49.642573964 +0000 UTC m=+2044.704262995" Jul 27 19:02:50.621175 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:02:50.621141 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-9aed4c03-openshift-default-5845bf9744-t4whw" Jul 27 19:02:50.622331 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:02:50.622307 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-9aed4c03-openshift-default-5845bf9744-t4whw" Jul 27 19:02:51.649907 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:02:51.649843 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:02:51.726992 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:02:51.726947 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:02:51.798846 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:02:51.798800 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:02:51.896073 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:02:51.896042 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:02:52.417582 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:02:52.417545 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:02:52.518818 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:02:52.518780 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:02:56.182160 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:02:56.182110 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:02:56.883941 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:02:56.883899 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:02:56.984760 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:02:56.984726 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:02:57.324546 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:02:57.324507 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:03:02.694810 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:03:02.694705 2579 generic.go:356] "Generic (PLEG): container finished" podID="bf98d888-f70a-414c-a6b8-60c11d79d76b" containerID="1bc92d1b59948f93add8776555233af8c3f86f1f86bb18f878f6564390e63d3d" exitCode=0 Jul 27 19:03:02.694810 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:03:02.694776 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9aed4c03-openshift-default-5845bf9744-t4whw" event={"ID":"bf98d888-f70a-414c-a6b8-60c11d79d76b","Type":"ContainerDied","Data":"1bc92d1b59948f93add8776555233af8c3f86f1f86bb18f878f6564390e63d3d"} Jul 27 19:03:02.695338 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:03:02.695320 2579 scope.go:117] "RemoveContainer" containerID="1bc92d1b59948f93add8776555233af8c3f86f1f86bb18f878f6564390e63d3d" Jul 27 19:03:02.696605 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:03:02.696570 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 27 19:03:02.696735 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:03:02.696663 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 27 19:03:02.696804 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:03:02.696751 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 27 19:03:03.700619 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:03:03.700549 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:03:03.701755 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:03:03.701729 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9aed4c03-openshift-default-5845bf9744-t4whw" event={"ID":"bf98d888-f70a-414c-a6b8-60c11d79d76b","Type":"ContainerStarted","Data":"4bc26bd1a2a8428b61ba34e4f1f82137e32be08cbc797ae8ca90d0b0e703bea6"} Jul 27 19:03:03.762571 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:03:03.762540 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:03:04.618736 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:03:04.618704 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-9aed4c03-openshift-default-5845bf9744-t4whw" Jul 27 19:03:04.619920 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:03:04.619891 2579 patch_prober.go:28] interesting pod/e2e-mt-9aed4c03-openshift-default-5845bf9744-t4whw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Jul 27 19:03:04.620046 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:03:04.619941 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9aed4c03-openshift-default-5845bf9744-t4whw" podUID="bf98d888-f70a-414c-a6b8-60c11d79d76b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Jul 27 19:03:05.619606 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:03:05.619510 2579 patch_prober.go:28] interesting pod/e2e-mt-9aed4c03-openshift-default-5845bf9744-t4whw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Jul 27 19:03:05.619606 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:03:05.619570 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9aed4c03-openshift-default-5845bf9744-t4whw" podUID="bf98d888-f70a-414c-a6b8-60c11d79d76b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Jul 27 19:03:05.621738 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:03:05.621658 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-9aed4c03-openshift-default-5845bf9744-t4whw" Jul 27 19:03:06.619466 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:03:06.619427 2579 patch_prober.go:28] interesting pod/e2e-mt-9aed4c03-openshift-default-5845bf9744-t4whw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Jul 27 19:03:06.619687 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:03:06.619512 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9aed4c03-openshift-default-5845bf9744-t4whw" podUID="bf98d888-f70a-414c-a6b8-60c11d79d76b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Jul 27 19:03:07.622981 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:03:07.622950 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-9aed4c03-openshift-default-5845bf9744-t4whw" Jul 27 19:03:07.624095 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:03:07.624070 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-9aed4c03-openshift-default-5845bf9744-t4whw" Jul 27 19:03:14.489970 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:03:14.489936 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:03:14.557674 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:03:14.557638 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:03:15.131349 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:03:15.131309 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:03:19.153896 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:03:19.153858 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:03:19.525291 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:03:19.525257 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:03:25.530406 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:03:25.530368 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:03:25.591009 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:03:25.590975 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:03:25.804369 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:03:25.804286 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-9aed4c03-openshift-default-5845bf9744-t4whw"] Jul 27 19:03:25.804578 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:03:25.804552 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-9aed4c03-openshift-default-5845bf9744-t4whw" podUID="bf98d888-f70a-414c-a6b8-60c11d79d76b" containerName="istio-proxy" containerID="cri-o://4bc26bd1a2a8428b61ba34e4f1f82137e32be08cbc797ae8ca90d0b0e703bea6" gracePeriod=30 Jul 27 19:03:26.743622 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:03:26.743585 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:03:26.834841 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:03:26.834797 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:03:31.053089 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:03:31.053062 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-9aed4c03-openshift-default-5845bf9744-t4whw" Jul 27 19:03:31.117369 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:03:31.117281 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf98d888-f70a-414c-a6b8-60c11d79d76b-istio-envoy\") pod \"bf98d888-f70a-414c-a6b8-60c11d79d76b\" (UID: \"bf98d888-f70a-414c-a6b8-60c11d79d76b\") " Jul 27 19:03:31.117369 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:03:31.117314 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf98d888-f70a-414c-a6b8-60c11d79d76b-credential-socket\") pod \"bf98d888-f70a-414c-a6b8-60c11d79d76b\" (UID: \"bf98d888-f70a-414c-a6b8-60c11d79d76b\") " Jul 27 19:03:31.117369 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:03:31.117336 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5zp49\" (UniqueName: \"kubernetes.io/projected/bf98d888-f70a-414c-a6b8-60c11d79d76b-kube-api-access-5zp49\") pod \"bf98d888-f70a-414c-a6b8-60c11d79d76b\" (UID: \"bf98d888-f70a-414c-a6b8-60c11d79d76b\") " Jul 27 19:03:31.117369 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:03:31.117353 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf98d888-f70a-414c-a6b8-60c11d79d76b-istio-podinfo\") pod \"bf98d888-f70a-414c-a6b8-60c11d79d76b\" (UID: \"bf98d888-f70a-414c-a6b8-60c11d79d76b\") " Jul 27 19:03:31.117696 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:03:31.117375 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf98d888-f70a-414c-a6b8-60c11d79d76b-istio-token\") pod \"bf98d888-f70a-414c-a6b8-60c11d79d76b\" (UID: \"bf98d888-f70a-414c-a6b8-60c11d79d76b\") " Jul 27 19:03:31.117696 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:03:31.117551 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf98d888-f70a-414c-a6b8-60c11d79d76b-istiod-ca-cert\") pod \"bf98d888-f70a-414c-a6b8-60c11d79d76b\" (UID: \"bf98d888-f70a-414c-a6b8-60c11d79d76b\") " Jul 27 19:03:31.117696 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:03:31.117604 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf98d888-f70a-414c-a6b8-60c11d79d76b-workload-socket\") pod \"bf98d888-f70a-414c-a6b8-60c11d79d76b\" (UID: \"bf98d888-f70a-414c-a6b8-60c11d79d76b\") " Jul 27 19:03:31.117696 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:03:31.117614 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf98d888-f70a-414c-a6b8-60c11d79d76b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "bf98d888-f70a-414c-a6b8-60c11d79d76b" (UID: "bf98d888-f70a-414c-a6b8-60c11d79d76b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 19:03:31.117696 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:03:31.117632 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf98d888-f70a-414c-a6b8-60c11d79d76b-workload-certs\") pod \"bf98d888-f70a-414c-a6b8-60c11d79d76b\" (UID: \"bf98d888-f70a-414c-a6b8-60c11d79d76b\") " Jul 27 19:03:31.117890 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:03:31.117725 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf98d888-f70a-414c-a6b8-60c11d79d76b-istio-data\") pod \"bf98d888-f70a-414c-a6b8-60c11d79d76b\" (UID: \"bf98d888-f70a-414c-a6b8-60c11d79d76b\") " Jul 27 19:03:31.117890 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:03:31.117811 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf98d888-f70a-414c-a6b8-60c11d79d76b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "bf98d888-f70a-414c-a6b8-60c11d79d76b" (UID: "bf98d888-f70a-414c-a6b8-60c11d79d76b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 19:03:31.118028 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:03:31.117995 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf98d888-f70a-414c-a6b8-60c11d79d76b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "bf98d888-f70a-414c-a6b8-60c11d79d76b" (UID: "bf98d888-f70a-414c-a6b8-60c11d79d76b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 19:03:31.118115 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:03:31.118097 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf98d888-f70a-414c-a6b8-60c11d79d76b-workload-certs\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 19:03:31.118176 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:03:31.118122 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf98d888-f70a-414c-a6b8-60c11d79d76b-credential-socket\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 19:03:31.118176 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:03:31.118139 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf98d888-f70a-414c-a6b8-60c11d79d76b-workload-socket\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 19:03:31.118539 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:03:31.118504 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf98d888-f70a-414c-a6b8-60c11d79d76b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "bf98d888-f70a-414c-a6b8-60c11d79d76b" (UID: "bf98d888-f70a-414c-a6b8-60c11d79d76b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 19:03:31.118598 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:03:31.118538 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf98d888-f70a-414c-a6b8-60c11d79d76b-istio-data" (OuterVolumeSpecName: "istio-data") pod "bf98d888-f70a-414c-a6b8-60c11d79d76b" (UID: "bf98d888-f70a-414c-a6b8-60c11d79d76b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 19:03:31.119902 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:03:31.119871 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bf98d888-f70a-414c-a6b8-60c11d79d76b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "bf98d888-f70a-414c-a6b8-60c11d79d76b" (UID: "bf98d888-f70a-414c-a6b8-60c11d79d76b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 27 19:03:31.119902 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:03:31.119884 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf98d888-f70a-414c-a6b8-60c11d79d76b-kube-api-access-5zp49" (OuterVolumeSpecName: "kube-api-access-5zp49") pod "bf98d888-f70a-414c-a6b8-60c11d79d76b" (UID: "bf98d888-f70a-414c-a6b8-60c11d79d76b"). InnerVolumeSpecName "kube-api-access-5zp49". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 19:03:31.120270 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:03:31.120247 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf98d888-f70a-414c-a6b8-60c11d79d76b-istio-token" (OuterVolumeSpecName: "istio-token") pod "bf98d888-f70a-414c-a6b8-60c11d79d76b" (UID: "bf98d888-f70a-414c-a6b8-60c11d79d76b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 19:03:31.120270 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:03:31.120223 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf98d888-f70a-414c-a6b8-60c11d79d76b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "bf98d888-f70a-414c-a6b8-60c11d79d76b" (UID: "bf98d888-f70a-414c-a6b8-60c11d79d76b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 19:03:31.218602 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:03:31.218560 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf98d888-f70a-414c-a6b8-60c11d79d76b-istio-data\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 19:03:31.218602 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:03:31.218598 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf98d888-f70a-414c-a6b8-60c11d79d76b-istio-envoy\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 19:03:31.218602 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:03:31.218611 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zp49\" (UniqueName: \"kubernetes.io/projected/bf98d888-f70a-414c-a6b8-60c11d79d76b-kube-api-access-5zp49\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 19:03:31.218828 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:03:31.218622 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf98d888-f70a-414c-a6b8-60c11d79d76b-istio-podinfo\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 19:03:31.218828 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:03:31.218630 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf98d888-f70a-414c-a6b8-60c11d79d76b-istio-token\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 19:03:31.218828 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:03:31.218637 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf98d888-f70a-414c-a6b8-60c11d79d76b-istiod-ca-cert\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 19:03:31.845194 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:03:31.845159 2579 generic.go:356] "Generic (PLEG): container finished" podID="bf98d888-f70a-414c-a6b8-60c11d79d76b" containerID="4bc26bd1a2a8428b61ba34e4f1f82137e32be08cbc797ae8ca90d0b0e703bea6" exitCode=0 Jul 27 19:03:31.845194 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:03:31.845198 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9aed4c03-openshift-default-5845bf9744-t4whw" event={"ID":"bf98d888-f70a-414c-a6b8-60c11d79d76b","Type":"ContainerDied","Data":"4bc26bd1a2a8428b61ba34e4f1f82137e32be08cbc797ae8ca90d0b0e703bea6"} Jul 27 19:03:31.845444 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:03:31.845215 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-9aed4c03-openshift-default-5845bf9744-t4whw" Jul 27 19:03:31.845444 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:03:31.845264 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9aed4c03-openshift-default-5845bf9744-t4whw" event={"ID":"bf98d888-f70a-414c-a6b8-60c11d79d76b","Type":"ContainerDied","Data":"60f5bdfc0bb2bbe668f12c64d4a51e3c52c4e4319e953b317ebc20d96282173f"} Jul 27 19:03:31.845444 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:03:31.845289 2579 scope.go:117] "RemoveContainer" containerID="4bc26bd1a2a8428b61ba34e4f1f82137e32be08cbc797ae8ca90d0b0e703bea6" Jul 27 19:03:31.854298 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:03:31.854280 2579 scope.go:117] "RemoveContainer" containerID="1bc92d1b59948f93add8776555233af8c3f86f1f86bb18f878f6564390e63d3d" Jul 27 19:03:31.862047 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:03:31.862015 2579 scope.go:117] "RemoveContainer" containerID="4bc26bd1a2a8428b61ba34e4f1f82137e32be08cbc797ae8ca90d0b0e703bea6" Jul 27 19:03:31.862360 ip-10-0-128-32 kubenswrapper[2579]: E0727 19:03:31.862336 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4bc26bd1a2a8428b61ba34e4f1f82137e32be08cbc797ae8ca90d0b0e703bea6\": container with ID starting with 4bc26bd1a2a8428b61ba34e4f1f82137e32be08cbc797ae8ca90d0b0e703bea6 not found: ID does not exist" containerID="4bc26bd1a2a8428b61ba34e4f1f82137e32be08cbc797ae8ca90d0b0e703bea6" Jul 27 19:03:31.862423 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:03:31.862371 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bc26bd1a2a8428b61ba34e4f1f82137e32be08cbc797ae8ca90d0b0e703bea6"} err="failed to get container status \"4bc26bd1a2a8428b61ba34e4f1f82137e32be08cbc797ae8ca90d0b0e703bea6\": rpc error: code = NotFound desc = could not find container \"4bc26bd1a2a8428b61ba34e4f1f82137e32be08cbc797ae8ca90d0b0e703bea6\": container with ID starting with 4bc26bd1a2a8428b61ba34e4f1f82137e32be08cbc797ae8ca90d0b0e703bea6 not found: ID does not exist" Jul 27 19:03:31.862423 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:03:31.862395 2579 scope.go:117] "RemoveContainer" containerID="1bc92d1b59948f93add8776555233af8c3f86f1f86bb18f878f6564390e63d3d" Jul 27 19:03:31.862655 ip-10-0-128-32 kubenswrapper[2579]: E0727 19:03:31.862628 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bc92d1b59948f93add8776555233af8c3f86f1f86bb18f878f6564390e63d3d\": container with ID starting with 1bc92d1b59948f93add8776555233af8c3f86f1f86bb18f878f6564390e63d3d not found: ID does not exist" containerID="1bc92d1b59948f93add8776555233af8c3f86f1f86bb18f878f6564390e63d3d" Jul 27 19:03:31.862727 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:03:31.862670 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bc92d1b59948f93add8776555233af8c3f86f1f86bb18f878f6564390e63d3d"} err="failed to get container status \"1bc92d1b59948f93add8776555233af8c3f86f1f86bb18f878f6564390e63d3d\": rpc error: code = NotFound desc = could not find container \"1bc92d1b59948f93add8776555233af8c3f86f1f86bb18f878f6564390e63d3d\": container with ID starting with 1bc92d1b59948f93add8776555233af8c3f86f1f86bb18f878f6564390e63d3d not found: ID does not exist" Jul 27 19:03:31.865182 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:03:31.865155 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-9aed4c03-openshift-default-5845bf9744-t4whw"] Jul 27 19:03:31.871090 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:03:31.871067 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-9aed4c03-openshift-default-5845bf9744-t4whw"] Jul 27 19:03:32.660596 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:03:32.660567 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:03:32.765215 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:03:32.765174 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:03:32.860535 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:03:32.860504 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:03:33.126947 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:03:33.126909 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:03:33.494118 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:03:33.494035 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bf98d888-f70a-414c-a6b8-60c11d79d76b" path="/var/lib/kubelet/pods/bf98d888-f70a-414c-a6b8-60c11d79d76b/volumes" Jul 27 19:03:39.348269 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:03:39.348219 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:03:39.446658 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:03:39.446624 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:03:45.556574 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:03:45.556542 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-mv4hb_715b2084-d79e-4cbf-a057-f2595b4e6b9b/console-operator/1.log" Jul 27 19:03:45.562424 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:03:45.562402 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-mv4hb_715b2084-d79e-4cbf-a057-f2595b4e6b9b/console-operator/1.log" Jul 27 19:03:46.220988 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:03:46.220953 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:03:46.491979 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:03:46.491904 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-fc9cd50f-openshift-default-5845ddbbd4-2hc7b"] Jul 27 19:03:46.492425 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:03:46.492407 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf98d888-f70a-414c-a6b8-60c11d79d76b" containerName="istio-proxy" Jul 27 19:03:46.492512 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:03:46.492426 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf98d888-f70a-414c-a6b8-60c11d79d76b" containerName="istio-proxy" Jul 27 19:03:46.492550 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:03:46.492520 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf98d888-f70a-414c-a6b8-60c11d79d76b" containerName="istio-proxy" Jul 27 19:03:46.492550 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:03:46.492528 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="b302badc-8586-4808-b8d6-3e77e5588803" containerName="istio-proxy" Jul 27 19:03:46.492550 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:03:46.492537 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf98d888-f70a-414c-a6b8-60c11d79d76b" containerName="istio-proxy" Jul 27 19:03:46.497153 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:03:46.497135 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-fc9cd50f-openshift-default-5845ddbbd4-2hc7b" Jul 27 19:03:46.500147 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:03:46.500125 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-fc9cd50f-openshift-default-dockercfg-grclp\"" Jul 27 19:03:46.506550 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:03:46.506525 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-fc9cd50f-openshift-default-5845ddbbd4-2hc7b"] Jul 27 19:03:46.521023 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:03:46.520999 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:03:46.557013 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:03:46.556979 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11107304-1eee-4798-9c56-eea3c514d384-istio-token\") pod \"e2e-mt-fc9cd50f-openshift-default-5845ddbbd4-2hc7b\" (UID: \"11107304-1eee-4798-9c56-eea3c514d384\") " pod="openshift-ingress/e2e-mt-fc9cd50f-openshift-default-5845ddbbd4-2hc7b" Jul 27 19:03:46.557588 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:03:46.557024 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11107304-1eee-4798-9c56-eea3c514d384-istio-envoy\") pod \"e2e-mt-fc9cd50f-openshift-default-5845ddbbd4-2hc7b\" (UID: \"11107304-1eee-4798-9c56-eea3c514d384\") " pod="openshift-ingress/e2e-mt-fc9cd50f-openshift-default-5845ddbbd4-2hc7b" Jul 27 19:03:46.557588 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:03:46.557116 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26ttt\" (UniqueName: \"kubernetes.io/projected/11107304-1eee-4798-9c56-eea3c514d384-kube-api-access-26ttt\") pod \"e2e-mt-fc9cd50f-openshift-default-5845ddbbd4-2hc7b\" (UID: \"11107304-1eee-4798-9c56-eea3c514d384\") " pod="openshift-ingress/e2e-mt-fc9cd50f-openshift-default-5845ddbbd4-2hc7b" Jul 27 19:03:46.557588 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:03:46.557149 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11107304-1eee-4798-9c56-eea3c514d384-istiod-ca-cert\") pod \"e2e-mt-fc9cd50f-openshift-default-5845ddbbd4-2hc7b\" (UID: \"11107304-1eee-4798-9c56-eea3c514d384\") " pod="openshift-ingress/e2e-mt-fc9cd50f-openshift-default-5845ddbbd4-2hc7b" Jul 27 19:03:46.557588 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:03:46.557177 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11107304-1eee-4798-9c56-eea3c514d384-workload-certs\") pod \"e2e-mt-fc9cd50f-openshift-default-5845ddbbd4-2hc7b\" (UID: \"11107304-1eee-4798-9c56-eea3c514d384\") " pod="openshift-ingress/e2e-mt-fc9cd50f-openshift-default-5845ddbbd4-2hc7b" Jul 27 19:03:46.557588 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:03:46.557202 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11107304-1eee-4798-9c56-eea3c514d384-credential-socket\") pod \"e2e-mt-fc9cd50f-openshift-default-5845ddbbd4-2hc7b\" (UID: \"11107304-1eee-4798-9c56-eea3c514d384\") " pod="openshift-ingress/e2e-mt-fc9cd50f-openshift-default-5845ddbbd4-2hc7b" Jul 27 19:03:46.557588 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:03:46.557308 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11107304-1eee-4798-9c56-eea3c514d384-istio-data\") pod \"e2e-mt-fc9cd50f-openshift-default-5845ddbbd4-2hc7b\" (UID: \"11107304-1eee-4798-9c56-eea3c514d384\") " pod="openshift-ingress/e2e-mt-fc9cd50f-openshift-default-5845ddbbd4-2hc7b" Jul 27 19:03:46.557588 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:03:46.557400 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11107304-1eee-4798-9c56-eea3c514d384-workload-socket\") pod \"e2e-mt-fc9cd50f-openshift-default-5845ddbbd4-2hc7b\" (UID: \"11107304-1eee-4798-9c56-eea3c514d384\") " pod="openshift-ingress/e2e-mt-fc9cd50f-openshift-default-5845ddbbd4-2hc7b" Jul 27 19:03:46.557588 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:03:46.557439 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11107304-1eee-4798-9c56-eea3c514d384-istio-podinfo\") pod \"e2e-mt-fc9cd50f-openshift-default-5845ddbbd4-2hc7b\" (UID: \"11107304-1eee-4798-9c56-eea3c514d384\") " pod="openshift-ingress/e2e-mt-fc9cd50f-openshift-default-5845ddbbd4-2hc7b" Jul 27 19:03:46.621763 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:03:46.621736 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:03:46.659086 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:03:46.659012 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26ttt\" (UniqueName: \"kubernetes.io/projected/11107304-1eee-4798-9c56-eea3c514d384-kube-api-access-26ttt\") pod \"e2e-mt-fc9cd50f-openshift-default-5845ddbbd4-2hc7b\" (UID: \"11107304-1eee-4798-9c56-eea3c514d384\") " pod="openshift-ingress/e2e-mt-fc9cd50f-openshift-default-5845ddbbd4-2hc7b" Jul 27 19:03:46.659086 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:03:46.659071 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11107304-1eee-4798-9c56-eea3c514d384-istiod-ca-cert\") pod \"e2e-mt-fc9cd50f-openshift-default-5845ddbbd4-2hc7b\" (UID: \"11107304-1eee-4798-9c56-eea3c514d384\") " pod="openshift-ingress/e2e-mt-fc9cd50f-openshift-default-5845ddbbd4-2hc7b" Jul 27 19:03:46.659373 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:03:46.659107 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11107304-1eee-4798-9c56-eea3c514d384-workload-certs\") pod \"e2e-mt-fc9cd50f-openshift-default-5845ddbbd4-2hc7b\" (UID: \"11107304-1eee-4798-9c56-eea3c514d384\") " pod="openshift-ingress/e2e-mt-fc9cd50f-openshift-default-5845ddbbd4-2hc7b" Jul 27 19:03:46.659373 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:03:46.659135 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11107304-1eee-4798-9c56-eea3c514d384-credential-socket\") pod \"e2e-mt-fc9cd50f-openshift-default-5845ddbbd4-2hc7b\" (UID: \"11107304-1eee-4798-9c56-eea3c514d384\") " pod="openshift-ingress/e2e-mt-fc9cd50f-openshift-default-5845ddbbd4-2hc7b" Jul 27 19:03:46.659373 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:03:46.659167 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11107304-1eee-4798-9c56-eea3c514d384-istio-data\") pod \"e2e-mt-fc9cd50f-openshift-default-5845ddbbd4-2hc7b\" (UID: \"11107304-1eee-4798-9c56-eea3c514d384\") " pod="openshift-ingress/e2e-mt-fc9cd50f-openshift-default-5845ddbbd4-2hc7b" Jul 27 19:03:46.659542 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:03:46.659518 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11107304-1eee-4798-9c56-eea3c514d384-workload-socket\") pod \"e2e-mt-fc9cd50f-openshift-default-5845ddbbd4-2hc7b\" (UID: \"11107304-1eee-4798-9c56-eea3c514d384\") " pod="openshift-ingress/e2e-mt-fc9cd50f-openshift-default-5845ddbbd4-2hc7b" Jul 27 19:03:46.659600 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:03:46.659575 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11107304-1eee-4798-9c56-eea3c514d384-istio-podinfo\") pod \"e2e-mt-fc9cd50f-openshift-default-5845ddbbd4-2hc7b\" (UID: \"11107304-1eee-4798-9c56-eea3c514d384\") " pod="openshift-ingress/e2e-mt-fc9cd50f-openshift-default-5845ddbbd4-2hc7b" Jul 27 19:03:46.659653 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:03:46.659628 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11107304-1eee-4798-9c56-eea3c514d384-workload-certs\") pod \"e2e-mt-fc9cd50f-openshift-default-5845ddbbd4-2hc7b\" (UID: \"11107304-1eee-4798-9c56-eea3c514d384\") " pod="openshift-ingress/e2e-mt-fc9cd50f-openshift-default-5845ddbbd4-2hc7b" Jul 27 19:03:46.659706 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:03:46.659658 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11107304-1eee-4798-9c56-eea3c514d384-istio-token\") pod \"e2e-mt-fc9cd50f-openshift-default-5845ddbbd4-2hc7b\" (UID: \"11107304-1eee-4798-9c56-eea3c514d384\") " pod="openshift-ingress/e2e-mt-fc9cd50f-openshift-default-5845ddbbd4-2hc7b" Jul 27 19:03:46.659763 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:03:46.659704 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11107304-1eee-4798-9c56-eea3c514d384-istio-envoy\") pod \"e2e-mt-fc9cd50f-openshift-default-5845ddbbd4-2hc7b\" (UID: \"11107304-1eee-4798-9c56-eea3c514d384\") " pod="openshift-ingress/e2e-mt-fc9cd50f-openshift-default-5845ddbbd4-2hc7b" Jul 27 19:03:46.659763 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:03:46.659736 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11107304-1eee-4798-9c56-eea3c514d384-credential-socket\") pod \"e2e-mt-fc9cd50f-openshift-default-5845ddbbd4-2hc7b\" (UID: \"11107304-1eee-4798-9c56-eea3c514d384\") " pod="openshift-ingress/e2e-mt-fc9cd50f-openshift-default-5845ddbbd4-2hc7b" Jul 27 19:03:46.659862 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:03:46.659839 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11107304-1eee-4798-9c56-eea3c514d384-workload-socket\") pod \"e2e-mt-fc9cd50f-openshift-default-5845ddbbd4-2hc7b\" (UID: \"11107304-1eee-4798-9c56-eea3c514d384\") " pod="openshift-ingress/e2e-mt-fc9cd50f-openshift-default-5845ddbbd4-2hc7b" Jul 27 19:03:46.660080 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:03:46.660055 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11107304-1eee-4798-9c56-eea3c514d384-istiod-ca-cert\") pod \"e2e-mt-fc9cd50f-openshift-default-5845ddbbd4-2hc7b\" (UID: \"11107304-1eee-4798-9c56-eea3c514d384\") " pod="openshift-ingress/e2e-mt-fc9cd50f-openshift-default-5845ddbbd4-2hc7b" Jul 27 19:03:46.660189 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:03:46.660142 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11107304-1eee-4798-9c56-eea3c514d384-istio-data\") pod \"e2e-mt-fc9cd50f-openshift-default-5845ddbbd4-2hc7b\" (UID: \"11107304-1eee-4798-9c56-eea3c514d384\") " pod="openshift-ingress/e2e-mt-fc9cd50f-openshift-default-5845ddbbd4-2hc7b" Jul 27 19:03:46.662270 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:03:46.662249 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11107304-1eee-4798-9c56-eea3c514d384-istio-envoy\") pod \"e2e-mt-fc9cd50f-openshift-default-5845ddbbd4-2hc7b\" (UID: \"11107304-1eee-4798-9c56-eea3c514d384\") " pod="openshift-ingress/e2e-mt-fc9cd50f-openshift-default-5845ddbbd4-2hc7b" Jul 27 19:03:46.662792 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:03:46.662763 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11107304-1eee-4798-9c56-eea3c514d384-istio-podinfo\") pod \"e2e-mt-fc9cd50f-openshift-default-5845ddbbd4-2hc7b\" (UID: \"11107304-1eee-4798-9c56-eea3c514d384\") " pod="openshift-ingress/e2e-mt-fc9cd50f-openshift-default-5845ddbbd4-2hc7b" Jul 27 19:03:46.668363 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:03:46.668340 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-26ttt\" (UniqueName: \"kubernetes.io/projected/11107304-1eee-4798-9c56-eea3c514d384-kube-api-access-26ttt\") pod \"e2e-mt-fc9cd50f-openshift-default-5845ddbbd4-2hc7b\" (UID: \"11107304-1eee-4798-9c56-eea3c514d384\") " pod="openshift-ingress/e2e-mt-fc9cd50f-openshift-default-5845ddbbd4-2hc7b" Jul 27 19:03:46.668478 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:03:46.668450 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11107304-1eee-4798-9c56-eea3c514d384-istio-token\") pod \"e2e-mt-fc9cd50f-openshift-default-5845ddbbd4-2hc7b\" (UID: \"11107304-1eee-4798-9c56-eea3c514d384\") " pod="openshift-ingress/e2e-mt-fc9cd50f-openshift-default-5845ddbbd4-2hc7b" Jul 27 19:03:46.807892 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:03:46.807859 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-fc9cd50f-openshift-default-5845ddbbd4-2hc7b" Jul 27 19:03:46.957837 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:03:46.957809 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-fc9cd50f-openshift-default-5845ddbbd4-2hc7b"] Jul 27 19:03:46.959881 ip-10-0-128-32 kubenswrapper[2579]: W0727 19:03:46.959845 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11107304_1eee_4798_9c56_eea3c514d384.slice/crio-e9b76fae283f4a77569992917970b7e2ff2b9a4b5fa6cd9b2f191c286523daf7 WatchSource:0}: Error finding container e9b76fae283f4a77569992917970b7e2ff2b9a4b5fa6cd9b2f191c286523daf7: Status 404 returned error can't find the container with id e9b76fae283f4a77569992917970b7e2ff2b9a4b5fa6cd9b2f191c286523daf7 Jul 27 19:03:46.962169 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:03:46.962134 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 27 19:03:46.962317 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:03:46.962204 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 27 19:03:46.962317 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:03:46.962263 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 27 19:03:47.927588 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:03:47.927556 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-fc9cd50f-openshift-default-5845ddbbd4-2hc7b" event={"ID":"11107304-1eee-4798-9c56-eea3c514d384","Type":"ContainerStarted","Data":"23cb89bac2dfb4c9e0ca40f49607f7944919e9d2a1343a7470e5923173b24255"} Jul 27 19:03:47.928005 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:03:47.927610 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-fc9cd50f-openshift-default-5845ddbbd4-2hc7b" event={"ID":"11107304-1eee-4798-9c56-eea3c514d384","Type":"ContainerStarted","Data":"e9b76fae283f4a77569992917970b7e2ff2b9a4b5fa6cd9b2f191c286523daf7"} Jul 27 19:03:47.950124 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:03:47.950071 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-fc9cd50f-openshift-default-5845ddbbd4-2hc7b" podStartSLOduration=1.950053741 podStartE2EDuration="1.950053741s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:03:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:03:47.947806832 +0000 UTC m=+2103.009495864" watchObservedRunningTime="2026-07-27 19:03:47.950053741 +0000 UTC m=+2103.011742772" Jul 27 19:03:48.808609 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:03:48.808572 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-fc9cd50f-openshift-default-5845ddbbd4-2hc7b" Jul 27 19:03:48.813793 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:03:48.813767 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-fc9cd50f-openshift-default-5845ddbbd4-2hc7b" Jul 27 19:03:48.933318 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:03:48.933289 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-fc9cd50f-openshift-default-5845ddbbd4-2hc7b" Jul 27 19:03:48.934391 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:03:48.934369 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-fc9cd50f-openshift-default-5845ddbbd4-2hc7b" Jul 27 19:03:49.908445 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:03:49.908403 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:03:49.989997 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:03:49.989964 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:03:50.064768 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:03:50.064730 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:03:50.218426 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:03:50.218343 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:03:50.587932 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:03:50.587899 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:03:50.692607 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:03:50.692571 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:03:54.413326 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:03:54.413286 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:04:04.693495 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:04:04.693455 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:04:04.916027 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:04:04.915994 2579 patch_prober.go:28] interesting pod/e2e-mt-fc9cd50f-openshift-default-5845ddbbd4-2hc7b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": read tcp 10.132.0.2:57992->10.132.0.62:15021: read: connection reset by peer" start-of-body= Jul 27 19:04:04.916364 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:04:04.916080 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-fc9cd50f-openshift-default-5845ddbbd4-2hc7b" podUID="11107304-1eee-4798-9c56-eea3c514d384" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": read tcp 10.132.0.2:57992->10.132.0.62:15021: read: connection reset by peer" Jul 27 19:04:05.019181 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:04:05.019102 2579 generic.go:356] "Generic (PLEG): container finished" podID="11107304-1eee-4798-9c56-eea3c514d384" containerID="23cb89bac2dfb4c9e0ca40f49607f7944919e9d2a1343a7470e5923173b24255" exitCode=0 Jul 27 19:04:05.019181 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:04:05.019169 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-fc9cd50f-openshift-default-5845ddbbd4-2hc7b" event={"ID":"11107304-1eee-4798-9c56-eea3c514d384","Type":"ContainerDied","Data":"23cb89bac2dfb4c9e0ca40f49607f7944919e9d2a1343a7470e5923173b24255"} Jul 27 19:04:05.019595 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:04:05.019582 2579 scope.go:117] "RemoveContainer" containerID="23cb89bac2dfb4c9e0ca40f49607f7944919e9d2a1343a7470e5923173b24255" Jul 27 19:04:05.020691 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:04:05.020666 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 27 19:04:05.020773 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:04:05.020731 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 27 19:04:05.020773 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:04:05.020762 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 27 19:04:06.029713 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:04:06.029679 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-fc9cd50f-openshift-default-5845ddbbd4-2hc7b" event={"ID":"11107304-1eee-4798-9c56-eea3c514d384","Type":"ContainerStarted","Data":"c589122348cd1c98fd65fe87fbcd842f863c5155f48ed8b99531c8bd20ffc5e7"} Jul 27 19:04:06.808989 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:04:06.808943 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-fc9cd50f-openshift-default-5845ddbbd4-2hc7b" Jul 27 19:04:06.809255 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:04:06.809205 2579 patch_prober.go:28] interesting pod/e2e-mt-fc9cd50f-openshift-default-5845ddbbd4-2hc7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Jul 27 19:04:06.809372 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:04:06.809305 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-fc9cd50f-openshift-default-5845ddbbd4-2hc7b" podUID="11107304-1eee-4798-9c56-eea3c514d384" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Jul 27 19:04:07.808666 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:04:07.808629 2579 patch_prober.go:28] interesting pod/e2e-mt-fc9cd50f-openshift-default-5845ddbbd4-2hc7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Jul 27 19:04:07.809048 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:04:07.808701 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-fc9cd50f-openshift-default-5845ddbbd4-2hc7b" podUID="11107304-1eee-4798-9c56-eea3c514d384" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Jul 27 19:04:08.809127 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:04:08.809093 2579 patch_prober.go:28] interesting pod/e2e-mt-fc9cd50f-openshift-default-5845ddbbd4-2hc7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Jul 27 19:04:08.809526 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:04:08.809152 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-fc9cd50f-openshift-default-5845ddbbd4-2hc7b" podUID="11107304-1eee-4798-9c56-eea3c514d384" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Jul 27 19:04:09.808459 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:04:09.808422 2579 patch_prober.go:28] interesting pod/e2e-mt-fc9cd50f-openshift-default-5845ddbbd4-2hc7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Jul 27 19:04:09.808642 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:04:09.808480 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-fc9cd50f-openshift-default-5845ddbbd4-2hc7b" podUID="11107304-1eee-4798-9c56-eea3c514d384" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Jul 27 19:04:10.812223 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:04:10.812193 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-fc9cd50f-openshift-default-5845ddbbd4-2hc7b" Jul 27 19:04:10.812720 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:04:10.812696 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-fc9cd50f-openshift-default-5845ddbbd4-2hc7b" Jul 27 19:04:10.813492 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:04:10.813471 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-fc9cd50f-openshift-default-5845ddbbd4-2hc7b" Jul 27 19:04:19.907019 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:04:19.906977 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:04:19.963709 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:04:19.963678 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:06:44.088810 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:06:44.088769 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:07:43.238930 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:07:43.238888 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:07:44.776500 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:07:44.776455 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:07:47.959501 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:07:47.959459 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:07:48.265697 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:07:48.265603 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:07:53.489885 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:07:53.489845 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:08:30.004831 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:08:30.004794 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:08:31.332425 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:08:31.332385 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:08:45.594821 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:08:45.594793 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-mv4hb_715b2084-d79e-4cbf-a057-f2595b4e6b9b/console-operator/1.log" Jul 27 19:08:45.600489 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:08:45.600464 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-mv4hb_715b2084-d79e-4cbf-a057-f2595b4e6b9b/console-operator/1.log" Jul 27 19:08:46.984796 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:08:46.984756 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:08:47.176813 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:08:47.176781 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:08:47.398097 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:08:47.398065 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-fc9cd50f-openshift-default-5845ddbbd4-2hc7b"] Jul 27 19:08:47.398392 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:08:47.398364 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-fc9cd50f-openshift-default-5845ddbbd4-2hc7b" podUID="11107304-1eee-4798-9c56-eea3c514d384" containerName="istio-proxy" containerID="cri-o://c589122348cd1c98fd65fe87fbcd842f863c5155f48ed8b99531c8bd20ffc5e7" gracePeriod=30 Jul 27 19:08:47.466767 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:08:47.466727 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:08:47.691965 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:08:47.691869 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:08:48.983945 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:08:48.983904 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:08:49.103276 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:08:49.103223 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:08:49.252969 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:08:49.252869 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:08:50.188851 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:08:50.188811 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:08:50.326462 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:08:50.326424 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:08:51.976047 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:08:51.976013 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:08:52.649689 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:08:52.649663 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-fc9cd50f-openshift-default-5845ddbbd4-2hc7b" Jul 27 19:08:52.750788 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:08:52.750697 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-26ttt\" (UniqueName: \"kubernetes.io/projected/11107304-1eee-4798-9c56-eea3c514d384-kube-api-access-26ttt\") pod \"11107304-1eee-4798-9c56-eea3c514d384\" (UID: \"11107304-1eee-4798-9c56-eea3c514d384\") " Jul 27 19:08:52.750788 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:08:52.750745 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11107304-1eee-4798-9c56-eea3c514d384-workload-certs\") pod \"11107304-1eee-4798-9c56-eea3c514d384\" (UID: \"11107304-1eee-4798-9c56-eea3c514d384\") " Jul 27 19:08:52.750788 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:08:52.750780 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11107304-1eee-4798-9c56-eea3c514d384-istio-data\") pod \"11107304-1eee-4798-9c56-eea3c514d384\" (UID: \"11107304-1eee-4798-9c56-eea3c514d384\") " Jul 27 19:08:52.751061 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:08:52.750823 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11107304-1eee-4798-9c56-eea3c514d384-credential-socket\") pod \"11107304-1eee-4798-9c56-eea3c514d384\" (UID: \"11107304-1eee-4798-9c56-eea3c514d384\") " Jul 27 19:08:52.751061 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:08:52.750861 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11107304-1eee-4798-9c56-eea3c514d384-workload-socket\") pod \"11107304-1eee-4798-9c56-eea3c514d384\" (UID: \"11107304-1eee-4798-9c56-eea3c514d384\") " Jul 27 19:08:52.751061 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:08:52.750913 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11107304-1eee-4798-9c56-eea3c514d384-istio-envoy\") pod \"11107304-1eee-4798-9c56-eea3c514d384\" (UID: \"11107304-1eee-4798-9c56-eea3c514d384\") " Jul 27 19:08:52.751061 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:08:52.750936 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11107304-1eee-4798-9c56-eea3c514d384-istiod-ca-cert\") pod \"11107304-1eee-4798-9c56-eea3c514d384\" (UID: \"11107304-1eee-4798-9c56-eea3c514d384\") " Jul 27 19:08:52.751061 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:08:52.751032 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11107304-1eee-4798-9c56-eea3c514d384-istio-token\") pod \"11107304-1eee-4798-9c56-eea3c514d384\" (UID: \"11107304-1eee-4798-9c56-eea3c514d384\") " Jul 27 19:08:52.751342 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:08:52.751065 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11107304-1eee-4798-9c56-eea3c514d384-istio-podinfo\") pod \"11107304-1eee-4798-9c56-eea3c514d384\" (UID: \"11107304-1eee-4798-9c56-eea3c514d384\") " Jul 27 19:08:52.751342 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:08:52.751155 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11107304-1eee-4798-9c56-eea3c514d384-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "11107304-1eee-4798-9c56-eea3c514d384" (UID: "11107304-1eee-4798-9c56-eea3c514d384"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 19:08:52.751342 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:08:52.751175 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11107304-1eee-4798-9c56-eea3c514d384-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "11107304-1eee-4798-9c56-eea3c514d384" (UID: "11107304-1eee-4798-9c56-eea3c514d384"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 19:08:52.751487 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:08:52.751470 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11107304-1eee-4798-9c56-eea3c514d384-workload-certs\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 19:08:52.751543 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:08:52.751490 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11107304-1eee-4798-9c56-eea3c514d384-credential-socket\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 19:08:52.751543 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:08:52.751497 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11107304-1eee-4798-9c56-eea3c514d384-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "11107304-1eee-4798-9c56-eea3c514d384" (UID: "11107304-1eee-4798-9c56-eea3c514d384"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 19:08:52.751667 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:08:52.751587 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11107304-1eee-4798-9c56-eea3c514d384-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "11107304-1eee-4798-9c56-eea3c514d384" (UID: "11107304-1eee-4798-9c56-eea3c514d384"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 19:08:52.751952 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:08:52.751917 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11107304-1eee-4798-9c56-eea3c514d384-istio-data" (OuterVolumeSpecName: "istio-data") pod "11107304-1eee-4798-9c56-eea3c514d384" (UID: "11107304-1eee-4798-9c56-eea3c514d384"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 19:08:52.753131 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:08:52.753104 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11107304-1eee-4798-9c56-eea3c514d384-kube-api-access-26ttt" (OuterVolumeSpecName: "kube-api-access-26ttt") pod "11107304-1eee-4798-9c56-eea3c514d384" (UID: "11107304-1eee-4798-9c56-eea3c514d384"). InnerVolumeSpecName "kube-api-access-26ttt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 19:08:52.753373 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:08:52.753346 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/11107304-1eee-4798-9c56-eea3c514d384-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "11107304-1eee-4798-9c56-eea3c514d384" (UID: "11107304-1eee-4798-9c56-eea3c514d384"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 27 19:08:52.753477 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:08:52.753456 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11107304-1eee-4798-9c56-eea3c514d384-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "11107304-1eee-4798-9c56-eea3c514d384" (UID: "11107304-1eee-4798-9c56-eea3c514d384"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 19:08:52.753539 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:08:52.753522 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11107304-1eee-4798-9c56-eea3c514d384-istio-token" (OuterVolumeSpecName: "istio-token") pod "11107304-1eee-4798-9c56-eea3c514d384" (UID: "11107304-1eee-4798-9c56-eea3c514d384"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 19:08:52.852330 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:08:52.852299 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11107304-1eee-4798-9c56-eea3c514d384-istio-token\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 19:08:52.852330 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:08:52.852324 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11107304-1eee-4798-9c56-eea3c514d384-istio-podinfo\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 19:08:52.852330 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:08:52.852334 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26ttt\" (UniqueName: \"kubernetes.io/projected/11107304-1eee-4798-9c56-eea3c514d384-kube-api-access-26ttt\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 19:08:52.852539 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:08:52.852343 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11107304-1eee-4798-9c56-eea3c514d384-istio-data\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 19:08:52.852539 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:08:52.852351 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11107304-1eee-4798-9c56-eea3c514d384-workload-socket\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 19:08:52.852539 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:08:52.852358 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11107304-1eee-4798-9c56-eea3c514d384-istio-envoy\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 19:08:52.852539 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:08:52.852367 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11107304-1eee-4798-9c56-eea3c514d384-istiod-ca-cert\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 19:08:53.486587 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:08:53.486547 2579 generic.go:356] "Generic (PLEG): container finished" podID="11107304-1eee-4798-9c56-eea3c514d384" containerID="c589122348cd1c98fd65fe87fbcd842f863c5155f48ed8b99531c8bd20ffc5e7" exitCode=0 Jul 27 19:08:53.487080 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:08:53.486867 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-fc9cd50f-openshift-default-5845ddbbd4-2hc7b" Jul 27 19:08:53.492723 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:08:53.492696 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-fc9cd50f-openshift-default-5845ddbbd4-2hc7b" event={"ID":"11107304-1eee-4798-9c56-eea3c514d384","Type":"ContainerDied","Data":"c589122348cd1c98fd65fe87fbcd842f863c5155f48ed8b99531c8bd20ffc5e7"} Jul 27 19:08:53.492853 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:08:53.492737 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-fc9cd50f-openshift-default-5845ddbbd4-2hc7b" event={"ID":"11107304-1eee-4798-9c56-eea3c514d384","Type":"ContainerDied","Data":"e9b76fae283f4a77569992917970b7e2ff2b9a4b5fa6cd9b2f191c286523daf7"} Jul 27 19:08:53.492853 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:08:53.492760 2579 scope.go:117] "RemoveContainer" containerID="c589122348cd1c98fd65fe87fbcd842f863c5155f48ed8b99531c8bd20ffc5e7" Jul 27 19:08:53.502205 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:08:53.502186 2579 scope.go:117] "RemoveContainer" containerID="23cb89bac2dfb4c9e0ca40f49607f7944919e9d2a1343a7470e5923173b24255" Jul 27 19:08:53.510832 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:08:53.510657 2579 scope.go:117] "RemoveContainer" containerID="c589122348cd1c98fd65fe87fbcd842f863c5155f48ed8b99531c8bd20ffc5e7" Jul 27 19:08:53.510966 ip-10-0-128-32 kubenswrapper[2579]: E0727 19:08:53.510947 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c589122348cd1c98fd65fe87fbcd842f863c5155f48ed8b99531c8bd20ffc5e7\": container with ID starting with c589122348cd1c98fd65fe87fbcd842f863c5155f48ed8b99531c8bd20ffc5e7 not found: ID does not exist" containerID="c589122348cd1c98fd65fe87fbcd842f863c5155f48ed8b99531c8bd20ffc5e7" Jul 27 19:08:53.511015 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:08:53.510973 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c589122348cd1c98fd65fe87fbcd842f863c5155f48ed8b99531c8bd20ffc5e7"} err="failed to get container status \"c589122348cd1c98fd65fe87fbcd842f863c5155f48ed8b99531c8bd20ffc5e7\": rpc error: code = NotFound desc = could not find container \"c589122348cd1c98fd65fe87fbcd842f863c5155f48ed8b99531c8bd20ffc5e7\": container with ID starting with c589122348cd1c98fd65fe87fbcd842f863c5155f48ed8b99531c8bd20ffc5e7 not found: ID does not exist" Jul 27 19:08:53.511015 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:08:53.510989 2579 scope.go:117] "RemoveContainer" containerID="23cb89bac2dfb4c9e0ca40f49607f7944919e9d2a1343a7470e5923173b24255" Jul 27 19:08:53.511330 ip-10-0-128-32 kubenswrapper[2579]: E0727 19:08:53.511253 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23cb89bac2dfb4c9e0ca40f49607f7944919e9d2a1343a7470e5923173b24255\": container with ID starting with 23cb89bac2dfb4c9e0ca40f49607f7944919e9d2a1343a7470e5923173b24255 not found: ID does not exist" containerID="23cb89bac2dfb4c9e0ca40f49607f7944919e9d2a1343a7470e5923173b24255" Jul 27 19:08:53.511330 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:08:53.511291 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23cb89bac2dfb4c9e0ca40f49607f7944919e9d2a1343a7470e5923173b24255"} err="failed to get container status \"23cb89bac2dfb4c9e0ca40f49607f7944919e9d2a1343a7470e5923173b24255\": rpc error: code = NotFound desc = could not find container \"23cb89bac2dfb4c9e0ca40f49607f7944919e9d2a1343a7470e5923173b24255\": container with ID starting with 23cb89bac2dfb4c9e0ca40f49607f7944919e9d2a1343a7470e5923173b24255 not found: ID does not exist" Jul 27 19:08:53.511703 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:08:53.511684 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-fc9cd50f-openshift-default-5845ddbbd4-2hc7b"] Jul 27 19:08:53.515217 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:08:53.515197 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-fc9cd50f-openshift-default-5845ddbbd4-2hc7b"] Jul 27 19:08:55.492899 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:08:55.492822 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="11107304-1eee-4798-9c56-eea3c514d384" path="/var/lib/kubelet/pods/11107304-1eee-4798-9c56-eea3c514d384/volumes" Jul 27 19:08:58.863272 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:08:58.863211 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:08:59.068768 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:08:59.068736 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:08:59.767651 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:08:59.767614 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:09:06.107039 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:09:06.106952 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:09:06.187778 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:09:06.187729 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:09:06.310151 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:09:06.310109 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:09:06.410632 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:09:06.410544 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:09:16.928697 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:09:16.928659 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:09:17.351155 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:09:17.351106 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:09:38.424407 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:09:38.424361 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:09:45.006461 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:09:45.006416 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:09:45.095628 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:09:45.095591 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:09:45.380479 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:09:45.380447 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:09:45.685736 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:09:45.685648 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-f9caec-openshift-default-74586474d-7dq9g"] Jul 27 19:09:45.686038 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:09:45.686027 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11107304-1eee-4798-9c56-eea3c514d384" containerName="istio-proxy" Jul 27 19:09:45.686088 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:09:45.686040 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="11107304-1eee-4798-9c56-eea3c514d384" containerName="istio-proxy" Jul 27 19:09:45.686088 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:09:45.686059 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf98d888-f70a-414c-a6b8-60c11d79d76b" containerName="istio-proxy" Jul 27 19:09:45.686088 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:09:45.686065 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf98d888-f70a-414c-a6b8-60c11d79d76b" containerName="istio-proxy" Jul 27 19:09:45.686183 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:09:45.686124 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="11107304-1eee-4798-9c56-eea3c514d384" containerName="istio-proxy" Jul 27 19:09:45.686183 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:09:45.686132 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="11107304-1eee-4798-9c56-eea3c514d384" containerName="istio-proxy" Jul 27 19:09:45.688164 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:09:45.688143 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f9caec-openshift-default-74586474d-7dq9g" Jul 27 19:09:45.692480 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:09:45.692445 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-f9caec-openshift-default-dockercfg-kpntl\"" Jul 27 19:09:45.702935 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:09:45.702913 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-f9caec-openshift-default-74586474d-7dq9g"] Jul 27 19:09:45.718245 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:09:45.718192 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:09:45.727922 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:09:45.727896 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24f7ec22-8bca-46b0-9bb4-b92ccab63840-workload-certs\") pod \"e2e-inf-a-f9caec-openshift-default-74586474d-7dq9g\" (UID: \"24f7ec22-8bca-46b0-9bb4-b92ccab63840\") " pod="openshift-ingress/e2e-inf-a-f9caec-openshift-default-74586474d-7dq9g" Jul 27 19:09:45.727922 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:09:45.727934 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24f7ec22-8bca-46b0-9bb4-b92ccab63840-credential-socket\") pod \"e2e-inf-a-f9caec-openshift-default-74586474d-7dq9g\" (UID: \"24f7ec22-8bca-46b0-9bb4-b92ccab63840\") " pod="openshift-ingress/e2e-inf-a-f9caec-openshift-default-74586474d-7dq9g" Jul 27 19:09:45.728135 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:09:45.728005 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dp64s\" (UniqueName: \"kubernetes.io/projected/24f7ec22-8bca-46b0-9bb4-b92ccab63840-kube-api-access-dp64s\") pod \"e2e-inf-a-f9caec-openshift-default-74586474d-7dq9g\" (UID: \"24f7ec22-8bca-46b0-9bb4-b92ccab63840\") " pod="openshift-ingress/e2e-inf-a-f9caec-openshift-default-74586474d-7dq9g" Jul 27 19:09:45.728135 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:09:45.728087 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24f7ec22-8bca-46b0-9bb4-b92ccab63840-istio-envoy\") pod \"e2e-inf-a-f9caec-openshift-default-74586474d-7dq9g\" (UID: \"24f7ec22-8bca-46b0-9bb4-b92ccab63840\") " pod="openshift-ingress/e2e-inf-a-f9caec-openshift-default-74586474d-7dq9g" Jul 27 19:09:45.728135 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:09:45.728130 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24f7ec22-8bca-46b0-9bb4-b92ccab63840-istio-token\") pod \"e2e-inf-a-f9caec-openshift-default-74586474d-7dq9g\" (UID: \"24f7ec22-8bca-46b0-9bb4-b92ccab63840\") " pod="openshift-ingress/e2e-inf-a-f9caec-openshift-default-74586474d-7dq9g" Jul 27 19:09:45.728290 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:09:45.728176 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24f7ec22-8bca-46b0-9bb4-b92ccab63840-istio-podinfo\") pod \"e2e-inf-a-f9caec-openshift-default-74586474d-7dq9g\" (UID: \"24f7ec22-8bca-46b0-9bb4-b92ccab63840\") " pod="openshift-ingress/e2e-inf-a-f9caec-openshift-default-74586474d-7dq9g" Jul 27 19:09:45.728345 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:09:45.728294 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24f7ec22-8bca-46b0-9bb4-b92ccab63840-istio-data\") pod \"e2e-inf-a-f9caec-openshift-default-74586474d-7dq9g\" (UID: \"24f7ec22-8bca-46b0-9bb4-b92ccab63840\") " pod="openshift-ingress/e2e-inf-a-f9caec-openshift-default-74586474d-7dq9g" Jul 27 19:09:45.728345 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:09:45.728334 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24f7ec22-8bca-46b0-9bb4-b92ccab63840-workload-socket\") pod \"e2e-inf-a-f9caec-openshift-default-74586474d-7dq9g\" (UID: \"24f7ec22-8bca-46b0-9bb4-b92ccab63840\") " pod="openshift-ingress/e2e-inf-a-f9caec-openshift-default-74586474d-7dq9g" Jul 27 19:09:45.728450 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:09:45.728356 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24f7ec22-8bca-46b0-9bb4-b92ccab63840-istiod-ca-cert\") pod \"e2e-inf-a-f9caec-openshift-default-74586474d-7dq9g\" (UID: \"24f7ec22-8bca-46b0-9bb4-b92ccab63840\") " pod="openshift-ingress/e2e-inf-a-f9caec-openshift-default-74586474d-7dq9g" Jul 27 19:09:45.828876 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:09:45.828841 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24f7ec22-8bca-46b0-9bb4-b92ccab63840-istio-data\") pod \"e2e-inf-a-f9caec-openshift-default-74586474d-7dq9g\" (UID: \"24f7ec22-8bca-46b0-9bb4-b92ccab63840\") " pod="openshift-ingress/e2e-inf-a-f9caec-openshift-default-74586474d-7dq9g" Jul 27 19:09:45.829076 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:09:45.828894 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24f7ec22-8bca-46b0-9bb4-b92ccab63840-workload-socket\") pod \"e2e-inf-a-f9caec-openshift-default-74586474d-7dq9g\" (UID: \"24f7ec22-8bca-46b0-9bb4-b92ccab63840\") " pod="openshift-ingress/e2e-inf-a-f9caec-openshift-default-74586474d-7dq9g" Jul 27 19:09:45.829076 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:09:45.828926 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24f7ec22-8bca-46b0-9bb4-b92ccab63840-istiod-ca-cert\") pod \"e2e-inf-a-f9caec-openshift-default-74586474d-7dq9g\" (UID: \"24f7ec22-8bca-46b0-9bb4-b92ccab63840\") " pod="openshift-ingress/e2e-inf-a-f9caec-openshift-default-74586474d-7dq9g" Jul 27 19:09:45.829200 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:09:45.829150 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24f7ec22-8bca-46b0-9bb4-b92ccab63840-workload-certs\") pod \"e2e-inf-a-f9caec-openshift-default-74586474d-7dq9g\" (UID: \"24f7ec22-8bca-46b0-9bb4-b92ccab63840\") " pod="openshift-ingress/e2e-inf-a-f9caec-openshift-default-74586474d-7dq9g" Jul 27 19:09:45.829285 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:09:45.829215 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24f7ec22-8bca-46b0-9bb4-b92ccab63840-credential-socket\") pod \"e2e-inf-a-f9caec-openshift-default-74586474d-7dq9g\" (UID: \"24f7ec22-8bca-46b0-9bb4-b92ccab63840\") " pod="openshift-ingress/e2e-inf-a-f9caec-openshift-default-74586474d-7dq9g" Jul 27 19:09:45.829285 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:09:45.829223 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24f7ec22-8bca-46b0-9bb4-b92ccab63840-istio-data\") pod \"e2e-inf-a-f9caec-openshift-default-74586474d-7dq9g\" (UID: \"24f7ec22-8bca-46b0-9bb4-b92ccab63840\") " pod="openshift-ingress/e2e-inf-a-f9caec-openshift-default-74586474d-7dq9g" Jul 27 19:09:45.829401 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:09:45.829297 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dp64s\" (UniqueName: \"kubernetes.io/projected/24f7ec22-8bca-46b0-9bb4-b92ccab63840-kube-api-access-dp64s\") pod \"e2e-inf-a-f9caec-openshift-default-74586474d-7dq9g\" (UID: \"24f7ec22-8bca-46b0-9bb4-b92ccab63840\") " pod="openshift-ingress/e2e-inf-a-f9caec-openshift-default-74586474d-7dq9g" Jul 27 19:09:45.829401 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:09:45.829319 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24f7ec22-8bca-46b0-9bb4-b92ccab63840-workload-socket\") pod \"e2e-inf-a-f9caec-openshift-default-74586474d-7dq9g\" (UID: \"24f7ec22-8bca-46b0-9bb4-b92ccab63840\") " pod="openshift-ingress/e2e-inf-a-f9caec-openshift-default-74586474d-7dq9g" Jul 27 19:09:45.829401 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:09:45.829327 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24f7ec22-8bca-46b0-9bb4-b92ccab63840-istio-envoy\") pod \"e2e-inf-a-f9caec-openshift-default-74586474d-7dq9g\" (UID: \"24f7ec22-8bca-46b0-9bb4-b92ccab63840\") " pod="openshift-ingress/e2e-inf-a-f9caec-openshift-default-74586474d-7dq9g" Jul 27 19:09:45.829401 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:09:45.829382 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24f7ec22-8bca-46b0-9bb4-b92ccab63840-istio-token\") pod \"e2e-inf-a-f9caec-openshift-default-74586474d-7dq9g\" (UID: \"24f7ec22-8bca-46b0-9bb4-b92ccab63840\") " pod="openshift-ingress/e2e-inf-a-f9caec-openshift-default-74586474d-7dq9g" Jul 27 19:09:45.829624 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:09:45.829432 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24f7ec22-8bca-46b0-9bb4-b92ccab63840-istio-podinfo\") pod \"e2e-inf-a-f9caec-openshift-default-74586474d-7dq9g\" (UID: \"24f7ec22-8bca-46b0-9bb4-b92ccab63840\") " pod="openshift-ingress/e2e-inf-a-f9caec-openshift-default-74586474d-7dq9g" Jul 27 19:09:45.829624 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:09:45.829529 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24f7ec22-8bca-46b0-9bb4-b92ccab63840-workload-certs\") pod \"e2e-inf-a-f9caec-openshift-default-74586474d-7dq9g\" (UID: \"24f7ec22-8bca-46b0-9bb4-b92ccab63840\") " pod="openshift-ingress/e2e-inf-a-f9caec-openshift-default-74586474d-7dq9g" Jul 27 19:09:45.829624 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:09:45.829606 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24f7ec22-8bca-46b0-9bb4-b92ccab63840-credential-socket\") pod \"e2e-inf-a-f9caec-openshift-default-74586474d-7dq9g\" (UID: \"24f7ec22-8bca-46b0-9bb4-b92ccab63840\") " pod="openshift-ingress/e2e-inf-a-f9caec-openshift-default-74586474d-7dq9g" Jul 27 19:09:45.829781 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:09:45.829762 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24f7ec22-8bca-46b0-9bb4-b92ccab63840-istiod-ca-cert\") pod \"e2e-inf-a-f9caec-openshift-default-74586474d-7dq9g\" (UID: \"24f7ec22-8bca-46b0-9bb4-b92ccab63840\") " pod="openshift-ingress/e2e-inf-a-f9caec-openshift-default-74586474d-7dq9g" Jul 27 19:09:45.831741 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:09:45.831713 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24f7ec22-8bca-46b0-9bb4-b92ccab63840-istio-envoy\") pod \"e2e-inf-a-f9caec-openshift-default-74586474d-7dq9g\" (UID: \"24f7ec22-8bca-46b0-9bb4-b92ccab63840\") " pod="openshift-ingress/e2e-inf-a-f9caec-openshift-default-74586474d-7dq9g" Jul 27 19:09:45.832183 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:09:45.832162 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24f7ec22-8bca-46b0-9bb4-b92ccab63840-istio-podinfo\") pod \"e2e-inf-a-f9caec-openshift-default-74586474d-7dq9g\" (UID: \"24f7ec22-8bca-46b0-9bb4-b92ccab63840\") " pod="openshift-ingress/e2e-inf-a-f9caec-openshift-default-74586474d-7dq9g" Jul 27 19:09:45.838492 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:09:45.838465 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24f7ec22-8bca-46b0-9bb4-b92ccab63840-istio-token\") pod \"e2e-inf-a-f9caec-openshift-default-74586474d-7dq9g\" (UID: \"24f7ec22-8bca-46b0-9bb4-b92ccab63840\") " pod="openshift-ingress/e2e-inf-a-f9caec-openshift-default-74586474d-7dq9g" Jul 27 19:09:45.838804 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:09:45.838783 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dp64s\" (UniqueName: \"kubernetes.io/projected/24f7ec22-8bca-46b0-9bb4-b92ccab63840-kube-api-access-dp64s\") pod \"e2e-inf-a-f9caec-openshift-default-74586474d-7dq9g\" (UID: \"24f7ec22-8bca-46b0-9bb4-b92ccab63840\") " pod="openshift-ingress/e2e-inf-a-f9caec-openshift-default-74586474d-7dq9g" Jul 27 19:09:45.938771 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:09:45.938667 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:09:45.998708 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:09:45.998654 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f9caec-openshift-default-74586474d-7dq9g" Jul 27 19:09:46.152430 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:09:46.152395 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-f9caec-openshift-default-74586474d-7dq9g"] Jul 27 19:09:46.155995 ip-10-0-128-32 kubenswrapper[2579]: W0727 19:09:46.155970 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24f7ec22_8bca_46b0_9bb4_b92ccab63840.slice/crio-22b59ab33c14853effbb4ffa43e833e6d546bd8b05e3eaf2a105349c3a50bc26 WatchSource:0}: Error finding container 22b59ab33c14853effbb4ffa43e833e6d546bd8b05e3eaf2a105349c3a50bc26: Status 404 returned error can't find the container with id 22b59ab33c14853effbb4ffa43e833e6d546bd8b05e3eaf2a105349c3a50bc26 Jul 27 19:09:46.158508 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:09:46.158468 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 27 19:09:46.158587 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:09:46.158540 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 27 19:09:46.158587 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:09:46.158568 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 27 19:09:46.756578 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:09:46.756546 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f9caec-openshift-default-74586474d-7dq9g" event={"ID":"24f7ec22-8bca-46b0-9bb4-b92ccab63840","Type":"ContainerStarted","Data":"5f5722d1d5d6e980deac1d8b6783f6caacb707002018b7b29f49232699a7fd77"} Jul 27 19:09:46.756808 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:09:46.756599 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f9caec-openshift-default-74586474d-7dq9g" event={"ID":"24f7ec22-8bca-46b0-9bb4-b92ccab63840","Type":"ContainerStarted","Data":"22b59ab33c14853effbb4ffa43e833e6d546bd8b05e3eaf2a105349c3a50bc26"} Jul 27 19:09:46.778508 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:09:46.778460 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-f9caec-openshift-default-74586474d-7dq9g" podStartSLOduration=1.778444365 podStartE2EDuration="1.778444365s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:09:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:09:46.777474169 +0000 UTC m=+2461.839163228" watchObservedRunningTime="2026-07-27 19:09:46.778444365 +0000 UTC m=+2461.840133396" Jul 27 19:09:46.999563 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:09:46.999531 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-f9caec-openshift-default-74586474d-7dq9g" Jul 27 19:09:48.004445 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:09:48.004416 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-f9caec-openshift-default-74586474d-7dq9g" Jul 27 19:09:48.767632 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:09:48.767599 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-f9caec-openshift-default-74586474d-7dq9g" Jul 27 19:09:48.768544 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:09:48.768520 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-f9caec-openshift-default-74586474d-7dq9g" Jul 27 19:09:48.799999 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:09:48.799963 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:09:48.897309 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:09:48.897272 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:09:48.973283 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:09:48.973248 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:09:49.076863 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:09:49.076825 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:09:49.532019 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:09:49.531982 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:09:49.697623 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:09:49.697572 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:09:53.392615 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:09:53.392576 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:09:53.757208 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:09:53.757123 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:09:54.067395 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:09:54.067362 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-9abe8e-openshift-default-7d4b9c5f84-4nxfh"] Jul 27 19:09:54.067807 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:09:54.067795 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11107304-1eee-4798-9c56-eea3c514d384" containerName="istio-proxy" Jul 27 19:09:54.067852 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:09:54.067809 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="11107304-1eee-4798-9c56-eea3c514d384" containerName="istio-proxy" Jul 27 19:09:54.070124 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:09:54.070109 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9abe8e-openshift-default-7d4b9c5f84-4nxfh" Jul 27 19:09:54.073214 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:09:54.073193 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-9abe8e-openshift-default-dockercfg-g6pnv\"" Jul 27 19:09:54.096954 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:09:54.096932 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-9abe8e-openshift-default-7d4b9c5f84-4nxfh"] Jul 27 19:09:54.112373 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:09:54.112348 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56d09ac8-e6cf-4605-b063-ed13073f2b82-istio-podinfo\") pod \"e2e-inf-b-9abe8e-openshift-default-7d4b9c5f84-4nxfh\" (UID: \"56d09ac8-e6cf-4605-b063-ed13073f2b82\") " pod="openshift-ingress/e2e-inf-b-9abe8e-openshift-default-7d4b9c5f84-4nxfh" Jul 27 19:09:54.112518 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:09:54.112387 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56d09ac8-e6cf-4605-b063-ed13073f2b82-workload-certs\") pod \"e2e-inf-b-9abe8e-openshift-default-7d4b9c5f84-4nxfh\" (UID: \"56d09ac8-e6cf-4605-b063-ed13073f2b82\") " pod="openshift-ingress/e2e-inf-b-9abe8e-openshift-default-7d4b9c5f84-4nxfh" Jul 27 19:09:54.112518 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:09:54.112416 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56d09ac8-e6cf-4605-b063-ed13073f2b82-istio-envoy\") pod \"e2e-inf-b-9abe8e-openshift-default-7d4b9c5f84-4nxfh\" (UID: \"56d09ac8-e6cf-4605-b063-ed13073f2b82\") " pod="openshift-ingress/e2e-inf-b-9abe8e-openshift-default-7d4b9c5f84-4nxfh" Jul 27 19:09:54.112619 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:09:54.112534 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56d09ac8-e6cf-4605-b063-ed13073f2b82-istio-token\") pod \"e2e-inf-b-9abe8e-openshift-default-7d4b9c5f84-4nxfh\" (UID: \"56d09ac8-e6cf-4605-b063-ed13073f2b82\") " pod="openshift-ingress/e2e-inf-b-9abe8e-openshift-default-7d4b9c5f84-4nxfh" Jul 27 19:09:54.112619 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:09:54.112579 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56d09ac8-e6cf-4605-b063-ed13073f2b82-istio-data\") pod \"e2e-inf-b-9abe8e-openshift-default-7d4b9c5f84-4nxfh\" (UID: \"56d09ac8-e6cf-4605-b063-ed13073f2b82\") " pod="openshift-ingress/e2e-inf-b-9abe8e-openshift-default-7d4b9c5f84-4nxfh" Jul 27 19:09:54.112693 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:09:54.112648 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56d09ac8-e6cf-4605-b063-ed13073f2b82-istiod-ca-cert\") pod \"e2e-inf-b-9abe8e-openshift-default-7d4b9c5f84-4nxfh\" (UID: \"56d09ac8-e6cf-4605-b063-ed13073f2b82\") " pod="openshift-ingress/e2e-inf-b-9abe8e-openshift-default-7d4b9c5f84-4nxfh" Jul 27 19:09:54.112693 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:09:54.112678 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56d09ac8-e6cf-4605-b063-ed13073f2b82-credential-socket\") pod \"e2e-inf-b-9abe8e-openshift-default-7d4b9c5f84-4nxfh\" (UID: \"56d09ac8-e6cf-4605-b063-ed13073f2b82\") " pod="openshift-ingress/e2e-inf-b-9abe8e-openshift-default-7d4b9c5f84-4nxfh" Jul 27 19:09:54.112777 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:09:54.112742 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56d09ac8-e6cf-4605-b063-ed13073f2b82-workload-socket\") pod \"e2e-inf-b-9abe8e-openshift-default-7d4b9c5f84-4nxfh\" (UID: \"56d09ac8-e6cf-4605-b063-ed13073f2b82\") " pod="openshift-ingress/e2e-inf-b-9abe8e-openshift-default-7d4b9c5f84-4nxfh" Jul 27 19:09:54.112827 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:09:54.112811 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4s2zs\" (UniqueName: \"kubernetes.io/projected/56d09ac8-e6cf-4605-b063-ed13073f2b82-kube-api-access-4s2zs\") pod \"e2e-inf-b-9abe8e-openshift-default-7d4b9c5f84-4nxfh\" (UID: \"56d09ac8-e6cf-4605-b063-ed13073f2b82\") " pod="openshift-ingress/e2e-inf-b-9abe8e-openshift-default-7d4b9c5f84-4nxfh" Jul 27 19:09:54.150211 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:09:54.150173 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:09:54.213689 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:09:54.213655 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56d09ac8-e6cf-4605-b063-ed13073f2b82-istiod-ca-cert\") pod \"e2e-inf-b-9abe8e-openshift-default-7d4b9c5f84-4nxfh\" (UID: \"56d09ac8-e6cf-4605-b063-ed13073f2b82\") " pod="openshift-ingress/e2e-inf-b-9abe8e-openshift-default-7d4b9c5f84-4nxfh" Jul 27 19:09:54.213689 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:09:54.213689 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56d09ac8-e6cf-4605-b063-ed13073f2b82-credential-socket\") pod \"e2e-inf-b-9abe8e-openshift-default-7d4b9c5f84-4nxfh\" (UID: \"56d09ac8-e6cf-4605-b063-ed13073f2b82\") " pod="openshift-ingress/e2e-inf-b-9abe8e-openshift-default-7d4b9c5f84-4nxfh" Jul 27 19:09:54.213955 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:09:54.213718 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56d09ac8-e6cf-4605-b063-ed13073f2b82-workload-socket\") pod \"e2e-inf-b-9abe8e-openshift-default-7d4b9c5f84-4nxfh\" (UID: \"56d09ac8-e6cf-4605-b063-ed13073f2b82\") " pod="openshift-ingress/e2e-inf-b-9abe8e-openshift-default-7d4b9c5f84-4nxfh" Jul 27 19:09:54.213955 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:09:54.213762 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4s2zs\" (UniqueName: \"kubernetes.io/projected/56d09ac8-e6cf-4605-b063-ed13073f2b82-kube-api-access-4s2zs\") pod \"e2e-inf-b-9abe8e-openshift-default-7d4b9c5f84-4nxfh\" (UID: \"56d09ac8-e6cf-4605-b063-ed13073f2b82\") " pod="openshift-ingress/e2e-inf-b-9abe8e-openshift-default-7d4b9c5f84-4nxfh" Jul 27 19:09:54.213955 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:09:54.213779 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56d09ac8-e6cf-4605-b063-ed13073f2b82-istio-podinfo\") pod \"e2e-inf-b-9abe8e-openshift-default-7d4b9c5f84-4nxfh\" (UID: \"56d09ac8-e6cf-4605-b063-ed13073f2b82\") " pod="openshift-ingress/e2e-inf-b-9abe8e-openshift-default-7d4b9c5f84-4nxfh" Jul 27 19:09:54.213955 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:09:54.213804 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56d09ac8-e6cf-4605-b063-ed13073f2b82-workload-certs\") pod \"e2e-inf-b-9abe8e-openshift-default-7d4b9c5f84-4nxfh\" (UID: \"56d09ac8-e6cf-4605-b063-ed13073f2b82\") " pod="openshift-ingress/e2e-inf-b-9abe8e-openshift-default-7d4b9c5f84-4nxfh" Jul 27 19:09:54.213955 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:09:54.213840 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56d09ac8-e6cf-4605-b063-ed13073f2b82-istio-envoy\") pod \"e2e-inf-b-9abe8e-openshift-default-7d4b9c5f84-4nxfh\" (UID: \"56d09ac8-e6cf-4605-b063-ed13073f2b82\") " pod="openshift-ingress/e2e-inf-b-9abe8e-openshift-default-7d4b9c5f84-4nxfh" Jul 27 19:09:54.213955 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:09:54.213894 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56d09ac8-e6cf-4605-b063-ed13073f2b82-istio-token\") pod \"e2e-inf-b-9abe8e-openshift-default-7d4b9c5f84-4nxfh\" (UID: \"56d09ac8-e6cf-4605-b063-ed13073f2b82\") " pod="openshift-ingress/e2e-inf-b-9abe8e-openshift-default-7d4b9c5f84-4nxfh" Jul 27 19:09:54.213955 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:09:54.213919 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56d09ac8-e6cf-4605-b063-ed13073f2b82-istio-data\") pod \"e2e-inf-b-9abe8e-openshift-default-7d4b9c5f84-4nxfh\" (UID: \"56d09ac8-e6cf-4605-b063-ed13073f2b82\") " pod="openshift-ingress/e2e-inf-b-9abe8e-openshift-default-7d4b9c5f84-4nxfh" Jul 27 19:09:54.214345 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:09:54.214223 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56d09ac8-e6cf-4605-b063-ed13073f2b82-credential-socket\") pod \"e2e-inf-b-9abe8e-openshift-default-7d4b9c5f84-4nxfh\" (UID: \"56d09ac8-e6cf-4605-b063-ed13073f2b82\") " pod="openshift-ingress/e2e-inf-b-9abe8e-openshift-default-7d4b9c5f84-4nxfh" Jul 27 19:09:54.214491 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:09:54.214472 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56d09ac8-e6cf-4605-b063-ed13073f2b82-workload-socket\") pod \"e2e-inf-b-9abe8e-openshift-default-7d4b9c5f84-4nxfh\" (UID: \"56d09ac8-e6cf-4605-b063-ed13073f2b82\") " pod="openshift-ingress/e2e-inf-b-9abe8e-openshift-default-7d4b9c5f84-4nxfh" Jul 27 19:09:54.214580 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:09:54.214515 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56d09ac8-e6cf-4605-b063-ed13073f2b82-istiod-ca-cert\") pod \"e2e-inf-b-9abe8e-openshift-default-7d4b9c5f84-4nxfh\" (UID: \"56d09ac8-e6cf-4605-b063-ed13073f2b82\") " pod="openshift-ingress/e2e-inf-b-9abe8e-openshift-default-7d4b9c5f84-4nxfh" Jul 27 19:09:54.214580 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:09:54.214524 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56d09ac8-e6cf-4605-b063-ed13073f2b82-workload-certs\") pod \"e2e-inf-b-9abe8e-openshift-default-7d4b9c5f84-4nxfh\" (UID: \"56d09ac8-e6cf-4605-b063-ed13073f2b82\") " pod="openshift-ingress/e2e-inf-b-9abe8e-openshift-default-7d4b9c5f84-4nxfh" Jul 27 19:09:54.214580 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:09:54.214574 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56d09ac8-e6cf-4605-b063-ed13073f2b82-istio-data\") pod \"e2e-inf-b-9abe8e-openshift-default-7d4b9c5f84-4nxfh\" (UID: \"56d09ac8-e6cf-4605-b063-ed13073f2b82\") " pod="openshift-ingress/e2e-inf-b-9abe8e-openshift-default-7d4b9c5f84-4nxfh" Jul 27 19:09:54.216324 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:09:54.216302 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56d09ac8-e6cf-4605-b063-ed13073f2b82-istio-envoy\") pod \"e2e-inf-b-9abe8e-openshift-default-7d4b9c5f84-4nxfh\" (UID: \"56d09ac8-e6cf-4605-b063-ed13073f2b82\") " pod="openshift-ingress/e2e-inf-b-9abe8e-openshift-default-7d4b9c5f84-4nxfh" Jul 27 19:09:54.216582 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:09:54.216554 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56d09ac8-e6cf-4605-b063-ed13073f2b82-istio-podinfo\") pod \"e2e-inf-b-9abe8e-openshift-default-7d4b9c5f84-4nxfh\" (UID: \"56d09ac8-e6cf-4605-b063-ed13073f2b82\") " pod="openshift-ingress/e2e-inf-b-9abe8e-openshift-default-7d4b9c5f84-4nxfh" Jul 27 19:09:54.224526 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:09:54.224502 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4s2zs\" (UniqueName: \"kubernetes.io/projected/56d09ac8-e6cf-4605-b063-ed13073f2b82-kube-api-access-4s2zs\") pod \"e2e-inf-b-9abe8e-openshift-default-7d4b9c5f84-4nxfh\" (UID: \"56d09ac8-e6cf-4605-b063-ed13073f2b82\") " pod="openshift-ingress/e2e-inf-b-9abe8e-openshift-default-7d4b9c5f84-4nxfh" Jul 27 19:09:54.225093 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:09:54.225079 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56d09ac8-e6cf-4605-b063-ed13073f2b82-istio-token\") pod \"e2e-inf-b-9abe8e-openshift-default-7d4b9c5f84-4nxfh\" (UID: \"56d09ac8-e6cf-4605-b063-ed13073f2b82\") " pod="openshift-ingress/e2e-inf-b-9abe8e-openshift-default-7d4b9c5f84-4nxfh" Jul 27 19:09:54.298843 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:09:54.298806 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:09:54.381023 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:09:54.380936 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9abe8e-openshift-default-7d4b9c5f84-4nxfh" Jul 27 19:09:54.532877 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:09:54.532849 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-9abe8e-openshift-default-7d4b9c5f84-4nxfh"] Jul 27 19:09:54.535125 ip-10-0-128-32 kubenswrapper[2579]: W0727 19:09:54.535086 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56d09ac8_e6cf_4605_b063_ed13073f2b82.slice/crio-6a1d48a3bc0b6270d3e648276ca442cc4f5aa72adb10aa3a9949145e872f58f3 WatchSource:0}: Error finding container 6a1d48a3bc0b6270d3e648276ca442cc4f5aa72adb10aa3a9949145e872f58f3: Status 404 returned error can't find the container with id 6a1d48a3bc0b6270d3e648276ca442cc4f5aa72adb10aa3a9949145e872f58f3 Jul 27 19:09:54.537778 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:09:54.537745 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 27 19:09:54.537878 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:09:54.537811 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 27 19:09:54.537878 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:09:54.537840 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 27 19:09:54.801303 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:09:54.801267 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9abe8e-openshift-default-7d4b9c5f84-4nxfh" event={"ID":"56d09ac8-e6cf-4605-b063-ed13073f2b82","Type":"ContainerStarted","Data":"b20b07fbb08adc2bdd7a486107db6e20ab9009522b870029d8767ed59002eb23"} Jul 27 19:09:54.801474 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:09:54.801332 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9abe8e-openshift-default-7d4b9c5f84-4nxfh" event={"ID":"56d09ac8-e6cf-4605-b063-ed13073f2b82","Type":"ContainerStarted","Data":"6a1d48a3bc0b6270d3e648276ca442cc4f5aa72adb10aa3a9949145e872f58f3"} Jul 27 19:09:54.832079 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:09:54.832023 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-9abe8e-openshift-default-7d4b9c5f84-4nxfh" podStartSLOduration=0.832003864 podStartE2EDuration="832.003864ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:09:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:09:54.830746855 +0000 UTC m=+2469.892435885" watchObservedRunningTime="2026-07-27 19:09:54.832003864 +0000 UTC m=+2469.893692899" Jul 27 19:09:55.302193 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:09:55.302159 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:09:55.381128 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:09:55.381090 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-9abe8e-openshift-default-7d4b9c5f84-4nxfh" Jul 27 19:09:55.382849 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:09:55.382822 2579 patch_prober.go:28] interesting pod/e2e-inf-b-9abe8e-openshift-default-7d4b9c5f84-4nxfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Jul 27 19:09:55.382990 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:09:55.382880 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9abe8e-openshift-default-7d4b9c5f84-4nxfh" podUID="56d09ac8-e6cf-4605-b063-ed13073f2b82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Jul 27 19:09:55.389340 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:09:55.389307 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:09:55.457925 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:09:55.457882 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:09:56.385204 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:09:56.385167 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-9abe8e-openshift-default-7d4b9c5f84-4nxfh" Jul 27 19:09:56.813650 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:09:56.813624 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-9abe8e-openshift-default-7d4b9c5f84-4nxfh" Jul 27 19:09:56.814531 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:09:56.814506 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-9abe8e-openshift-default-7d4b9c5f84-4nxfh" Jul 27 19:09:56.952475 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:09:56.952432 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:09:57.079715 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:09:57.079628 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:09:57.152114 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:09:57.152076 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:09:57.790804 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:09:57.790769 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:09:57.891423 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:09:57.891374 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:09:58.197173 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:09:58.197079 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:10:01.478698 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:10:01.478662 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:10:02.854353 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:10:02.854314 2579 generic.go:356] "Generic (PLEG): container finished" podID="24f7ec22-8bca-46b0-9bb4-b92ccab63840" containerID="5f5722d1d5d6e980deac1d8b6783f6caacb707002018b7b29f49232699a7fd77" exitCode=0 Jul 27 19:10:02.854794 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:10:02.854390 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f9caec-openshift-default-74586474d-7dq9g" event={"ID":"24f7ec22-8bca-46b0-9bb4-b92ccab63840","Type":"ContainerDied","Data":"5f5722d1d5d6e980deac1d8b6783f6caacb707002018b7b29f49232699a7fd77"} Jul 27 19:10:02.854897 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:10:02.854879 2579 scope.go:117] "RemoveContainer" containerID="5f5722d1d5d6e980deac1d8b6783f6caacb707002018b7b29f49232699a7fd77" Jul 27 19:10:02.855943 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:10:02.855917 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 27 19:10:02.856012 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:10:02.855986 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 27 19:10:02.856055 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:10:02.856016 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 27 19:10:03.768151 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:10:03.768121 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-f9caec-openshift-default-74586474d-7dq9g" Jul 27 19:10:03.861637 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:10:03.861603 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f9caec-openshift-default-74586474d-7dq9g" event={"ID":"24f7ec22-8bca-46b0-9bb4-b92ccab63840","Type":"ContainerStarted","Data":"9aea012ee64b0732f0359c69f961aa1f88b5c753d3dc9a9c5ac842b663ef3bf4"} Jul 27 19:10:03.999712 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:10:03.999675 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-f9caec-openshift-default-74586474d-7dq9g" Jul 27 19:10:04.000912 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:10:04.000886 2579 patch_prober.go:28] interesting pod/e2e-inf-a-f9caec-openshift-default-74586474d-7dq9g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Jul 27 19:10:04.001068 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:10:04.000950 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f9caec-openshift-default-74586474d-7dq9g" podUID="24f7ec22-8bca-46b0-9bb4-b92ccab63840" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Jul 27 19:10:04.999160 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:10:04.999117 2579 patch_prober.go:28] interesting pod/e2e-inf-a-f9caec-openshift-default-74586474d-7dq9g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Jul 27 19:10:04.999572 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:10:04.999179 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f9caec-openshift-default-74586474d-7dq9g" podUID="24f7ec22-8bca-46b0-9bb4-b92ccab63840" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Jul 27 19:10:05.998999 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:10:05.998960 2579 patch_prober.go:28] interesting pod/e2e-inf-a-f9caec-openshift-default-74586474d-7dq9g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Jul 27 19:10:05.999202 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:10:05.999019 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f9caec-openshift-default-74586474d-7dq9g" podUID="24f7ec22-8bca-46b0-9bb4-b92ccab63840" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Jul 27 19:10:06.999033 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:10:06.999000 2579 patch_prober.go:28] interesting pod/e2e-inf-a-f9caec-openshift-default-74586474d-7dq9g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Jul 27 19:10:06.999308 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:10:06.999285 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f9caec-openshift-default-74586474d-7dq9g" podUID="24f7ec22-8bca-46b0-9bb4-b92ccab63840" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Jul 27 19:10:08.003059 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:10:08.003026 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-f9caec-openshift-default-74586474d-7dq9g" Jul 27 19:10:08.003562 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:10:08.003544 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-f9caec-openshift-default-74586474d-7dq9g" Jul 27 19:10:08.004334 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:10:08.004313 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-f9caec-openshift-default-74586474d-7dq9g" Jul 27 19:10:11.923013 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:10:11.922969 2579 patch_prober.go:28] interesting pod/e2e-inf-b-9abe8e-openshift-default-7d4b9c5f84-4nxfh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": read tcp 10.132.0.2:57074->10.132.0.64:15021: read: connection reset by peer" start-of-body= Jul 27 19:10:11.923459 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:10:11.923046 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-9abe8e-openshift-default-7d4b9c5f84-4nxfh" podUID="56d09ac8-e6cf-4605-b063-ed13073f2b82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": read tcp 10.132.0.2:57074->10.132.0.64:15021: read: connection reset by peer" Jul 27 19:10:12.913412 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:10:12.913382 2579 generic.go:356] "Generic (PLEG): container finished" podID="56d09ac8-e6cf-4605-b063-ed13073f2b82" containerID="b20b07fbb08adc2bdd7a486107db6e20ab9009522b870029d8767ed59002eb23" exitCode=0 Jul 27 19:10:12.913584 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:10:12.913456 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9abe8e-openshift-default-7d4b9c5f84-4nxfh" event={"ID":"56d09ac8-e6cf-4605-b063-ed13073f2b82","Type":"ContainerDied","Data":"b20b07fbb08adc2bdd7a486107db6e20ab9009522b870029d8767ed59002eb23"} Jul 27 19:10:12.913880 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:10:12.913866 2579 scope.go:117] "RemoveContainer" containerID="b20b07fbb08adc2bdd7a486107db6e20ab9009522b870029d8767ed59002eb23" Jul 27 19:10:12.915515 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:10:12.915484 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 27 19:10:12.915784 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:10:12.915762 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 27 19:10:12.915954 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:10:12.915928 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 27 19:10:13.920739 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:10:13.920699 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9abe8e-openshift-default-7d4b9c5f84-4nxfh" event={"ID":"56d09ac8-e6cf-4605-b063-ed13073f2b82","Type":"ContainerStarted","Data":"608b0387115da6386bb8a0840bd5f8afdfdc22dfea4ae86f0c9c9f2774ffc78d"} Jul 27 19:10:14.381767 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:10:14.381732 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-9abe8e-openshift-default-7d4b9c5f84-4nxfh" Jul 27 19:10:14.381949 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:10:14.381918 2579 patch_prober.go:28] interesting pod/e2e-inf-b-9abe8e-openshift-default-7d4b9c5f84-4nxfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Jul 27 19:10:14.381996 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:10:14.381965 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9abe8e-openshift-default-7d4b9c5f84-4nxfh" podUID="56d09ac8-e6cf-4605-b063-ed13073f2b82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Jul 27 19:10:15.381630 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:10:15.381596 2579 patch_prober.go:28] interesting pod/e2e-inf-b-9abe8e-openshift-default-7d4b9c5f84-4nxfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Jul 27 19:10:15.382025 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:10:15.381657 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9abe8e-openshift-default-7d4b9c5f84-4nxfh" podUID="56d09ac8-e6cf-4605-b063-ed13073f2b82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Jul 27 19:10:16.381358 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:10:16.381319 2579 patch_prober.go:28] interesting pod/e2e-inf-b-9abe8e-openshift-default-7d4b9c5f84-4nxfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Jul 27 19:10:16.381539 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:10:16.381377 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9abe8e-openshift-default-7d4b9c5f84-4nxfh" podUID="56d09ac8-e6cf-4605-b063-ed13073f2b82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Jul 27 19:10:17.381330 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:10:17.381294 2579 patch_prober.go:28] interesting pod/e2e-inf-b-9abe8e-openshift-default-7d4b9c5f84-4nxfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Jul 27 19:10:17.381737 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:10:17.381357 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9abe8e-openshift-default-7d4b9c5f84-4nxfh" podUID="56d09ac8-e6cf-4605-b063-ed13073f2b82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Jul 27 19:10:17.614779 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:10:17.614717 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:10:18.386289 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:10:18.386255 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-9abe8e-openshift-default-7d4b9c5f84-4nxfh" Jul 27 19:10:18.386699 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:10:18.386670 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-9abe8e-openshift-default-7d4b9c5f84-4nxfh" Jul 27 19:10:18.387559 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:10:18.387537 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-9abe8e-openshift-default-7d4b9c5f84-4nxfh" Jul 27 19:10:34.936533 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:10:34.936487 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:11:16.401652 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:11:16.401610 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:11:17.470912 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:11:17.470877 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:11:27.995346 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:11:27.995309 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:11:28.091554 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:11:28.091511 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:11:32.629033 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:11:32.628987 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:11:33.057813 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:11:33.057741 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:11:39.951371 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:11:39.951323 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:11:40.310706 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:11:40.310670 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-f9caec-openshift-default-74586474d-7dq9g"] Jul 27 19:11:40.310908 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:11:40.310875 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-f9caec-openshift-default-74586474d-7dq9g" podUID="24f7ec22-8bca-46b0-9bb4-b92ccab63840" containerName="istio-proxy" containerID="cri-o://9aea012ee64b0732f0359c69f961aa1f88b5c753d3dc9a9c5ac842b663ef3bf4" gracePeriod=30 Jul 27 19:11:40.449821 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:11:40.449788 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:11:40.553568 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:11:40.553536 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:11:40.831821 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:11:40.831787 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:11:45.567976 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:11:45.567944 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f9caec-openshift-default-74586474d-7dq9g" Jul 27 19:11:45.611967 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:11:45.611886 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24f7ec22-8bca-46b0-9bb4-b92ccab63840-istio-token\") pod \"24f7ec22-8bca-46b0-9bb4-b92ccab63840\" (UID: \"24f7ec22-8bca-46b0-9bb4-b92ccab63840\") " Jul 27 19:11:45.611967 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:11:45.611946 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24f7ec22-8bca-46b0-9bb4-b92ccab63840-istiod-ca-cert\") pod \"24f7ec22-8bca-46b0-9bb4-b92ccab63840\" (UID: \"24f7ec22-8bca-46b0-9bb4-b92ccab63840\") " Jul 27 19:11:45.612190 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:11:45.611988 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dp64s\" (UniqueName: \"kubernetes.io/projected/24f7ec22-8bca-46b0-9bb4-b92ccab63840-kube-api-access-dp64s\") pod \"24f7ec22-8bca-46b0-9bb4-b92ccab63840\" (UID: \"24f7ec22-8bca-46b0-9bb4-b92ccab63840\") " Jul 27 19:11:45.612190 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:11:45.612059 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24f7ec22-8bca-46b0-9bb4-b92ccab63840-workload-socket\") pod \"24f7ec22-8bca-46b0-9bb4-b92ccab63840\" (UID: \"24f7ec22-8bca-46b0-9bb4-b92ccab63840\") " Jul 27 19:11:45.612190 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:11:45.612095 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24f7ec22-8bca-46b0-9bb4-b92ccab63840-istio-data\") pod \"24f7ec22-8bca-46b0-9bb4-b92ccab63840\" (UID: \"24f7ec22-8bca-46b0-9bb4-b92ccab63840\") " Jul 27 19:11:45.612190 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:11:45.612149 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24f7ec22-8bca-46b0-9bb4-b92ccab63840-workload-certs\") pod \"24f7ec22-8bca-46b0-9bb4-b92ccab63840\" (UID: \"24f7ec22-8bca-46b0-9bb4-b92ccab63840\") " Jul 27 19:11:45.612449 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:11:45.612201 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24f7ec22-8bca-46b0-9bb4-b92ccab63840-credential-socket\") pod \"24f7ec22-8bca-46b0-9bb4-b92ccab63840\" (UID: \"24f7ec22-8bca-46b0-9bb4-b92ccab63840\") " Jul 27 19:11:45.612449 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:11:45.612300 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24f7ec22-8bca-46b0-9bb4-b92ccab63840-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "24f7ec22-8bca-46b0-9bb4-b92ccab63840" (UID: "24f7ec22-8bca-46b0-9bb4-b92ccab63840"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 19:11:45.612449 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:11:45.612353 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24f7ec22-8bca-46b0-9bb4-b92ccab63840-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "24f7ec22-8bca-46b0-9bb4-b92ccab63840" (UID: "24f7ec22-8bca-46b0-9bb4-b92ccab63840"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 19:11:45.612621 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:11:45.612456 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24f7ec22-8bca-46b0-9bb4-b92ccab63840-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "24f7ec22-8bca-46b0-9bb4-b92ccab63840" (UID: "24f7ec22-8bca-46b0-9bb4-b92ccab63840"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 19:11:45.612621 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:11:45.612559 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24f7ec22-8bca-46b0-9bb4-b92ccab63840-istio-envoy\") pod \"24f7ec22-8bca-46b0-9bb4-b92ccab63840\" (UID: \"24f7ec22-8bca-46b0-9bb4-b92ccab63840\") " Jul 27 19:11:45.612621 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:11:45.612590 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24f7ec22-8bca-46b0-9bb4-b92ccab63840-istio-podinfo\") pod \"24f7ec22-8bca-46b0-9bb4-b92ccab63840\" (UID: \"24f7ec22-8bca-46b0-9bb4-b92ccab63840\") " Jul 27 19:11:45.612789 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:11:45.612562 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24f7ec22-8bca-46b0-9bb4-b92ccab63840-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "24f7ec22-8bca-46b0-9bb4-b92ccab63840" (UID: "24f7ec22-8bca-46b0-9bb4-b92ccab63840"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 19:11:45.612939 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:11:45.612913 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24f7ec22-8bca-46b0-9bb4-b92ccab63840-workload-socket\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 19:11:45.613259 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:11:45.612942 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24f7ec22-8bca-46b0-9bb4-b92ccab63840-workload-certs\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 19:11:45.613259 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:11:45.612960 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24f7ec22-8bca-46b0-9bb4-b92ccab63840-credential-socket\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 19:11:45.613259 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:11:45.612978 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24f7ec22-8bca-46b0-9bb4-b92ccab63840-istiod-ca-cert\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 19:11:45.613449 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:11:45.613330 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24f7ec22-8bca-46b0-9bb4-b92ccab63840-istio-data" (OuterVolumeSpecName: "istio-data") pod "24f7ec22-8bca-46b0-9bb4-b92ccab63840" (UID: "24f7ec22-8bca-46b0-9bb4-b92ccab63840"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 19:11:45.614437 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:11:45.614408 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24f7ec22-8bca-46b0-9bb4-b92ccab63840-kube-api-access-dp64s" (OuterVolumeSpecName: "kube-api-access-dp64s") pod "24f7ec22-8bca-46b0-9bb4-b92ccab63840" (UID: "24f7ec22-8bca-46b0-9bb4-b92ccab63840"). InnerVolumeSpecName "kube-api-access-dp64s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 19:11:45.614720 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:11:45.614697 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/24f7ec22-8bca-46b0-9bb4-b92ccab63840-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "24f7ec22-8bca-46b0-9bb4-b92ccab63840" (UID: "24f7ec22-8bca-46b0-9bb4-b92ccab63840"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 27 19:11:45.614855 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:11:45.614840 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24f7ec22-8bca-46b0-9bb4-b92ccab63840-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "24f7ec22-8bca-46b0-9bb4-b92ccab63840" (UID: "24f7ec22-8bca-46b0-9bb4-b92ccab63840"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 19:11:45.615099 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:11:45.615077 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24f7ec22-8bca-46b0-9bb4-b92ccab63840-istio-token" (OuterVolumeSpecName: "istio-token") pod "24f7ec22-8bca-46b0-9bb4-b92ccab63840" (UID: "24f7ec22-8bca-46b0-9bb4-b92ccab63840"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 19:11:45.713790 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:11:45.713763 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24f7ec22-8bca-46b0-9bb4-b92ccab63840-istio-data\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 19:11:45.713790 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:11:45.713788 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24f7ec22-8bca-46b0-9bb4-b92ccab63840-istio-envoy\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 19:11:45.713790 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:11:45.713797 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24f7ec22-8bca-46b0-9bb4-b92ccab63840-istio-podinfo\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 19:11:45.714009 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:11:45.713806 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24f7ec22-8bca-46b0-9bb4-b92ccab63840-istio-token\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 19:11:45.714009 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:11:45.713815 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dp64s\" (UniqueName: \"kubernetes.io/projected/24f7ec22-8bca-46b0-9bb4-b92ccab63840-kube-api-access-dp64s\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 19:11:46.420752 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:11:46.420717 2579 generic.go:356] "Generic (PLEG): container finished" podID="24f7ec22-8bca-46b0-9bb4-b92ccab63840" containerID="9aea012ee64b0732f0359c69f961aa1f88b5c753d3dc9a9c5ac842b663ef3bf4" exitCode=0 Jul 27 19:11:46.420941 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:11:46.420762 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f9caec-openshift-default-74586474d-7dq9g" event={"ID":"24f7ec22-8bca-46b0-9bb4-b92ccab63840","Type":"ContainerDied","Data":"9aea012ee64b0732f0359c69f961aa1f88b5c753d3dc9a9c5ac842b663ef3bf4"} Jul 27 19:11:46.420941 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:11:46.420785 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f9caec-openshift-default-74586474d-7dq9g" Jul 27 19:11:46.420941 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:11:46.420829 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f9caec-openshift-default-74586474d-7dq9g" event={"ID":"24f7ec22-8bca-46b0-9bb4-b92ccab63840","Type":"ContainerDied","Data":"22b59ab33c14853effbb4ffa43e833e6d546bd8b05e3eaf2a105349c3a50bc26"} Jul 27 19:11:46.420941 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:11:46.420852 2579 scope.go:117] "RemoveContainer" containerID="9aea012ee64b0732f0359c69f961aa1f88b5c753d3dc9a9c5ac842b663ef3bf4" Jul 27 19:11:46.430945 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:11:46.430781 2579 scope.go:117] "RemoveContainer" containerID="5f5722d1d5d6e980deac1d8b6783f6caacb707002018b7b29f49232699a7fd77" Jul 27 19:11:46.439071 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:11:46.439051 2579 scope.go:117] "RemoveContainer" containerID="9aea012ee64b0732f0359c69f961aa1f88b5c753d3dc9a9c5ac842b663ef3bf4" Jul 27 19:11:46.439401 ip-10-0-128-32 kubenswrapper[2579]: E0727 19:11:46.439379 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9aea012ee64b0732f0359c69f961aa1f88b5c753d3dc9a9c5ac842b663ef3bf4\": container with ID starting with 9aea012ee64b0732f0359c69f961aa1f88b5c753d3dc9a9c5ac842b663ef3bf4 not found: ID does not exist" containerID="9aea012ee64b0732f0359c69f961aa1f88b5c753d3dc9a9c5ac842b663ef3bf4" Jul 27 19:11:46.439468 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:11:46.439414 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9aea012ee64b0732f0359c69f961aa1f88b5c753d3dc9a9c5ac842b663ef3bf4"} err="failed to get container status \"9aea012ee64b0732f0359c69f961aa1f88b5c753d3dc9a9c5ac842b663ef3bf4\": rpc error: code = NotFound desc = could not find container \"9aea012ee64b0732f0359c69f961aa1f88b5c753d3dc9a9c5ac842b663ef3bf4\": container with ID starting with 9aea012ee64b0732f0359c69f961aa1f88b5c753d3dc9a9c5ac842b663ef3bf4 not found: ID does not exist" Jul 27 19:11:46.439468 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:11:46.439434 2579 scope.go:117] "RemoveContainer" containerID="5f5722d1d5d6e980deac1d8b6783f6caacb707002018b7b29f49232699a7fd77" Jul 27 19:11:46.439702 ip-10-0-128-32 kubenswrapper[2579]: E0727 19:11:46.439681 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f5722d1d5d6e980deac1d8b6783f6caacb707002018b7b29f49232699a7fd77\": container with ID starting with 5f5722d1d5d6e980deac1d8b6783f6caacb707002018b7b29f49232699a7fd77 not found: ID does not exist" containerID="5f5722d1d5d6e980deac1d8b6783f6caacb707002018b7b29f49232699a7fd77" Jul 27 19:11:46.439757 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:11:46.439706 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f5722d1d5d6e980deac1d8b6783f6caacb707002018b7b29f49232699a7fd77"} err="failed to get container status \"5f5722d1d5d6e980deac1d8b6783f6caacb707002018b7b29f49232699a7fd77\": rpc error: code = NotFound desc = could not find container \"5f5722d1d5d6e980deac1d8b6783f6caacb707002018b7b29f49232699a7fd77\": container with ID starting with 5f5722d1d5d6e980deac1d8b6783f6caacb707002018b7b29f49232699a7fd77 not found: ID does not exist" Jul 27 19:11:46.453780 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:11:46.453753 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-f9caec-openshift-default-74586474d-7dq9g"] Jul 27 19:11:46.456704 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:11:46.456677 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-f9caec-openshift-default-74586474d-7dq9g"] Jul 27 19:11:47.501978 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:11:47.501945 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="24f7ec22-8bca-46b0-9bb4-b92ccab63840" path="/var/lib/kubelet/pods/24f7ec22-8bca-46b0-9bb4-b92ccab63840/volumes" Jul 27 19:11:51.448988 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:11:51.448955 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:11:51.534100 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:11:51.534057 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:11:55.885837 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:11:55.885794 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:11:56.342680 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:11:56.342623 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:12:02.798190 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:02.798148 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:12:02.900822 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:02.900775 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:12:03.061258 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:03.061156 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-9abe8e-openshift-default-7d4b9c5f84-4nxfh"] Jul 27 19:12:03.061423 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:03.061396 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-9abe8e-openshift-default-7d4b9c5f84-4nxfh" podUID="56d09ac8-e6cf-4605-b063-ed13073f2b82" containerName="istio-proxy" containerID="cri-o://608b0387115da6386bb8a0840bd5f8afdfdc22dfea4ae86f0c9c9f2774ffc78d" gracePeriod=30 Jul 27 19:12:03.095538 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:03.095499 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:12:03.304014 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:03.303976 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:12:03.389049 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:03.388962 2579 patch_prober.go:28] interesting pod/e2e-inf-b-9abe8e-openshift-default-7d4b9c5f84-4nxfh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 27 19:12:03.389049 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:03.389023 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-9abe8e-openshift-default-7d4b9c5f84-4nxfh" podUID="56d09ac8-e6cf-4605-b063-ed13073f2b82" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 27 19:12:03.652323 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:03.652207 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:12:03.937310 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:03.937216 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-bf852f-openshift-default-cbcc79f9d-w7kv2"] Jul 27 19:12:03.937715 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:03.937625 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24f7ec22-8bca-46b0-9bb4-b92ccab63840" containerName="istio-proxy" Jul 27 19:12:03.937715 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:03.937635 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="24f7ec22-8bca-46b0-9bb4-b92ccab63840" containerName="istio-proxy" Jul 27 19:12:03.937715 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:03.937702 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="24f7ec22-8bca-46b0-9bb4-b92ccab63840" containerName="istio-proxy" Jul 27 19:12:03.941143 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:03.941121 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-bf852f-openshift-default-cbcc79f9d-w7kv2" Jul 27 19:12:03.944159 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:03.944138 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-bf852f-openshift-default-dockercfg-bws49\"" Jul 27 19:12:03.952221 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:03.952195 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-bf852f-openshift-default-cbcc79f9d-w7kv2"] Jul 27 19:12:03.964207 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:03.964178 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:12:04.086989 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:04.086950 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/600e9c9d-063e-42ce-9045-527f17d838da-istio-podinfo\") pod \"e2e-api-a-bf852f-openshift-default-cbcc79f9d-w7kv2\" (UID: \"600e9c9d-063e-42ce-9045-527f17d838da\") " pod="openshift-ingress/e2e-api-a-bf852f-openshift-default-cbcc79f9d-w7kv2" Jul 27 19:12:04.087182 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:04.086993 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/600e9c9d-063e-42ce-9045-527f17d838da-credential-socket\") pod \"e2e-api-a-bf852f-openshift-default-cbcc79f9d-w7kv2\" (UID: \"600e9c9d-063e-42ce-9045-527f17d838da\") " pod="openshift-ingress/e2e-api-a-bf852f-openshift-default-cbcc79f9d-w7kv2" Jul 27 19:12:04.087182 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:04.087015 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/600e9c9d-063e-42ce-9045-527f17d838da-istio-data\") pod \"e2e-api-a-bf852f-openshift-default-cbcc79f9d-w7kv2\" (UID: \"600e9c9d-063e-42ce-9045-527f17d838da\") " pod="openshift-ingress/e2e-api-a-bf852f-openshift-default-cbcc79f9d-w7kv2" Jul 27 19:12:04.087182 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:04.087111 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/600e9c9d-063e-42ce-9045-527f17d838da-workload-certs\") pod \"e2e-api-a-bf852f-openshift-default-cbcc79f9d-w7kv2\" (UID: \"600e9c9d-063e-42ce-9045-527f17d838da\") " pod="openshift-ingress/e2e-api-a-bf852f-openshift-default-cbcc79f9d-w7kv2" Jul 27 19:12:04.087182 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:04.087146 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/600e9c9d-063e-42ce-9045-527f17d838da-istiod-ca-cert\") pod \"e2e-api-a-bf852f-openshift-default-cbcc79f9d-w7kv2\" (UID: \"600e9c9d-063e-42ce-9045-527f17d838da\") " pod="openshift-ingress/e2e-api-a-bf852f-openshift-default-cbcc79f9d-w7kv2" Jul 27 19:12:04.087182 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:04.087179 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4nqq\" (UniqueName: \"kubernetes.io/projected/600e9c9d-063e-42ce-9045-527f17d838da-kube-api-access-m4nqq\") pod \"e2e-api-a-bf852f-openshift-default-cbcc79f9d-w7kv2\" (UID: \"600e9c9d-063e-42ce-9045-527f17d838da\") " pod="openshift-ingress/e2e-api-a-bf852f-openshift-default-cbcc79f9d-w7kv2" Jul 27 19:12:04.087482 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:04.087284 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/600e9c9d-063e-42ce-9045-527f17d838da-workload-socket\") pod \"e2e-api-a-bf852f-openshift-default-cbcc79f9d-w7kv2\" (UID: \"600e9c9d-063e-42ce-9045-527f17d838da\") " pod="openshift-ingress/e2e-api-a-bf852f-openshift-default-cbcc79f9d-w7kv2" Jul 27 19:12:04.087482 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:04.087306 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/600e9c9d-063e-42ce-9045-527f17d838da-istio-token\") pod \"e2e-api-a-bf852f-openshift-default-cbcc79f9d-w7kv2\" (UID: \"600e9c9d-063e-42ce-9045-527f17d838da\") " pod="openshift-ingress/e2e-api-a-bf852f-openshift-default-cbcc79f9d-w7kv2" Jul 27 19:12:04.087482 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:04.087339 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/600e9c9d-063e-42ce-9045-527f17d838da-istio-envoy\") pod \"e2e-api-a-bf852f-openshift-default-cbcc79f9d-w7kv2\" (UID: \"600e9c9d-063e-42ce-9045-527f17d838da\") " pod="openshift-ingress/e2e-api-a-bf852f-openshift-default-cbcc79f9d-w7kv2" Jul 27 19:12:04.188607 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:04.188522 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/600e9c9d-063e-42ce-9045-527f17d838da-istio-podinfo\") pod \"e2e-api-a-bf852f-openshift-default-cbcc79f9d-w7kv2\" (UID: \"600e9c9d-063e-42ce-9045-527f17d838da\") " pod="openshift-ingress/e2e-api-a-bf852f-openshift-default-cbcc79f9d-w7kv2" Jul 27 19:12:04.188607 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:04.188579 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/600e9c9d-063e-42ce-9045-527f17d838da-credential-socket\") pod \"e2e-api-a-bf852f-openshift-default-cbcc79f9d-w7kv2\" (UID: \"600e9c9d-063e-42ce-9045-527f17d838da\") " pod="openshift-ingress/e2e-api-a-bf852f-openshift-default-cbcc79f9d-w7kv2" Jul 27 19:12:04.188832 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:04.188610 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/600e9c9d-063e-42ce-9045-527f17d838da-istio-data\") pod \"e2e-api-a-bf852f-openshift-default-cbcc79f9d-w7kv2\" (UID: \"600e9c9d-063e-42ce-9045-527f17d838da\") " pod="openshift-ingress/e2e-api-a-bf852f-openshift-default-cbcc79f9d-w7kv2" Jul 27 19:12:04.188832 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:04.188668 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/600e9c9d-063e-42ce-9045-527f17d838da-workload-certs\") pod \"e2e-api-a-bf852f-openshift-default-cbcc79f9d-w7kv2\" (UID: \"600e9c9d-063e-42ce-9045-527f17d838da\") " pod="openshift-ingress/e2e-api-a-bf852f-openshift-default-cbcc79f9d-w7kv2" Jul 27 19:12:04.188832 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:04.188694 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/600e9c9d-063e-42ce-9045-527f17d838da-istiod-ca-cert\") pod \"e2e-api-a-bf852f-openshift-default-cbcc79f9d-w7kv2\" (UID: \"600e9c9d-063e-42ce-9045-527f17d838da\") " pod="openshift-ingress/e2e-api-a-bf852f-openshift-default-cbcc79f9d-w7kv2" Jul 27 19:12:04.188832 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:04.188731 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4nqq\" (UniqueName: \"kubernetes.io/projected/600e9c9d-063e-42ce-9045-527f17d838da-kube-api-access-m4nqq\") pod \"e2e-api-a-bf852f-openshift-default-cbcc79f9d-w7kv2\" (UID: \"600e9c9d-063e-42ce-9045-527f17d838da\") " pod="openshift-ingress/e2e-api-a-bf852f-openshift-default-cbcc79f9d-w7kv2" Jul 27 19:12:04.188832 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:04.188776 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/600e9c9d-063e-42ce-9045-527f17d838da-workload-socket\") pod \"e2e-api-a-bf852f-openshift-default-cbcc79f9d-w7kv2\" (UID: \"600e9c9d-063e-42ce-9045-527f17d838da\") " pod="openshift-ingress/e2e-api-a-bf852f-openshift-default-cbcc79f9d-w7kv2" Jul 27 19:12:04.188832 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:04.188803 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/600e9c9d-063e-42ce-9045-527f17d838da-istio-token\") pod \"e2e-api-a-bf852f-openshift-default-cbcc79f9d-w7kv2\" (UID: \"600e9c9d-063e-42ce-9045-527f17d838da\") " pod="openshift-ingress/e2e-api-a-bf852f-openshift-default-cbcc79f9d-w7kv2" Jul 27 19:12:04.189141 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:04.188840 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/600e9c9d-063e-42ce-9045-527f17d838da-istio-envoy\") pod \"e2e-api-a-bf852f-openshift-default-cbcc79f9d-w7kv2\" (UID: \"600e9c9d-063e-42ce-9045-527f17d838da\") " pod="openshift-ingress/e2e-api-a-bf852f-openshift-default-cbcc79f9d-w7kv2" Jul 27 19:12:04.189141 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:04.189071 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/600e9c9d-063e-42ce-9045-527f17d838da-credential-socket\") pod \"e2e-api-a-bf852f-openshift-default-cbcc79f9d-w7kv2\" (UID: \"600e9c9d-063e-42ce-9045-527f17d838da\") " pod="openshift-ingress/e2e-api-a-bf852f-openshift-default-cbcc79f9d-w7kv2" Jul 27 19:12:04.189141 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:04.189078 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/600e9c9d-063e-42ce-9045-527f17d838da-workload-certs\") pod \"e2e-api-a-bf852f-openshift-default-cbcc79f9d-w7kv2\" (UID: \"600e9c9d-063e-42ce-9045-527f17d838da\") " pod="openshift-ingress/e2e-api-a-bf852f-openshift-default-cbcc79f9d-w7kv2" Jul 27 19:12:04.189420 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:04.189387 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/600e9c9d-063e-42ce-9045-527f17d838da-istio-data\") pod \"e2e-api-a-bf852f-openshift-default-cbcc79f9d-w7kv2\" (UID: \"600e9c9d-063e-42ce-9045-527f17d838da\") " pod="openshift-ingress/e2e-api-a-bf852f-openshift-default-cbcc79f9d-w7kv2" Jul 27 19:12:04.189666 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:04.189636 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/600e9c9d-063e-42ce-9045-527f17d838da-workload-socket\") pod \"e2e-api-a-bf852f-openshift-default-cbcc79f9d-w7kv2\" (UID: \"600e9c9d-063e-42ce-9045-527f17d838da\") " pod="openshift-ingress/e2e-api-a-bf852f-openshift-default-cbcc79f9d-w7kv2" Jul 27 19:12:04.189781 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:04.189642 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/600e9c9d-063e-42ce-9045-527f17d838da-istiod-ca-cert\") pod \"e2e-api-a-bf852f-openshift-default-cbcc79f9d-w7kv2\" (UID: \"600e9c9d-063e-42ce-9045-527f17d838da\") " pod="openshift-ingress/e2e-api-a-bf852f-openshift-default-cbcc79f9d-w7kv2" Jul 27 19:12:04.191321 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:04.191302 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/600e9c9d-063e-42ce-9045-527f17d838da-istio-envoy\") pod \"e2e-api-a-bf852f-openshift-default-cbcc79f9d-w7kv2\" (UID: \"600e9c9d-063e-42ce-9045-527f17d838da\") " pod="openshift-ingress/e2e-api-a-bf852f-openshift-default-cbcc79f9d-w7kv2" Jul 27 19:12:04.191321 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:04.191299 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/600e9c9d-063e-42ce-9045-527f17d838da-istio-podinfo\") pod \"e2e-api-a-bf852f-openshift-default-cbcc79f9d-w7kv2\" (UID: \"600e9c9d-063e-42ce-9045-527f17d838da\") " pod="openshift-ingress/e2e-api-a-bf852f-openshift-default-cbcc79f9d-w7kv2" Jul 27 19:12:04.197619 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:04.197593 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4nqq\" (UniqueName: \"kubernetes.io/projected/600e9c9d-063e-42ce-9045-527f17d838da-kube-api-access-m4nqq\") pod \"e2e-api-a-bf852f-openshift-default-cbcc79f9d-w7kv2\" (UID: \"600e9c9d-063e-42ce-9045-527f17d838da\") " pod="openshift-ingress/e2e-api-a-bf852f-openshift-default-cbcc79f9d-w7kv2" Jul 27 19:12:04.198391 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:04.198371 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/600e9c9d-063e-42ce-9045-527f17d838da-istio-token\") pod \"e2e-api-a-bf852f-openshift-default-cbcc79f9d-w7kv2\" (UID: \"600e9c9d-063e-42ce-9045-527f17d838da\") " pod="openshift-ingress/e2e-api-a-bf852f-openshift-default-cbcc79f9d-w7kv2" Jul 27 19:12:04.229239 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:04.229198 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:12:04.252201 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:04.252173 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-bf852f-openshift-default-cbcc79f9d-w7kv2" Jul 27 19:12:04.398729 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:04.398643 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-bf852f-openshift-default-cbcc79f9d-w7kv2"] Jul 27 19:12:04.401138 ip-10-0-128-32 kubenswrapper[2579]: W0727 19:12:04.401111 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod600e9c9d_063e_42ce_9045_527f17d838da.slice/crio-649522f682116d06d40194e4418a25bb8814f5911a979d5a8daf5b95cdcf1aff WatchSource:0}: Error finding container 649522f682116d06d40194e4418a25bb8814f5911a979d5a8daf5b95cdcf1aff: Status 404 returned error can't find the container with id 649522f682116d06d40194e4418a25bb8814f5911a979d5a8daf5b95cdcf1aff Jul 27 19:12:04.403511 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:04.403470 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 27 19:12:04.403596 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:04.403569 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 27 19:12:04.403640 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:04.403613 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 27 19:12:04.513183 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:04.513147 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-bf852f-openshift-default-cbcc79f9d-w7kv2" event={"ID":"600e9c9d-063e-42ce-9045-527f17d838da","Type":"ContainerStarted","Data":"4df13aa44e890e3202d4ab988ff44a0470123eff09fb3e42a9245930fba1c196"} Jul 27 19:12:04.513415 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:04.513212 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-bf852f-openshift-default-cbcc79f9d-w7kv2" event={"ID":"600e9c9d-063e-42ce-9045-527f17d838da","Type":"ContainerStarted","Data":"649522f682116d06d40194e4418a25bb8814f5911a979d5a8daf5b95cdcf1aff"} Jul 27 19:12:04.541441 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:04.541379 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-bf852f-openshift-default-cbcc79f9d-w7kv2" podStartSLOduration=1.5413620639999999 podStartE2EDuration="1.541362064s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:12:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:12:04.53816258 +0000 UTC m=+2599.599851625" watchObservedRunningTime="2026-07-27 19:12:04.541362064 +0000 UTC m=+2599.603051094" Jul 27 19:12:05.253079 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:05.253039 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-bf852f-openshift-default-cbcc79f9d-w7kv2" Jul 27 19:12:05.257951 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:05.257926 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-bf852f-openshift-default-cbcc79f9d-w7kv2" Jul 27 19:12:05.518739 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:05.518631 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-bf852f-openshift-default-cbcc79f9d-w7kv2" Jul 27 19:12:05.519677 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:05.519658 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-bf852f-openshift-default-cbcc79f9d-w7kv2" Jul 27 19:12:06.516021 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:06.515955 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:12:06.602033 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:06.601990 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:12:06.709733 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:06.709691 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:12:06.801407 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:06.801376 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:12:07.320880 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:07.320852 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:12:07.460653 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:07.460619 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:12:08.327445 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:08.327364 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9abe8e-openshift-default-7d4b9c5f84-4nxfh" Jul 27 19:12:08.431037 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:08.431000 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56d09ac8-e6cf-4605-b063-ed13073f2b82-workload-certs\") pod \"56d09ac8-e6cf-4605-b063-ed13073f2b82\" (UID: \"56d09ac8-e6cf-4605-b063-ed13073f2b82\") " Jul 27 19:12:08.431037 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:08.431038 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56d09ac8-e6cf-4605-b063-ed13073f2b82-istio-envoy\") pod \"56d09ac8-e6cf-4605-b063-ed13073f2b82\" (UID: \"56d09ac8-e6cf-4605-b063-ed13073f2b82\") " Jul 27 19:12:08.431341 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:08.431076 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56d09ac8-e6cf-4605-b063-ed13073f2b82-workload-socket\") pod \"56d09ac8-e6cf-4605-b063-ed13073f2b82\" (UID: \"56d09ac8-e6cf-4605-b063-ed13073f2b82\") " Jul 27 19:12:08.431341 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:08.431127 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56d09ac8-e6cf-4605-b063-ed13073f2b82-istio-token\") pod \"56d09ac8-e6cf-4605-b063-ed13073f2b82\" (UID: \"56d09ac8-e6cf-4605-b063-ed13073f2b82\") " Jul 27 19:12:08.431341 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:08.431181 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56d09ac8-e6cf-4605-b063-ed13073f2b82-istiod-ca-cert\") pod \"56d09ac8-e6cf-4605-b063-ed13073f2b82\" (UID: \"56d09ac8-e6cf-4605-b063-ed13073f2b82\") " Jul 27 19:12:08.431341 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:08.431255 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56d09ac8-e6cf-4605-b063-ed13073f2b82-istio-data\") pod \"56d09ac8-e6cf-4605-b063-ed13073f2b82\" (UID: \"56d09ac8-e6cf-4605-b063-ed13073f2b82\") " Jul 27 19:12:08.431341 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:08.431288 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4s2zs\" (UniqueName: \"kubernetes.io/projected/56d09ac8-e6cf-4605-b063-ed13073f2b82-kube-api-access-4s2zs\") pod \"56d09ac8-e6cf-4605-b063-ed13073f2b82\" (UID: \"56d09ac8-e6cf-4605-b063-ed13073f2b82\") " Jul 27 19:12:08.431614 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:08.431359 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56d09ac8-e6cf-4605-b063-ed13073f2b82-istio-podinfo\") pod \"56d09ac8-e6cf-4605-b063-ed13073f2b82\" (UID: \"56d09ac8-e6cf-4605-b063-ed13073f2b82\") " Jul 27 19:12:08.431614 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:08.431384 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56d09ac8-e6cf-4605-b063-ed13073f2b82-credential-socket\") pod \"56d09ac8-e6cf-4605-b063-ed13073f2b82\" (UID: \"56d09ac8-e6cf-4605-b063-ed13073f2b82\") " Jul 27 19:12:08.431614 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:08.431402 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56d09ac8-e6cf-4605-b063-ed13073f2b82-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "56d09ac8-e6cf-4605-b063-ed13073f2b82" (UID: "56d09ac8-e6cf-4605-b063-ed13073f2b82"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 19:12:08.431614 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:08.431464 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56d09ac8-e6cf-4605-b063-ed13073f2b82-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "56d09ac8-e6cf-4605-b063-ed13073f2b82" (UID: "56d09ac8-e6cf-4605-b063-ed13073f2b82"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 19:12:08.431821 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:08.431661 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56d09ac8-e6cf-4605-b063-ed13073f2b82-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "56d09ac8-e6cf-4605-b063-ed13073f2b82" (UID: "56d09ac8-e6cf-4605-b063-ed13073f2b82"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 19:12:08.431821 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:08.431733 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56d09ac8-e6cf-4605-b063-ed13073f2b82-workload-socket\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 19:12:08.431821 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:08.431754 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56d09ac8-e6cf-4605-b063-ed13073f2b82-istiod-ca-cert\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 19:12:08.431821 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:08.431769 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56d09ac8-e6cf-4605-b063-ed13073f2b82-workload-certs\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 19:12:08.432013 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:08.431897 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56d09ac8-e6cf-4605-b063-ed13073f2b82-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "56d09ac8-e6cf-4605-b063-ed13073f2b82" (UID: "56d09ac8-e6cf-4605-b063-ed13073f2b82"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 19:12:08.432363 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:08.432336 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56d09ac8-e6cf-4605-b063-ed13073f2b82-istio-data" (OuterVolumeSpecName: "istio-data") pod "56d09ac8-e6cf-4605-b063-ed13073f2b82" (UID: "56d09ac8-e6cf-4605-b063-ed13073f2b82"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 19:12:08.433774 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:08.433752 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56d09ac8-e6cf-4605-b063-ed13073f2b82-istio-token" (OuterVolumeSpecName: "istio-token") pod "56d09ac8-e6cf-4605-b063-ed13073f2b82" (UID: "56d09ac8-e6cf-4605-b063-ed13073f2b82"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 19:12:08.433922 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:08.433907 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56d09ac8-e6cf-4605-b063-ed13073f2b82-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "56d09ac8-e6cf-4605-b063-ed13073f2b82" (UID: "56d09ac8-e6cf-4605-b063-ed13073f2b82"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 19:12:08.434185 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:08.434166 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/56d09ac8-e6cf-4605-b063-ed13073f2b82-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "56d09ac8-e6cf-4605-b063-ed13073f2b82" (UID: "56d09ac8-e6cf-4605-b063-ed13073f2b82"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 27 19:12:08.434500 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:08.434468 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56d09ac8-e6cf-4605-b063-ed13073f2b82-kube-api-access-4s2zs" (OuterVolumeSpecName: "kube-api-access-4s2zs") pod "56d09ac8-e6cf-4605-b063-ed13073f2b82" (UID: "56d09ac8-e6cf-4605-b063-ed13073f2b82"). InnerVolumeSpecName "kube-api-access-4s2zs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 19:12:08.532889 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:08.532855 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56d09ac8-e6cf-4605-b063-ed13073f2b82-istio-token\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 19:12:08.532889 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:08.532883 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56d09ac8-e6cf-4605-b063-ed13073f2b82-istio-data\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 19:12:08.532889 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:08.532893 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4s2zs\" (UniqueName: \"kubernetes.io/projected/56d09ac8-e6cf-4605-b063-ed13073f2b82-kube-api-access-4s2zs\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 19:12:08.533106 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:08.532903 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56d09ac8-e6cf-4605-b063-ed13073f2b82-istio-podinfo\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 19:12:08.533106 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:08.532914 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56d09ac8-e6cf-4605-b063-ed13073f2b82-credential-socket\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 19:12:08.533106 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:08.532921 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56d09ac8-e6cf-4605-b063-ed13073f2b82-istio-envoy\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 19:12:08.541111 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:08.541079 2579 generic.go:356] "Generic (PLEG): container finished" podID="56d09ac8-e6cf-4605-b063-ed13073f2b82" containerID="608b0387115da6386bb8a0840bd5f8afdfdc22dfea4ae86f0c9c9f2774ffc78d" exitCode=0 Jul 27 19:12:08.541288 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:08.541135 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9abe8e-openshift-default-7d4b9c5f84-4nxfh" Jul 27 19:12:08.541288 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:08.541172 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9abe8e-openshift-default-7d4b9c5f84-4nxfh" event={"ID":"56d09ac8-e6cf-4605-b063-ed13073f2b82","Type":"ContainerDied","Data":"608b0387115da6386bb8a0840bd5f8afdfdc22dfea4ae86f0c9c9f2774ffc78d"} Jul 27 19:12:08.541389 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:08.541284 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9abe8e-openshift-default-7d4b9c5f84-4nxfh" event={"ID":"56d09ac8-e6cf-4605-b063-ed13073f2b82","Type":"ContainerDied","Data":"6a1d48a3bc0b6270d3e648276ca442cc4f5aa72adb10aa3a9949145e872f58f3"} Jul 27 19:12:08.541389 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:08.541325 2579 scope.go:117] "RemoveContainer" containerID="608b0387115da6386bb8a0840bd5f8afdfdc22dfea4ae86f0c9c9f2774ffc78d" Jul 27 19:12:08.551422 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:08.551404 2579 scope.go:117] "RemoveContainer" containerID="b20b07fbb08adc2bdd7a486107db6e20ab9009522b870029d8767ed59002eb23" Jul 27 19:12:08.559484 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:08.559460 2579 scope.go:117] "RemoveContainer" containerID="608b0387115da6386bb8a0840bd5f8afdfdc22dfea4ae86f0c9c9f2774ffc78d" Jul 27 19:12:08.559728 ip-10-0-128-32 kubenswrapper[2579]: E0727 19:12:08.559708 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"608b0387115da6386bb8a0840bd5f8afdfdc22dfea4ae86f0c9c9f2774ffc78d\": container with ID starting with 608b0387115da6386bb8a0840bd5f8afdfdc22dfea4ae86f0c9c9f2774ffc78d not found: ID does not exist" containerID="608b0387115da6386bb8a0840bd5f8afdfdc22dfea4ae86f0c9c9f2774ffc78d" Jul 27 19:12:08.559779 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:08.559737 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"608b0387115da6386bb8a0840bd5f8afdfdc22dfea4ae86f0c9c9f2774ffc78d"} err="failed to get container status \"608b0387115da6386bb8a0840bd5f8afdfdc22dfea4ae86f0c9c9f2774ffc78d\": rpc error: code = NotFound desc = could not find container \"608b0387115da6386bb8a0840bd5f8afdfdc22dfea4ae86f0c9c9f2774ffc78d\": container with ID starting with 608b0387115da6386bb8a0840bd5f8afdfdc22dfea4ae86f0c9c9f2774ffc78d not found: ID does not exist" Jul 27 19:12:08.559779 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:08.559755 2579 scope.go:117] "RemoveContainer" containerID="b20b07fbb08adc2bdd7a486107db6e20ab9009522b870029d8767ed59002eb23" Jul 27 19:12:08.560001 ip-10-0-128-32 kubenswrapper[2579]: E0727 19:12:08.559981 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b20b07fbb08adc2bdd7a486107db6e20ab9009522b870029d8767ed59002eb23\": container with ID starting with b20b07fbb08adc2bdd7a486107db6e20ab9009522b870029d8767ed59002eb23 not found: ID does not exist" containerID="b20b07fbb08adc2bdd7a486107db6e20ab9009522b870029d8767ed59002eb23" Jul 27 19:12:08.560051 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:08.560007 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b20b07fbb08adc2bdd7a486107db6e20ab9009522b870029d8767ed59002eb23"} err="failed to get container status \"b20b07fbb08adc2bdd7a486107db6e20ab9009522b870029d8767ed59002eb23\": rpc error: code = NotFound desc = could not find container \"b20b07fbb08adc2bdd7a486107db6e20ab9009522b870029d8767ed59002eb23\": container with ID starting with b20b07fbb08adc2bdd7a486107db6e20ab9009522b870029d8767ed59002eb23 not found: ID does not exist" Jul 27 19:12:08.566010 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:08.565986 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-9abe8e-openshift-default-7d4b9c5f84-4nxfh"] Jul 27 19:12:08.573604 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:08.573583 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-9abe8e-openshift-default-7d4b9c5f84-4nxfh"] Jul 27 19:12:09.493440 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:09.493403 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="56d09ac8-e6cf-4605-b063-ed13073f2b82" path="/var/lib/kubelet/pods/56d09ac8-e6cf-4605-b063-ed13073f2b82/volumes" Jul 27 19:12:11.070624 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:11.070584 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:12:11.578290 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:11.578216 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:12:11.858588 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:11.858491 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-e1ba46-openshift-default-94f679f5b-rbdg4"] Jul 27 19:12:11.859438 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:11.859410 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56d09ac8-e6cf-4605-b063-ed13073f2b82" containerName="istio-proxy" Jul 27 19:12:11.859602 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:11.859588 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="56d09ac8-e6cf-4605-b063-ed13073f2b82" containerName="istio-proxy" Jul 27 19:12:11.859717 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:11.859707 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24f7ec22-8bca-46b0-9bb4-b92ccab63840" containerName="istio-proxy" Jul 27 19:12:11.859801 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:11.859792 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="24f7ec22-8bca-46b0-9bb4-b92ccab63840" containerName="istio-proxy" Jul 27 19:12:11.859901 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:11.859891 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56d09ac8-e6cf-4605-b063-ed13073f2b82" containerName="istio-proxy" Jul 27 19:12:11.859970 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:11.859961 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="56d09ac8-e6cf-4605-b063-ed13073f2b82" containerName="istio-proxy" Jul 27 19:12:11.860281 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:11.860264 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="56d09ac8-e6cf-4605-b063-ed13073f2b82" containerName="istio-proxy" Jul 27 19:12:11.860419 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:11.860409 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="24f7ec22-8bca-46b0-9bb4-b92ccab63840" containerName="istio-proxy" Jul 27 19:12:11.865857 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:11.865838 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e1ba46-openshift-default-94f679f5b-rbdg4" Jul 27 19:12:11.869121 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:11.869103 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-e1ba46-openshift-default-dockercfg-zncsz\"" Jul 27 19:12:11.874847 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:11.874821 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-e1ba46-openshift-default-94f679f5b-rbdg4"] Jul 27 19:12:11.893497 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:11.893467 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:12:11.965764 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:11.965716 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8af1d827-36b6-48f4-a93c-26228a71f784-credential-socket\") pod \"e2e-api-b-e1ba46-openshift-default-94f679f5b-rbdg4\" (UID: \"8af1d827-36b6-48f4-a93c-26228a71f784\") " pod="openshift-ingress/e2e-api-b-e1ba46-openshift-default-94f679f5b-rbdg4" Jul 27 19:12:11.965948 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:11.965776 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8af1d827-36b6-48f4-a93c-26228a71f784-istio-data\") pod \"e2e-api-b-e1ba46-openshift-default-94f679f5b-rbdg4\" (UID: \"8af1d827-36b6-48f4-a93c-26228a71f784\") " pod="openshift-ingress/e2e-api-b-e1ba46-openshift-default-94f679f5b-rbdg4" Jul 27 19:12:11.965948 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:11.965829 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8af1d827-36b6-48f4-a93c-26228a71f784-workload-certs\") pod \"e2e-api-b-e1ba46-openshift-default-94f679f5b-rbdg4\" (UID: \"8af1d827-36b6-48f4-a93c-26228a71f784\") " pod="openshift-ingress/e2e-api-b-e1ba46-openshift-default-94f679f5b-rbdg4" Jul 27 19:12:11.965948 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:11.965862 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8af1d827-36b6-48f4-a93c-26228a71f784-istio-envoy\") pod \"e2e-api-b-e1ba46-openshift-default-94f679f5b-rbdg4\" (UID: \"8af1d827-36b6-48f4-a93c-26228a71f784\") " pod="openshift-ingress/e2e-api-b-e1ba46-openshift-default-94f679f5b-rbdg4" Jul 27 19:12:11.965948 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:11.965887 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8af1d827-36b6-48f4-a93c-26228a71f784-istiod-ca-cert\") pod \"e2e-api-b-e1ba46-openshift-default-94f679f5b-rbdg4\" (UID: \"8af1d827-36b6-48f4-a93c-26228a71f784\") " pod="openshift-ingress/e2e-api-b-e1ba46-openshift-default-94f679f5b-rbdg4" Jul 27 19:12:11.965948 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:11.965918 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8af1d827-36b6-48f4-a93c-26228a71f784-workload-socket\") pod \"e2e-api-b-e1ba46-openshift-default-94f679f5b-rbdg4\" (UID: \"8af1d827-36b6-48f4-a93c-26228a71f784\") " pod="openshift-ingress/e2e-api-b-e1ba46-openshift-default-94f679f5b-rbdg4" Jul 27 19:12:11.966194 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:11.965949 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8af1d827-36b6-48f4-a93c-26228a71f784-istio-token\") pod \"e2e-api-b-e1ba46-openshift-default-94f679f5b-rbdg4\" (UID: \"8af1d827-36b6-48f4-a93c-26228a71f784\") " pod="openshift-ingress/e2e-api-b-e1ba46-openshift-default-94f679f5b-rbdg4" Jul 27 19:12:11.966194 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:11.966015 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8af1d827-36b6-48f4-a93c-26228a71f784-istio-podinfo\") pod \"e2e-api-b-e1ba46-openshift-default-94f679f5b-rbdg4\" (UID: \"8af1d827-36b6-48f4-a93c-26228a71f784\") " pod="openshift-ingress/e2e-api-b-e1ba46-openshift-default-94f679f5b-rbdg4" Jul 27 19:12:11.966194 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:11.966056 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhhl9\" (UniqueName: \"kubernetes.io/projected/8af1d827-36b6-48f4-a93c-26228a71f784-kube-api-access-vhhl9\") pod \"e2e-api-b-e1ba46-openshift-default-94f679f5b-rbdg4\" (UID: \"8af1d827-36b6-48f4-a93c-26228a71f784\") " pod="openshift-ingress/e2e-api-b-e1ba46-openshift-default-94f679f5b-rbdg4" Jul 27 19:12:12.067331 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:12.067294 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhhl9\" (UniqueName: \"kubernetes.io/projected/8af1d827-36b6-48f4-a93c-26228a71f784-kube-api-access-vhhl9\") pod \"e2e-api-b-e1ba46-openshift-default-94f679f5b-rbdg4\" (UID: \"8af1d827-36b6-48f4-a93c-26228a71f784\") " pod="openshift-ingress/e2e-api-b-e1ba46-openshift-default-94f679f5b-rbdg4" Jul 27 19:12:12.067526 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:12.067367 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8af1d827-36b6-48f4-a93c-26228a71f784-credential-socket\") pod \"e2e-api-b-e1ba46-openshift-default-94f679f5b-rbdg4\" (UID: \"8af1d827-36b6-48f4-a93c-26228a71f784\") " pod="openshift-ingress/e2e-api-b-e1ba46-openshift-default-94f679f5b-rbdg4" Jul 27 19:12:12.067526 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:12.067397 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8af1d827-36b6-48f4-a93c-26228a71f784-istio-data\") pod \"e2e-api-b-e1ba46-openshift-default-94f679f5b-rbdg4\" (UID: \"8af1d827-36b6-48f4-a93c-26228a71f784\") " pod="openshift-ingress/e2e-api-b-e1ba46-openshift-default-94f679f5b-rbdg4" Jul 27 19:12:12.067526 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:12.067444 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8af1d827-36b6-48f4-a93c-26228a71f784-workload-certs\") pod \"e2e-api-b-e1ba46-openshift-default-94f679f5b-rbdg4\" (UID: \"8af1d827-36b6-48f4-a93c-26228a71f784\") " pod="openshift-ingress/e2e-api-b-e1ba46-openshift-default-94f679f5b-rbdg4" Jul 27 19:12:12.067526 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:12.067476 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8af1d827-36b6-48f4-a93c-26228a71f784-istio-envoy\") pod \"e2e-api-b-e1ba46-openshift-default-94f679f5b-rbdg4\" (UID: \"8af1d827-36b6-48f4-a93c-26228a71f784\") " pod="openshift-ingress/e2e-api-b-e1ba46-openshift-default-94f679f5b-rbdg4" Jul 27 19:12:12.067526 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:12.067503 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8af1d827-36b6-48f4-a93c-26228a71f784-istiod-ca-cert\") pod \"e2e-api-b-e1ba46-openshift-default-94f679f5b-rbdg4\" (UID: \"8af1d827-36b6-48f4-a93c-26228a71f784\") " pod="openshift-ingress/e2e-api-b-e1ba46-openshift-default-94f679f5b-rbdg4" Jul 27 19:12:12.067797 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:12.067537 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8af1d827-36b6-48f4-a93c-26228a71f784-workload-socket\") pod \"e2e-api-b-e1ba46-openshift-default-94f679f5b-rbdg4\" (UID: \"8af1d827-36b6-48f4-a93c-26228a71f784\") " pod="openshift-ingress/e2e-api-b-e1ba46-openshift-default-94f679f5b-rbdg4" Jul 27 19:12:12.067797 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:12.067566 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8af1d827-36b6-48f4-a93c-26228a71f784-istio-token\") pod \"e2e-api-b-e1ba46-openshift-default-94f679f5b-rbdg4\" (UID: \"8af1d827-36b6-48f4-a93c-26228a71f784\") " pod="openshift-ingress/e2e-api-b-e1ba46-openshift-default-94f679f5b-rbdg4" Jul 27 19:12:12.067797 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:12.067665 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8af1d827-36b6-48f4-a93c-26228a71f784-istio-podinfo\") pod \"e2e-api-b-e1ba46-openshift-default-94f679f5b-rbdg4\" (UID: \"8af1d827-36b6-48f4-a93c-26228a71f784\") " pod="openshift-ingress/e2e-api-b-e1ba46-openshift-default-94f679f5b-rbdg4" Jul 27 19:12:12.067797 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:12.067763 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8af1d827-36b6-48f4-a93c-26228a71f784-credential-socket\") pod \"e2e-api-b-e1ba46-openshift-default-94f679f5b-rbdg4\" (UID: \"8af1d827-36b6-48f4-a93c-26228a71f784\") " pod="openshift-ingress/e2e-api-b-e1ba46-openshift-default-94f679f5b-rbdg4" Jul 27 19:12:12.067969 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:12.067832 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8af1d827-36b6-48f4-a93c-26228a71f784-workload-certs\") pod \"e2e-api-b-e1ba46-openshift-default-94f679f5b-rbdg4\" (UID: \"8af1d827-36b6-48f4-a93c-26228a71f784\") " pod="openshift-ingress/e2e-api-b-e1ba46-openshift-default-94f679f5b-rbdg4" Jul 27 19:12:12.067969 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:12.067947 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8af1d827-36b6-48f4-a93c-26228a71f784-istio-data\") pod \"e2e-api-b-e1ba46-openshift-default-94f679f5b-rbdg4\" (UID: \"8af1d827-36b6-48f4-a93c-26228a71f784\") " pod="openshift-ingress/e2e-api-b-e1ba46-openshift-default-94f679f5b-rbdg4" Jul 27 19:12:12.068079 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:12.068057 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8af1d827-36b6-48f4-a93c-26228a71f784-workload-socket\") pod \"e2e-api-b-e1ba46-openshift-default-94f679f5b-rbdg4\" (UID: \"8af1d827-36b6-48f4-a93c-26228a71f784\") " pod="openshift-ingress/e2e-api-b-e1ba46-openshift-default-94f679f5b-rbdg4" Jul 27 19:12:12.068397 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:12.068326 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8af1d827-36b6-48f4-a93c-26228a71f784-istiod-ca-cert\") pod \"e2e-api-b-e1ba46-openshift-default-94f679f5b-rbdg4\" (UID: \"8af1d827-36b6-48f4-a93c-26228a71f784\") " pod="openshift-ingress/e2e-api-b-e1ba46-openshift-default-94f679f5b-rbdg4" Jul 27 19:12:12.070117 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:12.070088 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8af1d827-36b6-48f4-a93c-26228a71f784-istio-envoy\") pod \"e2e-api-b-e1ba46-openshift-default-94f679f5b-rbdg4\" (UID: \"8af1d827-36b6-48f4-a93c-26228a71f784\") " pod="openshift-ingress/e2e-api-b-e1ba46-openshift-default-94f679f5b-rbdg4" Jul 27 19:12:12.070316 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:12.070297 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8af1d827-36b6-48f4-a93c-26228a71f784-istio-podinfo\") pod \"e2e-api-b-e1ba46-openshift-default-94f679f5b-rbdg4\" (UID: \"8af1d827-36b6-48f4-a93c-26228a71f784\") " pod="openshift-ingress/e2e-api-b-e1ba46-openshift-default-94f679f5b-rbdg4" Jul 27 19:12:12.076675 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:12.076649 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhhl9\" (UniqueName: \"kubernetes.io/projected/8af1d827-36b6-48f4-a93c-26228a71f784-kube-api-access-vhhl9\") pod \"e2e-api-b-e1ba46-openshift-default-94f679f5b-rbdg4\" (UID: \"8af1d827-36b6-48f4-a93c-26228a71f784\") " pod="openshift-ingress/e2e-api-b-e1ba46-openshift-default-94f679f5b-rbdg4" Jul 27 19:12:12.077013 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:12.076864 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8af1d827-36b6-48f4-a93c-26228a71f784-istio-token\") pod \"e2e-api-b-e1ba46-openshift-default-94f679f5b-rbdg4\" (UID: \"8af1d827-36b6-48f4-a93c-26228a71f784\") " pod="openshift-ingress/e2e-api-b-e1ba46-openshift-default-94f679f5b-rbdg4" Jul 27 19:12:12.109592 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:12.109528 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:12:12.177501 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:12.177463 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e1ba46-openshift-default-94f679f5b-rbdg4" Jul 27 19:12:12.322697 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:12.322672 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-e1ba46-openshift-default-94f679f5b-rbdg4"] Jul 27 19:12:12.324803 ip-10-0-128-32 kubenswrapper[2579]: W0727 19:12:12.324763 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8af1d827_36b6_48f4_a93c_26228a71f784.slice/crio-06c0b5ea2c5911495a9cd07bf66881731e31c57108c58a4753c2fd064c509672 WatchSource:0}: Error finding container 06c0b5ea2c5911495a9cd07bf66881731e31c57108c58a4753c2fd064c509672: Status 404 returned error can't find the container with id 06c0b5ea2c5911495a9cd07bf66881731e31c57108c58a4753c2fd064c509672 Jul 27 19:12:12.327221 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:12.327188 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 27 19:12:12.327331 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:12.327293 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 27 19:12:12.327331 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:12.327326 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 27 19:12:12.565314 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:12.565277 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e1ba46-openshift-default-94f679f5b-rbdg4" event={"ID":"8af1d827-36b6-48f4-a93c-26228a71f784","Type":"ContainerStarted","Data":"94e8f0a9547a8f8c9cdd8f8f069c28576a3ac78a77a58b8891da637f04c21fe8"} Jul 27 19:12:12.565497 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:12.565334 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e1ba46-openshift-default-94f679f5b-rbdg4" event={"ID":"8af1d827-36b6-48f4-a93c-26228a71f784","Type":"ContainerStarted","Data":"06c0b5ea2c5911495a9cd07bf66881731e31c57108c58a4753c2fd064c509672"} Jul 27 19:12:12.594132 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:12.594083 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-e1ba46-openshift-default-94f679f5b-rbdg4" podStartSLOduration=1.594066872 podStartE2EDuration="1.594066872s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:12:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:12:12.591905847 +0000 UTC m=+2607.653594879" watchObservedRunningTime="2026-07-27 19:12:12.594066872 +0000 UTC m=+2607.655755903" Jul 27 19:12:13.178423 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:13.178379 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-e1ba46-openshift-default-94f679f5b-rbdg4" Jul 27 19:12:13.184419 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:13.184389 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-e1ba46-openshift-default-94f679f5b-rbdg4" Jul 27 19:12:13.571090 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:13.571058 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-e1ba46-openshift-default-94f679f5b-rbdg4" Jul 27 19:12:13.571952 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:13.571933 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-e1ba46-openshift-default-94f679f5b-rbdg4" Jul 27 19:12:14.645891 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:14.645856 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:12:14.796539 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:14.796511 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:12:14.961782 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:14.961687 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:12:15.330432 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:15.330386 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:12:15.982660 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:15.982626 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:12:16.142887 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:16.142848 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:12:19.156775 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:19.156730 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:12:21.561385 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:21.561338 2579 patch_prober.go:28] interesting pod/e2e-api-a-bf852f-openshift-default-cbcc79f9d-w7kv2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 27 19:12:21.561800 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:21.561402 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-bf852f-openshift-default-cbcc79f9d-w7kv2" podUID="600e9c9d-063e-42ce-9045-527f17d838da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 27 19:12:22.622564 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:22.622475 2579 generic.go:356] "Generic (PLEG): container finished" podID="600e9c9d-063e-42ce-9045-527f17d838da" containerID="4df13aa44e890e3202d4ab988ff44a0470123eff09fb3e42a9245930fba1c196" exitCode=0 Jul 27 19:12:22.622564 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:22.622541 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-bf852f-openshift-default-cbcc79f9d-w7kv2" event={"ID":"600e9c9d-063e-42ce-9045-527f17d838da","Type":"ContainerDied","Data":"4df13aa44e890e3202d4ab988ff44a0470123eff09fb3e42a9245930fba1c196"} Jul 27 19:12:22.623151 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:22.623061 2579 scope.go:117] "RemoveContainer" containerID="4df13aa44e890e3202d4ab988ff44a0470123eff09fb3e42a9245930fba1c196" Jul 27 19:12:22.624312 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:22.624283 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 27 19:12:22.624437 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:22.624351 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 27 19:12:22.624437 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:22.624382 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 27 19:12:23.632205 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:23.632168 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-bf852f-openshift-default-cbcc79f9d-w7kv2" event={"ID":"600e9c9d-063e-42ce-9045-527f17d838da","Type":"ContainerStarted","Data":"76e6a58892dd553cc97de5ae2161876ef465996a6e67d6fdeebbb9bf3761cdf7"} Jul 27 19:12:24.253249 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:24.253201 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-bf852f-openshift-default-cbcc79f9d-w7kv2" Jul 27 19:12:24.253458 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:24.253434 2579 patch_prober.go:28] interesting pod/e2e-api-a-bf852f-openshift-default-cbcc79f9d-w7kv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Jul 27 19:12:24.253524 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:24.253503 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-bf852f-openshift-default-cbcc79f9d-w7kv2" podUID="600e9c9d-063e-42ce-9045-527f17d838da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Jul 27 19:12:25.253037 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:25.252990 2579 patch_prober.go:28] interesting pod/e2e-api-a-bf852f-openshift-default-cbcc79f9d-w7kv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Jul 27 19:12:25.253580 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:25.253062 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-bf852f-openshift-default-cbcc79f9d-w7kv2" podUID="600e9c9d-063e-42ce-9045-527f17d838da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Jul 27 19:12:26.253402 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:26.253352 2579 patch_prober.go:28] interesting pod/e2e-api-a-bf852f-openshift-default-cbcc79f9d-w7kv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Jul 27 19:12:26.253918 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:26.253428 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-bf852f-openshift-default-cbcc79f9d-w7kv2" podUID="600e9c9d-063e-42ce-9045-527f17d838da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Jul 27 19:12:27.252979 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:27.252935 2579 patch_prober.go:28] interesting pod/e2e-api-a-bf852f-openshift-default-cbcc79f9d-w7kv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Jul 27 19:12:27.253312 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:27.253278 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-bf852f-openshift-default-cbcc79f9d-w7kv2" podUID="600e9c9d-063e-42ce-9045-527f17d838da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Jul 27 19:12:28.253722 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:28.253678 2579 patch_prober.go:28] interesting pod/e2e-api-a-bf852f-openshift-default-cbcc79f9d-w7kv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Jul 27 19:12:28.254280 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:28.253753 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-bf852f-openshift-default-cbcc79f9d-w7kv2" podUID="600e9c9d-063e-42ce-9045-527f17d838da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Jul 27 19:12:29.256687 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:29.256648 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-bf852f-openshift-default-cbcc79f9d-w7kv2" Jul 27 19:12:29.257217 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:29.257193 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-bf852f-openshift-default-cbcc79f9d-w7kv2" Jul 27 19:12:29.258209 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:29.258186 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-bf852f-openshift-default-cbcc79f9d-w7kv2" Jul 27 19:12:29.612356 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:29.612322 2579 patch_prober.go:28] interesting pod/e2e-api-b-e1ba46-openshift-default-94f679f5b-rbdg4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 27 19:12:29.612528 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:29.612403 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-e1ba46-openshift-default-94f679f5b-rbdg4" podUID="8af1d827-36b6-48f4-a93c-26228a71f784" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 27 19:12:31.690112 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:31.690030 2579 generic.go:356] "Generic (PLEG): container finished" podID="8af1d827-36b6-48f4-a93c-26228a71f784" containerID="94e8f0a9547a8f8c9cdd8f8f069c28576a3ac78a77a58b8891da637f04c21fe8" exitCode=0 Jul 27 19:12:31.690561 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:31.690104 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e1ba46-openshift-default-94f679f5b-rbdg4" event={"ID":"8af1d827-36b6-48f4-a93c-26228a71f784","Type":"ContainerDied","Data":"94e8f0a9547a8f8c9cdd8f8f069c28576a3ac78a77a58b8891da637f04c21fe8"} Jul 27 19:12:31.690561 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:31.690520 2579 scope.go:117] "RemoveContainer" containerID="94e8f0a9547a8f8c9cdd8f8f069c28576a3ac78a77a58b8891da637f04c21fe8" Jul 27 19:12:31.691553 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:31.691523 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 27 19:12:31.691658 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:31.691610 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 27 19:12:31.691727 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:31.691661 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 27 19:12:32.704251 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:32.704200 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e1ba46-openshift-default-94f679f5b-rbdg4" event={"ID":"8af1d827-36b6-48f4-a93c-26228a71f784","Type":"ContainerStarted","Data":"e033968965c5ec98d26bd7511e7d1d55464659b39d638735d6dce2c13fc83a75"} Jul 27 19:12:33.177805 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:33.177763 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-e1ba46-openshift-default-94f679f5b-rbdg4" Jul 27 19:12:33.178055 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:33.177968 2579 patch_prober.go:28] interesting pod/e2e-api-b-e1ba46-openshift-default-94f679f5b-rbdg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Jul 27 19:12:33.178055 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:33.178024 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e1ba46-openshift-default-94f679f5b-rbdg4" podUID="8af1d827-36b6-48f4-a93c-26228a71f784" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Jul 27 19:12:34.177977 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:34.177937 2579 patch_prober.go:28] interesting pod/e2e-api-b-e1ba46-openshift-default-94f679f5b-rbdg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Jul 27 19:12:34.178395 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:34.177999 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e1ba46-openshift-default-94f679f5b-rbdg4" podUID="8af1d827-36b6-48f4-a93c-26228a71f784" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Jul 27 19:12:35.178510 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:35.178469 2579 patch_prober.go:28] interesting pod/e2e-api-b-e1ba46-openshift-default-94f679f5b-rbdg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Jul 27 19:12:35.178898 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:35.178527 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e1ba46-openshift-default-94f679f5b-rbdg4" podUID="8af1d827-36b6-48f4-a93c-26228a71f784" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Jul 27 19:12:36.178395 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:36.178359 2579 patch_prober.go:28] interesting pod/e2e-api-b-e1ba46-openshift-default-94f679f5b-rbdg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Jul 27 19:12:36.178547 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:36.178420 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e1ba46-openshift-default-94f679f5b-rbdg4" podUID="8af1d827-36b6-48f4-a93c-26228a71f784" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Jul 27 19:12:37.181460 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:37.181431 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-e1ba46-openshift-default-94f679f5b-rbdg4" Jul 27 19:12:37.181910 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:37.181848 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-e1ba46-openshift-default-94f679f5b-rbdg4" Jul 27 19:12:37.182708 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:37.182684 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-e1ba46-openshift-default-94f679f5b-rbdg4" Jul 27 19:12:42.823531 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:42.823489 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:12:42.936786 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:42.936746 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:12:43.035053 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:43.035003 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:12:47.641458 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:47.641416 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:12:49.255956 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:49.255911 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:12:54.863307 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:54.863264 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:12:54.960505 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:54.960435 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:12:55.146056 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:55.145970 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-bf852f-openshift-default-cbcc79f9d-w7kv2"] Jul 27 19:12:55.146221 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:55.146166 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-bf852f-openshift-default-cbcc79f9d-w7kv2" podUID="600e9c9d-063e-42ce-9045-527f17d838da" containerName="istio-proxy" containerID="cri-o://76e6a58892dd553cc97de5ae2161876ef465996a6e67d6fdeebbb9bf3761cdf7" gracePeriod=30 Jul 27 19:12:55.169532 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:55.169496 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:12:55.413039 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:55.412942 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:12:59.260288 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:59.260251 2579 patch_prober.go:28] interesting pod/e2e-api-a-bf852f-openshift-default-cbcc79f9d-w7kv2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 27 19:12:59.260663 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:12:59.260320 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-bf852f-openshift-default-cbcc79f9d-w7kv2" podUID="600e9c9d-063e-42ce-9045-527f17d838da" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 27 19:13:00.408917 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:00.408858 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-bf852f-openshift-default-cbcc79f9d-w7kv2" Jul 27 19:13:00.471749 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:00.471716 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/600e9c9d-063e-42ce-9045-527f17d838da-workload-socket\") pod \"600e9c9d-063e-42ce-9045-527f17d838da\" (UID: \"600e9c9d-063e-42ce-9045-527f17d838da\") " Jul 27 19:13:00.471749 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:00.471751 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/600e9c9d-063e-42ce-9045-527f17d838da-istiod-ca-cert\") pod \"600e9c9d-063e-42ce-9045-527f17d838da\" (UID: \"600e9c9d-063e-42ce-9045-527f17d838da\") " Jul 27 19:13:00.472004 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:00.471784 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/600e9c9d-063e-42ce-9045-527f17d838da-istio-envoy\") pod \"600e9c9d-063e-42ce-9045-527f17d838da\" (UID: \"600e9c9d-063e-42ce-9045-527f17d838da\") " Jul 27 19:13:00.472004 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:00.471833 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/600e9c9d-063e-42ce-9045-527f17d838da-istio-token\") pod \"600e9c9d-063e-42ce-9045-527f17d838da\" (UID: \"600e9c9d-063e-42ce-9045-527f17d838da\") " Jul 27 19:13:00.472004 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:00.471852 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/600e9c9d-063e-42ce-9045-527f17d838da-istio-podinfo\") pod \"600e9c9d-063e-42ce-9045-527f17d838da\" (UID: \"600e9c9d-063e-42ce-9045-527f17d838da\") " Jul 27 19:13:00.472004 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:00.471868 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/600e9c9d-063e-42ce-9045-527f17d838da-credential-socket\") pod \"600e9c9d-063e-42ce-9045-527f17d838da\" (UID: \"600e9c9d-063e-42ce-9045-527f17d838da\") " Jul 27 19:13:00.472004 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:00.471889 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m4nqq\" (UniqueName: \"kubernetes.io/projected/600e9c9d-063e-42ce-9045-527f17d838da-kube-api-access-m4nqq\") pod \"600e9c9d-063e-42ce-9045-527f17d838da\" (UID: \"600e9c9d-063e-42ce-9045-527f17d838da\") " Jul 27 19:13:00.472004 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:00.471924 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/600e9c9d-063e-42ce-9045-527f17d838da-istio-data\") pod \"600e9c9d-063e-42ce-9045-527f17d838da\" (UID: \"600e9c9d-063e-42ce-9045-527f17d838da\") " Jul 27 19:13:00.472004 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:00.471945 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/600e9c9d-063e-42ce-9045-527f17d838da-workload-certs\") pod \"600e9c9d-063e-42ce-9045-527f17d838da\" (UID: \"600e9c9d-063e-42ce-9045-527f17d838da\") " Jul 27 19:13:00.472441 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:00.472410 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/600e9c9d-063e-42ce-9045-527f17d838da-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "600e9c9d-063e-42ce-9045-527f17d838da" (UID: "600e9c9d-063e-42ce-9045-527f17d838da"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 19:13:00.472616 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:00.472583 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/600e9c9d-063e-42ce-9045-527f17d838da-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "600e9c9d-063e-42ce-9045-527f17d838da" (UID: "600e9c9d-063e-42ce-9045-527f17d838da"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 19:13:00.472733 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:00.472694 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/600e9c9d-063e-42ce-9045-527f17d838da-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "600e9c9d-063e-42ce-9045-527f17d838da" (UID: "600e9c9d-063e-42ce-9045-527f17d838da"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 19:13:00.472867 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:00.472845 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/600e9c9d-063e-42ce-9045-527f17d838da-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "600e9c9d-063e-42ce-9045-527f17d838da" (UID: "600e9c9d-063e-42ce-9045-527f17d838da"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 19:13:00.474277 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:00.473480 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/600e9c9d-063e-42ce-9045-527f17d838da-workload-certs\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 19:13:00.474277 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:00.473506 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/600e9c9d-063e-42ce-9045-527f17d838da-workload-socket\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 19:13:00.474277 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:00.473522 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/600e9c9d-063e-42ce-9045-527f17d838da-istiod-ca-cert\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 19:13:00.474277 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:00.473537 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/600e9c9d-063e-42ce-9045-527f17d838da-credential-socket\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 19:13:00.474277 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:00.473552 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/600e9c9d-063e-42ce-9045-527f17d838da-istio-data" (OuterVolumeSpecName: "istio-data") pod "600e9c9d-063e-42ce-9045-527f17d838da" (UID: "600e9c9d-063e-42ce-9045-527f17d838da"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 19:13:00.481914 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:00.481879 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/600e9c9d-063e-42ce-9045-527f17d838da-istio-token" (OuterVolumeSpecName: "istio-token") pod "600e9c9d-063e-42ce-9045-527f17d838da" (UID: "600e9c9d-063e-42ce-9045-527f17d838da"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 19:13:00.482306 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:00.482286 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/600e9c9d-063e-42ce-9045-527f17d838da-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "600e9c9d-063e-42ce-9045-527f17d838da" (UID: "600e9c9d-063e-42ce-9045-527f17d838da"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 19:13:00.482306 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:00.482293 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/600e9c9d-063e-42ce-9045-527f17d838da-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "600e9c9d-063e-42ce-9045-527f17d838da" (UID: "600e9c9d-063e-42ce-9045-527f17d838da"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 27 19:13:00.482466 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:00.482293 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/600e9c9d-063e-42ce-9045-527f17d838da-kube-api-access-m4nqq" (OuterVolumeSpecName: "kube-api-access-m4nqq") pod "600e9c9d-063e-42ce-9045-527f17d838da" (UID: "600e9c9d-063e-42ce-9045-527f17d838da"). InnerVolumeSpecName "kube-api-access-m4nqq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 19:13:00.574970 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:00.574933 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/600e9c9d-063e-42ce-9045-527f17d838da-istio-token\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 19:13:00.574970 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:00.574968 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/600e9c9d-063e-42ce-9045-527f17d838da-istio-podinfo\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 19:13:00.575167 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:00.574982 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4nqq\" (UniqueName: \"kubernetes.io/projected/600e9c9d-063e-42ce-9045-527f17d838da-kube-api-access-m4nqq\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 19:13:00.575167 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:00.574996 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/600e9c9d-063e-42ce-9045-527f17d838da-istio-data\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 19:13:00.575167 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:00.575008 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/600e9c9d-063e-42ce-9045-527f17d838da-istio-envoy\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 19:13:00.850924 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:00.850891 2579 generic.go:356] "Generic (PLEG): container finished" podID="600e9c9d-063e-42ce-9045-527f17d838da" containerID="76e6a58892dd553cc97de5ae2161876ef465996a6e67d6fdeebbb9bf3761cdf7" exitCode=0 Jul 27 19:13:00.851106 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:00.850956 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-bf852f-openshift-default-cbcc79f9d-w7kv2" Jul 27 19:13:00.851106 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:00.850977 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-bf852f-openshift-default-cbcc79f9d-w7kv2" event={"ID":"600e9c9d-063e-42ce-9045-527f17d838da","Type":"ContainerDied","Data":"76e6a58892dd553cc97de5ae2161876ef465996a6e67d6fdeebbb9bf3761cdf7"} Jul 27 19:13:00.851106 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:00.851025 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-bf852f-openshift-default-cbcc79f9d-w7kv2" event={"ID":"600e9c9d-063e-42ce-9045-527f17d838da","Type":"ContainerDied","Data":"649522f682116d06d40194e4418a25bb8814f5911a979d5a8daf5b95cdcf1aff"} Jul 27 19:13:00.851106 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:00.851048 2579 scope.go:117] "RemoveContainer" containerID="76e6a58892dd553cc97de5ae2161876ef465996a6e67d6fdeebbb9bf3761cdf7" Jul 27 19:13:00.860739 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:00.860720 2579 scope.go:117] "RemoveContainer" containerID="4df13aa44e890e3202d4ab988ff44a0470123eff09fb3e42a9245930fba1c196" Jul 27 19:13:00.868831 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:00.868815 2579 scope.go:117] "RemoveContainer" containerID="76e6a58892dd553cc97de5ae2161876ef465996a6e67d6fdeebbb9bf3761cdf7" Jul 27 19:13:00.869085 ip-10-0-128-32 kubenswrapper[2579]: E0727 19:13:00.869057 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76e6a58892dd553cc97de5ae2161876ef465996a6e67d6fdeebbb9bf3761cdf7\": container with ID starting with 76e6a58892dd553cc97de5ae2161876ef465996a6e67d6fdeebbb9bf3761cdf7 not found: ID does not exist" containerID="76e6a58892dd553cc97de5ae2161876ef465996a6e67d6fdeebbb9bf3761cdf7" Jul 27 19:13:00.869178 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:00.869086 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76e6a58892dd553cc97de5ae2161876ef465996a6e67d6fdeebbb9bf3761cdf7"} err="failed to get container status \"76e6a58892dd553cc97de5ae2161876ef465996a6e67d6fdeebbb9bf3761cdf7\": rpc error: code = NotFound desc = could not find container \"76e6a58892dd553cc97de5ae2161876ef465996a6e67d6fdeebbb9bf3761cdf7\": container with ID starting with 76e6a58892dd553cc97de5ae2161876ef465996a6e67d6fdeebbb9bf3761cdf7 not found: ID does not exist" Jul 27 19:13:00.869178 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:00.869102 2579 scope.go:117] "RemoveContainer" containerID="4df13aa44e890e3202d4ab988ff44a0470123eff09fb3e42a9245930fba1c196" Jul 27 19:13:00.869342 ip-10-0-128-32 kubenswrapper[2579]: E0727 19:13:00.869325 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4df13aa44e890e3202d4ab988ff44a0470123eff09fb3e42a9245930fba1c196\": container with ID starting with 4df13aa44e890e3202d4ab988ff44a0470123eff09fb3e42a9245930fba1c196 not found: ID does not exist" containerID="4df13aa44e890e3202d4ab988ff44a0470123eff09fb3e42a9245930fba1c196" Jul 27 19:13:00.869382 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:00.869352 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4df13aa44e890e3202d4ab988ff44a0470123eff09fb3e42a9245930fba1c196"} err="failed to get container status \"4df13aa44e890e3202d4ab988ff44a0470123eff09fb3e42a9245930fba1c196\": rpc error: code = NotFound desc = could not find container \"4df13aa44e890e3202d4ab988ff44a0470123eff09fb3e42a9245930fba1c196\": container with ID starting with 4df13aa44e890e3202d4ab988ff44a0470123eff09fb3e42a9245930fba1c196 not found: ID does not exist" Jul 27 19:13:00.876073 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:00.876052 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-bf852f-openshift-default-cbcc79f9d-w7kv2"] Jul 27 19:13:00.882972 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:00.882951 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-bf852f-openshift-default-cbcc79f9d-w7kv2"] Jul 27 19:13:01.494899 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:01.494869 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="600e9c9d-063e-42ce-9045-527f17d838da" path="/var/lib/kubelet/pods/600e9c9d-063e-42ce-9045-527f17d838da/volumes" Jul 27 19:13:11.319697 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:11.319657 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:13:11.420206 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:11.420164 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:13:11.522754 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:11.522716 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:13:15.635661 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:15.635603 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:13:16.086797 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:16.086744 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:13:22.957186 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:22.957146 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:13:23.069163 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:23.069126 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:13:23.299264 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:23.299207 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-e1ba46-openshift-default-94f679f5b-rbdg4"] Jul 27 19:13:23.299524 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:23.299500 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-e1ba46-openshift-default-94f679f5b-rbdg4" podUID="8af1d827-36b6-48f4-a93c-26228a71f784" containerName="istio-proxy" containerID="cri-o://e033968965c5ec98d26bd7511e7d1d55464659b39d638735d6dce2c13fc83a75" gracePeriod=30 Jul 27 19:13:23.352682 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:23.352644 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:13:23.576782 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:23.576674 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:13:23.908203 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:23.908113 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:13:24.261342 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:24.261259 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-7f7473-openshift-default-65db98ff7b-dgl96"] Jul 27 19:13:24.261686 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:24.261674 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="600e9c9d-063e-42ce-9045-527f17d838da" containerName="istio-proxy" Jul 27 19:13:24.261686 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:24.261684 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="600e9c9d-063e-42ce-9045-527f17d838da" containerName="istio-proxy" Jul 27 19:13:24.261763 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:24.261754 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="600e9c9d-063e-42ce-9045-527f17d838da" containerName="istio-proxy" Jul 27 19:13:24.261797 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:24.261765 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="56d09ac8-e6cf-4605-b063-ed13073f2b82" containerName="istio-proxy" Jul 27 19:13:24.261797 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:24.261772 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="600e9c9d-063e-42ce-9045-527f17d838da" containerName="istio-proxy" Jul 27 19:13:24.265014 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:24.264998 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-7f7473-openshift-default-65db98ff7b-dgl96" Jul 27 19:13:24.272006 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:24.271762 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-7f7473-openshift-default-dockercfg-6g4cc\"" Jul 27 19:13:24.285756 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:24.285734 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-7f7473-openshift-default-65db98ff7b-dgl96"] Jul 27 19:13:24.310101 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:24.310068 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:13:24.405752 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:24.405716 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2223e056-27a1-44c6-ac23-924e117185fd-credential-socket\") pod \"e2e-shared-a-7f7473-openshift-default-65db98ff7b-dgl96\" (UID: \"2223e056-27a1-44c6-ac23-924e117185fd\") " pod="openshift-ingress/e2e-shared-a-7f7473-openshift-default-65db98ff7b-dgl96" Jul 27 19:13:24.405924 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:24.405776 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2223e056-27a1-44c6-ac23-924e117185fd-workload-socket\") pod \"e2e-shared-a-7f7473-openshift-default-65db98ff7b-dgl96\" (UID: \"2223e056-27a1-44c6-ac23-924e117185fd\") " pod="openshift-ingress/e2e-shared-a-7f7473-openshift-default-65db98ff7b-dgl96" Jul 27 19:13:24.405924 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:24.405837 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2223e056-27a1-44c6-ac23-924e117185fd-istio-envoy\") pod \"e2e-shared-a-7f7473-openshift-default-65db98ff7b-dgl96\" (UID: \"2223e056-27a1-44c6-ac23-924e117185fd\") " pod="openshift-ingress/e2e-shared-a-7f7473-openshift-default-65db98ff7b-dgl96" Jul 27 19:13:24.405924 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:24.405904 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2223e056-27a1-44c6-ac23-924e117185fd-istio-podinfo\") pod \"e2e-shared-a-7f7473-openshift-default-65db98ff7b-dgl96\" (UID: \"2223e056-27a1-44c6-ac23-924e117185fd\") " pod="openshift-ingress/e2e-shared-a-7f7473-openshift-default-65db98ff7b-dgl96" Jul 27 19:13:24.406037 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:24.405938 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2223e056-27a1-44c6-ac23-924e117185fd-workload-certs\") pod \"e2e-shared-a-7f7473-openshift-default-65db98ff7b-dgl96\" (UID: \"2223e056-27a1-44c6-ac23-924e117185fd\") " pod="openshift-ingress/e2e-shared-a-7f7473-openshift-default-65db98ff7b-dgl96" Jul 27 19:13:24.406037 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:24.405971 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2223e056-27a1-44c6-ac23-924e117185fd-istio-data\") pod \"e2e-shared-a-7f7473-openshift-default-65db98ff7b-dgl96\" (UID: \"2223e056-27a1-44c6-ac23-924e117185fd\") " pod="openshift-ingress/e2e-shared-a-7f7473-openshift-default-65db98ff7b-dgl96" Jul 27 19:13:24.406037 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:24.406006 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ljsf\" (UniqueName: \"kubernetes.io/projected/2223e056-27a1-44c6-ac23-924e117185fd-kube-api-access-7ljsf\") pod \"e2e-shared-a-7f7473-openshift-default-65db98ff7b-dgl96\" (UID: \"2223e056-27a1-44c6-ac23-924e117185fd\") " pod="openshift-ingress/e2e-shared-a-7f7473-openshift-default-65db98ff7b-dgl96" Jul 27 19:13:24.406037 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:24.406024 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2223e056-27a1-44c6-ac23-924e117185fd-istio-token\") pod \"e2e-shared-a-7f7473-openshift-default-65db98ff7b-dgl96\" (UID: \"2223e056-27a1-44c6-ac23-924e117185fd\") " pod="openshift-ingress/e2e-shared-a-7f7473-openshift-default-65db98ff7b-dgl96" Jul 27 19:13:24.406167 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:24.406040 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2223e056-27a1-44c6-ac23-924e117185fd-istiod-ca-cert\") pod \"e2e-shared-a-7f7473-openshift-default-65db98ff7b-dgl96\" (UID: \"2223e056-27a1-44c6-ac23-924e117185fd\") " pod="openshift-ingress/e2e-shared-a-7f7473-openshift-default-65db98ff7b-dgl96" Jul 27 19:13:24.506687 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:24.506657 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2223e056-27a1-44c6-ac23-924e117185fd-workload-certs\") pod \"e2e-shared-a-7f7473-openshift-default-65db98ff7b-dgl96\" (UID: \"2223e056-27a1-44c6-ac23-924e117185fd\") " pod="openshift-ingress/e2e-shared-a-7f7473-openshift-default-65db98ff7b-dgl96" Jul 27 19:13:24.506848 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:24.506699 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2223e056-27a1-44c6-ac23-924e117185fd-istio-data\") pod \"e2e-shared-a-7f7473-openshift-default-65db98ff7b-dgl96\" (UID: \"2223e056-27a1-44c6-ac23-924e117185fd\") " pod="openshift-ingress/e2e-shared-a-7f7473-openshift-default-65db98ff7b-dgl96" Jul 27 19:13:24.506848 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:24.506722 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ljsf\" (UniqueName: \"kubernetes.io/projected/2223e056-27a1-44c6-ac23-924e117185fd-kube-api-access-7ljsf\") pod \"e2e-shared-a-7f7473-openshift-default-65db98ff7b-dgl96\" (UID: \"2223e056-27a1-44c6-ac23-924e117185fd\") " pod="openshift-ingress/e2e-shared-a-7f7473-openshift-default-65db98ff7b-dgl96" Jul 27 19:13:24.506848 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:24.506762 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2223e056-27a1-44c6-ac23-924e117185fd-istio-token\") pod \"e2e-shared-a-7f7473-openshift-default-65db98ff7b-dgl96\" (UID: \"2223e056-27a1-44c6-ac23-924e117185fd\") " pod="openshift-ingress/e2e-shared-a-7f7473-openshift-default-65db98ff7b-dgl96" Jul 27 19:13:24.506848 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:24.506791 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2223e056-27a1-44c6-ac23-924e117185fd-istiod-ca-cert\") pod \"e2e-shared-a-7f7473-openshift-default-65db98ff7b-dgl96\" (UID: \"2223e056-27a1-44c6-ac23-924e117185fd\") " pod="openshift-ingress/e2e-shared-a-7f7473-openshift-default-65db98ff7b-dgl96" Jul 27 19:13:24.506848 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:24.506829 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2223e056-27a1-44c6-ac23-924e117185fd-credential-socket\") pod \"e2e-shared-a-7f7473-openshift-default-65db98ff7b-dgl96\" (UID: \"2223e056-27a1-44c6-ac23-924e117185fd\") " pod="openshift-ingress/e2e-shared-a-7f7473-openshift-default-65db98ff7b-dgl96" Jul 27 19:13:24.507134 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:24.506891 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2223e056-27a1-44c6-ac23-924e117185fd-workload-socket\") pod \"e2e-shared-a-7f7473-openshift-default-65db98ff7b-dgl96\" (UID: \"2223e056-27a1-44c6-ac23-924e117185fd\") " pod="openshift-ingress/e2e-shared-a-7f7473-openshift-default-65db98ff7b-dgl96" Jul 27 19:13:24.507134 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:24.506919 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2223e056-27a1-44c6-ac23-924e117185fd-istio-envoy\") pod \"e2e-shared-a-7f7473-openshift-default-65db98ff7b-dgl96\" (UID: \"2223e056-27a1-44c6-ac23-924e117185fd\") " pod="openshift-ingress/e2e-shared-a-7f7473-openshift-default-65db98ff7b-dgl96" Jul 27 19:13:24.507280 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:24.507198 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2223e056-27a1-44c6-ac23-924e117185fd-istio-podinfo\") pod \"e2e-shared-a-7f7473-openshift-default-65db98ff7b-dgl96\" (UID: \"2223e056-27a1-44c6-ac23-924e117185fd\") " pod="openshift-ingress/e2e-shared-a-7f7473-openshift-default-65db98ff7b-dgl96" Jul 27 19:13:24.507280 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:24.507243 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2223e056-27a1-44c6-ac23-924e117185fd-istio-data\") pod \"e2e-shared-a-7f7473-openshift-default-65db98ff7b-dgl96\" (UID: \"2223e056-27a1-44c6-ac23-924e117185fd\") " pod="openshift-ingress/e2e-shared-a-7f7473-openshift-default-65db98ff7b-dgl96" Jul 27 19:13:24.507396 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:24.507345 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2223e056-27a1-44c6-ac23-924e117185fd-workload-socket\") pod \"e2e-shared-a-7f7473-openshift-default-65db98ff7b-dgl96\" (UID: \"2223e056-27a1-44c6-ac23-924e117185fd\") " pod="openshift-ingress/e2e-shared-a-7f7473-openshift-default-65db98ff7b-dgl96" Jul 27 19:13:24.507457 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:24.507434 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2223e056-27a1-44c6-ac23-924e117185fd-workload-certs\") pod \"e2e-shared-a-7f7473-openshift-default-65db98ff7b-dgl96\" (UID: \"2223e056-27a1-44c6-ac23-924e117185fd\") " pod="openshift-ingress/e2e-shared-a-7f7473-openshift-default-65db98ff7b-dgl96" Jul 27 19:13:24.507593 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:24.507575 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2223e056-27a1-44c6-ac23-924e117185fd-credential-socket\") pod \"e2e-shared-a-7f7473-openshift-default-65db98ff7b-dgl96\" (UID: \"2223e056-27a1-44c6-ac23-924e117185fd\") " pod="openshift-ingress/e2e-shared-a-7f7473-openshift-default-65db98ff7b-dgl96" Jul 27 19:13:24.507722 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:24.507704 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2223e056-27a1-44c6-ac23-924e117185fd-istiod-ca-cert\") pod \"e2e-shared-a-7f7473-openshift-default-65db98ff7b-dgl96\" (UID: \"2223e056-27a1-44c6-ac23-924e117185fd\") " pod="openshift-ingress/e2e-shared-a-7f7473-openshift-default-65db98ff7b-dgl96" Jul 27 19:13:24.509511 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:24.509493 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2223e056-27a1-44c6-ac23-924e117185fd-istio-envoy\") pod \"e2e-shared-a-7f7473-openshift-default-65db98ff7b-dgl96\" (UID: \"2223e056-27a1-44c6-ac23-924e117185fd\") " pod="openshift-ingress/e2e-shared-a-7f7473-openshift-default-65db98ff7b-dgl96" Jul 27 19:13:24.509714 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:24.509698 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2223e056-27a1-44c6-ac23-924e117185fd-istio-podinfo\") pod \"e2e-shared-a-7f7473-openshift-default-65db98ff7b-dgl96\" (UID: \"2223e056-27a1-44c6-ac23-924e117185fd\") " pod="openshift-ingress/e2e-shared-a-7f7473-openshift-default-65db98ff7b-dgl96" Jul 27 19:13:24.516324 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:24.516247 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2223e056-27a1-44c6-ac23-924e117185fd-istio-token\") pod \"e2e-shared-a-7f7473-openshift-default-65db98ff7b-dgl96\" (UID: \"2223e056-27a1-44c6-ac23-924e117185fd\") " pod="openshift-ingress/e2e-shared-a-7f7473-openshift-default-65db98ff7b-dgl96" Jul 27 19:13:24.516568 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:24.516552 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ljsf\" (UniqueName: \"kubernetes.io/projected/2223e056-27a1-44c6-ac23-924e117185fd-kube-api-access-7ljsf\") pod \"e2e-shared-a-7f7473-openshift-default-65db98ff7b-dgl96\" (UID: \"2223e056-27a1-44c6-ac23-924e117185fd\") " pod="openshift-ingress/e2e-shared-a-7f7473-openshift-default-65db98ff7b-dgl96" Jul 27 19:13:24.575460 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:24.575430 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-7f7473-openshift-default-65db98ff7b-dgl96" Jul 27 19:13:24.727657 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:24.727603 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-7f7473-openshift-default-65db98ff7b-dgl96"] Jul 27 19:13:24.730632 ip-10-0-128-32 kubenswrapper[2579]: W0727 19:13:24.730605 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2223e056_27a1_44c6_ac23_924e117185fd.slice/crio-bb1d6735b95cbf5bd9a4dac0af9b3f103ae7b273ed15b5c99a9ac35254f0ae31 WatchSource:0}: Error finding container bb1d6735b95cbf5bd9a4dac0af9b3f103ae7b273ed15b5c99a9ac35254f0ae31: Status 404 returned error can't find the container with id bb1d6735b95cbf5bd9a4dac0af9b3f103ae7b273ed15b5c99a9ac35254f0ae31 Jul 27 19:13:24.733501 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:24.733463 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 27 19:13:24.733604 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:24.733549 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 27 19:13:24.733604 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:24.733590 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 27 19:13:24.973178 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:24.973138 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-7f7473-openshift-default-65db98ff7b-dgl96" event={"ID":"2223e056-27a1-44c6-ac23-924e117185fd","Type":"ContainerStarted","Data":"058bdf3c55e20e88fca5ec412826e8e288ad5c0f0130ea4ca6d7d7da4b64b4ec"} Jul 27 19:13:24.973353 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:24.973196 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-7f7473-openshift-default-65db98ff7b-dgl96" event={"ID":"2223e056-27a1-44c6-ac23-924e117185fd","Type":"ContainerStarted","Data":"bb1d6735b95cbf5bd9a4dac0af9b3f103ae7b273ed15b5c99a9ac35254f0ae31"} Jul 27 19:13:25.008248 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:25.008164 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-7f7473-openshift-default-65db98ff7b-dgl96" podStartSLOduration=1.008141025 podStartE2EDuration="1.008141025s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:13:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:13:25.003953097 +0000 UTC m=+2680.065642128" watchObservedRunningTime="2026-07-27 19:13:25.008141025 +0000 UTC m=+2680.069830056" Jul 27 19:13:25.095120 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:25.095085 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:13:25.575780 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:25.575746 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-7f7473-openshift-default-65db98ff7b-dgl96" Jul 27 19:13:25.580839 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:25.580811 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-7f7473-openshift-default-65db98ff7b-dgl96" Jul 27 19:13:25.978286 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:25.978173 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-7f7473-openshift-default-65db98ff7b-dgl96" Jul 27 19:13:25.979218 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:25.979198 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-7f7473-openshift-default-65db98ff7b-dgl96" Jul 27 19:13:27.061659 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:27.061612 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:13:27.174047 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:27.174011 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:13:27.439023 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:27.438897 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:13:27.984723 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:27.984682 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:13:28.503765 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:28.503703 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:13:28.568185 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:28.568127 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e1ba46-openshift-default-94f679f5b-rbdg4" Jul 27 19:13:28.646176 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:28.646143 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8af1d827-36b6-48f4-a93c-26228a71f784-istiod-ca-cert\") pod \"8af1d827-36b6-48f4-a93c-26228a71f784\" (UID: \"8af1d827-36b6-48f4-a93c-26228a71f784\") " Jul 27 19:13:28.646374 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:28.646192 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8af1d827-36b6-48f4-a93c-26228a71f784-istio-podinfo\") pod \"8af1d827-36b6-48f4-a93c-26228a71f784\" (UID: \"8af1d827-36b6-48f4-a93c-26228a71f784\") " Jul 27 19:13:28.646374 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:28.646217 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8af1d827-36b6-48f4-a93c-26228a71f784-workload-certs\") pod \"8af1d827-36b6-48f4-a93c-26228a71f784\" (UID: \"8af1d827-36b6-48f4-a93c-26228a71f784\") " Jul 27 19:13:28.646374 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:28.646259 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vhhl9\" (UniqueName: \"kubernetes.io/projected/8af1d827-36b6-48f4-a93c-26228a71f784-kube-api-access-vhhl9\") pod \"8af1d827-36b6-48f4-a93c-26228a71f784\" (UID: \"8af1d827-36b6-48f4-a93c-26228a71f784\") " Jul 27 19:13:28.646374 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:28.646284 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8af1d827-36b6-48f4-a93c-26228a71f784-istio-data\") pod \"8af1d827-36b6-48f4-a93c-26228a71f784\" (UID: \"8af1d827-36b6-48f4-a93c-26228a71f784\") " Jul 27 19:13:28.646374 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:28.646327 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8af1d827-36b6-48f4-a93c-26228a71f784-credential-socket\") pod \"8af1d827-36b6-48f4-a93c-26228a71f784\" (UID: \"8af1d827-36b6-48f4-a93c-26228a71f784\") " Jul 27 19:13:28.646643 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:28.646396 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8af1d827-36b6-48f4-a93c-26228a71f784-istio-token\") pod \"8af1d827-36b6-48f4-a93c-26228a71f784\" (UID: \"8af1d827-36b6-48f4-a93c-26228a71f784\") " Jul 27 19:13:28.646643 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:28.646430 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8af1d827-36b6-48f4-a93c-26228a71f784-workload-socket\") pod \"8af1d827-36b6-48f4-a93c-26228a71f784\" (UID: \"8af1d827-36b6-48f4-a93c-26228a71f784\") " Jul 27 19:13:28.646643 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:28.646449 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8af1d827-36b6-48f4-a93c-26228a71f784-istio-envoy\") pod \"8af1d827-36b6-48f4-a93c-26228a71f784\" (UID: \"8af1d827-36b6-48f4-a93c-26228a71f784\") " Jul 27 19:13:28.646643 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:28.646530 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8af1d827-36b6-48f4-a93c-26228a71f784-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8af1d827-36b6-48f4-a93c-26228a71f784" (UID: "8af1d827-36b6-48f4-a93c-26228a71f784"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 19:13:28.646850 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:28.646661 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8af1d827-36b6-48f4-a93c-26228a71f784-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8af1d827-36b6-48f4-a93c-26228a71f784" (UID: "8af1d827-36b6-48f4-a93c-26228a71f784"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 19:13:28.646850 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:28.646754 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8af1d827-36b6-48f4-a93c-26228a71f784-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8af1d827-36b6-48f4-a93c-26228a71f784" (UID: "8af1d827-36b6-48f4-a93c-26228a71f784"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 19:13:28.646850 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:28.646829 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8af1d827-36b6-48f4-a93c-26228a71f784-istiod-ca-cert\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 19:13:28.646850 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:28.646850 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8af1d827-36b6-48f4-a93c-26228a71f784-workload-certs\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 19:13:28.647180 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:28.647134 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8af1d827-36b6-48f4-a93c-26228a71f784-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8af1d827-36b6-48f4-a93c-26228a71f784" (UID: "8af1d827-36b6-48f4-a93c-26228a71f784"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 19:13:28.648401 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:28.648152 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8af1d827-36b6-48f4-a93c-26228a71f784-istio-data" (OuterVolumeSpecName: "istio-data") pod "8af1d827-36b6-48f4-a93c-26228a71f784" (UID: "8af1d827-36b6-48f4-a93c-26228a71f784"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 19:13:28.649527 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:28.649472 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8af1d827-36b6-48f4-a93c-26228a71f784-istio-token" (OuterVolumeSpecName: "istio-token") pod "8af1d827-36b6-48f4-a93c-26228a71f784" (UID: "8af1d827-36b6-48f4-a93c-26228a71f784"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 19:13:28.649527 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:28.649488 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8af1d827-36b6-48f4-a93c-26228a71f784-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8af1d827-36b6-48f4-a93c-26228a71f784" (UID: "8af1d827-36b6-48f4-a93c-26228a71f784"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 19:13:28.649892 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:28.649723 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8af1d827-36b6-48f4-a93c-26228a71f784-kube-api-access-vhhl9" (OuterVolumeSpecName: "kube-api-access-vhhl9") pod "8af1d827-36b6-48f4-a93c-26228a71f784" (UID: "8af1d827-36b6-48f4-a93c-26228a71f784"). InnerVolumeSpecName "kube-api-access-vhhl9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 19:13:28.649968 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:28.649932 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8af1d827-36b6-48f4-a93c-26228a71f784-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8af1d827-36b6-48f4-a93c-26228a71f784" (UID: "8af1d827-36b6-48f4-a93c-26228a71f784"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 27 19:13:28.702695 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:28.702660 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:13:28.748087 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:28.748058 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhhl9\" (UniqueName: \"kubernetes.io/projected/8af1d827-36b6-48f4-a93c-26228a71f784-kube-api-access-vhhl9\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 19:13:28.748087 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:28.748085 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8af1d827-36b6-48f4-a93c-26228a71f784-istio-data\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 19:13:28.748292 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:28.748095 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8af1d827-36b6-48f4-a93c-26228a71f784-credential-socket\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 19:13:28.748292 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:28.748106 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8af1d827-36b6-48f4-a93c-26228a71f784-istio-token\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 19:13:28.748292 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:28.748120 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8af1d827-36b6-48f4-a93c-26228a71f784-workload-socket\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 19:13:28.748292 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:28.748129 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8af1d827-36b6-48f4-a93c-26228a71f784-istio-envoy\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 19:13:28.748292 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:28.748136 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8af1d827-36b6-48f4-a93c-26228a71f784-istio-podinfo\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 19:13:28.996682 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:28.996598 2579 generic.go:356] "Generic (PLEG): container finished" podID="8af1d827-36b6-48f4-a93c-26228a71f784" containerID="e033968965c5ec98d26bd7511e7d1d55464659b39d638735d6dce2c13fc83a75" exitCode=0 Jul 27 19:13:28.996682 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:28.996648 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e1ba46-openshift-default-94f679f5b-rbdg4" Jul 27 19:13:28.996851 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:28.996687 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e1ba46-openshift-default-94f679f5b-rbdg4" event={"ID":"8af1d827-36b6-48f4-a93c-26228a71f784","Type":"ContainerDied","Data":"e033968965c5ec98d26bd7511e7d1d55464659b39d638735d6dce2c13fc83a75"} Jul 27 19:13:28.996851 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:28.996784 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e1ba46-openshift-default-94f679f5b-rbdg4" event={"ID":"8af1d827-36b6-48f4-a93c-26228a71f784","Type":"ContainerDied","Data":"06c0b5ea2c5911495a9cd07bf66881731e31c57108c58a4753c2fd064c509672"} Jul 27 19:13:28.996851 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:28.996812 2579 scope.go:117] "RemoveContainer" containerID="e033968965c5ec98d26bd7511e7d1d55464659b39d638735d6dce2c13fc83a75" Jul 27 19:13:29.007332 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:29.007311 2579 scope.go:117] "RemoveContainer" containerID="94e8f0a9547a8f8c9cdd8f8f069c28576a3ac78a77a58b8891da637f04c21fe8" Jul 27 19:13:29.015587 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:29.015572 2579 scope.go:117] "RemoveContainer" containerID="e033968965c5ec98d26bd7511e7d1d55464659b39d638735d6dce2c13fc83a75" Jul 27 19:13:29.015842 ip-10-0-128-32 kubenswrapper[2579]: E0727 19:13:29.015821 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e033968965c5ec98d26bd7511e7d1d55464659b39d638735d6dce2c13fc83a75\": container with ID starting with e033968965c5ec98d26bd7511e7d1d55464659b39d638735d6dce2c13fc83a75 not found: ID does not exist" containerID="e033968965c5ec98d26bd7511e7d1d55464659b39d638735d6dce2c13fc83a75" Jul 27 19:13:29.015900 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:29.015851 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e033968965c5ec98d26bd7511e7d1d55464659b39d638735d6dce2c13fc83a75"} err="failed to get container status \"e033968965c5ec98d26bd7511e7d1d55464659b39d638735d6dce2c13fc83a75\": rpc error: code = NotFound desc = could not find container \"e033968965c5ec98d26bd7511e7d1d55464659b39d638735d6dce2c13fc83a75\": container with ID starting with e033968965c5ec98d26bd7511e7d1d55464659b39d638735d6dce2c13fc83a75 not found: ID does not exist" Jul 27 19:13:29.015900 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:29.015868 2579 scope.go:117] "RemoveContainer" containerID="94e8f0a9547a8f8c9cdd8f8f069c28576a3ac78a77a58b8891da637f04c21fe8" Jul 27 19:13:29.016096 ip-10-0-128-32 kubenswrapper[2579]: E0727 19:13:29.016077 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94e8f0a9547a8f8c9cdd8f8f069c28576a3ac78a77a58b8891da637f04c21fe8\": container with ID starting with 94e8f0a9547a8f8c9cdd8f8f069c28576a3ac78a77a58b8891da637f04c21fe8 not found: ID does not exist" containerID="94e8f0a9547a8f8c9cdd8f8f069c28576a3ac78a77a58b8891da637f04c21fe8" Jul 27 19:13:29.016135 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:29.016102 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94e8f0a9547a8f8c9cdd8f8f069c28576a3ac78a77a58b8891da637f04c21fe8"} err="failed to get container status \"94e8f0a9547a8f8c9cdd8f8f069c28576a3ac78a77a58b8891da637f04c21fe8\": rpc error: code = NotFound desc = could not find container \"94e8f0a9547a8f8c9cdd8f8f069c28576a3ac78a77a58b8891da637f04c21fe8\": container with ID starting with 94e8f0a9547a8f8c9cdd8f8f069c28576a3ac78a77a58b8891da637f04c21fe8 not found: ID does not exist" Jul 27 19:13:29.037705 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:29.037678 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-e1ba46-openshift-default-94f679f5b-rbdg4"] Jul 27 19:13:29.050756 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:29.050730 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-e1ba46-openshift-default-94f679f5b-rbdg4"] Jul 27 19:13:29.494298 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:29.494264 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8af1d827-36b6-48f4-a93c-26228a71f784" path="/var/lib/kubelet/pods/8af1d827-36b6-48f4-a93c-26228a71f784/volumes" Jul 27 19:13:31.585830 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:31.585798 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:13:32.063004 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:32.062966 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:13:32.315345 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:32.315259 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-6710a9-openshift-default-7d4765f448-58wm4"] Jul 27 19:13:32.315662 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:32.315650 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="600e9c9d-063e-42ce-9045-527f17d838da" containerName="istio-proxy" Jul 27 19:13:32.315723 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:32.315664 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="600e9c9d-063e-42ce-9045-527f17d838da" containerName="istio-proxy" Jul 27 19:13:32.315723 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:32.315673 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8af1d827-36b6-48f4-a93c-26228a71f784" containerName="istio-proxy" Jul 27 19:13:32.315723 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:32.315678 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="8af1d827-36b6-48f4-a93c-26228a71f784" containerName="istio-proxy" Jul 27 19:13:32.315723 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:32.315688 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8af1d827-36b6-48f4-a93c-26228a71f784" containerName="istio-proxy" Jul 27 19:13:32.315723 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:32.315693 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="8af1d827-36b6-48f4-a93c-26228a71f784" containerName="istio-proxy" Jul 27 19:13:32.315903 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:32.315766 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="8af1d827-36b6-48f4-a93c-26228a71f784" containerName="istio-proxy" Jul 27 19:13:32.315903 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:32.315776 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="8af1d827-36b6-48f4-a93c-26228a71f784" containerName="istio-proxy" Jul 27 19:13:32.319177 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:32.319155 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-6710a9-openshift-default-7d4765f448-58wm4" Jul 27 19:13:32.323253 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:32.323220 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-6710a9-openshift-default-dockercfg-blp25\"" Jul 27 19:13:32.350181 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:32.350154 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-6710a9-openshift-default-7d4765f448-58wm4"] Jul 27 19:13:32.376405 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:32.376365 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:13:32.385142 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:32.385115 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ea60e2a-42c9-4459-9c2e-e1809921f6c1-istio-data\") pod \"e2e-shared-b-6710a9-openshift-default-7d4765f448-58wm4\" (UID: \"6ea60e2a-42c9-4459-9c2e-e1809921f6c1\") " pod="openshift-ingress/e2e-shared-b-6710a9-openshift-default-7d4765f448-58wm4" Jul 27 19:13:32.385330 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:32.385157 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ea60e2a-42c9-4459-9c2e-e1809921f6c1-istio-envoy\") pod \"e2e-shared-b-6710a9-openshift-default-7d4765f448-58wm4\" (UID: \"6ea60e2a-42c9-4459-9c2e-e1809921f6c1\") " pod="openshift-ingress/e2e-shared-b-6710a9-openshift-default-7d4765f448-58wm4" Jul 27 19:13:32.385330 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:32.385183 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ea60e2a-42c9-4459-9c2e-e1809921f6c1-istio-token\") pod \"e2e-shared-b-6710a9-openshift-default-7d4765f448-58wm4\" (UID: \"6ea60e2a-42c9-4459-9c2e-e1809921f6c1\") " pod="openshift-ingress/e2e-shared-b-6710a9-openshift-default-7d4765f448-58wm4" Jul 27 19:13:32.385330 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:32.385208 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nkk4\" (UniqueName: \"kubernetes.io/projected/6ea60e2a-42c9-4459-9c2e-e1809921f6c1-kube-api-access-5nkk4\") pod \"e2e-shared-b-6710a9-openshift-default-7d4765f448-58wm4\" (UID: \"6ea60e2a-42c9-4459-9c2e-e1809921f6c1\") " pod="openshift-ingress/e2e-shared-b-6710a9-openshift-default-7d4765f448-58wm4" Jul 27 19:13:32.385330 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:32.385302 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ea60e2a-42c9-4459-9c2e-e1809921f6c1-istio-podinfo\") pod \"e2e-shared-b-6710a9-openshift-default-7d4765f448-58wm4\" (UID: \"6ea60e2a-42c9-4459-9c2e-e1809921f6c1\") " pod="openshift-ingress/e2e-shared-b-6710a9-openshift-default-7d4765f448-58wm4" Jul 27 19:13:32.385489 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:32.385339 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ea60e2a-42c9-4459-9c2e-e1809921f6c1-workload-socket\") pod \"e2e-shared-b-6710a9-openshift-default-7d4765f448-58wm4\" (UID: \"6ea60e2a-42c9-4459-9c2e-e1809921f6c1\") " pod="openshift-ingress/e2e-shared-b-6710a9-openshift-default-7d4765f448-58wm4" Jul 27 19:13:32.385489 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:32.385360 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ea60e2a-42c9-4459-9c2e-e1809921f6c1-credential-socket\") pod \"e2e-shared-b-6710a9-openshift-default-7d4765f448-58wm4\" (UID: \"6ea60e2a-42c9-4459-9c2e-e1809921f6c1\") " pod="openshift-ingress/e2e-shared-b-6710a9-openshift-default-7d4765f448-58wm4" Jul 27 19:13:32.385489 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:32.385444 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ea60e2a-42c9-4459-9c2e-e1809921f6c1-workload-certs\") pod \"e2e-shared-b-6710a9-openshift-default-7d4765f448-58wm4\" (UID: \"6ea60e2a-42c9-4459-9c2e-e1809921f6c1\") " pod="openshift-ingress/e2e-shared-b-6710a9-openshift-default-7d4765f448-58wm4" Jul 27 19:13:32.385489 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:32.385470 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ea60e2a-42c9-4459-9c2e-e1809921f6c1-istiod-ca-cert\") pod \"e2e-shared-b-6710a9-openshift-default-7d4765f448-58wm4\" (UID: \"6ea60e2a-42c9-4459-9c2e-e1809921f6c1\") " pod="openshift-ingress/e2e-shared-b-6710a9-openshift-default-7d4765f448-58wm4" Jul 27 19:13:32.486374 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:32.486344 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ea60e2a-42c9-4459-9c2e-e1809921f6c1-workload-certs\") pod \"e2e-shared-b-6710a9-openshift-default-7d4765f448-58wm4\" (UID: \"6ea60e2a-42c9-4459-9c2e-e1809921f6c1\") " pod="openshift-ingress/e2e-shared-b-6710a9-openshift-default-7d4765f448-58wm4" Jul 27 19:13:32.486374 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:32.486375 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ea60e2a-42c9-4459-9c2e-e1809921f6c1-istiod-ca-cert\") pod \"e2e-shared-b-6710a9-openshift-default-7d4765f448-58wm4\" (UID: \"6ea60e2a-42c9-4459-9c2e-e1809921f6c1\") " pod="openshift-ingress/e2e-shared-b-6710a9-openshift-default-7d4765f448-58wm4" Jul 27 19:13:32.486643 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:32.486401 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ea60e2a-42c9-4459-9c2e-e1809921f6c1-istio-data\") pod \"e2e-shared-b-6710a9-openshift-default-7d4765f448-58wm4\" (UID: \"6ea60e2a-42c9-4459-9c2e-e1809921f6c1\") " pod="openshift-ingress/e2e-shared-b-6710a9-openshift-default-7d4765f448-58wm4" Jul 27 19:13:32.486643 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:32.486438 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ea60e2a-42c9-4459-9c2e-e1809921f6c1-istio-envoy\") pod \"e2e-shared-b-6710a9-openshift-default-7d4765f448-58wm4\" (UID: \"6ea60e2a-42c9-4459-9c2e-e1809921f6c1\") " pod="openshift-ingress/e2e-shared-b-6710a9-openshift-default-7d4765f448-58wm4" Jul 27 19:13:32.486643 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:32.486465 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ea60e2a-42c9-4459-9c2e-e1809921f6c1-istio-token\") pod \"e2e-shared-b-6710a9-openshift-default-7d4765f448-58wm4\" (UID: \"6ea60e2a-42c9-4459-9c2e-e1809921f6c1\") " pod="openshift-ingress/e2e-shared-b-6710a9-openshift-default-7d4765f448-58wm4" Jul 27 19:13:32.486643 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:32.486491 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nkk4\" (UniqueName: \"kubernetes.io/projected/6ea60e2a-42c9-4459-9c2e-e1809921f6c1-kube-api-access-5nkk4\") pod \"e2e-shared-b-6710a9-openshift-default-7d4765f448-58wm4\" (UID: \"6ea60e2a-42c9-4459-9c2e-e1809921f6c1\") " pod="openshift-ingress/e2e-shared-b-6710a9-openshift-default-7d4765f448-58wm4" Jul 27 19:13:32.486643 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:32.486553 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ea60e2a-42c9-4459-9c2e-e1809921f6c1-istio-podinfo\") pod \"e2e-shared-b-6710a9-openshift-default-7d4765f448-58wm4\" (UID: \"6ea60e2a-42c9-4459-9c2e-e1809921f6c1\") " pod="openshift-ingress/e2e-shared-b-6710a9-openshift-default-7d4765f448-58wm4" Jul 27 19:13:32.486643 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:32.486607 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ea60e2a-42c9-4459-9c2e-e1809921f6c1-workload-socket\") pod \"e2e-shared-b-6710a9-openshift-default-7d4765f448-58wm4\" (UID: \"6ea60e2a-42c9-4459-9c2e-e1809921f6c1\") " pod="openshift-ingress/e2e-shared-b-6710a9-openshift-default-7d4765f448-58wm4" Jul 27 19:13:32.486643 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:32.486637 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ea60e2a-42c9-4459-9c2e-e1809921f6c1-credential-socket\") pod \"e2e-shared-b-6710a9-openshift-default-7d4765f448-58wm4\" (UID: \"6ea60e2a-42c9-4459-9c2e-e1809921f6c1\") " pod="openshift-ingress/e2e-shared-b-6710a9-openshift-default-7d4765f448-58wm4" Jul 27 19:13:32.486975 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:32.486853 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ea60e2a-42c9-4459-9c2e-e1809921f6c1-workload-certs\") pod \"e2e-shared-b-6710a9-openshift-default-7d4765f448-58wm4\" (UID: \"6ea60e2a-42c9-4459-9c2e-e1809921f6c1\") " pod="openshift-ingress/e2e-shared-b-6710a9-openshift-default-7d4765f448-58wm4" Jul 27 19:13:32.486975 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:32.486892 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ea60e2a-42c9-4459-9c2e-e1809921f6c1-istio-data\") pod \"e2e-shared-b-6710a9-openshift-default-7d4765f448-58wm4\" (UID: \"6ea60e2a-42c9-4459-9c2e-e1809921f6c1\") " pod="openshift-ingress/e2e-shared-b-6710a9-openshift-default-7d4765f448-58wm4" Jul 27 19:13:32.487057 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:32.487036 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ea60e2a-42c9-4459-9c2e-e1809921f6c1-credential-socket\") pod \"e2e-shared-b-6710a9-openshift-default-7d4765f448-58wm4\" (UID: \"6ea60e2a-42c9-4459-9c2e-e1809921f6c1\") " pod="openshift-ingress/e2e-shared-b-6710a9-openshift-default-7d4765f448-58wm4" Jul 27 19:13:32.487117 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:32.487090 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ea60e2a-42c9-4459-9c2e-e1809921f6c1-workload-socket\") pod \"e2e-shared-b-6710a9-openshift-default-7d4765f448-58wm4\" (UID: \"6ea60e2a-42c9-4459-9c2e-e1809921f6c1\") " pod="openshift-ingress/e2e-shared-b-6710a9-openshift-default-7d4765f448-58wm4" Jul 27 19:13:32.487314 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:32.487296 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ea60e2a-42c9-4459-9c2e-e1809921f6c1-istiod-ca-cert\") pod \"e2e-shared-b-6710a9-openshift-default-7d4765f448-58wm4\" (UID: \"6ea60e2a-42c9-4459-9c2e-e1809921f6c1\") " pod="openshift-ingress/e2e-shared-b-6710a9-openshift-default-7d4765f448-58wm4" Jul 27 19:13:32.488918 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:32.488898 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ea60e2a-42c9-4459-9c2e-e1809921f6c1-istio-envoy\") pod \"e2e-shared-b-6710a9-openshift-default-7d4765f448-58wm4\" (UID: \"6ea60e2a-42c9-4459-9c2e-e1809921f6c1\") " pod="openshift-ingress/e2e-shared-b-6710a9-openshift-default-7d4765f448-58wm4" Jul 27 19:13:32.489268 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:32.489223 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ea60e2a-42c9-4459-9c2e-e1809921f6c1-istio-podinfo\") pod \"e2e-shared-b-6710a9-openshift-default-7d4765f448-58wm4\" (UID: \"6ea60e2a-42c9-4459-9c2e-e1809921f6c1\") " pod="openshift-ingress/e2e-shared-b-6710a9-openshift-default-7d4765f448-58wm4" Jul 27 19:13:32.505853 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:32.505823 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ea60e2a-42c9-4459-9c2e-e1809921f6c1-istio-token\") pod \"e2e-shared-b-6710a9-openshift-default-7d4765f448-58wm4\" (UID: \"6ea60e2a-42c9-4459-9c2e-e1809921f6c1\") " pod="openshift-ingress/e2e-shared-b-6710a9-openshift-default-7d4765f448-58wm4" Jul 27 19:13:32.507637 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:32.507616 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nkk4\" (UniqueName: \"kubernetes.io/projected/6ea60e2a-42c9-4459-9c2e-e1809921f6c1-kube-api-access-5nkk4\") pod \"e2e-shared-b-6710a9-openshift-default-7d4765f448-58wm4\" (UID: \"6ea60e2a-42c9-4459-9c2e-e1809921f6c1\") " pod="openshift-ingress/e2e-shared-b-6710a9-openshift-default-7d4765f448-58wm4" Jul 27 19:13:32.629896 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:32.629799 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-6710a9-openshift-default-7d4765f448-58wm4" Jul 27 19:13:32.837109 ip-10-0-128-32 kubenswrapper[2579]: W0727 19:13:32.837070 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ea60e2a_42c9_4459_9c2e_e1809921f6c1.slice/crio-ec66bd861b078546ac3b35342f894747b61c0eec8f99adba7d4cd25533b5e5c4 WatchSource:0}: Error finding container ec66bd861b078546ac3b35342f894747b61c0eec8f99adba7d4cd25533b5e5c4: Status 404 returned error can't find the container with id ec66bd861b078546ac3b35342f894747b61c0eec8f99adba7d4cd25533b5e5c4 Jul 27 19:13:32.840343 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:32.840317 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-6710a9-openshift-default-7d4765f448-58wm4"] Jul 27 19:13:32.840447 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:32.840351 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 27 19:13:32.840447 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:32.840413 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 27 19:13:32.840447 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:32.840440 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 27 19:13:32.884442 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:32.884406 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:13:33.020691 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:33.020644 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-6710a9-openshift-default-7d4765f448-58wm4" event={"ID":"6ea60e2a-42c9-4459-9c2e-e1809921f6c1","Type":"ContainerStarted","Data":"bd7fa69c680cdd8e514320c3c626f47b5680fb71aa59fafd98136758770f6602"} Jul 27 19:13:33.020882 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:33.020712 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-6710a9-openshift-default-7d4765f448-58wm4" event={"ID":"6ea60e2a-42c9-4459-9c2e-e1809921f6c1","Type":"ContainerStarted","Data":"ec66bd861b078546ac3b35342f894747b61c0eec8f99adba7d4cd25533b5e5c4"} Jul 27 19:13:33.058433 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:33.058372 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-6710a9-openshift-default-7d4765f448-58wm4" podStartSLOduration=1.058352821 podStartE2EDuration="1.058352821s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:13:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:13:33.057838539 +0000 UTC m=+2688.119527570" watchObservedRunningTime="2026-07-27 19:13:33.058352821 +0000 UTC m=+2688.120041851" Jul 27 19:13:33.630894 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:33.630856 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-6710a9-openshift-default-7d4765f448-58wm4" Jul 27 19:13:33.636019 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:33.635997 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-6710a9-openshift-default-7d4765f448-58wm4" Jul 27 19:13:34.028041 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:34.028007 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-6710a9-openshift-default-7d4765f448-58wm4" Jul 27 19:13:34.029124 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:34.029099 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-6710a9-openshift-default-7d4765f448-58wm4" Jul 27 19:13:35.364590 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:35.364549 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:13:35.827175 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:35.827135 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:13:36.242154 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:36.242056 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:13:37.375781 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:37.375742 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:13:37.633050 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:37.632912 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:13:39.662667 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:39.662618 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:13:42.020401 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:42.020358 2579 patch_prober.go:28] interesting pod/e2e-shared-a-7f7473-openshift-default-65db98ff7b-dgl96 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 27 19:13:42.020848 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:42.020419 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-7f7473-openshift-default-65db98ff7b-dgl96" podUID="2223e056-27a1-44c6-ac23-924e117185fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 27 19:13:43.084293 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:43.084248 2579 generic.go:356] "Generic (PLEG): container finished" podID="2223e056-27a1-44c6-ac23-924e117185fd" containerID="058bdf3c55e20e88fca5ec412826e8e288ad5c0f0130ea4ca6d7d7da4b64b4ec" exitCode=0 Jul 27 19:13:43.084777 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:43.084306 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-7f7473-openshift-default-65db98ff7b-dgl96" event={"ID":"2223e056-27a1-44c6-ac23-924e117185fd","Type":"ContainerDied","Data":"058bdf3c55e20e88fca5ec412826e8e288ad5c0f0130ea4ca6d7d7da4b64b4ec"} Jul 27 19:13:43.084777 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:43.084733 2579 scope.go:117] "RemoveContainer" containerID="058bdf3c55e20e88fca5ec412826e8e288ad5c0f0130ea4ca6d7d7da4b64b4ec" Jul 27 19:13:43.085838 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:43.085811 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 27 19:13:43.085903 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:43.085878 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 27 19:13:43.085940 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:43.085910 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 27 19:13:44.091112 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:44.091070 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-7f7473-openshift-default-65db98ff7b-dgl96" event={"ID":"2223e056-27a1-44c6-ac23-924e117185fd","Type":"ContainerStarted","Data":"d60fe785dc599b076e94609faef8399f3473f3c6a233c56c01f854be295af402"} Jul 27 19:13:44.576267 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:44.575873 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-7f7473-openshift-default-65db98ff7b-dgl96" Jul 27 19:13:44.576677 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:44.576639 2579 patch_prober.go:28] interesting pod/e2e-shared-a-7f7473-openshift-default-65db98ff7b-dgl96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Jul 27 19:13:44.576878 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:44.576845 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-7f7473-openshift-default-65db98ff7b-dgl96" podUID="2223e056-27a1-44c6-ac23-924e117185fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Jul 27 19:13:45.576688 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:45.576637 2579 patch_prober.go:28] interesting pod/e2e-shared-a-7f7473-openshift-default-65db98ff7b-dgl96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Jul 27 19:13:45.577188 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:45.576733 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-7f7473-openshift-default-65db98ff7b-dgl96" podUID="2223e056-27a1-44c6-ac23-924e117185fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Jul 27 19:13:45.636853 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:45.636816 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-mv4hb_715b2084-d79e-4cbf-a057-f2595b4e6b9b/console-operator/1.log" Jul 27 19:13:45.656254 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:45.654992 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-mv4hb_715b2084-d79e-4cbf-a057-f2595b4e6b9b/console-operator/1.log" Jul 27 19:13:46.576801 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:46.576750 2579 patch_prober.go:28] interesting pod/e2e-shared-a-7f7473-openshift-default-65db98ff7b-dgl96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Jul 27 19:13:46.577307 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:46.576848 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-7f7473-openshift-default-65db98ff7b-dgl96" podUID="2223e056-27a1-44c6-ac23-924e117185fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Jul 27 19:13:47.576520 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:47.576482 2579 patch_prober.go:28] interesting pod/e2e-shared-a-7f7473-openshift-default-65db98ff7b-dgl96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Jul 27 19:13:47.576827 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:47.576795 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-7f7473-openshift-default-65db98ff7b-dgl96" podUID="2223e056-27a1-44c6-ac23-924e117185fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Jul 27 19:13:48.580284 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:48.580253 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-7f7473-openshift-default-65db98ff7b-dgl96" Jul 27 19:13:48.580734 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:48.580656 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-7f7473-openshift-default-65db98ff7b-dgl96" Jul 27 19:13:48.581461 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:48.581441 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-7f7473-openshift-default-65db98ff7b-dgl96" Jul 27 19:13:50.070401 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:50.070357 2579 patch_prober.go:28] interesting pod/e2e-shared-b-6710a9-openshift-default-7d4765f448-58wm4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 27 19:13:50.070902 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:50.070471 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-6710a9-openshift-default-7d4765f448-58wm4" podUID="6ea60e2a-42c9-4459-9c2e-e1809921f6c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 27 19:13:51.277323 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:51.277279 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:13:52.135201 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:52.135162 2579 generic.go:356] "Generic (PLEG): container finished" podID="6ea60e2a-42c9-4459-9c2e-e1809921f6c1" containerID="bd7fa69c680cdd8e514320c3c626f47b5680fb71aa59fafd98136758770f6602" exitCode=0 Jul 27 19:13:52.135402 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:52.135264 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-6710a9-openshift-default-7d4765f448-58wm4" event={"ID":"6ea60e2a-42c9-4459-9c2e-e1809921f6c1","Type":"ContainerDied","Data":"bd7fa69c680cdd8e514320c3c626f47b5680fb71aa59fafd98136758770f6602"} Jul 27 19:13:52.135811 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:52.135793 2579 scope.go:117] "RemoveContainer" containerID="bd7fa69c680cdd8e514320c3c626f47b5680fb71aa59fafd98136758770f6602" Jul 27 19:13:52.137083 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:52.137054 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 27 19:13:52.137195 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:52.137138 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 27 19:13:52.137195 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:52.137183 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 27 19:13:53.145861 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:53.145817 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-6710a9-openshift-default-7d4765f448-58wm4" event={"ID":"6ea60e2a-42c9-4459-9c2e-e1809921f6c1","Type":"ContainerStarted","Data":"79f38ce48191b585b088167202ced3261d5067276a9d7b1ec173e275dfcb7189"} Jul 27 19:13:53.631178 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:53.631143 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-6710a9-openshift-default-7d4765f448-58wm4" Jul 27 19:13:53.631554 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:53.631529 2579 patch_prober.go:28] interesting pod/e2e-shared-b-6710a9-openshift-default-7d4765f448-58wm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Jul 27 19:13:53.631738 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:53.631713 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-6710a9-openshift-default-7d4765f448-58wm4" podUID="6ea60e2a-42c9-4459-9c2e-e1809921f6c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Jul 27 19:13:54.630823 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:54.630776 2579 patch_prober.go:28] interesting pod/e2e-shared-b-6710a9-openshift-default-7d4765f448-58wm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Jul 27 19:13:54.631366 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:54.630847 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-6710a9-openshift-default-7d4765f448-58wm4" podUID="6ea60e2a-42c9-4459-9c2e-e1809921f6c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Jul 27 19:13:55.631071 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:55.631005 2579 patch_prober.go:28] interesting pod/e2e-shared-b-6710a9-openshift-default-7d4765f448-58wm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Jul 27 19:13:55.631572 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:55.631108 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-6710a9-openshift-default-7d4765f448-58wm4" podUID="6ea60e2a-42c9-4459-9c2e-e1809921f6c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Jul 27 19:13:56.631157 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:56.631111 2579 patch_prober.go:28] interesting pod/e2e-shared-b-6710a9-openshift-default-7d4765f448-58wm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Jul 27 19:13:56.631825 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:56.631789 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-6710a9-openshift-default-7d4765f448-58wm4" podUID="6ea60e2a-42c9-4459-9c2e-e1809921f6c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Jul 27 19:13:57.635068 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:57.635038 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-6710a9-openshift-default-7d4765f448-58wm4" Jul 27 19:13:57.635532 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:57.635417 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-6710a9-openshift-default-7d4765f448-58wm4" Jul 27 19:13:57.636139 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:13:57.636122 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-6710a9-openshift-default-7d4765f448-58wm4" Jul 27 19:14:33.481370 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:14:33.481316 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:14:45.966015 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:14:45.965972 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:15:25.964900 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:15:25.964810 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:16:19.924300 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:16:19.924266 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:16:20.113821 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:16:20.113780 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:16:57.259454 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:16:57.259417 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:17:30.272920 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:17:30.272868 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:18:32.307429 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:18:32.307342 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:18:33.407592 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:18:33.407551 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:18:45.683495 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:18:45.683466 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-mv4hb_715b2084-d79e-4cbf-a057-f2595b4e6b9b/console-operator/1.log" Jul 27 19:18:45.701057 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:18:45.701031 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-mv4hb_715b2084-d79e-4cbf-a057-f2595b4e6b9b/console-operator/1.log" Jul 27 19:18:50.526786 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:18:50.526747 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:18:59.119815 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:18:59.119772 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:19:00.136818 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:19:00.136779 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:19:18.048100 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:19:18.048063 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-7f7473/sub-test-model-7f7473-kserve-6f96848bf9-cshlt"] Jul 27 19:19:18.050582 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:19:18.050562 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-7f7473/sub-test-model-7f7473-kserve-6f96848bf9-cshlt" Jul 27 19:19:18.059225 ip-10-0-128-32 kubenswrapper[2579]: E0727 19:19:18.059191 2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"default-dockercfg-5ncxf\" is forbidden: User \"system:node:ip-10-0-128-32.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"ai-tenant-e2e-shared-a-7f7473\": no relationship found between node 'ip-10-0-128-32.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"ai-tenant-e2e-shared-a-7f7473\"/\"default-dockercfg-5ncxf\"" type="*v1.Secret" Jul 27 19:19:18.067139 ip-10-0-128-32 kubenswrapper[2579]: E0727 19:19:18.067103 2579 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"kube-root-ca.crt\" is forbidden: User \"system:node:ip-10-0-128-32.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"ai-tenant-e2e-shared-a-7f7473\": no relationship found between node 'ip-10-0-128-32.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"ai-tenant-e2e-shared-a-7f7473\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" Jul 27 19:19:18.067274 ip-10-0-128-32 kubenswrapper[2579]: E0727 19:19:18.067155 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"sub-test-model-7f7473-kserve-6f96848bf9-cshlt\" is forbidden: User \"system:node:ip-10-0-128-32.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"ai-tenant-e2e-shared-a-7f7473\": no relationship found between node 'ip-10-0-128-32.ec2.internal' and this object" podUID="1ac9a81c-b5c7-4241-8074-f5bf4ffda0eb" pod="ai-tenant-e2e-shared-a-7f7473/sub-test-model-7f7473-kserve-6f96848bf9-cshlt" Jul 27 19:19:18.067274 ip-10-0-128-32 kubenswrapper[2579]: E0727 19:19:18.067169 2579 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"openshift-service-ca.crt\" is forbidden: User \"system:node:ip-10-0-128-32.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"ai-tenant-e2e-shared-a-7f7473\": no relationship found between node 'ip-10-0-128-32.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"ai-tenant-e2e-shared-a-7f7473\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" Jul 27 19:19:18.079011 ip-10-0-128-32 kubenswrapper[2579]: E0727 19:19:18.078986 2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"sub-test-model-7f7473-kserve-self-signed-certs\" is forbidden: User \"system:node:ip-10-0-128-32.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"ai-tenant-e2e-shared-a-7f7473\": no relationship found between node 'ip-10-0-128-32.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"ai-tenant-e2e-shared-a-7f7473\"/\"sub-test-model-7f7473-kserve-self-signed-certs\"" type="*v1.Secret" Jul 27 19:19:18.125266 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:19:18.125214 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-7f7473/sub-test-model-7f7473-kserve-6f96848bf9-cshlt"] Jul 27 19:19:18.125724 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:19:18.125678 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ac9a81c-b5c7-4241-8074-f5bf4ffda0eb-model-cache\") pod \"sub-test-model-7f7473-kserve-6f96848bf9-cshlt\" (UID: \"1ac9a81c-b5c7-4241-8074-f5bf4ffda0eb\") " pod="ai-tenant-e2e-shared-a-7f7473/sub-test-model-7f7473-kserve-6f96848bf9-cshlt" Jul 27 19:19:18.125841 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:19:18.125757 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ac9a81c-b5c7-4241-8074-f5bf4ffda0eb-tls-certs\") pod \"sub-test-model-7f7473-kserve-6f96848bf9-cshlt\" (UID: \"1ac9a81c-b5c7-4241-8074-f5bf4ffda0eb\") " pod="ai-tenant-e2e-shared-a-7f7473/sub-test-model-7f7473-kserve-6f96848bf9-cshlt" Jul 27 19:19:18.125841 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:19:18.125832 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbbjz\" (UniqueName: \"kubernetes.io/projected/1ac9a81c-b5c7-4241-8074-f5bf4ffda0eb-kube-api-access-tbbjz\") pod \"sub-test-model-7f7473-kserve-6f96848bf9-cshlt\" (UID: \"1ac9a81c-b5c7-4241-8074-f5bf4ffda0eb\") " pod="ai-tenant-e2e-shared-a-7f7473/sub-test-model-7f7473-kserve-6f96848bf9-cshlt" Jul 27 19:19:18.126170 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:19:18.125906 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ac9a81c-b5c7-4241-8074-f5bf4ffda0eb-home\") pod \"sub-test-model-7f7473-kserve-6f96848bf9-cshlt\" (UID: \"1ac9a81c-b5c7-4241-8074-f5bf4ffda0eb\") " pod="ai-tenant-e2e-shared-a-7f7473/sub-test-model-7f7473-kserve-6f96848bf9-cshlt" Jul 27 19:19:18.126170 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:19:18.125929 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ac9a81c-b5c7-4241-8074-f5bf4ffda0eb-tmp-dir\") pod \"sub-test-model-7f7473-kserve-6f96848bf9-cshlt\" (UID: \"1ac9a81c-b5c7-4241-8074-f5bf4ffda0eb\") " pod="ai-tenant-e2e-shared-a-7f7473/sub-test-model-7f7473-kserve-6f96848bf9-cshlt" Jul 27 19:19:18.126170 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:19:18.126001 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ac9a81c-b5c7-4241-8074-f5bf4ffda0eb-dshm\") pod \"sub-test-model-7f7473-kserve-6f96848bf9-cshlt\" (UID: \"1ac9a81c-b5c7-4241-8074-f5bf4ffda0eb\") " pod="ai-tenant-e2e-shared-a-7f7473/sub-test-model-7f7473-kserve-6f96848bf9-cshlt" Jul 27 19:19:18.145271 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:19:18.145204 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:19:18.227404 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:19:18.227369 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ac9a81c-b5c7-4241-8074-f5bf4ffda0eb-home\") pod \"sub-test-model-7f7473-kserve-6f96848bf9-cshlt\" (UID: \"1ac9a81c-b5c7-4241-8074-f5bf4ffda0eb\") " pod="ai-tenant-e2e-shared-a-7f7473/sub-test-model-7f7473-kserve-6f96848bf9-cshlt" Jul 27 19:19:18.227404 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:19:18.227404 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ac9a81c-b5c7-4241-8074-f5bf4ffda0eb-tmp-dir\") pod \"sub-test-model-7f7473-kserve-6f96848bf9-cshlt\" (UID: \"1ac9a81c-b5c7-4241-8074-f5bf4ffda0eb\") " pod="ai-tenant-e2e-shared-a-7f7473/sub-test-model-7f7473-kserve-6f96848bf9-cshlt" Jul 27 19:19:18.227675 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:19:18.227436 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ac9a81c-b5c7-4241-8074-f5bf4ffda0eb-dshm\") pod \"sub-test-model-7f7473-kserve-6f96848bf9-cshlt\" (UID: \"1ac9a81c-b5c7-4241-8074-f5bf4ffda0eb\") " pod="ai-tenant-e2e-shared-a-7f7473/sub-test-model-7f7473-kserve-6f96848bf9-cshlt" Jul 27 19:19:18.227675 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:19:18.227559 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ac9a81c-b5c7-4241-8074-f5bf4ffda0eb-model-cache\") pod \"sub-test-model-7f7473-kserve-6f96848bf9-cshlt\" (UID: \"1ac9a81c-b5c7-4241-8074-f5bf4ffda0eb\") " pod="ai-tenant-e2e-shared-a-7f7473/sub-test-model-7f7473-kserve-6f96848bf9-cshlt" Jul 27 19:19:18.227675 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:19:18.227611 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ac9a81c-b5c7-4241-8074-f5bf4ffda0eb-tls-certs\") pod \"sub-test-model-7f7473-kserve-6f96848bf9-cshlt\" (UID: \"1ac9a81c-b5c7-4241-8074-f5bf4ffda0eb\") " pod="ai-tenant-e2e-shared-a-7f7473/sub-test-model-7f7473-kserve-6f96848bf9-cshlt" Jul 27 19:19:18.227850 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:19:18.227675 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbbjz\" (UniqueName: \"kubernetes.io/projected/1ac9a81c-b5c7-4241-8074-f5bf4ffda0eb-kube-api-access-tbbjz\") pod \"sub-test-model-7f7473-kserve-6f96848bf9-cshlt\" (UID: \"1ac9a81c-b5c7-4241-8074-f5bf4ffda0eb\") " pod="ai-tenant-e2e-shared-a-7f7473/sub-test-model-7f7473-kserve-6f96848bf9-cshlt" Jul 27 19:19:18.227850 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:19:18.227820 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ac9a81c-b5c7-4241-8074-f5bf4ffda0eb-home\") pod \"sub-test-model-7f7473-kserve-6f96848bf9-cshlt\" (UID: \"1ac9a81c-b5c7-4241-8074-f5bf4ffda0eb\") " pod="ai-tenant-e2e-shared-a-7f7473/sub-test-model-7f7473-kserve-6f96848bf9-cshlt" Jul 27 19:19:18.227957 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:19:18.227922 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ac9a81c-b5c7-4241-8074-f5bf4ffda0eb-tmp-dir\") pod \"sub-test-model-7f7473-kserve-6f96848bf9-cshlt\" (UID: \"1ac9a81c-b5c7-4241-8074-f5bf4ffda0eb\") " pod="ai-tenant-e2e-shared-a-7f7473/sub-test-model-7f7473-kserve-6f96848bf9-cshlt" Jul 27 19:19:18.228157 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:19:18.228133 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ac9a81c-b5c7-4241-8074-f5bf4ffda0eb-model-cache\") pod \"sub-test-model-7f7473-kserve-6f96848bf9-cshlt\" (UID: \"1ac9a81c-b5c7-4241-8074-f5bf4ffda0eb\") " pod="ai-tenant-e2e-shared-a-7f7473/sub-test-model-7f7473-kserve-6f96848bf9-cshlt" Jul 27 19:19:18.230000 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:19:18.229948 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ac9a81c-b5c7-4241-8074-f5bf4ffda0eb-dshm\") pod \"sub-test-model-7f7473-kserve-6f96848bf9-cshlt\" (UID: \"1ac9a81c-b5c7-4241-8074-f5bf4ffda0eb\") " pod="ai-tenant-e2e-shared-a-7f7473/sub-test-model-7f7473-kserve-6f96848bf9-cshlt" Jul 27 19:19:18.888391 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:19:18.888363 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-7f7473\"/\"default-dockercfg-5ncxf\"" Jul 27 19:19:18.908508 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:19:18.908476 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-7f7473\"/\"openshift-service-ca.crt\"" Jul 27 19:19:19.228174 ip-10-0-128-32 kubenswrapper[2579]: E0727 19:19:19.228089 2579 secret.go:189] Couldn't get secret ai-tenant-e2e-shared-a-7f7473/sub-test-model-7f7473-kserve-self-signed-certs: failed to sync secret cache: timed out waiting for the condition Jul 27 19:19:19.228557 ip-10-0-128-32 kubenswrapper[2579]: E0727 19:19:19.228183 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ac9a81c-b5c7-4241-8074-f5bf4ffda0eb-tls-certs podName:1ac9a81c-b5c7-4241-8074-f5bf4ffda0eb nodeName:}" failed. No retries permitted until 2026-07-27 19:19:19.728166733 +0000 UTC m=+3034.789855742 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/1ac9a81c-b5c7-4241-8074-f5bf4ffda0eb-tls-certs") pod "sub-test-model-7f7473-kserve-6f96848bf9-cshlt" (UID: "1ac9a81c-b5c7-4241-8074-f5bf4ffda0eb") : failed to sync secret cache: timed out waiting for the condition Jul 27 19:19:19.256012 ip-10-0-128-32 kubenswrapper[2579]: E0727 19:19:19.255981 2579 projected.go:291] Couldn't get configMap ai-tenant-e2e-shared-a-7f7473/kube-root-ca.crt: failed to sync configmap cache: timed out waiting for the condition Jul 27 19:19:19.256012 ip-10-0-128-32 kubenswrapper[2579]: E0727 19:19:19.256017 2579 projected.go:196] Error preparing data for projected volume kube-api-access-tbbjz for pod ai-tenant-e2e-shared-a-7f7473/sub-test-model-7f7473-kserve-6f96848bf9-cshlt: failed to sync configmap cache: timed out waiting for the condition Jul 27 19:19:19.256305 ip-10-0-128-32 kubenswrapper[2579]: E0727 19:19:19.256087 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1ac9a81c-b5c7-4241-8074-f5bf4ffda0eb-kube-api-access-tbbjz podName:1ac9a81c-b5c7-4241-8074-f5bf4ffda0eb nodeName:}" failed. No retries permitted until 2026-07-27 19:19:19.756070951 +0000 UTC m=+3034.817759960 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-tbbjz" (UniqueName: "kubernetes.io/projected/1ac9a81c-b5c7-4241-8074-f5bf4ffda0eb-kube-api-access-tbbjz") pod "sub-test-model-7f7473-kserve-6f96848bf9-cshlt" (UID: "1ac9a81c-b5c7-4241-8074-f5bf4ffda0eb") : failed to sync configmap cache: timed out waiting for the condition Jul 27 19:19:19.263344 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:19:19.263325 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-7f7473\"/\"sub-test-model-7f7473-kserve-self-signed-certs\"" Jul 27 19:19:19.307384 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:19:19.307358 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-7f7473\"/\"kube-root-ca.crt\"" Jul 27 19:19:19.741727 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:19:19.741686 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ac9a81c-b5c7-4241-8074-f5bf4ffda0eb-tls-certs\") pod \"sub-test-model-7f7473-kserve-6f96848bf9-cshlt\" (UID: \"1ac9a81c-b5c7-4241-8074-f5bf4ffda0eb\") " pod="ai-tenant-e2e-shared-a-7f7473/sub-test-model-7f7473-kserve-6f96848bf9-cshlt" Jul 27 19:19:19.744405 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:19:19.744383 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ac9a81c-b5c7-4241-8074-f5bf4ffda0eb-tls-certs\") pod \"sub-test-model-7f7473-kserve-6f96848bf9-cshlt\" (UID: \"1ac9a81c-b5c7-4241-8074-f5bf4ffda0eb\") " pod="ai-tenant-e2e-shared-a-7f7473/sub-test-model-7f7473-kserve-6f96848bf9-cshlt" Jul 27 19:19:19.843216 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:19:19.843176 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbbjz\" (UniqueName: \"kubernetes.io/projected/1ac9a81c-b5c7-4241-8074-f5bf4ffda0eb-kube-api-access-tbbjz\") pod \"sub-test-model-7f7473-kserve-6f96848bf9-cshlt\" (UID: \"1ac9a81c-b5c7-4241-8074-f5bf4ffda0eb\") " pod="ai-tenant-e2e-shared-a-7f7473/sub-test-model-7f7473-kserve-6f96848bf9-cshlt" Jul 27 19:19:19.845909 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:19:19.845887 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbbjz\" (UniqueName: \"kubernetes.io/projected/1ac9a81c-b5c7-4241-8074-f5bf4ffda0eb-kube-api-access-tbbjz\") pod \"sub-test-model-7f7473-kserve-6f96848bf9-cshlt\" (UID: \"1ac9a81c-b5c7-4241-8074-f5bf4ffda0eb\") " pod="ai-tenant-e2e-shared-a-7f7473/sub-test-model-7f7473-kserve-6f96848bf9-cshlt" Jul 27 19:19:19.860214 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:19:19.860185 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-7f7473/sub-test-model-7f7473-kserve-6f96848bf9-cshlt" Jul 27 19:19:20.030022 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:19:20.029993 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-7f7473/sub-test-model-7f7473-kserve-6f96848bf9-cshlt"] Jul 27 19:19:20.032139 ip-10-0-128-32 kubenswrapper[2579]: W0727 19:19:20.032099 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1ac9a81c_b5c7_4241_8074_f5bf4ffda0eb.slice/crio-58ccf99a9e3d47b110e3780cff7f56b759db77f33a50667e30f3d8bf9732ae69 WatchSource:0}: Error finding container 58ccf99a9e3d47b110e3780cff7f56b759db77f33a50667e30f3d8bf9732ae69: Status 404 returned error can't find the container with id 58ccf99a9e3d47b110e3780cff7f56b759db77f33a50667e30f3d8bf9732ae69 Jul 27 19:19:20.882384 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:19:20.882348 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-7f7473/sub-test-model-7f7473-kserve-6f96848bf9-cshlt" event={"ID":"1ac9a81c-b5c7-4241-8074-f5bf4ffda0eb","Type":"ContainerStarted","Data":"d403e4270f7c8b5af4271dbc90a34b7e2abacbd566dfec78aa0aac81a17a75cb"} Jul 27 19:19:20.882384 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:19:20.882388 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-7f7473/sub-test-model-7f7473-kserve-6f96848bf9-cshlt" event={"ID":"1ac9a81c-b5c7-4241-8074-f5bf4ffda0eb","Type":"ContainerStarted","Data":"58ccf99a9e3d47b110e3780cff7f56b759db77f33a50667e30f3d8bf9732ae69"} Jul 27 19:19:20.920347 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:19:20.920290 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-7f7473/sub-test-model-7f7473-kserve-6f96848bf9-cshlt" podStartSLOduration=3.920275246 podStartE2EDuration="3.920275246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:19:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:19:20.917327196 +0000 UTC m=+3035.979016229" watchObservedRunningTime="2026-07-27 19:19:20.920275246 +0000 UTC m=+3035.981964277" Jul 27 19:19:26.499184 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:19:26.499142 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:19:29.860803 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:19:29.860757 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-7f7473/sub-test-model-7f7473-kserve-6f96848bf9-cshlt" Jul 27 19:19:29.860803 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:19:29.860812 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-7f7473/sub-test-model-7f7473-kserve-6f96848bf9-cshlt" Jul 27 19:19:29.873760 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:19:29.873732 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-7f7473/sub-test-model-7f7473-kserve-6f96848bf9-cshlt" Jul 27 19:19:29.943100 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:19:29.943068 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-7f7473/sub-test-model-7f7473-kserve-6f96848bf9-cshlt" Jul 27 19:19:50.287340 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:19:50.287298 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:21:27.805198 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:21:27.805119 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-7f7473/sub-test-model-7f7473-kserve-6f96848bf9-cshlt"] Jul 27 19:21:27.862950 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:21:27.862913 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:21:28.742543 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:21:28.742508 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:21:36.376134 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:21:36.376073 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:21:42.826641 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:21:42.826593 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-7f7473/sub-test-model-7f7473-kserve-6f96848bf9-cshlt" podUID="1ac9a81c-b5c7-4241-8074-f5bf4ffda0eb" containerName="main" containerID="cri-o://d403e4270f7c8b5af4271dbc90a34b7e2abacbd566dfec78aa0aac81a17a75cb" gracePeriod=45 Jul 27 19:21:43.077269 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:21:43.077181 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-7f7473/sub-test-model-7f7473-kserve-6f96848bf9-cshlt" Jul 27 19:21:43.156242 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:21:43.156195 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tbbjz\" (UniqueName: \"kubernetes.io/projected/1ac9a81c-b5c7-4241-8074-f5bf4ffda0eb-kube-api-access-tbbjz\") pod \"1ac9a81c-b5c7-4241-8074-f5bf4ffda0eb\" (UID: \"1ac9a81c-b5c7-4241-8074-f5bf4ffda0eb\") " Jul 27 19:21:43.156404 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:21:43.156258 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ac9a81c-b5c7-4241-8074-f5bf4ffda0eb-tmp-dir\") pod \"1ac9a81c-b5c7-4241-8074-f5bf4ffda0eb\" (UID: \"1ac9a81c-b5c7-4241-8074-f5bf4ffda0eb\") " Jul 27 19:21:43.156404 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:21:43.156296 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ac9a81c-b5c7-4241-8074-f5bf4ffda0eb-dshm\") pod \"1ac9a81c-b5c7-4241-8074-f5bf4ffda0eb\" (UID: \"1ac9a81c-b5c7-4241-8074-f5bf4ffda0eb\") " Jul 27 19:21:43.156404 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:21:43.156336 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ac9a81c-b5c7-4241-8074-f5bf4ffda0eb-model-cache\") pod \"1ac9a81c-b5c7-4241-8074-f5bf4ffda0eb\" (UID: \"1ac9a81c-b5c7-4241-8074-f5bf4ffda0eb\") " Jul 27 19:21:43.156404 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:21:43.156352 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ac9a81c-b5c7-4241-8074-f5bf4ffda0eb-tls-certs\") pod \"1ac9a81c-b5c7-4241-8074-f5bf4ffda0eb\" (UID: \"1ac9a81c-b5c7-4241-8074-f5bf4ffda0eb\") " Jul 27 19:21:43.156404 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:21:43.156385 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ac9a81c-b5c7-4241-8074-f5bf4ffda0eb-home\") pod \"1ac9a81c-b5c7-4241-8074-f5bf4ffda0eb\" (UID: \"1ac9a81c-b5c7-4241-8074-f5bf4ffda0eb\") " Jul 27 19:21:43.156678 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:21:43.156587 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ac9a81c-b5c7-4241-8074-f5bf4ffda0eb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1ac9a81c-b5c7-4241-8074-f5bf4ffda0eb" (UID: "1ac9a81c-b5c7-4241-8074-f5bf4ffda0eb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 19:21:43.156678 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:21:43.156607 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ac9a81c-b5c7-4241-8074-f5bf4ffda0eb-model-cache" (OuterVolumeSpecName: "model-cache") pod "1ac9a81c-b5c7-4241-8074-f5bf4ffda0eb" (UID: "1ac9a81c-b5c7-4241-8074-f5bf4ffda0eb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 19:21:43.156678 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:21:43.156653 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ac9a81c-b5c7-4241-8074-f5bf4ffda0eb-home" (OuterVolumeSpecName: "home") pod "1ac9a81c-b5c7-4241-8074-f5bf4ffda0eb" (UID: "1ac9a81c-b5c7-4241-8074-f5bf4ffda0eb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 19:21:43.156793 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:21:43.156780 2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ac9a81c-b5c7-4241-8074-f5bf4ffda0eb-model-cache\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 19:21:43.156832 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:21:43.156800 2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ac9a81c-b5c7-4241-8074-f5bf4ffda0eb-home\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 19:21:43.156832 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:21:43.156813 2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ac9a81c-b5c7-4241-8074-f5bf4ffda0eb-tmp-dir\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 19:21:43.158682 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:21:43.158658 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ac9a81c-b5c7-4241-8074-f5bf4ffda0eb-dshm" (OuterVolumeSpecName: "dshm") pod "1ac9a81c-b5c7-4241-8074-f5bf4ffda0eb" (UID: "1ac9a81c-b5c7-4241-8074-f5bf4ffda0eb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 19:21:43.159102 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:21:43.159079 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ac9a81c-b5c7-4241-8074-f5bf4ffda0eb-kube-api-access-tbbjz" (OuterVolumeSpecName: "kube-api-access-tbbjz") pod "1ac9a81c-b5c7-4241-8074-f5bf4ffda0eb" (UID: "1ac9a81c-b5c7-4241-8074-f5bf4ffda0eb"). InnerVolumeSpecName "kube-api-access-tbbjz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 19:21:43.159202 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:21:43.159129 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ac9a81c-b5c7-4241-8074-f5bf4ffda0eb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1ac9a81c-b5c7-4241-8074-f5bf4ffda0eb" (UID: "1ac9a81c-b5c7-4241-8074-f5bf4ffda0eb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 19:21:43.257606 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:21:43.257569 2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ac9a81c-b5c7-4241-8074-f5bf4ffda0eb-dshm\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 19:21:43.257606 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:21:43.257599 2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ac9a81c-b5c7-4241-8074-f5bf4ffda0eb-tls-certs\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 19:21:43.257606 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:21:43.257610 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbbjz\" (UniqueName: \"kubernetes.io/projected/1ac9a81c-b5c7-4241-8074-f5bf4ffda0eb-kube-api-access-tbbjz\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 19:21:43.642476 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:21:43.642440 2579 generic.go:356] "Generic (PLEG): container finished" podID="1ac9a81c-b5c7-4241-8074-f5bf4ffda0eb" containerID="d403e4270f7c8b5af4271dbc90a34b7e2abacbd566dfec78aa0aac81a17a75cb" exitCode=0 Jul 27 19:21:43.642667 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:21:43.642487 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-7f7473/sub-test-model-7f7473-kserve-6f96848bf9-cshlt" Jul 27 19:21:43.642667 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:21:43.642517 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-7f7473/sub-test-model-7f7473-kserve-6f96848bf9-cshlt" event={"ID":"1ac9a81c-b5c7-4241-8074-f5bf4ffda0eb","Type":"ContainerDied","Data":"d403e4270f7c8b5af4271dbc90a34b7e2abacbd566dfec78aa0aac81a17a75cb"} Jul 27 19:21:43.642667 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:21:43.642558 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-7f7473/sub-test-model-7f7473-kserve-6f96848bf9-cshlt" event={"ID":"1ac9a81c-b5c7-4241-8074-f5bf4ffda0eb","Type":"ContainerDied","Data":"58ccf99a9e3d47b110e3780cff7f56b759db77f33a50667e30f3d8bf9732ae69"} Jul 27 19:21:43.642667 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:21:43.642578 2579 scope.go:117] "RemoveContainer" containerID="d403e4270f7c8b5af4271dbc90a34b7e2abacbd566dfec78aa0aac81a17a75cb" Jul 27 19:21:43.651576 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:21:43.651559 2579 scope.go:117] "RemoveContainer" containerID="d403e4270f7c8b5af4271dbc90a34b7e2abacbd566dfec78aa0aac81a17a75cb" Jul 27 19:21:43.651844 ip-10-0-128-32 kubenswrapper[2579]: E0727 19:21:43.651826 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d403e4270f7c8b5af4271dbc90a34b7e2abacbd566dfec78aa0aac81a17a75cb\": container with ID starting with d403e4270f7c8b5af4271dbc90a34b7e2abacbd566dfec78aa0aac81a17a75cb not found: ID does not exist" containerID="d403e4270f7c8b5af4271dbc90a34b7e2abacbd566dfec78aa0aac81a17a75cb" Jul 27 19:21:43.651910 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:21:43.651855 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d403e4270f7c8b5af4271dbc90a34b7e2abacbd566dfec78aa0aac81a17a75cb"} err="failed to get container status \"d403e4270f7c8b5af4271dbc90a34b7e2abacbd566dfec78aa0aac81a17a75cb\": rpc error: code = NotFound desc = could not find container \"d403e4270f7c8b5af4271dbc90a34b7e2abacbd566dfec78aa0aac81a17a75cb\": container with ID starting with d403e4270f7c8b5af4271dbc90a34b7e2abacbd566dfec78aa0aac81a17a75cb not found: ID does not exist" Jul 27 19:21:43.685091 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:21:43.685060 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-7f7473/sub-test-model-7f7473-kserve-6f96848bf9-cshlt"] Jul 27 19:21:43.689032 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:21:43.689009 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-7f7473/sub-test-model-7f7473-kserve-6f96848bf9-cshlt"] Jul 27 19:21:45.493748 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:21:45.493715 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1ac9a81c-b5c7-4241-8074-f5bf4ffda0eb" path="/var/lib/kubelet/pods/1ac9a81c-b5c7-4241-8074-f5bf4ffda0eb/volumes" Jul 27 19:22:01.872517 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:22:01.872482 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-6710a9/rate-test-model-6710a9-kserve-7867b54999-nnvrj"] Jul 27 19:22:01.872924 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:22:01.872879 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ac9a81c-b5c7-4241-8074-f5bf4ffda0eb" containerName="main" Jul 27 19:22:01.872924 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:22:01.872890 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ac9a81c-b5c7-4241-8074-f5bf4ffda0eb" containerName="main" Jul 27 19:22:01.872998 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:22:01.872956 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ac9a81c-b5c7-4241-8074-f5bf4ffda0eb" containerName="main" Jul 27 19:22:01.875657 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:22:01.875640 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-6710a9/rate-test-model-6710a9-kserve-7867b54999-nnvrj" Jul 27 19:22:01.879442 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:22:01.879411 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-6710a9\"/\"rate-test-model-6710a9-kserve-self-signed-certs\"" Jul 27 19:22:01.880864 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:22:01.880840 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-6710a9\"/\"kube-root-ca.crt\"" Jul 27 19:22:01.880981 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:22:01.880883 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-6710a9\"/\"default-dockercfg-r8f6c\"" Jul 27 19:22:01.883384 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:22:01.883364 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-6710a9\"/\"openshift-service-ca.crt\"" Jul 27 19:22:01.893719 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:22:01.893684 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-6710a9/rate-test-model-6710a9-kserve-7867b54999-nnvrj"] Jul 27 19:22:01.912557 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:22:01.912524 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:22:02.043565 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:22:02.043534 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a64cf674-dded-4466-8522-b363512207ba-tmp-dir\") pod \"rate-test-model-6710a9-kserve-7867b54999-nnvrj\" (UID: \"a64cf674-dded-4466-8522-b363512207ba\") " pod="ai-tenant-e2e-shared-b-6710a9/rate-test-model-6710a9-kserve-7867b54999-nnvrj" Jul 27 19:22:02.043565 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:22:02.043573 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cpxz\" (UniqueName: \"kubernetes.io/projected/a64cf674-dded-4466-8522-b363512207ba-kube-api-access-4cpxz\") pod \"rate-test-model-6710a9-kserve-7867b54999-nnvrj\" (UID: \"a64cf674-dded-4466-8522-b363512207ba\") " pod="ai-tenant-e2e-shared-b-6710a9/rate-test-model-6710a9-kserve-7867b54999-nnvrj" Jul 27 19:22:02.043789 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:22:02.043600 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a64cf674-dded-4466-8522-b363512207ba-model-cache\") pod \"rate-test-model-6710a9-kserve-7867b54999-nnvrj\" (UID: \"a64cf674-dded-4466-8522-b363512207ba\") " pod="ai-tenant-e2e-shared-b-6710a9/rate-test-model-6710a9-kserve-7867b54999-nnvrj" Jul 27 19:22:02.043789 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:22:02.043702 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a64cf674-dded-4466-8522-b363512207ba-home\") pod \"rate-test-model-6710a9-kserve-7867b54999-nnvrj\" (UID: \"a64cf674-dded-4466-8522-b363512207ba\") " pod="ai-tenant-e2e-shared-b-6710a9/rate-test-model-6710a9-kserve-7867b54999-nnvrj" Jul 27 19:22:02.043789 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:22:02.043730 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a64cf674-dded-4466-8522-b363512207ba-dshm\") pod \"rate-test-model-6710a9-kserve-7867b54999-nnvrj\" (UID: \"a64cf674-dded-4466-8522-b363512207ba\") " pod="ai-tenant-e2e-shared-b-6710a9/rate-test-model-6710a9-kserve-7867b54999-nnvrj" Jul 27 19:22:02.043789 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:22:02.043746 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a64cf674-dded-4466-8522-b363512207ba-tls-certs\") pod \"rate-test-model-6710a9-kserve-7867b54999-nnvrj\" (UID: \"a64cf674-dded-4466-8522-b363512207ba\") " pod="ai-tenant-e2e-shared-b-6710a9/rate-test-model-6710a9-kserve-7867b54999-nnvrj" Jul 27 19:22:02.145075 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:22:02.144991 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a64cf674-dded-4466-8522-b363512207ba-tmp-dir\") pod \"rate-test-model-6710a9-kserve-7867b54999-nnvrj\" (UID: \"a64cf674-dded-4466-8522-b363512207ba\") " pod="ai-tenant-e2e-shared-b-6710a9/rate-test-model-6710a9-kserve-7867b54999-nnvrj" Jul 27 19:22:02.145075 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:22:02.145036 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cpxz\" (UniqueName: \"kubernetes.io/projected/a64cf674-dded-4466-8522-b363512207ba-kube-api-access-4cpxz\") pod \"rate-test-model-6710a9-kserve-7867b54999-nnvrj\" (UID: \"a64cf674-dded-4466-8522-b363512207ba\") " pod="ai-tenant-e2e-shared-b-6710a9/rate-test-model-6710a9-kserve-7867b54999-nnvrj" Jul 27 19:22:02.145314 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:22:02.145088 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a64cf674-dded-4466-8522-b363512207ba-model-cache\") pod \"rate-test-model-6710a9-kserve-7867b54999-nnvrj\" (UID: \"a64cf674-dded-4466-8522-b363512207ba\") " pod="ai-tenant-e2e-shared-b-6710a9/rate-test-model-6710a9-kserve-7867b54999-nnvrj" Jul 27 19:22:02.145314 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:22:02.145185 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a64cf674-dded-4466-8522-b363512207ba-home\") pod \"rate-test-model-6710a9-kserve-7867b54999-nnvrj\" (UID: \"a64cf674-dded-4466-8522-b363512207ba\") " pod="ai-tenant-e2e-shared-b-6710a9/rate-test-model-6710a9-kserve-7867b54999-nnvrj" Jul 27 19:22:02.145314 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:22:02.145268 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a64cf674-dded-4466-8522-b363512207ba-dshm\") pod \"rate-test-model-6710a9-kserve-7867b54999-nnvrj\" (UID: \"a64cf674-dded-4466-8522-b363512207ba\") " pod="ai-tenant-e2e-shared-b-6710a9/rate-test-model-6710a9-kserve-7867b54999-nnvrj" Jul 27 19:22:02.145314 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:22:02.145300 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a64cf674-dded-4466-8522-b363512207ba-tls-certs\") pod \"rate-test-model-6710a9-kserve-7867b54999-nnvrj\" (UID: \"a64cf674-dded-4466-8522-b363512207ba\") " pod="ai-tenant-e2e-shared-b-6710a9/rate-test-model-6710a9-kserve-7867b54999-nnvrj" Jul 27 19:22:02.145551 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:22:02.145532 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a64cf674-dded-4466-8522-b363512207ba-tmp-dir\") pod \"rate-test-model-6710a9-kserve-7867b54999-nnvrj\" (UID: \"a64cf674-dded-4466-8522-b363512207ba\") " pod="ai-tenant-e2e-shared-b-6710a9/rate-test-model-6710a9-kserve-7867b54999-nnvrj" Jul 27 19:22:02.145614 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:22:02.145583 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a64cf674-dded-4466-8522-b363512207ba-model-cache\") pod \"rate-test-model-6710a9-kserve-7867b54999-nnvrj\" (UID: \"a64cf674-dded-4466-8522-b363512207ba\") " pod="ai-tenant-e2e-shared-b-6710a9/rate-test-model-6710a9-kserve-7867b54999-nnvrj" Jul 27 19:22:02.145735 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:22:02.145610 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a64cf674-dded-4466-8522-b363512207ba-home\") pod \"rate-test-model-6710a9-kserve-7867b54999-nnvrj\" (UID: \"a64cf674-dded-4466-8522-b363512207ba\") " pod="ai-tenant-e2e-shared-b-6710a9/rate-test-model-6710a9-kserve-7867b54999-nnvrj" Jul 27 19:22:02.147744 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:22:02.147725 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a64cf674-dded-4466-8522-b363512207ba-dshm\") pod \"rate-test-model-6710a9-kserve-7867b54999-nnvrj\" (UID: \"a64cf674-dded-4466-8522-b363512207ba\") " pod="ai-tenant-e2e-shared-b-6710a9/rate-test-model-6710a9-kserve-7867b54999-nnvrj" Jul 27 19:22:02.148143 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:22:02.148120 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a64cf674-dded-4466-8522-b363512207ba-tls-certs\") pod \"rate-test-model-6710a9-kserve-7867b54999-nnvrj\" (UID: \"a64cf674-dded-4466-8522-b363512207ba\") " pod="ai-tenant-e2e-shared-b-6710a9/rate-test-model-6710a9-kserve-7867b54999-nnvrj" Jul 27 19:22:02.159593 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:22:02.159564 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cpxz\" (UniqueName: \"kubernetes.io/projected/a64cf674-dded-4466-8522-b363512207ba-kube-api-access-4cpxz\") pod \"rate-test-model-6710a9-kserve-7867b54999-nnvrj\" (UID: \"a64cf674-dded-4466-8522-b363512207ba\") " pod="ai-tenant-e2e-shared-b-6710a9/rate-test-model-6710a9-kserve-7867b54999-nnvrj" Jul 27 19:22:02.185916 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:22:02.185890 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-6710a9/rate-test-model-6710a9-kserve-7867b54999-nnvrj" Jul 27 19:22:02.331768 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:22:02.331740 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-6710a9/rate-test-model-6710a9-kserve-7867b54999-nnvrj"] Jul 27 19:22:02.333479 ip-10-0-128-32 kubenswrapper[2579]: W0727 19:22:02.333441 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda64cf674_dded_4466_8522_b363512207ba.slice/crio-c7f02f0b1661248ec08d17551e500b9eff75154c54baf1d58d3daa273e93c773 WatchSource:0}: Error finding container c7f02f0b1661248ec08d17551e500b9eff75154c54baf1d58d3daa273e93c773: Status 404 returned error can't find the container with id c7f02f0b1661248ec08d17551e500b9eff75154c54baf1d58d3daa273e93c773 Jul 27 19:22:02.750774 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:22:02.750739 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-6710a9/rate-test-model-6710a9-kserve-7867b54999-nnvrj" event={"ID":"a64cf674-dded-4466-8522-b363512207ba","Type":"ContainerStarted","Data":"eeba89d5997f2bae2fb4f378e9263b5a1db90ce279be3f856f342483112fb9a6"} Jul 27 19:22:02.750774 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:22:02.750783 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-6710a9/rate-test-model-6710a9-kserve-7867b54999-nnvrj" event={"ID":"a64cf674-dded-4466-8522-b363512207ba","Type":"ContainerStarted","Data":"c7f02f0b1661248ec08d17551e500b9eff75154c54baf1d58d3daa273e93c773"} Jul 27 19:22:02.770710 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:22:02.770662 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-6710a9/rate-test-model-6710a9-kserve-7867b54999-nnvrj" podStartSLOduration=1.770642286 podStartE2EDuration="1.770642286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:22:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:22:02.769486069 +0000 UTC m=+3197.831175102" watchObservedRunningTime="2026-07-27 19:22:02.770642286 +0000 UTC m=+3197.832331316" Jul 27 19:22:12.186668 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:22:12.186624 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-6710a9/rate-test-model-6710a9-kserve-7867b54999-nnvrj" Jul 27 19:22:12.187218 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:22:12.186684 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-6710a9/rate-test-model-6710a9-kserve-7867b54999-nnvrj" Jul 27 19:22:12.199882 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:22:12.199859 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-6710a9/rate-test-model-6710a9-kserve-7867b54999-nnvrj" Jul 27 19:22:12.815200 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:22:12.815159 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-6710a9/rate-test-model-6710a9-kserve-7867b54999-nnvrj" Jul 27 19:23:15.203166 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:15.203130 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:23:16.155828 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:16.155768 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-6710a9/rate-test-model-6710a9-kserve-7867b54999-nnvrj"] Jul 27 19:23:16.177001 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:16.176972 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:23:31.175934 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:31.175883 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-6710a9/rate-test-model-6710a9-kserve-7867b54999-nnvrj" podUID="a64cf674-dded-4466-8522-b363512207ba" containerName="main" containerID="cri-o://eeba89d5997f2bae2fb4f378e9263b5a1db90ce279be3f856f342483112fb9a6" gracePeriod=45 Jul 27 19:23:31.432704 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:31.432639 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-6710a9/rate-test-model-6710a9-kserve-7867b54999-nnvrj" Jul 27 19:23:31.529858 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:31.529828 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a64cf674-dded-4466-8522-b363512207ba-tls-certs\") pod \"a64cf674-dded-4466-8522-b363512207ba\" (UID: \"a64cf674-dded-4466-8522-b363512207ba\") " Jul 27 19:23:31.530037 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:31.529896 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a64cf674-dded-4466-8522-b363512207ba-tmp-dir\") pod \"a64cf674-dded-4466-8522-b363512207ba\" (UID: \"a64cf674-dded-4466-8522-b363512207ba\") " Jul 27 19:23:31.530037 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:31.529920 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a64cf674-dded-4466-8522-b363512207ba-dshm\") pod \"a64cf674-dded-4466-8522-b363512207ba\" (UID: \"a64cf674-dded-4466-8522-b363512207ba\") " Jul 27 19:23:31.530037 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:31.529946 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a64cf674-dded-4466-8522-b363512207ba-home\") pod \"a64cf674-dded-4466-8522-b363512207ba\" (UID: \"a64cf674-dded-4466-8522-b363512207ba\") " Jul 27 19:23:31.530211 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:31.530104 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4cpxz\" (UniqueName: \"kubernetes.io/projected/a64cf674-dded-4466-8522-b363512207ba-kube-api-access-4cpxz\") pod \"a64cf674-dded-4466-8522-b363512207ba\" (UID: \"a64cf674-dded-4466-8522-b363512207ba\") " Jul 27 19:23:31.530211 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:31.530134 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a64cf674-dded-4466-8522-b363512207ba-model-cache\") pod \"a64cf674-dded-4466-8522-b363512207ba\" (UID: \"a64cf674-dded-4466-8522-b363512207ba\") " Jul 27 19:23:31.530418 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:31.530378 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a64cf674-dded-4466-8522-b363512207ba-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a64cf674-dded-4466-8522-b363512207ba" (UID: "a64cf674-dded-4466-8522-b363512207ba"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 19:23:31.530501 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:31.530479 2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a64cf674-dded-4466-8522-b363512207ba-tmp-dir\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 19:23:31.530641 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:31.530615 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a64cf674-dded-4466-8522-b363512207ba-model-cache" (OuterVolumeSpecName: "model-cache") pod "a64cf674-dded-4466-8522-b363512207ba" (UID: "a64cf674-dded-4466-8522-b363512207ba"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 19:23:31.530694 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:31.530678 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a64cf674-dded-4466-8522-b363512207ba-home" (OuterVolumeSpecName: "home") pod "a64cf674-dded-4466-8522-b363512207ba" (UID: "a64cf674-dded-4466-8522-b363512207ba"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 19:23:31.532414 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:31.532388 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a64cf674-dded-4466-8522-b363512207ba-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a64cf674-dded-4466-8522-b363512207ba" (UID: "a64cf674-dded-4466-8522-b363512207ba"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 19:23:31.532607 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:31.532588 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a64cf674-dded-4466-8522-b363512207ba-kube-api-access-4cpxz" (OuterVolumeSpecName: "kube-api-access-4cpxz") pod "a64cf674-dded-4466-8522-b363512207ba" (UID: "a64cf674-dded-4466-8522-b363512207ba"). InnerVolumeSpecName "kube-api-access-4cpxz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 19:23:31.532935 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:31.532916 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a64cf674-dded-4466-8522-b363512207ba-dshm" (OuterVolumeSpecName: "dshm") pod "a64cf674-dded-4466-8522-b363512207ba" (UID: "a64cf674-dded-4466-8522-b363512207ba"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 19:23:31.631244 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:31.631187 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4cpxz\" (UniqueName: \"kubernetes.io/projected/a64cf674-dded-4466-8522-b363512207ba-kube-api-access-4cpxz\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 19:23:31.631244 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:31.631241 2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a64cf674-dded-4466-8522-b363512207ba-model-cache\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 19:23:31.631439 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:31.631257 2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a64cf674-dded-4466-8522-b363512207ba-tls-certs\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 19:23:31.631439 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:31.631270 2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a64cf674-dded-4466-8522-b363512207ba-dshm\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 19:23:31.631439 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:31.631280 2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a64cf674-dded-4466-8522-b363512207ba-home\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 19:23:31.965939 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:31.965907 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7701-140320378424320"] Jul 27 19:23:31.966351 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:31.966336 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a64cf674-dded-4466-8522-b363512207ba" containerName="main" Jul 27 19:23:31.966351 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:31.966352 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a64cf674-dded-4466-8522-b363512207ba" containerName="main" Jul 27 19:23:31.966448 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:31.966418 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a64cf674-dded-4466-8522-b363512207ba" containerName="main" Jul 27 19:23:31.969607 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:31.969590 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7701-140320378424320" Jul 27 19:23:31.974523 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:31.974504 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-5jdw9\"" Jul 27 19:23:31.983457 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:31.983432 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7701-140320378424320"] Jul 27 19:23:32.035908 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:32.035872 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2525\" (UniqueName: \"kubernetes.io/projected/374585bf-2aad-4c88-85ae-e995d28069da-kube-api-access-l2525\") pod \"test-curl-7701-140320378424320\" (UID: \"374585bf-2aad-4c88-85ae-e995d28069da\") " pod="opendatahub/test-curl-7701-140320378424320" Jul 27 19:23:32.136662 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:32.136613 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2525\" (UniqueName: \"kubernetes.io/projected/374585bf-2aad-4c88-85ae-e995d28069da-kube-api-access-l2525\") pod \"test-curl-7701-140320378424320\" (UID: \"374585bf-2aad-4c88-85ae-e995d28069da\") " pod="opendatahub/test-curl-7701-140320378424320" Jul 27 19:23:32.146274 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:32.146242 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2525\" (UniqueName: \"kubernetes.io/projected/374585bf-2aad-4c88-85ae-e995d28069da-kube-api-access-l2525\") pod \"test-curl-7701-140320378424320\" (UID: \"374585bf-2aad-4c88-85ae-e995d28069da\") " pod="opendatahub/test-curl-7701-140320378424320" Jul 27 19:23:32.241668 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:32.241581 2579 generic.go:356] "Generic (PLEG): container finished" podID="a64cf674-dded-4466-8522-b363512207ba" containerID="eeba89d5997f2bae2fb4f378e9263b5a1db90ce279be3f856f342483112fb9a6" exitCode=0 Jul 27 19:23:32.241668 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:32.241646 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-6710a9/rate-test-model-6710a9-kserve-7867b54999-nnvrj" Jul 27 19:23:32.242140 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:32.241670 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-6710a9/rate-test-model-6710a9-kserve-7867b54999-nnvrj" event={"ID":"a64cf674-dded-4466-8522-b363512207ba","Type":"ContainerDied","Data":"eeba89d5997f2bae2fb4f378e9263b5a1db90ce279be3f856f342483112fb9a6"} Jul 27 19:23:32.242140 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:32.241720 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-6710a9/rate-test-model-6710a9-kserve-7867b54999-nnvrj" event={"ID":"a64cf674-dded-4466-8522-b363512207ba","Type":"ContainerDied","Data":"c7f02f0b1661248ec08d17551e500b9eff75154c54baf1d58d3daa273e93c773"} Jul 27 19:23:32.242140 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:32.241743 2579 scope.go:117] "RemoveContainer" containerID="eeba89d5997f2bae2fb4f378e9263b5a1db90ce279be3f856f342483112fb9a6" Jul 27 19:23:32.251599 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:32.251580 2579 scope.go:117] "RemoveContainer" containerID="eeba89d5997f2bae2fb4f378e9263b5a1db90ce279be3f856f342483112fb9a6" Jul 27 19:23:32.251868 ip-10-0-128-32 kubenswrapper[2579]: E0727 19:23:32.251850 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eeba89d5997f2bae2fb4f378e9263b5a1db90ce279be3f856f342483112fb9a6\": container with ID starting with eeba89d5997f2bae2fb4f378e9263b5a1db90ce279be3f856f342483112fb9a6 not found: ID does not exist" containerID="eeba89d5997f2bae2fb4f378e9263b5a1db90ce279be3f856f342483112fb9a6" Jul 27 19:23:32.251918 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:32.251876 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eeba89d5997f2bae2fb4f378e9263b5a1db90ce279be3f856f342483112fb9a6"} err="failed to get container status \"eeba89d5997f2bae2fb4f378e9263b5a1db90ce279be3f856f342483112fb9a6\": rpc error: code = NotFound desc = could not find container \"eeba89d5997f2bae2fb4f378e9263b5a1db90ce279be3f856f342483112fb9a6\": container with ID starting with eeba89d5997f2bae2fb4f378e9263b5a1db90ce279be3f856f342483112fb9a6 not found: ID does not exist" Jul 27 19:23:32.268388 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:32.268366 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-6710a9/rate-test-model-6710a9-kserve-7867b54999-nnvrj"] Jul 27 19:23:32.273446 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:32.273425 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-6710a9/rate-test-model-6710a9-kserve-7867b54999-nnvrj"] Jul 27 19:23:32.278457 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:32.278440 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7701-140320378424320" Jul 27 19:23:32.432738 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:32.432705 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7701-140320378424320"] Jul 27 19:23:32.433912 ip-10-0-128-32 kubenswrapper[2579]: W0727 19:23:32.433887 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod374585bf_2aad_4c88_85ae_e995d28069da.slice/crio-d24ec6184b4c6726aca87267a2c60b511e0be9357fd06600291728c3d450dba9 WatchSource:0}: Error finding container d24ec6184b4c6726aca87267a2c60b511e0be9357fd06600291728c3d450dba9: Status 404 returned error can't find the container with id d24ec6184b4c6726aca87267a2c60b511e0be9357fd06600291728c3d450dba9 Jul 27 19:23:32.435180 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:32.435164 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 27 19:23:33.249192 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:33.249155 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7701-140320378424320" event={"ID":"374585bf-2aad-4c88-85ae-e995d28069da","Type":"ContainerStarted","Data":"4d8b78b7dc977903b1624437d6e4d4abdcd7877d099e33e6f41681c60fa7ada3"} Jul 27 19:23:33.249192 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:33.249196 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7701-140320378424320" event={"ID":"374585bf-2aad-4c88-85ae-e995d28069da","Type":"ContainerStarted","Data":"d24ec6184b4c6726aca87267a2c60b511e0be9357fd06600291728c3d450dba9"} Jul 27 19:23:33.271896 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:33.271853 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7701-140320378424320" podStartSLOduration=1.558786346 podStartE2EDuration="2.271837439s" totalImagesPullingTime="713.051093ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:23:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:23:33.271000639 +0000 UTC m=+3288.332689670" watchObservedRunningTime="2026-07-27 19:23:33.271837439 +0000 UTC m=+3288.333526469" Jul 27 19:23:33.421735 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:33.421648 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7701-140320378424320_374585bf-2aad-4c88-85ae-e995d28069da/test-curl-7701-140320378424320/0.log" Jul 27 19:23:33.493309 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:33.493267 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a64cf674-dded-4466-8522-b363512207ba" path="/var/lib/kubelet/pods/a64cf674-dded-4466-8522-b363512207ba/volumes" Jul 27 19:23:34.255505 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:34.255472 2579 generic.go:356] "Generic (PLEG): container finished" podID="374585bf-2aad-4c88-85ae-e995d28069da" containerID="4d8b78b7dc977903b1624437d6e4d4abdcd7877d099e33e6f41681c60fa7ada3" exitCode=0 Jul 27 19:23:34.255505 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:34.255507 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7701-140320378424320" event={"ID":"374585bf-2aad-4c88-85ae-e995d28069da","Type":"ContainerDied","Data":"4d8b78b7dc977903b1624437d6e4d4abdcd7877d099e33e6f41681c60fa7ada3"} Jul 27 19:23:35.400083 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:35.400052 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7701-140320378424320" Jul 27 19:23:35.443640 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:35.443611 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7701-140320378424320"] Jul 27 19:23:35.445923 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:35.445898 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7701-140320378424320"] Jul 27 19:23:35.468902 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:35.468872 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l2525\" (UniqueName: \"kubernetes.io/projected/374585bf-2aad-4c88-85ae-e995d28069da-kube-api-access-l2525\") pod \"374585bf-2aad-4c88-85ae-e995d28069da\" (UID: \"374585bf-2aad-4c88-85ae-e995d28069da\") " Jul 27 19:23:35.471122 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:35.471098 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/374585bf-2aad-4c88-85ae-e995d28069da-kube-api-access-l2525" (OuterVolumeSpecName: "kube-api-access-l2525") pod "374585bf-2aad-4c88-85ae-e995d28069da" (UID: "374585bf-2aad-4c88-85ae-e995d28069da"). InnerVolumeSpecName "kube-api-access-l2525". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 19:23:35.495122 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:35.495093 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="374585bf-2aad-4c88-85ae-e995d28069da" path="/var/lib/kubelet/pods/374585bf-2aad-4c88-85ae-e995d28069da/volumes" Jul 27 19:23:35.569726 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:35.569696 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2525\" (UniqueName: \"kubernetes.io/projected/374585bf-2aad-4c88-85ae-e995d28069da-kube-api-access-l2525\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 19:23:36.268394 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:36.268323 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7701-140320378424320" Jul 27 19:23:36.268394 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:36.268349 2579 scope.go:117] "RemoveContainer" containerID="4d8b78b7dc977903b1624437d6e4d4abdcd7877d099e33e6f41681c60fa7ada3" Jul 27 19:23:36.324404 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:36.324371 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-1d1107-openshift-default-5bfc5f8cb6-mcf9c"] Jul 27 19:23:36.325141 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:36.325115 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="374585bf-2aad-4c88-85ae-e995d28069da" containerName="test-curl-7701-140320378424320" Jul 27 19:23:36.325365 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:36.325347 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="374585bf-2aad-4c88-85ae-e995d28069da" containerName="test-curl-7701-140320378424320" Jul 27 19:23:36.325652 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:36.325631 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="374585bf-2aad-4c88-85ae-e995d28069da" containerName="test-curl-7701-140320378424320" Jul 27 19:23:36.330424 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:36.330402 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-1d1107-openshift-default-5bfc5f8cb6-mcf9c" Jul 27 19:23:36.333552 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:36.333521 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-1d1107-openshift-default-dockercfg-nct6f\"" Jul 27 19:23:36.347626 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:36.347602 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-1d1107-openshift-default-5bfc5f8cb6-mcf9c"] Jul 27 19:23:36.478189 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:36.478145 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e07be686-8abf-43d8-8d57-ec992ee9cd35-workload-socket\") pod \"e2e-ipp-a-1d1107-openshift-default-5bfc5f8cb6-mcf9c\" (UID: \"e07be686-8abf-43d8-8d57-ec992ee9cd35\") " pod="openshift-ingress/e2e-ipp-a-1d1107-openshift-default-5bfc5f8cb6-mcf9c" Jul 27 19:23:36.478189 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:36.478191 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e07be686-8abf-43d8-8d57-ec992ee9cd35-istio-envoy\") pod \"e2e-ipp-a-1d1107-openshift-default-5bfc5f8cb6-mcf9c\" (UID: \"e07be686-8abf-43d8-8d57-ec992ee9cd35\") " pod="openshift-ingress/e2e-ipp-a-1d1107-openshift-default-5bfc5f8cb6-mcf9c" Jul 27 19:23:36.478636 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:36.478284 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e07be686-8abf-43d8-8d57-ec992ee9cd35-istio-token\") pod \"e2e-ipp-a-1d1107-openshift-default-5bfc5f8cb6-mcf9c\" (UID: \"e07be686-8abf-43d8-8d57-ec992ee9cd35\") " pod="openshift-ingress/e2e-ipp-a-1d1107-openshift-default-5bfc5f8cb6-mcf9c" Jul 27 19:23:36.478636 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:36.478339 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e07be686-8abf-43d8-8d57-ec992ee9cd35-workload-certs\") pod \"e2e-ipp-a-1d1107-openshift-default-5bfc5f8cb6-mcf9c\" (UID: \"e07be686-8abf-43d8-8d57-ec992ee9cd35\") " pod="openshift-ingress/e2e-ipp-a-1d1107-openshift-default-5bfc5f8cb6-mcf9c" Jul 27 19:23:36.478636 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:36.478401 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e07be686-8abf-43d8-8d57-ec992ee9cd35-credential-socket\") pod \"e2e-ipp-a-1d1107-openshift-default-5bfc5f8cb6-mcf9c\" (UID: \"e07be686-8abf-43d8-8d57-ec992ee9cd35\") " pod="openshift-ingress/e2e-ipp-a-1d1107-openshift-default-5bfc5f8cb6-mcf9c" Jul 27 19:23:36.478636 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:36.478439 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pdnk\" (UniqueName: \"kubernetes.io/projected/e07be686-8abf-43d8-8d57-ec992ee9cd35-kube-api-access-8pdnk\") pod \"e2e-ipp-a-1d1107-openshift-default-5bfc5f8cb6-mcf9c\" (UID: \"e07be686-8abf-43d8-8d57-ec992ee9cd35\") " pod="openshift-ingress/e2e-ipp-a-1d1107-openshift-default-5bfc5f8cb6-mcf9c" Jul 27 19:23:36.478636 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:36.478460 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e07be686-8abf-43d8-8d57-ec992ee9cd35-istio-podinfo\") pod \"e2e-ipp-a-1d1107-openshift-default-5bfc5f8cb6-mcf9c\" (UID: \"e07be686-8abf-43d8-8d57-ec992ee9cd35\") " pod="openshift-ingress/e2e-ipp-a-1d1107-openshift-default-5bfc5f8cb6-mcf9c" Jul 27 19:23:36.478636 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:36.478508 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e07be686-8abf-43d8-8d57-ec992ee9cd35-istio-data\") pod \"e2e-ipp-a-1d1107-openshift-default-5bfc5f8cb6-mcf9c\" (UID: \"e07be686-8abf-43d8-8d57-ec992ee9cd35\") " pod="openshift-ingress/e2e-ipp-a-1d1107-openshift-default-5bfc5f8cb6-mcf9c" Jul 27 19:23:36.478636 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:36.478532 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e07be686-8abf-43d8-8d57-ec992ee9cd35-istiod-ca-cert\") pod \"e2e-ipp-a-1d1107-openshift-default-5bfc5f8cb6-mcf9c\" (UID: \"e07be686-8abf-43d8-8d57-ec992ee9cd35\") " pod="openshift-ingress/e2e-ipp-a-1d1107-openshift-default-5bfc5f8cb6-mcf9c" Jul 27 19:23:36.578958 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:36.578924 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e07be686-8abf-43d8-8d57-ec992ee9cd35-istiod-ca-cert\") pod \"e2e-ipp-a-1d1107-openshift-default-5bfc5f8cb6-mcf9c\" (UID: \"e07be686-8abf-43d8-8d57-ec992ee9cd35\") " pod="openshift-ingress/e2e-ipp-a-1d1107-openshift-default-5bfc5f8cb6-mcf9c" Jul 27 19:23:36.579126 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:36.578970 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e07be686-8abf-43d8-8d57-ec992ee9cd35-workload-socket\") pod \"e2e-ipp-a-1d1107-openshift-default-5bfc5f8cb6-mcf9c\" (UID: \"e07be686-8abf-43d8-8d57-ec992ee9cd35\") " pod="openshift-ingress/e2e-ipp-a-1d1107-openshift-default-5bfc5f8cb6-mcf9c" Jul 27 19:23:36.579126 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:36.579007 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e07be686-8abf-43d8-8d57-ec992ee9cd35-istio-envoy\") pod \"e2e-ipp-a-1d1107-openshift-default-5bfc5f8cb6-mcf9c\" (UID: \"e07be686-8abf-43d8-8d57-ec992ee9cd35\") " pod="openshift-ingress/e2e-ipp-a-1d1107-openshift-default-5bfc5f8cb6-mcf9c" Jul 27 19:23:36.579126 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:36.579045 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e07be686-8abf-43d8-8d57-ec992ee9cd35-istio-token\") pod \"e2e-ipp-a-1d1107-openshift-default-5bfc5f8cb6-mcf9c\" (UID: \"e07be686-8abf-43d8-8d57-ec992ee9cd35\") " pod="openshift-ingress/e2e-ipp-a-1d1107-openshift-default-5bfc5f8cb6-mcf9c" Jul 27 19:23:36.579306 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:36.579200 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e07be686-8abf-43d8-8d57-ec992ee9cd35-workload-certs\") pod \"e2e-ipp-a-1d1107-openshift-default-5bfc5f8cb6-mcf9c\" (UID: \"e07be686-8abf-43d8-8d57-ec992ee9cd35\") " pod="openshift-ingress/e2e-ipp-a-1d1107-openshift-default-5bfc5f8cb6-mcf9c" Jul 27 19:23:36.579306 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:36.579291 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e07be686-8abf-43d8-8d57-ec992ee9cd35-credential-socket\") pod \"e2e-ipp-a-1d1107-openshift-default-5bfc5f8cb6-mcf9c\" (UID: \"e07be686-8abf-43d8-8d57-ec992ee9cd35\") " pod="openshift-ingress/e2e-ipp-a-1d1107-openshift-default-5bfc5f8cb6-mcf9c" Jul 27 19:23:36.579411 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:36.579333 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pdnk\" (UniqueName: \"kubernetes.io/projected/e07be686-8abf-43d8-8d57-ec992ee9cd35-kube-api-access-8pdnk\") pod \"e2e-ipp-a-1d1107-openshift-default-5bfc5f8cb6-mcf9c\" (UID: \"e07be686-8abf-43d8-8d57-ec992ee9cd35\") " pod="openshift-ingress/e2e-ipp-a-1d1107-openshift-default-5bfc5f8cb6-mcf9c" Jul 27 19:23:36.579411 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:36.579363 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e07be686-8abf-43d8-8d57-ec992ee9cd35-istio-podinfo\") pod \"e2e-ipp-a-1d1107-openshift-default-5bfc5f8cb6-mcf9c\" (UID: \"e07be686-8abf-43d8-8d57-ec992ee9cd35\") " pod="openshift-ingress/e2e-ipp-a-1d1107-openshift-default-5bfc5f8cb6-mcf9c" Jul 27 19:23:36.579521 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:36.579415 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e07be686-8abf-43d8-8d57-ec992ee9cd35-istio-data\") pod \"e2e-ipp-a-1d1107-openshift-default-5bfc5f8cb6-mcf9c\" (UID: \"e07be686-8abf-43d8-8d57-ec992ee9cd35\") " pod="openshift-ingress/e2e-ipp-a-1d1107-openshift-default-5bfc5f8cb6-mcf9c" Jul 27 19:23:36.579589 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:36.579565 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e07be686-8abf-43d8-8d57-ec992ee9cd35-workload-socket\") pod \"e2e-ipp-a-1d1107-openshift-default-5bfc5f8cb6-mcf9c\" (UID: \"e07be686-8abf-43d8-8d57-ec992ee9cd35\") " pod="openshift-ingress/e2e-ipp-a-1d1107-openshift-default-5bfc5f8cb6-mcf9c" Jul 27 19:23:36.579644 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:36.579607 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e07be686-8abf-43d8-8d57-ec992ee9cd35-workload-certs\") pod \"e2e-ipp-a-1d1107-openshift-default-5bfc5f8cb6-mcf9c\" (UID: \"e07be686-8abf-43d8-8d57-ec992ee9cd35\") " pod="openshift-ingress/e2e-ipp-a-1d1107-openshift-default-5bfc5f8cb6-mcf9c" Jul 27 19:23:36.579736 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:36.579713 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e07be686-8abf-43d8-8d57-ec992ee9cd35-credential-socket\") pod \"e2e-ipp-a-1d1107-openshift-default-5bfc5f8cb6-mcf9c\" (UID: \"e07be686-8abf-43d8-8d57-ec992ee9cd35\") " pod="openshift-ingress/e2e-ipp-a-1d1107-openshift-default-5bfc5f8cb6-mcf9c" Jul 27 19:23:36.579815 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:36.579793 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e07be686-8abf-43d8-8d57-ec992ee9cd35-istio-data\") pod \"e2e-ipp-a-1d1107-openshift-default-5bfc5f8cb6-mcf9c\" (UID: \"e07be686-8abf-43d8-8d57-ec992ee9cd35\") " pod="openshift-ingress/e2e-ipp-a-1d1107-openshift-default-5bfc5f8cb6-mcf9c" Jul 27 19:23:36.579869 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:36.579806 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e07be686-8abf-43d8-8d57-ec992ee9cd35-istiod-ca-cert\") pod \"e2e-ipp-a-1d1107-openshift-default-5bfc5f8cb6-mcf9c\" (UID: \"e07be686-8abf-43d8-8d57-ec992ee9cd35\") " pod="openshift-ingress/e2e-ipp-a-1d1107-openshift-default-5bfc5f8cb6-mcf9c" Jul 27 19:23:36.581563 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:36.581538 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e07be686-8abf-43d8-8d57-ec992ee9cd35-istio-envoy\") pod \"e2e-ipp-a-1d1107-openshift-default-5bfc5f8cb6-mcf9c\" (UID: \"e07be686-8abf-43d8-8d57-ec992ee9cd35\") " pod="openshift-ingress/e2e-ipp-a-1d1107-openshift-default-5bfc5f8cb6-mcf9c" Jul 27 19:23:36.582161 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:36.582145 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e07be686-8abf-43d8-8d57-ec992ee9cd35-istio-podinfo\") pod \"e2e-ipp-a-1d1107-openshift-default-5bfc5f8cb6-mcf9c\" (UID: \"e07be686-8abf-43d8-8d57-ec992ee9cd35\") " pod="openshift-ingress/e2e-ipp-a-1d1107-openshift-default-5bfc5f8cb6-mcf9c" Jul 27 19:23:36.593141 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:36.593122 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e07be686-8abf-43d8-8d57-ec992ee9cd35-istio-token\") pod \"e2e-ipp-a-1d1107-openshift-default-5bfc5f8cb6-mcf9c\" (UID: \"e07be686-8abf-43d8-8d57-ec992ee9cd35\") " pod="openshift-ingress/e2e-ipp-a-1d1107-openshift-default-5bfc5f8cb6-mcf9c" Jul 27 19:23:36.599483 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:36.599461 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pdnk\" (UniqueName: \"kubernetes.io/projected/e07be686-8abf-43d8-8d57-ec992ee9cd35-kube-api-access-8pdnk\") pod \"e2e-ipp-a-1d1107-openshift-default-5bfc5f8cb6-mcf9c\" (UID: \"e07be686-8abf-43d8-8d57-ec992ee9cd35\") " pod="openshift-ingress/e2e-ipp-a-1d1107-openshift-default-5bfc5f8cb6-mcf9c" Jul 27 19:23:36.643542 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:36.643510 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-1d1107-openshift-default-5bfc5f8cb6-mcf9c" Jul 27 19:23:36.805852 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:36.805819 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-1d1107-openshift-default-5bfc5f8cb6-mcf9c"] Jul 27 19:23:36.807316 ip-10-0-128-32 kubenswrapper[2579]: W0727 19:23:36.807288 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode07be686_8abf_43d8_8d57_ec992ee9cd35.slice/crio-dd44dab4daa5cd07eeabd18c534e04e2650c7eedc8447ab70dbea7c6a09d32e1 WatchSource:0}: Error finding container dd44dab4daa5cd07eeabd18c534e04e2650c7eedc8447ab70dbea7c6a09d32e1: Status 404 returned error can't find the container with id dd44dab4daa5cd07eeabd18c534e04e2650c7eedc8447ab70dbea7c6a09d32e1 Jul 27 19:23:36.809872 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:36.809828 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 27 19:23:36.809973 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:36.809908 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 27 19:23:36.809973 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:36.809945 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 27 19:23:36.840003 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:36.839942 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:23:37.276569 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:37.276528 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-1d1107-openshift-default-5bfc5f8cb6-mcf9c" event={"ID":"e07be686-8abf-43d8-8d57-ec992ee9cd35","Type":"ContainerStarted","Data":"ffeb57794d62f458acf4d533ccc0472e01418617982ae3f05b6d22d9a4f4207f"} Jul 27 19:23:37.276726 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:37.276615 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-1d1107-openshift-default-5bfc5f8cb6-mcf9c" event={"ID":"e07be686-8abf-43d8-8d57-ec992ee9cd35","Type":"ContainerStarted","Data":"dd44dab4daa5cd07eeabd18c534e04e2650c7eedc8447ab70dbea7c6a09d32e1"} Jul 27 19:23:37.307419 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:37.307359 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-1d1107-openshift-default-5bfc5f8cb6-mcf9c" podStartSLOduration=1.30734274 podStartE2EDuration="1.30734274s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:23:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:23:37.306212875 +0000 UTC m=+3292.367901906" watchObservedRunningTime="2026-07-27 19:23:37.30734274 +0000 UTC m=+3292.369031770" Jul 27 19:23:37.644347 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:37.644259 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-1d1107-openshift-default-5bfc5f8cb6-mcf9c" Jul 27 19:23:37.649571 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:37.649542 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-1d1107-openshift-default-5bfc5f8cb6-mcf9c" Jul 27 19:23:37.881156 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:37.881119 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:23:38.282145 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:38.282111 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-1d1107-openshift-default-5bfc5f8cb6-mcf9c" Jul 27 19:23:38.283158 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:38.283130 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-1d1107-openshift-default-5bfc5f8cb6-mcf9c" Jul 27 19:23:39.298752 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:39.298707 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:23:39.402057 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:39.402024 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:23:40.561732 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:40.561692 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:23:40.787545 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:40.787509 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:23:41.025733 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:41.025693 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:23:41.131713 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:41.131679 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:23:43.790489 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:43.790457 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:23:45.718433 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:45.718410 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-mv4hb_715b2084-d79e-4cbf-a057-f2595b4e6b9b/console-operator/1.log" Jul 27 19:23:45.737337 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:45.737315 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-mv4hb_715b2084-d79e-4cbf-a057-f2595b4e6b9b/console-operator/1.log" Jul 27 19:23:49.661177 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:49.661120 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:23:49.765379 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:49.765341 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:23:54.323433 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:54.323388 2579 patch_prober.go:28] interesting pod/e2e-ipp-a-1d1107-openshift-default-5bfc5f8cb6-mcf9c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 27 19:23:54.323855 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:54.323472 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-1d1107-openshift-default-5bfc5f8cb6-mcf9c" podUID="e07be686-8abf-43d8-8d57-ec992ee9cd35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 27 19:23:54.788461 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:54.788425 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:23:55.098175 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:55.098090 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-0bd532-openshift-default-59595d6b48-s4vm2"] Jul 27 19:23:55.104788 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:55.104763 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-0bd532-openshift-default-59595d6b48-s4vm2" Jul 27 19:23:55.107873 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:55.107854 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-0bd532-openshift-default-dockercfg-rd9dt\"" Jul 27 19:23:55.116545 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:55.116513 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-0bd532-openshift-default-59595d6b48-s4vm2"] Jul 27 19:23:55.124965 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:55.124864 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:23:55.268836 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:55.268799 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a467cc20-a0ce-4fcf-b369-4c1ee4b3ed85-istio-token\") pod \"e2e-ipp-b-0bd532-openshift-default-59595d6b48-s4vm2\" (UID: \"a467cc20-a0ce-4fcf-b369-4c1ee4b3ed85\") " pod="openshift-ingress/e2e-ipp-b-0bd532-openshift-default-59595d6b48-s4vm2" Jul 27 19:23:55.269014 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:55.268891 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a467cc20-a0ce-4fcf-b369-4c1ee4b3ed85-istio-envoy\") pod \"e2e-ipp-b-0bd532-openshift-default-59595d6b48-s4vm2\" (UID: \"a467cc20-a0ce-4fcf-b369-4c1ee4b3ed85\") " pod="openshift-ingress/e2e-ipp-b-0bd532-openshift-default-59595d6b48-s4vm2" Jul 27 19:23:55.269014 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:55.268955 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a467cc20-a0ce-4fcf-b369-4c1ee4b3ed85-workload-socket\") pod \"e2e-ipp-b-0bd532-openshift-default-59595d6b48-s4vm2\" (UID: \"a467cc20-a0ce-4fcf-b369-4c1ee4b3ed85\") " pod="openshift-ingress/e2e-ipp-b-0bd532-openshift-default-59595d6b48-s4vm2" Jul 27 19:23:55.269014 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:55.269004 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a467cc20-a0ce-4fcf-b369-4c1ee4b3ed85-istio-podinfo\") pod \"e2e-ipp-b-0bd532-openshift-default-59595d6b48-s4vm2\" (UID: \"a467cc20-a0ce-4fcf-b369-4c1ee4b3ed85\") " pod="openshift-ingress/e2e-ipp-b-0bd532-openshift-default-59595d6b48-s4vm2" Jul 27 19:23:55.269129 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:55.269027 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a467cc20-a0ce-4fcf-b369-4c1ee4b3ed85-credential-socket\") pod \"e2e-ipp-b-0bd532-openshift-default-59595d6b48-s4vm2\" (UID: \"a467cc20-a0ce-4fcf-b369-4c1ee4b3ed85\") " pod="openshift-ingress/e2e-ipp-b-0bd532-openshift-default-59595d6b48-s4vm2" Jul 27 19:23:55.269129 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:55.269049 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zftz6\" (UniqueName: \"kubernetes.io/projected/a467cc20-a0ce-4fcf-b369-4c1ee4b3ed85-kube-api-access-zftz6\") pod \"e2e-ipp-b-0bd532-openshift-default-59595d6b48-s4vm2\" (UID: \"a467cc20-a0ce-4fcf-b369-4c1ee4b3ed85\") " pod="openshift-ingress/e2e-ipp-b-0bd532-openshift-default-59595d6b48-s4vm2" Jul 27 19:23:55.269195 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:55.269123 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a467cc20-a0ce-4fcf-b369-4c1ee4b3ed85-istio-data\") pod \"e2e-ipp-b-0bd532-openshift-default-59595d6b48-s4vm2\" (UID: \"a467cc20-a0ce-4fcf-b369-4c1ee4b3ed85\") " pod="openshift-ingress/e2e-ipp-b-0bd532-openshift-default-59595d6b48-s4vm2" Jul 27 19:23:55.269195 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:55.269168 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a467cc20-a0ce-4fcf-b369-4c1ee4b3ed85-workload-certs\") pod \"e2e-ipp-b-0bd532-openshift-default-59595d6b48-s4vm2\" (UID: \"a467cc20-a0ce-4fcf-b369-4c1ee4b3ed85\") " pod="openshift-ingress/e2e-ipp-b-0bd532-openshift-default-59595d6b48-s4vm2" Jul 27 19:23:55.269316 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:55.269268 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a467cc20-a0ce-4fcf-b369-4c1ee4b3ed85-istiod-ca-cert\") pod \"e2e-ipp-b-0bd532-openshift-default-59595d6b48-s4vm2\" (UID: \"a467cc20-a0ce-4fcf-b369-4c1ee4b3ed85\") " pod="openshift-ingress/e2e-ipp-b-0bd532-openshift-default-59595d6b48-s4vm2" Jul 27 19:23:55.292499 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:55.292457 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:23:55.370766 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:55.370680 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a467cc20-a0ce-4fcf-b369-4c1ee4b3ed85-istiod-ca-cert\") pod \"e2e-ipp-b-0bd532-openshift-default-59595d6b48-s4vm2\" (UID: \"a467cc20-a0ce-4fcf-b369-4c1ee4b3ed85\") " pod="openshift-ingress/e2e-ipp-b-0bd532-openshift-default-59595d6b48-s4vm2" Jul 27 19:23:55.370766 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:55.370733 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a467cc20-a0ce-4fcf-b369-4c1ee4b3ed85-istio-token\") pod \"e2e-ipp-b-0bd532-openshift-default-59595d6b48-s4vm2\" (UID: \"a467cc20-a0ce-4fcf-b369-4c1ee4b3ed85\") " pod="openshift-ingress/e2e-ipp-b-0bd532-openshift-default-59595d6b48-s4vm2" Jul 27 19:23:55.371309 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:55.370772 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a467cc20-a0ce-4fcf-b369-4c1ee4b3ed85-istio-envoy\") pod \"e2e-ipp-b-0bd532-openshift-default-59595d6b48-s4vm2\" (UID: \"a467cc20-a0ce-4fcf-b369-4c1ee4b3ed85\") " pod="openshift-ingress/e2e-ipp-b-0bd532-openshift-default-59595d6b48-s4vm2" Jul 27 19:23:55.371309 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:55.370797 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a467cc20-a0ce-4fcf-b369-4c1ee4b3ed85-workload-socket\") pod \"e2e-ipp-b-0bd532-openshift-default-59595d6b48-s4vm2\" (UID: \"a467cc20-a0ce-4fcf-b369-4c1ee4b3ed85\") " pod="openshift-ingress/e2e-ipp-b-0bd532-openshift-default-59595d6b48-s4vm2" Jul 27 19:23:55.371309 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:55.370830 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a467cc20-a0ce-4fcf-b369-4c1ee4b3ed85-istio-podinfo\") pod \"e2e-ipp-b-0bd532-openshift-default-59595d6b48-s4vm2\" (UID: \"a467cc20-a0ce-4fcf-b369-4c1ee4b3ed85\") " pod="openshift-ingress/e2e-ipp-b-0bd532-openshift-default-59595d6b48-s4vm2" Jul 27 19:23:55.371309 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:55.370862 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a467cc20-a0ce-4fcf-b369-4c1ee4b3ed85-credential-socket\") pod \"e2e-ipp-b-0bd532-openshift-default-59595d6b48-s4vm2\" (UID: \"a467cc20-a0ce-4fcf-b369-4c1ee4b3ed85\") " pod="openshift-ingress/e2e-ipp-b-0bd532-openshift-default-59595d6b48-s4vm2" Jul 27 19:23:55.371309 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:55.371072 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zftz6\" (UniqueName: \"kubernetes.io/projected/a467cc20-a0ce-4fcf-b369-4c1ee4b3ed85-kube-api-access-zftz6\") pod \"e2e-ipp-b-0bd532-openshift-default-59595d6b48-s4vm2\" (UID: \"a467cc20-a0ce-4fcf-b369-4c1ee4b3ed85\") " pod="openshift-ingress/e2e-ipp-b-0bd532-openshift-default-59595d6b48-s4vm2" Jul 27 19:23:55.371309 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:55.371126 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a467cc20-a0ce-4fcf-b369-4c1ee4b3ed85-istio-data\") pod \"e2e-ipp-b-0bd532-openshift-default-59595d6b48-s4vm2\" (UID: \"a467cc20-a0ce-4fcf-b369-4c1ee4b3ed85\") " pod="openshift-ingress/e2e-ipp-b-0bd532-openshift-default-59595d6b48-s4vm2" Jul 27 19:23:55.371309 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:55.371175 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a467cc20-a0ce-4fcf-b369-4c1ee4b3ed85-workload-certs\") pod \"e2e-ipp-b-0bd532-openshift-default-59595d6b48-s4vm2\" (UID: \"a467cc20-a0ce-4fcf-b369-4c1ee4b3ed85\") " pod="openshift-ingress/e2e-ipp-b-0bd532-openshift-default-59595d6b48-s4vm2" Jul 27 19:23:55.371309 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:55.371285 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a467cc20-a0ce-4fcf-b369-4c1ee4b3ed85-credential-socket\") pod \"e2e-ipp-b-0bd532-openshift-default-59595d6b48-s4vm2\" (UID: \"a467cc20-a0ce-4fcf-b369-4c1ee4b3ed85\") " pod="openshift-ingress/e2e-ipp-b-0bd532-openshift-default-59595d6b48-s4vm2" Jul 27 19:23:55.371976 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:55.371517 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a467cc20-a0ce-4fcf-b369-4c1ee4b3ed85-istio-data\") pod \"e2e-ipp-b-0bd532-openshift-default-59595d6b48-s4vm2\" (UID: \"a467cc20-a0ce-4fcf-b369-4c1ee4b3ed85\") " pod="openshift-ingress/e2e-ipp-b-0bd532-openshift-default-59595d6b48-s4vm2" Jul 27 19:23:55.371976 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:55.371556 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a467cc20-a0ce-4fcf-b369-4c1ee4b3ed85-workload-socket\") pod \"e2e-ipp-b-0bd532-openshift-default-59595d6b48-s4vm2\" (UID: \"a467cc20-a0ce-4fcf-b369-4c1ee4b3ed85\") " pod="openshift-ingress/e2e-ipp-b-0bd532-openshift-default-59595d6b48-s4vm2" Jul 27 19:23:55.371976 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:55.371577 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a467cc20-a0ce-4fcf-b369-4c1ee4b3ed85-workload-certs\") pod \"e2e-ipp-b-0bd532-openshift-default-59595d6b48-s4vm2\" (UID: \"a467cc20-a0ce-4fcf-b369-4c1ee4b3ed85\") " pod="openshift-ingress/e2e-ipp-b-0bd532-openshift-default-59595d6b48-s4vm2" Jul 27 19:23:55.371976 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:55.371619 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a467cc20-a0ce-4fcf-b369-4c1ee4b3ed85-istiod-ca-cert\") pod \"e2e-ipp-b-0bd532-openshift-default-59595d6b48-s4vm2\" (UID: \"a467cc20-a0ce-4fcf-b369-4c1ee4b3ed85\") " pod="openshift-ingress/e2e-ipp-b-0bd532-openshift-default-59595d6b48-s4vm2" Jul 27 19:23:55.373599 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:55.373575 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a467cc20-a0ce-4fcf-b369-4c1ee4b3ed85-istio-envoy\") pod \"e2e-ipp-b-0bd532-openshift-default-59595d6b48-s4vm2\" (UID: \"a467cc20-a0ce-4fcf-b369-4c1ee4b3ed85\") " pod="openshift-ingress/e2e-ipp-b-0bd532-openshift-default-59595d6b48-s4vm2" Jul 27 19:23:55.373845 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:55.373819 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a467cc20-a0ce-4fcf-b369-4c1ee4b3ed85-istio-podinfo\") pod \"e2e-ipp-b-0bd532-openshift-default-59595d6b48-s4vm2\" (UID: \"a467cc20-a0ce-4fcf-b369-4c1ee4b3ed85\") " pod="openshift-ingress/e2e-ipp-b-0bd532-openshift-default-59595d6b48-s4vm2" Jul 27 19:23:55.379828 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:55.379786 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zftz6\" (UniqueName: \"kubernetes.io/projected/a467cc20-a0ce-4fcf-b369-4c1ee4b3ed85-kube-api-access-zftz6\") pod \"e2e-ipp-b-0bd532-openshift-default-59595d6b48-s4vm2\" (UID: \"a467cc20-a0ce-4fcf-b369-4c1ee4b3ed85\") " pod="openshift-ingress/e2e-ipp-b-0bd532-openshift-default-59595d6b48-s4vm2" Jul 27 19:23:55.380025 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:55.380009 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a467cc20-a0ce-4fcf-b369-4c1ee4b3ed85-istio-token\") pod \"e2e-ipp-b-0bd532-openshift-default-59595d6b48-s4vm2\" (UID: \"a467cc20-a0ce-4fcf-b369-4c1ee4b3ed85\") " pod="openshift-ingress/e2e-ipp-b-0bd532-openshift-default-59595d6b48-s4vm2" Jul 27 19:23:55.417442 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:55.417417 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-0bd532-openshift-default-59595d6b48-s4vm2" Jul 27 19:23:55.591485 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:55.591462 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-0bd532-openshift-default-59595d6b48-s4vm2"] Jul 27 19:23:55.594089 ip-10-0-128-32 kubenswrapper[2579]: W0727 19:23:55.594053 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda467cc20_a0ce_4fcf_b369_4c1ee4b3ed85.slice/crio-0d21be373b6f12b7794b528a2f1b83a01f0f570d6dd1bd68489d9bc46082c1b2 WatchSource:0}: Error finding container 0d21be373b6f12b7794b528a2f1b83a01f0f570d6dd1bd68489d9bc46082c1b2: Status 404 returned error can't find the container with id 0d21be373b6f12b7794b528a2f1b83a01f0f570d6dd1bd68489d9bc46082c1b2 Jul 27 19:23:55.596506 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:55.596476 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 27 19:23:55.596591 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:55.596544 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 27 19:23:55.596591 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:55.596576 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 27 19:23:56.392055 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:56.392025 2579 generic.go:356] "Generic (PLEG): container finished" podID="e07be686-8abf-43d8-8d57-ec992ee9cd35" containerID="ffeb57794d62f458acf4d533ccc0472e01418617982ae3f05b6d22d9a4f4207f" exitCode=0 Jul 27 19:23:56.392572 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:56.392104 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-1d1107-openshift-default-5bfc5f8cb6-mcf9c" event={"ID":"e07be686-8abf-43d8-8d57-ec992ee9cd35","Type":"ContainerDied","Data":"ffeb57794d62f458acf4d533ccc0472e01418617982ae3f05b6d22d9a4f4207f"} Jul 27 19:23:56.392652 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:56.392632 2579 scope.go:117] "RemoveContainer" containerID="ffeb57794d62f458acf4d533ccc0472e01418617982ae3f05b6d22d9a4f4207f" Jul 27 19:23:56.393812 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:56.393789 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-0bd532-openshift-default-59595d6b48-s4vm2" event={"ID":"a467cc20-a0ce-4fcf-b369-4c1ee4b3ed85","Type":"ContainerStarted","Data":"9817d3e6b8de940ea71b6fe5234415ac4d062c5a5165d399e23f8716e7d18873"} Jul 27 19:23:56.393901 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:56.393837 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-0bd532-openshift-default-59595d6b48-s4vm2" event={"ID":"a467cc20-a0ce-4fcf-b369-4c1ee4b3ed85","Type":"ContainerStarted","Data":"0d21be373b6f12b7794b528a2f1b83a01f0f570d6dd1bd68489d9bc46082c1b2"} Jul 27 19:23:56.393901 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:56.393790 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 27 19:23:56.393974 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:56.393935 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 27 19:23:56.394009 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:56.393989 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 27 19:23:56.417922 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:56.417896 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-0bd532-openshift-default-59595d6b48-s4vm2" Jul 27 19:23:56.423584 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:56.423558 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-0bd532-openshift-default-59595d6b48-s4vm2" Jul 27 19:23:56.466820 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:56.466764 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-0bd532-openshift-default-59595d6b48-s4vm2" podStartSLOduration=1.466743935 podStartE2EDuration="1.466743935s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:23:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:23:56.463754319 +0000 UTC m=+3311.525443350" watchObservedRunningTime="2026-07-27 19:23:56.466743935 +0000 UTC m=+3311.528432968" Jul 27 19:23:57.037920 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:57.037877 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:23:57.400737 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:57.400655 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-1d1107-openshift-default-5bfc5f8cb6-mcf9c" event={"ID":"e07be686-8abf-43d8-8d57-ec992ee9cd35","Type":"ContainerStarted","Data":"2bf86b10219cf48884d9786c1775b4146ecd26542a1f178a6e0092a78c06e1a2"} Jul 27 19:23:57.401164 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:57.401060 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-0bd532-openshift-default-59595d6b48-s4vm2" Jul 27 19:23:57.401834 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:57.401810 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-0bd532-openshift-default-59595d6b48-s4vm2" Jul 27 19:23:57.643842 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:57.643751 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-1d1107-openshift-default-5bfc5f8cb6-mcf9c" Jul 27 19:23:57.644070 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:57.644038 2579 patch_prober.go:28] interesting pod/e2e-ipp-a-1d1107-openshift-default-5bfc5f8cb6-mcf9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Jul 27 19:23:57.644157 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:57.644131 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-1d1107-openshift-default-5bfc5f8cb6-mcf9c" podUID="e07be686-8abf-43d8-8d57-ec992ee9cd35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Jul 27 19:23:58.545252 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:58.545188 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:23:58.644648 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:58.644604 2579 patch_prober.go:28] interesting pod/e2e-ipp-a-1d1107-openshift-default-5bfc5f8cb6-mcf9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Jul 27 19:23:58.644846 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:58.644690 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-1d1107-openshift-default-5bfc5f8cb6-mcf9c" podUID="e07be686-8abf-43d8-8d57-ec992ee9cd35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Jul 27 19:23:58.656977 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:58.656931 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:23:59.644424 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:59.644379 2579 patch_prober.go:28] interesting pod/e2e-ipp-a-1d1107-openshift-default-5bfc5f8cb6-mcf9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Jul 27 19:23:59.644954 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:23:59.644471 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-1d1107-openshift-default-5bfc5f8cb6-mcf9c" podUID="e07be686-8abf-43d8-8d57-ec992ee9cd35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Jul 27 19:24:00.235113 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:24:00.235078 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:24:00.360809 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:24:00.360770 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:24:00.644006 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:24:00.643967 2579 patch_prober.go:28] interesting pod/e2e-ipp-a-1d1107-openshift-default-5bfc5f8cb6-mcf9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Jul 27 19:24:00.644305 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:24:00.644272 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-1d1107-openshift-default-5bfc5f8cb6-mcf9c" podUID="e07be686-8abf-43d8-8d57-ec992ee9cd35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Jul 27 19:24:00.976281 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:24:00.976152 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:24:01.077362 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:24:01.077320 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:24:01.648443 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:24:01.648411 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-1d1107-openshift-default-5bfc5f8cb6-mcf9c" Jul 27 19:24:01.648856 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:24:01.648835 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-1d1107-openshift-default-5bfc5f8cb6-mcf9c" Jul 27 19:24:01.649582 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:24:01.649563 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-1d1107-openshift-default-5bfc5f8cb6-mcf9c" Jul 27 19:24:02.991345 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:24:02.991304 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:24:16.508308 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:24:16.508271 2579 generic.go:356] "Generic (PLEG): container finished" podID="a467cc20-a0ce-4fcf-b369-4c1ee4b3ed85" containerID="9817d3e6b8de940ea71b6fe5234415ac4d062c5a5165d399e23f8716e7d18873" exitCode=0 Jul 27 19:24:16.508726 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:24:16.508383 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-0bd532-openshift-default-59595d6b48-s4vm2" event={"ID":"a467cc20-a0ce-4fcf-b369-4c1ee4b3ed85","Type":"ContainerDied","Data":"9817d3e6b8de940ea71b6fe5234415ac4d062c5a5165d399e23f8716e7d18873"} Jul 27 19:24:16.508788 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:24:16.508771 2579 scope.go:117] "RemoveContainer" containerID="9817d3e6b8de940ea71b6fe5234415ac4d062c5a5165d399e23f8716e7d18873" Jul 27 19:24:16.509881 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:24:16.509850 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 27 19:24:16.509988 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:24:16.509945 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 27 19:24:16.510052 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:24:16.509992 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 27 19:24:17.516176 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:24:17.516134 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-0bd532-openshift-default-59595d6b48-s4vm2" event={"ID":"a467cc20-a0ce-4fcf-b369-4c1ee4b3ed85","Type":"ContainerStarted","Data":"f1a59c256b482a9669a91884d935d216f22e3a4d30767bf8672acd711d3796b3"} Jul 27 19:24:18.417596 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:24:18.417558 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-0bd532-openshift-default-59595d6b48-s4vm2" Jul 27 19:24:18.417777 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:24:18.417716 2579 patch_prober.go:28] interesting pod/e2e-ipp-b-0bd532-openshift-default-59595d6b48-s4vm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Jul 27 19:24:18.417777 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:24:18.417767 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-0bd532-openshift-default-59595d6b48-s4vm2" podUID="a467cc20-a0ce-4fcf-b369-4c1ee4b3ed85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Jul 27 19:24:19.418672 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:24:19.418617 2579 patch_prober.go:28] interesting pod/e2e-ipp-b-0bd532-openshift-default-59595d6b48-s4vm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Jul 27 19:24:19.419069 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:24:19.418691 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-0bd532-openshift-default-59595d6b48-s4vm2" podUID="a467cc20-a0ce-4fcf-b369-4c1ee4b3ed85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Jul 27 19:24:20.295922 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:24:20.295849 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-8rfpl_77a26a98-9824-423f-be2e-c316fc57d11b/payload-processing/0.log" Jul 27 19:24:20.418298 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:24:20.418261 2579 patch_prober.go:28] interesting pod/e2e-ipp-b-0bd532-openshift-default-59595d6b48-s4vm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Jul 27 19:24:20.418469 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:24:20.418321 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-0bd532-openshift-default-59595d6b48-s4vm2" podUID="a467cc20-a0ce-4fcf-b369-4c1ee4b3ed85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Jul 27 19:24:21.418019 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:24:21.417978 2579 patch_prober.go:28] interesting pod/e2e-ipp-b-0bd532-openshift-default-59595d6b48-s4vm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Jul 27 19:24:21.418632 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:24:21.418048 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-0bd532-openshift-default-59595d6b48-s4vm2" podUID="a467cc20-a0ce-4fcf-b369-4c1ee4b3ed85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Jul 27 19:24:22.421546 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:24:22.421515 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-0bd532-openshift-default-59595d6b48-s4vm2" Jul 27 19:24:22.422041 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:24:22.422027 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-0bd532-openshift-default-59595d6b48-s4vm2" Jul 27 19:24:22.422865 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:24:22.422841 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-0bd532-openshift-default-59595d6b48-s4vm2" Jul 27 19:24:28.545524 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:24:28.545487 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:24:29.476324 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:24:29.476279 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:24:29.576487 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:24:29.576448 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:24:42.486796 ip-10-0-128-32 kubenswrapper[2579]: E0727 19:24:42.486752 2579 controller.go:195] "Failed to update lease" err="Put \"https://a21232e2c335645078a0372e432f61e3-d6350540945ea601.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-128-32.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Jul 27 19:24:44.448105 ip-10-0-128-32 kubenswrapper[2579]: E0727 19:24:44.448063 2579 kubelet_node_status.go:536] "Error updating node status, will retry" err="failed to patch status \"{\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"type\\\":\\\"DiskPressure\\\"},{\\\"type\\\":\\\"PIDPressure\\\"},{\\\"type\\\":\\\"Ready\\\"}],\\\"conditions\\\":[{\\\"lastHeartbeatTime\\\":\\\"2026-07-27T19:24:34Z\\\",\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-27T19:24:34Z\\\",\\\"type\\\":\\\"DiskPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-27T19:24:34Z\\\",\\\"type\\\":\\\"PIDPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-27T19:24:34Z\\\",\\\"type\\\":\\\"Ready\\\"}],\\\"runtimeHandlers\\\":[{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"crun\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"runc\\\"}]}}\" for node \"ip-10-0-128-32.ec2.internal\": Patch \"https://a21232e2c335645078a0372e432f61e3-d6350540945ea601.elb.us-east-1.amazonaws.com:6443/api/v1/nodes/ip-10-0-128-32.ec2.internal/status?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Jul 27 19:24:52.487116 ip-10-0-128-32 kubenswrapper[2579]: E0727 19:24:52.487070 2579 controller.go:195] "Failed to update lease" err="Put \"https://a21232e2c335645078a0372e432f61e3-d6350540945ea601.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-128-32.ec2.internal?timeout=10s\": context deadline exceeded" Jul 27 19:24:54.448455 ip-10-0-128-32 kubenswrapper[2579]: E0727 19:24:54.448423 2579 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-128-32.ec2.internal\": Get \"https://a21232e2c335645078a0372e432f61e3-d6350540945ea601.elb.us-east-1.amazonaws.com:6443/api/v1/nodes/ip-10-0-128-32.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Jul 27 19:25:02.487568 ip-10-0-128-32 kubenswrapper[2579]: E0727 19:25:02.487523 2579 controller.go:195] "Failed to update lease" err="Put \"https://a21232e2c335645078a0372e432f61e3-d6350540945ea601.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-128-32.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Jul 27 19:25:04.449167 ip-10-0-128-32 kubenswrapper[2579]: E0727 19:25:04.449136 2579 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-128-32.ec2.internal\": Get \"https://a21232e2c335645078a0372e432f61e3-d6350540945ea601.elb.us-east-1.amazonaws.com:6443/api/v1/nodes/ip-10-0-128-32.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Jul 27 19:25:04.579042 ip-10-0-128-32 kubenswrapper[2579]: E0727 19:25:04.579003 2579 webhook.go:269] Failed to make webhook authorizer request: Post "https://a21232e2c335645078a0372e432f61e3-d6350540945ea601.elb.us-east-1.amazonaws.com:6443/apis/authorization.k8s.io/v1/subjectaccessreviews": context canceled Jul 27 19:25:04.579042 ip-10-0-128-32 kubenswrapper[2579]: E0727 19:25:04.579041 2579 server.go:347] "Authorization error" err="Post \"https://a21232e2c335645078a0372e432f61e3-d6350540945ea601.elb.us-east-1.amazonaws.com:6443/apis/authorization.k8s.io/v1/subjectaccessreviews\": context canceled" user="system:serviceaccount:openshift-monitoring:prometheus-k8s" verb="get" resource="nodes" subresource="metrics" Jul 27 19:25:12.349913 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:12.349869 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 27 19:25:12.349913 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:12.349894 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 27 19:25:12.350509 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:12.349935 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 27 19:25:12.350509 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:12.349973 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 27 19:25:12.350509 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:12.349983 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 27 19:25:12.350509 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:12.350003 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 27 19:25:12.350509 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:12.350026 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"router-dockercfg-wvq8l\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 27 19:25:12.350509 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:12.350055 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 27 19:25:12.350509 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:12.350063 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 27 19:25:12.350509 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:12.350063 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 27 19:25:12.350509 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:12.350087 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 27 19:25:12.350509 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:12.350094 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 27 19:25:12.350509 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:12.350095 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 27 19:25:12.350509 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:12.350084 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-0bd532-openshift-default-dockercfg-rd9dt\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 27 19:25:12.350509 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:12.350133 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 27 19:25:12.350509 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:12.350150 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 27 19:25:12.350509 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:12.350165 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"llm\"/\"default-dockercfg-6w4tt\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 27 19:25:12.350509 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:12.350216 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 27 19:25:12.350509 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:12.350259 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 27 19:25:12.350509 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:12.350260 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 27 19:25:12.351362 ip-10-0-128-32 kubenswrapper[2579]: E0727 19:25:12.350254 2579 token_manager.go:124] "Couldn't update token" err="Post \"https://a21232e2c335645078a0372e432f61e3-d6350540945ea601.elb.us-east-1.amazonaws.com:6443/api/v1/namespaces/opendatahub/serviceaccounts/llmisvc-controller-manager/token\": http2: client connection lost" cacheKey="\"llmisvc-controller-manager\"/\"opendatahub\"/[]string(nil)/3607/v1.BoundObjectReference{Kind:\"Pod\", APIVersion:\"v1\", Name:\"llmisvc-controller-manager-777c6bd7d5-w9d4l\", UID:\"52b6da00-246b-4954-adcc-5b482d1c41bc\"}/\"\"" Jul 27 19:25:12.351362 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:12.350286 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 27 19:25:12.351362 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:12.350299 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 27 19:25:12.351362 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:12.350308 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 27 19:25:12.351362 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:12.350322 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 27 19:25:12.351362 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:12.350325 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-584h7\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 27 19:25:12.351362 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:12.350334 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-vsnt9\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 27 19:25:12.351362 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:12.350363 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 27 19:25:12.351362 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:12.350380 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-glqgk\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 27 19:25:12.351362 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:12.350384 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"dns-dockercfg-9pm47\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 27 19:25:12.351362 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:12.350388 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 27 19:25:12.351362 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:12.350389 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-swtr8\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 27 19:25:12.351362 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:12.350408 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 27 19:25:12.351362 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:12.350412 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 27 19:25:12.351362 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:12.350436 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-2v6bt\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 27 19:25:12.351362 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:12.350459 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 27 19:25:12.351362 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:12.350459 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"kube-system\"/\"default-dockercfg-dxz2n\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 27 19:25:12.352037 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:12.350469 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-5w26j\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 27 19:25:12.352037 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:12.350486 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 27 19:25:12.352037 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:12.350490 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"dns-default\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 27 19:25:12.352037 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:12.350502 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 27 19:25:12.352037 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:12.350510 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 27 19:25:12.352037 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:12.350518 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 27 19:25:12.352037 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:12.350533 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 27 19:25:12.352037 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:12.350556 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 27 19:25:12.352037 ip-10-0-128-32 kubenswrapper[2579]: E0727 19:25:12.350555 2579 token_manager.go:124] "Couldn't update token" err="Post \"https://a21232e2c335645078a0372e432f61e3-d6350540945ea601.elb.us-east-1.amazonaws.com:6443/api/v1/namespaces/openshift-ingress/serviceaccounts/data-science-gateway-data-science-gateway-class/token\": http2: client connection lost" cacheKey="\"data-science-gateway-data-science-gateway-class\"/\"openshift-ingress\"/[]string(nil)/3607/v1.BoundObjectReference{Kind:\"Pod\", APIVersion:\"v1\", Name:\"data-science-gateway-data-science-gateway-class-d66cd8c7c-w6ltt\", UID:\"2f32486f-5cb4-48a9-96e0-6ee08e76933a\"}/\"\"" Jul 27 19:25:12.352037 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:12.350571 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"kuadrant-system\"/\"default-dockercfg-jv8qx\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 27 19:25:12.352037 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:12.350588 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-mcct4\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 27 19:25:12.352037 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:12.350611 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 27 19:25:12.352037 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:12.350623 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 27 19:25:12.352037 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:12.350624 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-q4btw\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 27 19:25:12.352037 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:12.350620 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 27 19:25:12.352037 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:12.350662 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 27 19:25:12.352037 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:12.350674 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 27 19:25:12.352649 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:12.350690 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-qsr4x\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 27 19:25:12.352649 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:12.350697 2579 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 27 19:25:12.352649 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:12.350705 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 27 19:25:12.352649 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:12.350712 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 27 19:25:12.352649 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:12.350744 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 27 19:25:12.352649 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:12.350767 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"kube-system\"/\"original-pull-secret\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 27 19:25:12.352649 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:12.350779 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 27 19:25:12.352649 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:12.350781 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-network-console\"/\"default-dockercfg-v85bm\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 27 19:25:12.352649 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:12.350788 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 27 19:25:12.352649 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:12.350799 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-stdww\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 27 19:25:12.352649 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:12.350818 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 27 19:25:12.352649 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:12.350819 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 27 19:25:12.352649 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:12.350825 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 27 19:25:12.352649 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:12.350835 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 27 19:25:12.352649 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:12.350847 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"llm\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 27 19:25:12.352649 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:12.350849 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 27 19:25:12.352649 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:12.350860 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 27 19:25:12.352649 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:12.350870 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 27 19:25:12.353184 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:12.350875 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 27 19:25:12.353184 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:12.350891 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 27 19:25:12.353184 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:12.350898 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 27 19:25:12.353184 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:12.350915 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-zlwxd\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 27 19:25:12.353184 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:12.350936 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 27 19:25:12.353184 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:12.350943 2579 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 27 19:25:12.353184 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:12.350948 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-xfn8t\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 27 19:25:12.353184 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:12.350956 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 27 19:25:12.353184 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:12.350970 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 27 19:25:12.353184 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:12.350973 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-nrk2j\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 27 19:25:12.353184 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:12.350979 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-rg85g\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 27 19:25:12.353184 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:12.350992 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-cc5kb\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 27 19:25:12.353184 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:12.350995 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 27 19:25:12.353184 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:12.351000 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 27 19:25:12.353184 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:12.351015 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 27 19:25:12.353184 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:12.351017 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 27 19:25:12.353184 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:12.351024 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 27 19:25:12.353184 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:12.351033 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 27 19:25:12.353813 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:12.351034 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-8kmxg\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 27 19:25:12.353813 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:12.351036 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 27 19:25:12.353813 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:12.351053 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 27 19:25:12.353813 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:12.351064 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 27 19:25:12.353813 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:12.351072 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 27 19:25:12.353813 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:12.351078 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 27 19:25:12.353813 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:12.351084 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 27 19:25:12.353813 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:12.351078 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-cqfq8\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 27 19:25:12.353813 ip-10-0-128-32 kubenswrapper[2579]: E0727 19:25:12.351092 2579 controller.go:195] "Failed to update lease" err="Put \"https://a21232e2c335645078a0372e432f61e3-d6350540945ea601.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-128-32.ec2.internal?timeout=10s\": http2: client connection lost" Jul 27 19:25:12.353813 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:12.351106 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-8dt4n\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 27 19:25:12.353813 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:12.351113 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 27 19:25:12.353813 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:12.351122 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 27 19:25:12.353813 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:12.351143 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 27 19:25:12.353813 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:12.351158 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 27 19:25:12.353813 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:12.351160 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 27 19:25:12.353813 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:12.351171 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 27 19:25:12.353813 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:12.351175 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"default-dockercfg-vmnbn\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 27 19:25:12.353813 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:12.351187 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 27 19:25:12.354593 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:12.351196 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-9j445\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 27 19:25:12.354593 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:12.351204 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 27 19:25:12.354593 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:12.351211 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 27 19:25:12.354593 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:12.351206 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 27 19:25:12.354593 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:12.351211 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 27 19:25:12.354593 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:12.351221 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-fn8tb\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 27 19:25:12.354593 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:12.351261 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"kube-system\"/\"konnectivity-agent\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 27 19:25:12.354593 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:12.351288 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 27 19:25:12.354593 ip-10-0-128-32 kubenswrapper[2579]: E0727 19:25:12.351283 2579 token_manager.go:124] "Couldn't update token" err="Post \"https://a21232e2c335645078a0372e432f61e3-d6350540945ea601.elb.us-east-1.amazonaws.com:6443/api/v1/namespaces/opendatahub/serviceaccounts/odh-model-controller/token\": http2: client connection lost" cacheKey="\"odh-model-controller\"/\"opendatahub\"/[]string(nil)/3607/v1.BoundObjectReference{Kind:\"Pod\", APIVersion:\"v1\", Name:\"odh-model-controller-7998c67dc4-sgg44\", UID:\"87423332-edfc-4e36-a11b-56324b7db773\"}/\"\"" Jul 27 19:25:12.354593 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:12.351318 2579 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 27 19:25:12.354593 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:12.351344 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 27 19:25:12.354593 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:12.351355 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-nc52h\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 27 19:25:12.354593 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:12.351368 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 27 19:25:12.354593 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:12.351387 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-7f7473-openshift-default-dockercfg-6g4cc\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 27 19:25:12.354593 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:12.351403 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 27 19:25:12.354593 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:12.351413 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-chdqz\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 27 19:25:12.354593 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:12.351417 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 27 19:25:12.355106 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:12.351428 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 27 19:25:12.355106 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:12.351451 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-1d1107-openshift-default-dockercfg-nct6f\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 27 19:25:12.355106 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:12.351460 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-msf2r\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 27 19:25:12.355106 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:12.351471 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 27 19:25:12.355106 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:12.351481 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-grgzp\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 27 19:25:12.355106 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:12.351487 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-d4rxx\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 27 19:25:12.355106 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:12.351498 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 27 19:25:12.355106 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:12.351507 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 27 19:25:12.355106 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:12.351518 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 27 19:25:12.355106 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:12.351522 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-n97zp\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 27 19:25:12.355106 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:12.351536 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 27 19:25:12.355106 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:12.351539 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 27 19:25:12.355106 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:12.351549 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-7gqrv\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 27 19:25:12.355106 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:12.351558 2579 reflector.go:571] "Warning: watch ended with error" reflector="pkg/kubelet/config/apiserver.go:66" type="*v1.Pod" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 27 19:25:12.355106 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:12.351568 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 27 19:25:12.355106 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:12.351578 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 27 19:25:12.355106 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:12.351582 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 27 19:25:12.355106 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:12.351601 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 27 19:25:12.355713 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:12.351605 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-6710a9-openshift-default-dockercfg-blp25\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 27 19:25:12.355713 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:12.351603 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 27 19:25:12.355713 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:12.351618 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 27 19:25:12.355713 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:12.351621 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 27 19:25:12.355713 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:12.351626 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-dvvg4\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 27 19:25:12.355713 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:12.351646 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-xvqxn\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 27 19:25:12.355713 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:12.351653 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 27 19:25:12.355713 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:12.351651 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"operator-dockercfg-dxzx5\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 27 19:25:12.355713 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:12.351667 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-d5vgr\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 27 19:25:12.355713 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:12.351673 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-bwbqc\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 27 19:25:12.355713 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:12.351669 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 27 19:25:12.355713 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:12.351690 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 27 19:25:12.355713 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:12.351698 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-6p8s5\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 27 19:25:12.355713 ip-10-0-128-32 kubenswrapper[2579]: E0727 19:25:12.351708 2579 token_manager.go:124] "Couldn't update token" err="Post \"https://a21232e2c335645078a0372e432f61e3-d6350540945ea601.elb.us-east-1.amazonaws.com:6443/api/v1/namespaces/openshift-ingress/serviceaccounts/kube-auth-proxy/token\": http2: client connection lost" cacheKey="\"kube-auth-proxy\"/\"openshift-ingress\"/[]string(nil)/3607/v1.BoundObjectReference{Kind:\"Pod\", APIVersion:\"v1\", Name:\"kube-auth-proxy-7896769b64-tkbg9\", UID:\"2bfb0e76-8207-48bb-810a-6f865001eeb1\"}/\"\"" Jul 27 19:25:12.355713 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:12.351899 2579 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 27 19:25:13.162377 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:13.162346 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 27 19:25:13.166794 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:13.166770 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 27 19:25:13.173805 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:13.173787 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 27 19:25:13.174397 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:13.174379 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-1d1107-openshift-default-dockercfg-nct6f\"" Jul 27 19:25:13.189388 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:13.189368 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 27 19:25:13.194763 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:13.194747 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 27 19:25:13.196575 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:13.196556 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-grgzp\"" Jul 27 19:25:13.204741 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:13.204718 2579 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 27 19:25:13.231085 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:13.231059 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-dvvg4\"" Jul 27 19:25:13.240405 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:13.240388 2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 27 19:25:13.246186 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:13.246171 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Jul 27 19:25:13.250464 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:13.250450 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 27 19:25:13.253777 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:13.253755 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 27 19:25:13.255122 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:13.255106 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-swtr8\"" Jul 27 19:25:13.258092 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:13.258076 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 27 19:25:13.264717 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:13.264700 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 27 19:25:13.272777 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:13.272760 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Jul 27 19:25:13.273835 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:13.273821 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-dxz2n\"" Jul 27 19:25:13.283861 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:13.283834 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 27 19:25:13.286441 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:13.286426 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-wvq8l\"" Jul 27 19:25:13.286683 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:13.286670 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 27 19:25:13.288036 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:13.288018 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-8kmxg\"" Jul 27 19:25:13.289473 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:13.289458 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 27 19:25:13.290105 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:13.290092 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-xfn8t\"" Jul 27 19:25:13.298159 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:13.298141 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 27 19:25:13.303126 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:13.303112 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 27 19:25:13.304061 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:13.304047 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 27 19:25:13.315592 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:13.315573 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 27 19:25:13.315688 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:13.315670 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-d5vgr\"" Jul 27 19:25:13.315922 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:13.315907 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 27 19:25:13.316008 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:13.315993 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-bwbqc\"" Jul 27 19:25:13.322354 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:13.322335 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 27 19:25:13.323434 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:13.323415 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 27 19:25:13.323580 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:13.323557 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-glqgk\"" Jul 27 19:25:13.327973 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:13.327949 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 27 19:25:13.330611 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:13.330596 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 27 19:25:13.333139 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:13.333114 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-v85bm\"" Jul 27 19:25:13.336329 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:13.336315 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 27 19:25:13.336385 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:13.336322 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-7gqrv\"" Jul 27 19:25:13.340175 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:13.340158 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Jul 27 19:25:13.343213 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:13.343198 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 27 19:25:13.357887 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:13.357864 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 27 19:25:13.359516 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:13.359499 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-vmnbn\"" Jul 27 19:25:13.361606 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:13.361591 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-6710a9-openshift-default-dockercfg-blp25\"" Jul 27 19:25:13.365279 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:13.365264 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 27 19:25:13.372953 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:13.372939 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 27 19:25:13.373880 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:13.373866 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-q4btw\"" Jul 27 19:25:13.384121 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:13.384105 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-vsnt9\"" Jul 27 19:25:13.396082 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:13.396065 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 27 19:25:13.398944 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:13.398924 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 27 19:25:13.401051 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:13.401035 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-rg85g\"" Jul 27 19:25:13.405556 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:13.405540 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 27 19:25:13.413357 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:13.413305 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 27 19:25:13.413357 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:13.413320 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 27 19:25:13.413850 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:13.413836 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-0bd532-openshift-default-dockercfg-rd9dt\"" Jul 27 19:25:13.417664 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:13.417646 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 27 19:25:13.425315 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:13.425297 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 27 19:25:13.427455 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:13.427440 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 27 19:25:13.429496 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:13.429480 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 27 19:25:13.435651 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:13.435631 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-9j445\"" Jul 27 19:25:13.440862 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:13.440845 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 27 19:25:13.446710 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:13.446694 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 27 19:25:13.449957 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:13.449940 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 27 19:25:13.450712 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:13.450696 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-2v6bt\"" Jul 27 19:25:13.453753 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:13.453739 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-msf2r\"" Jul 27 19:25:13.454564 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:13.454546 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 27 19:25:13.471780 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:13.471756 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-dxzx5\"" Jul 27 19:25:13.475869 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:13.475852 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 27 19:25:13.494535 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:13.494515 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 27 19:25:13.497397 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:13.497367 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 27 19:25:13.515045 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:13.515024 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-zlwxd\"" Jul 27 19:25:13.517068 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:13.517051 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 27 19:25:13.519768 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:13.519750 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 27 19:25:13.537694 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:13.537668 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 27 19:25:13.537694 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:13.537692 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 27 19:25:13.546507 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:13.546483 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 27 19:25:13.554324 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:13.554307 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-7f7473-openshift-default-dockercfg-6g4cc\"" Jul 27 19:25:13.555006 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:13.554992 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 27 19:25:13.555563 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:13.555548 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 27 19:25:13.557593 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:13.557574 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 27 19:25:13.557677 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:13.557664 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 27 19:25:13.564213 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:13.564197 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 27 19:25:13.582486 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:13.582468 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 27 19:25:13.583532 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:13.583509 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 27 19:25:13.591358 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:13.591342 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-chdqz\"" Jul 27 19:25:13.591629 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:13.591612 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-nrk2j\"" Jul 27 19:25:13.597040 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:13.597024 2579 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 27 19:25:13.597040 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:13.597036 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 27 19:25:13.597714 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:13.597700 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 27 19:25:13.598538 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:13.598525 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 27 19:25:13.599036 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:13.599023 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 27 19:25:13.622190 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:13.622165 2579 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 27 19:25:13.631203 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:13.631181 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Jul 27 19:25:13.633004 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:13.632989 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 27 19:25:13.634177 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:13.634160 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 27 19:25:13.634714 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:13.634700 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-cqfq8\"" Jul 27 19:25:13.636974 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:13.636961 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 27 19:25:13.645831 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:13.645812 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 27 19:25:13.653678 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:13.653660 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-9pm47\"" Jul 27 19:25:13.655663 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:13.655644 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 27 19:25:13.659780 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:13.659763 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 27 19:25:13.667654 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:13.667597 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-8dt4n\"" Jul 27 19:25:13.675101 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:13.675084 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 27 19:25:13.681194 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:13.681176 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 27 19:25:13.691552 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:13.691535 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 27 19:25:13.692497 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:13.692482 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 27 19:25:13.707476 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:13.707458 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 27 19:25:13.715055 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:13.715033 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Jul 27 19:25:13.716486 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:13.716469 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-mcct4\"" Jul 27 19:25:13.720066 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:13.720046 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-stdww\"" Jul 27 19:25:13.727812 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:13.727788 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 27 19:25:13.729982 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:13.729965 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 27 19:25:13.740485 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:13.740464 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 27 19:25:13.741749 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:13.741733 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 27 19:25:13.751459 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:13.751435 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 27 19:25:13.753787 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:13.753771 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-6w4tt\"" Jul 27 19:25:13.754898 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:13.754884 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 27 19:25:13.755093 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:13.755079 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 27 19:25:13.756174 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:13.756161 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 27 19:25:13.768794 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:13.768778 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 27 19:25:13.773562 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:13.773547 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-jv8qx\"" Jul 27 19:25:13.779325 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:13.779310 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 27 19:25:13.784068 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:13.784054 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 27 19:25:13.793252 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:13.793218 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 27 19:25:13.796490 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:13.796475 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 27 19:25:13.797712 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:13.797693 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 27 19:25:13.804454 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:13.804433 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 27 19:25:13.804536 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:13.804481 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-xvqxn\"" Jul 27 19:25:13.805872 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:13.805853 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 27 19:25:13.816920 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:13.816896 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-qsr4x\"" Jul 27 19:25:13.817725 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:13.817708 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 27 19:25:13.828422 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:13.828404 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-5w26j\"" Jul 27 19:25:13.830560 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:13.830547 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 27 19:25:13.844542 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:13.844519 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 27 19:25:13.862442 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:13.862420 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 27 19:25:13.881821 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:13.881798 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 27 19:25:13.902815 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:13.902788 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-6p8s5\"" Jul 27 19:25:13.923404 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:13.923331 2579 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 27 19:25:13.942601 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:13.942574 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 27 19:25:13.963071 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:13.963050 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-nc52h\"" Jul 27 19:25:13.982958 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:13.982940 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 27 19:25:14.003994 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:14.003974 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 27 19:25:14.023142 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:14.023123 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-fn8tb\"" Jul 27 19:25:14.042531 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:14.042514 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 27 19:25:14.062083 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:14.062064 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 27 19:25:14.082721 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:14.082697 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 27 19:25:14.102653 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:14.102630 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-d4rxx\"" Jul 27 19:25:14.122886 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:14.122863 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 27 19:25:14.142918 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:14.142895 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 27 19:25:14.162732 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:14.162708 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-n97zp\"" Jul 27 19:25:14.182638 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:14.182570 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 27 19:25:14.201946 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:14.201920 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-cc5kb\"" Jul 27 19:25:14.223408 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:14.223384 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-584h7\"" Jul 27 19:25:44.429159 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:44.429125 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-8rfpl_77a26a98-9824-423f-be2e-c316fc57d11b/payload-processing/0.log" Jul 27 19:25:44.918789 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:44.918747 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:25:45.265330 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:45.265250 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-fa6952-openshift-default-6f7dfcbd94-dlfq5"] Jul 27 19:25:45.269606 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:45.269586 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-fa6952-openshift-default-6f7dfcbd94-dlfq5" Jul 27 19:25:45.272880 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:45.272859 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-fa6952-openshift-default-dockercfg-6rm7n\"" Jul 27 19:25:45.282562 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:45.282538 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-fa6952-openshift-default-6f7dfcbd94-dlfq5"] Jul 27 19:25:45.312578 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:45.312547 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:25:45.402335 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:45.402259 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/254c91ed-447c-435f-8573-8b3400da6f2c-istio-data\") pod \"e2e-ipp-cleanup-fa6952-openshift-default-6f7dfcbd94-dlfq5\" (UID: \"254c91ed-447c-435f-8573-8b3400da6f2c\") " pod="openshift-ingress/e2e-ipp-cleanup-fa6952-openshift-default-6f7dfcbd94-dlfq5" Jul 27 19:25:45.402555 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:45.402388 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/254c91ed-447c-435f-8573-8b3400da6f2c-istio-envoy\") pod \"e2e-ipp-cleanup-fa6952-openshift-default-6f7dfcbd94-dlfq5\" (UID: \"254c91ed-447c-435f-8573-8b3400da6f2c\") " pod="openshift-ingress/e2e-ipp-cleanup-fa6952-openshift-default-6f7dfcbd94-dlfq5" Jul 27 19:25:45.402555 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:45.402432 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/254c91ed-447c-435f-8573-8b3400da6f2c-istio-token\") pod \"e2e-ipp-cleanup-fa6952-openshift-default-6f7dfcbd94-dlfq5\" (UID: \"254c91ed-447c-435f-8573-8b3400da6f2c\") " pod="openshift-ingress/e2e-ipp-cleanup-fa6952-openshift-default-6f7dfcbd94-dlfq5" Jul 27 19:25:45.402555 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:45.402469 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/254c91ed-447c-435f-8573-8b3400da6f2c-workload-certs\") pod \"e2e-ipp-cleanup-fa6952-openshift-default-6f7dfcbd94-dlfq5\" (UID: \"254c91ed-447c-435f-8573-8b3400da6f2c\") " pod="openshift-ingress/e2e-ipp-cleanup-fa6952-openshift-default-6f7dfcbd94-dlfq5" Jul 27 19:25:45.402555 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:45.402510 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-225m5\" (UniqueName: \"kubernetes.io/projected/254c91ed-447c-435f-8573-8b3400da6f2c-kube-api-access-225m5\") pod \"e2e-ipp-cleanup-fa6952-openshift-default-6f7dfcbd94-dlfq5\" (UID: \"254c91ed-447c-435f-8573-8b3400da6f2c\") " pod="openshift-ingress/e2e-ipp-cleanup-fa6952-openshift-default-6f7dfcbd94-dlfq5" Jul 27 19:25:45.402555 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:45.402548 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/254c91ed-447c-435f-8573-8b3400da6f2c-istio-podinfo\") pod \"e2e-ipp-cleanup-fa6952-openshift-default-6f7dfcbd94-dlfq5\" (UID: \"254c91ed-447c-435f-8573-8b3400da6f2c\") " pod="openshift-ingress/e2e-ipp-cleanup-fa6952-openshift-default-6f7dfcbd94-dlfq5" Jul 27 19:25:45.402774 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:45.402587 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/254c91ed-447c-435f-8573-8b3400da6f2c-istiod-ca-cert\") pod \"e2e-ipp-cleanup-fa6952-openshift-default-6f7dfcbd94-dlfq5\" (UID: \"254c91ed-447c-435f-8573-8b3400da6f2c\") " pod="openshift-ingress/e2e-ipp-cleanup-fa6952-openshift-default-6f7dfcbd94-dlfq5" Jul 27 19:25:45.402774 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:45.402627 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/254c91ed-447c-435f-8573-8b3400da6f2c-workload-socket\") pod \"e2e-ipp-cleanup-fa6952-openshift-default-6f7dfcbd94-dlfq5\" (UID: \"254c91ed-447c-435f-8573-8b3400da6f2c\") " pod="openshift-ingress/e2e-ipp-cleanup-fa6952-openshift-default-6f7dfcbd94-dlfq5" Jul 27 19:25:45.402774 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:45.402645 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/254c91ed-447c-435f-8573-8b3400da6f2c-credential-socket\") pod \"e2e-ipp-cleanup-fa6952-openshift-default-6f7dfcbd94-dlfq5\" (UID: \"254c91ed-447c-435f-8573-8b3400da6f2c\") " pod="openshift-ingress/e2e-ipp-cleanup-fa6952-openshift-default-6f7dfcbd94-dlfq5" Jul 27 19:25:45.503822 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:45.503786 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-225m5\" (UniqueName: \"kubernetes.io/projected/254c91ed-447c-435f-8573-8b3400da6f2c-kube-api-access-225m5\") pod \"e2e-ipp-cleanup-fa6952-openshift-default-6f7dfcbd94-dlfq5\" (UID: \"254c91ed-447c-435f-8573-8b3400da6f2c\") " pod="openshift-ingress/e2e-ipp-cleanup-fa6952-openshift-default-6f7dfcbd94-dlfq5" Jul 27 19:25:45.504197 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:45.503833 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/254c91ed-447c-435f-8573-8b3400da6f2c-istio-podinfo\") pod \"e2e-ipp-cleanup-fa6952-openshift-default-6f7dfcbd94-dlfq5\" (UID: \"254c91ed-447c-435f-8573-8b3400da6f2c\") " pod="openshift-ingress/e2e-ipp-cleanup-fa6952-openshift-default-6f7dfcbd94-dlfq5" Jul 27 19:25:45.504197 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:45.503998 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/254c91ed-447c-435f-8573-8b3400da6f2c-istiod-ca-cert\") pod \"e2e-ipp-cleanup-fa6952-openshift-default-6f7dfcbd94-dlfq5\" (UID: \"254c91ed-447c-435f-8573-8b3400da6f2c\") " pod="openshift-ingress/e2e-ipp-cleanup-fa6952-openshift-default-6f7dfcbd94-dlfq5" Jul 27 19:25:45.504197 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:45.504072 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/254c91ed-447c-435f-8573-8b3400da6f2c-workload-socket\") pod \"e2e-ipp-cleanup-fa6952-openshift-default-6f7dfcbd94-dlfq5\" (UID: \"254c91ed-447c-435f-8573-8b3400da6f2c\") " pod="openshift-ingress/e2e-ipp-cleanup-fa6952-openshift-default-6f7dfcbd94-dlfq5" Jul 27 19:25:45.504197 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:45.504102 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/254c91ed-447c-435f-8573-8b3400da6f2c-credential-socket\") pod \"e2e-ipp-cleanup-fa6952-openshift-default-6f7dfcbd94-dlfq5\" (UID: \"254c91ed-447c-435f-8573-8b3400da6f2c\") " pod="openshift-ingress/e2e-ipp-cleanup-fa6952-openshift-default-6f7dfcbd94-dlfq5" Jul 27 19:25:45.504441 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:45.504197 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/254c91ed-447c-435f-8573-8b3400da6f2c-istio-data\") pod \"e2e-ipp-cleanup-fa6952-openshift-default-6f7dfcbd94-dlfq5\" (UID: \"254c91ed-447c-435f-8573-8b3400da6f2c\") " pod="openshift-ingress/e2e-ipp-cleanup-fa6952-openshift-default-6f7dfcbd94-dlfq5" Jul 27 19:25:45.504441 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:45.504280 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/254c91ed-447c-435f-8573-8b3400da6f2c-istio-envoy\") pod \"e2e-ipp-cleanup-fa6952-openshift-default-6f7dfcbd94-dlfq5\" (UID: \"254c91ed-447c-435f-8573-8b3400da6f2c\") " pod="openshift-ingress/e2e-ipp-cleanup-fa6952-openshift-default-6f7dfcbd94-dlfq5" Jul 27 19:25:45.504441 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:45.504314 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/254c91ed-447c-435f-8573-8b3400da6f2c-istio-token\") pod \"e2e-ipp-cleanup-fa6952-openshift-default-6f7dfcbd94-dlfq5\" (UID: \"254c91ed-447c-435f-8573-8b3400da6f2c\") " pod="openshift-ingress/e2e-ipp-cleanup-fa6952-openshift-default-6f7dfcbd94-dlfq5" Jul 27 19:25:45.504441 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:45.504367 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/254c91ed-447c-435f-8573-8b3400da6f2c-workload-certs\") pod \"e2e-ipp-cleanup-fa6952-openshift-default-6f7dfcbd94-dlfq5\" (UID: \"254c91ed-447c-435f-8573-8b3400da6f2c\") " pod="openshift-ingress/e2e-ipp-cleanup-fa6952-openshift-default-6f7dfcbd94-dlfq5" Jul 27 19:25:45.504659 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:45.504483 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/254c91ed-447c-435f-8573-8b3400da6f2c-workload-socket\") pod \"e2e-ipp-cleanup-fa6952-openshift-default-6f7dfcbd94-dlfq5\" (UID: \"254c91ed-447c-435f-8573-8b3400da6f2c\") " pod="openshift-ingress/e2e-ipp-cleanup-fa6952-openshift-default-6f7dfcbd94-dlfq5" Jul 27 19:25:45.504659 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:45.504578 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/254c91ed-447c-435f-8573-8b3400da6f2c-credential-socket\") pod \"e2e-ipp-cleanup-fa6952-openshift-default-6f7dfcbd94-dlfq5\" (UID: \"254c91ed-447c-435f-8573-8b3400da6f2c\") " pod="openshift-ingress/e2e-ipp-cleanup-fa6952-openshift-default-6f7dfcbd94-dlfq5" Jul 27 19:25:45.504659 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:45.504599 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/254c91ed-447c-435f-8573-8b3400da6f2c-istio-data\") pod \"e2e-ipp-cleanup-fa6952-openshift-default-6f7dfcbd94-dlfq5\" (UID: \"254c91ed-447c-435f-8573-8b3400da6f2c\") " pod="openshift-ingress/e2e-ipp-cleanup-fa6952-openshift-default-6f7dfcbd94-dlfq5" Jul 27 19:25:45.504768 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:45.504737 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/254c91ed-447c-435f-8573-8b3400da6f2c-istiod-ca-cert\") pod \"e2e-ipp-cleanup-fa6952-openshift-default-6f7dfcbd94-dlfq5\" (UID: \"254c91ed-447c-435f-8573-8b3400da6f2c\") " pod="openshift-ingress/e2e-ipp-cleanup-fa6952-openshift-default-6f7dfcbd94-dlfq5" Jul 27 19:25:45.504836 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:45.504818 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/254c91ed-447c-435f-8573-8b3400da6f2c-workload-certs\") pod \"e2e-ipp-cleanup-fa6952-openshift-default-6f7dfcbd94-dlfq5\" (UID: \"254c91ed-447c-435f-8573-8b3400da6f2c\") " pod="openshift-ingress/e2e-ipp-cleanup-fa6952-openshift-default-6f7dfcbd94-dlfq5" Jul 27 19:25:45.506483 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:45.506460 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/254c91ed-447c-435f-8573-8b3400da6f2c-istio-podinfo\") pod \"e2e-ipp-cleanup-fa6952-openshift-default-6f7dfcbd94-dlfq5\" (UID: \"254c91ed-447c-435f-8573-8b3400da6f2c\") " pod="openshift-ingress/e2e-ipp-cleanup-fa6952-openshift-default-6f7dfcbd94-dlfq5" Jul 27 19:25:45.506679 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:45.506661 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/254c91ed-447c-435f-8573-8b3400da6f2c-istio-envoy\") pod \"e2e-ipp-cleanup-fa6952-openshift-default-6f7dfcbd94-dlfq5\" (UID: \"254c91ed-447c-435f-8573-8b3400da6f2c\") " pod="openshift-ingress/e2e-ipp-cleanup-fa6952-openshift-default-6f7dfcbd94-dlfq5" Jul 27 19:25:45.514905 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:45.514880 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-225m5\" (UniqueName: \"kubernetes.io/projected/254c91ed-447c-435f-8573-8b3400da6f2c-kube-api-access-225m5\") pod \"e2e-ipp-cleanup-fa6952-openshift-default-6f7dfcbd94-dlfq5\" (UID: \"254c91ed-447c-435f-8573-8b3400da6f2c\") " pod="openshift-ingress/e2e-ipp-cleanup-fa6952-openshift-default-6f7dfcbd94-dlfq5" Jul 27 19:25:45.516284 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:45.516213 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/254c91ed-447c-435f-8573-8b3400da6f2c-istio-token\") pod \"e2e-ipp-cleanup-fa6952-openshift-default-6f7dfcbd94-dlfq5\" (UID: \"254c91ed-447c-435f-8573-8b3400da6f2c\") " pod="openshift-ingress/e2e-ipp-cleanup-fa6952-openshift-default-6f7dfcbd94-dlfq5" Jul 27 19:25:45.584545 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:45.584511 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-fa6952-openshift-default-dockercfg-6rm7n\"" Jul 27 19:25:45.590920 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:45.590897 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-fa6952-openshift-default-6f7dfcbd94-dlfq5" Jul 27 19:25:45.651281 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:45.651191 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:25:45.770590 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:45.770560 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-fa6952-openshift-default-6f7dfcbd94-dlfq5"] Jul 27 19:25:45.771120 ip-10-0-128-32 kubenswrapper[2579]: W0727 19:25:45.771093 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod254c91ed_447c_435f_8573_8b3400da6f2c.slice/crio-70f138e9b41ac2050aba171df05446cbac5d6ae900365c2acb7fd61359215e25 WatchSource:0}: Error finding container 70f138e9b41ac2050aba171df05446cbac5d6ae900365c2acb7fd61359215e25: Status 404 returned error can't find the container with id 70f138e9b41ac2050aba171df05446cbac5d6ae900365c2acb7fd61359215e25 Jul 27 19:25:45.773560 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:45.773526 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 27 19:25:45.773653 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:45.773588 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 27 19:25:45.773653 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:45.773615 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 27 19:25:46.004884 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:46.004843 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-fa6952-openshift-default-6f7dfcbd94-dlfq5" event={"ID":"254c91ed-447c-435f-8573-8b3400da6f2c","Type":"ContainerStarted","Data":"bcfcb765a6183f739d2a9b60f9c8eaba2f0eac4ab0c250c87085601b1b5671c9"} Jul 27 19:25:46.005084 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:46.004937 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-fa6952-openshift-default-6f7dfcbd94-dlfq5" event={"ID":"254c91ed-447c-435f-8573-8b3400da6f2c","Type":"ContainerStarted","Data":"70f138e9b41ac2050aba171df05446cbac5d6ae900365c2acb7fd61359215e25"} Jul 27 19:25:46.035018 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:46.034961 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-fa6952-openshift-default-6f7dfcbd94-dlfq5" podStartSLOduration=1.034943469 podStartE2EDuration="1.034943469s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:25:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:25:46.033605655 +0000 UTC m=+3421.095294687" watchObservedRunningTime="2026-07-27 19:25:46.034943469 +0000 UTC m=+3421.096632499" Jul 27 19:25:46.591995 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:46.591955 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-fa6952-openshift-default-6f7dfcbd94-dlfq5" Jul 27 19:25:46.597167 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:46.597142 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-fa6952-openshift-default-6f7dfcbd94-dlfq5" Jul 27 19:25:47.010440 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:47.010357 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-fa6952-openshift-default-6f7dfcbd94-dlfq5" Jul 27 19:25:47.011305 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:47.011287 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-fa6952-openshift-default-6f7dfcbd94-dlfq5" Jul 27 19:25:48.134405 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:48.134371 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:25:48.230279 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:48.230223 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:25:50.329022 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:50.328987 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:25:50.689872 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:50.689779 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:25:50.807819 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:50.807788 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:25:50.911110 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:50.911077 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:25:51.276173 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:51.276144 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:25:51.357746 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:51.357695 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:25:53.343902 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:25:53.343862 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:26:09.145880 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:26:09.145832 2579 generic.go:356] "Generic (PLEG): container finished" podID="254c91ed-447c-435f-8573-8b3400da6f2c" containerID="bcfcb765a6183f739d2a9b60f9c8eaba2f0eac4ab0c250c87085601b1b5671c9" exitCode=0 Jul 27 19:26:09.146399 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:26:09.145907 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-fa6952-openshift-default-6f7dfcbd94-dlfq5" event={"ID":"254c91ed-447c-435f-8573-8b3400da6f2c","Type":"ContainerDied","Data":"bcfcb765a6183f739d2a9b60f9c8eaba2f0eac4ab0c250c87085601b1b5671c9"} Jul 27 19:26:09.146399 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:26:09.146382 2579 scope.go:117] "RemoveContainer" containerID="bcfcb765a6183f739d2a9b60f9c8eaba2f0eac4ab0c250c87085601b1b5671c9" Jul 27 19:26:09.147587 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:26:09.147562 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 27 19:26:09.147695 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:26:09.147643 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 27 19:26:09.147695 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:26:09.147680 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 27 19:26:10.156131 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:26:10.156093 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-fa6952-openshift-default-6f7dfcbd94-dlfq5" event={"ID":"254c91ed-447c-435f-8573-8b3400da6f2c","Type":"ContainerStarted","Data":"5adad03613a7fc8a823742329a667ed38dca494e35aa6f37ae03e81c1bf6667c"} Jul 27 19:26:10.591807 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:26:10.591769 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-fa6952-openshift-default-6f7dfcbd94-dlfq5" Jul 27 19:26:10.592005 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:26:10.591982 2579 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-fa6952-openshift-default-6f7dfcbd94-dlfq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Jul 27 19:26:10.592058 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:26:10.592043 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-fa6952-openshift-default-6f7dfcbd94-dlfq5" podUID="254c91ed-447c-435f-8573-8b3400da6f2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Jul 27 19:26:11.591754 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:26:11.591714 2579 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-fa6952-openshift-default-6f7dfcbd94-dlfq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Jul 27 19:26:11.592154 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:26:11.591807 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-fa6952-openshift-default-6f7dfcbd94-dlfq5" podUID="254c91ed-447c-435f-8573-8b3400da6f2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Jul 27 19:26:12.591487 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:26:12.591451 2579 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-fa6952-openshift-default-6f7dfcbd94-dlfq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Jul 27 19:26:12.591658 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:26:12.591523 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-fa6952-openshift-default-6f7dfcbd94-dlfq5" podUID="254c91ed-447c-435f-8573-8b3400da6f2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Jul 27 19:26:13.591749 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:26:13.591703 2579 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-fa6952-openshift-default-6f7dfcbd94-dlfq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Jul 27 19:26:13.592138 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:26:13.591768 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-fa6952-openshift-default-6f7dfcbd94-dlfq5" podUID="254c91ed-447c-435f-8573-8b3400da6f2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Jul 27 19:26:14.595980 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:26:14.595945 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-fa6952-openshift-default-6f7dfcbd94-dlfq5" Jul 27 19:26:14.596372 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:26:14.596332 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-fa6952-openshift-default-6f7dfcbd94-dlfq5" Jul 27 19:26:14.597114 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:26:14.597096 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-fa6952-openshift-default-6f7dfcbd94-dlfq5" Jul 27 19:26:15.915710 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:26:15.915674 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:26:16.005803 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:26:16.005766 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:26:16.103695 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:26:16.103655 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:26:21.128464 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:26:21.128423 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:26:21.525933 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:26:21.525899 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:26:28.331317 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:26:28.331277 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:26:28.433338 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:26:28.433288 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:26:28.633184 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:26:28.633097 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-fa6952-openshift-default-6f7dfcbd94-dlfq5"] Jul 27 19:26:28.633396 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:26:28.633364 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-fa6952-openshift-default-6f7dfcbd94-dlfq5" podUID="254c91ed-447c-435f-8573-8b3400da6f2c" containerName="istio-proxy" containerID="cri-o://5adad03613a7fc8a823742329a667ed38dca494e35aa6f37ae03e81c1bf6667c" gracePeriod=30 Jul 27 19:26:28.674981 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:26:28.674949 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:26:28.896842 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:26:28.896750 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:26:29.599064 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:26:29.599026 2579 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-fa6952-openshift-default-6f7dfcbd94-dlfq5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 27 19:26:29.599508 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:26:29.599082 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-cleanup-fa6952-openshift-default-6f7dfcbd94-dlfq5" podUID="254c91ed-447c-435f-8573-8b3400da6f2c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 27 19:26:33.893481 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:26:33.893456 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-fa6952-openshift-default-6f7dfcbd94-dlfq5" Jul 27 19:26:34.009786 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:26:34.009702 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/254c91ed-447c-435f-8573-8b3400da6f2c-istio-envoy\") pod \"254c91ed-447c-435f-8573-8b3400da6f2c\" (UID: \"254c91ed-447c-435f-8573-8b3400da6f2c\") " Jul 27 19:26:34.009967 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:26:34.009789 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/254c91ed-447c-435f-8573-8b3400da6f2c-istio-token\") pod \"254c91ed-447c-435f-8573-8b3400da6f2c\" (UID: \"254c91ed-447c-435f-8573-8b3400da6f2c\") " Jul 27 19:26:34.009967 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:26:34.009814 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/254c91ed-447c-435f-8573-8b3400da6f2c-credential-socket\") pod \"254c91ed-447c-435f-8573-8b3400da6f2c\" (UID: \"254c91ed-447c-435f-8573-8b3400da6f2c\") " Jul 27 19:26:34.009967 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:26:34.009848 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/254c91ed-447c-435f-8573-8b3400da6f2c-workload-socket\") pod \"254c91ed-447c-435f-8573-8b3400da6f2c\" (UID: \"254c91ed-447c-435f-8573-8b3400da6f2c\") " Jul 27 19:26:34.009967 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:26:34.009885 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/254c91ed-447c-435f-8573-8b3400da6f2c-istio-podinfo\") pod \"254c91ed-447c-435f-8573-8b3400da6f2c\" (UID: \"254c91ed-447c-435f-8573-8b3400da6f2c\") " Jul 27 19:26:34.009967 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:26:34.009916 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/254c91ed-447c-435f-8573-8b3400da6f2c-istio-data\") pod \"254c91ed-447c-435f-8573-8b3400da6f2c\" (UID: \"254c91ed-447c-435f-8573-8b3400da6f2c\") " Jul 27 19:26:34.009967 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:26:34.009942 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/254c91ed-447c-435f-8573-8b3400da6f2c-workload-certs\") pod \"254c91ed-447c-435f-8573-8b3400da6f2c\" (UID: \"254c91ed-447c-435f-8573-8b3400da6f2c\") " Jul 27 19:26:34.010309 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:26:34.009969 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-225m5\" (UniqueName: \"kubernetes.io/projected/254c91ed-447c-435f-8573-8b3400da6f2c-kube-api-access-225m5\") pod \"254c91ed-447c-435f-8573-8b3400da6f2c\" (UID: \"254c91ed-447c-435f-8573-8b3400da6f2c\") " Jul 27 19:26:34.010309 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:26:34.010014 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/254c91ed-447c-435f-8573-8b3400da6f2c-istiod-ca-cert\") pod \"254c91ed-447c-435f-8573-8b3400da6f2c\" (UID: \"254c91ed-447c-435f-8573-8b3400da6f2c\") " Jul 27 19:26:34.010309 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:26:34.010128 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/254c91ed-447c-435f-8573-8b3400da6f2c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "254c91ed-447c-435f-8573-8b3400da6f2c" (UID: "254c91ed-447c-435f-8573-8b3400da6f2c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 19:26:34.010309 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:26:34.010196 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/254c91ed-447c-435f-8573-8b3400da6f2c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "254c91ed-447c-435f-8573-8b3400da6f2c" (UID: "254c91ed-447c-435f-8573-8b3400da6f2c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 19:26:34.010681 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:26:34.010422 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/254c91ed-447c-435f-8573-8b3400da6f2c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "254c91ed-447c-435f-8573-8b3400da6f2c" (UID: "254c91ed-447c-435f-8573-8b3400da6f2c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 19:26:34.010681 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:26:34.010434 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/254c91ed-447c-435f-8573-8b3400da6f2c-workload-certs\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 19:26:34.010681 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:26:34.010451 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/254c91ed-447c-435f-8573-8b3400da6f2c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "254c91ed-447c-435f-8573-8b3400da6f2c" (UID: "254c91ed-447c-435f-8573-8b3400da6f2c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 19:26:34.010681 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:26:34.010457 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/254c91ed-447c-435f-8573-8b3400da6f2c-credential-socket\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 19:26:34.010898 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:26:34.010802 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/254c91ed-447c-435f-8573-8b3400da6f2c-istio-data" (OuterVolumeSpecName: "istio-data") pod "254c91ed-447c-435f-8573-8b3400da6f2c" (UID: "254c91ed-447c-435f-8573-8b3400da6f2c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 19:26:34.012243 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:26:34.012199 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/254c91ed-447c-435f-8573-8b3400da6f2c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "254c91ed-447c-435f-8573-8b3400da6f2c" (UID: "254c91ed-447c-435f-8573-8b3400da6f2c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 19:26:34.012365 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:26:34.012224 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/254c91ed-447c-435f-8573-8b3400da6f2c-istio-token" (OuterVolumeSpecName: "istio-token") pod "254c91ed-447c-435f-8573-8b3400da6f2c" (UID: "254c91ed-447c-435f-8573-8b3400da6f2c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 19:26:34.012427 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:26:34.012407 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/254c91ed-447c-435f-8573-8b3400da6f2c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "254c91ed-447c-435f-8573-8b3400da6f2c" (UID: "254c91ed-447c-435f-8573-8b3400da6f2c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 27 19:26:34.012704 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:26:34.012686 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/254c91ed-447c-435f-8573-8b3400da6f2c-kube-api-access-225m5" (OuterVolumeSpecName: "kube-api-access-225m5") pod "254c91ed-447c-435f-8573-8b3400da6f2c" (UID: "254c91ed-447c-435f-8573-8b3400da6f2c"). InnerVolumeSpecName "kube-api-access-225m5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 19:26:34.111895 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:26:34.111851 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/254c91ed-447c-435f-8573-8b3400da6f2c-istio-token\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 19:26:34.111895 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:26:34.111892 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/254c91ed-447c-435f-8573-8b3400da6f2c-workload-socket\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 19:26:34.111895 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:26:34.111901 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/254c91ed-447c-435f-8573-8b3400da6f2c-istio-podinfo\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 19:26:34.111895 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:26:34.111910 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/254c91ed-447c-435f-8573-8b3400da6f2c-istio-data\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 19:26:34.112156 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:26:34.111920 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-225m5\" (UniqueName: \"kubernetes.io/projected/254c91ed-447c-435f-8573-8b3400da6f2c-kube-api-access-225m5\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 19:26:34.112156 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:26:34.111928 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/254c91ed-447c-435f-8573-8b3400da6f2c-istiod-ca-cert\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 19:26:34.112156 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:26:34.111937 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/254c91ed-447c-435f-8573-8b3400da6f2c-istio-envoy\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 19:26:34.293010 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:26:34.292973 2579 generic.go:356] "Generic (PLEG): container finished" podID="254c91ed-447c-435f-8573-8b3400da6f2c" containerID="5adad03613a7fc8a823742329a667ed38dca494e35aa6f37ae03e81c1bf6667c" exitCode=0 Jul 27 19:26:34.293204 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:26:34.293033 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-fa6952-openshift-default-6f7dfcbd94-dlfq5" Jul 27 19:26:34.293204 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:26:34.293041 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-fa6952-openshift-default-6f7dfcbd94-dlfq5" event={"ID":"254c91ed-447c-435f-8573-8b3400da6f2c","Type":"ContainerDied","Data":"5adad03613a7fc8a823742329a667ed38dca494e35aa6f37ae03e81c1bf6667c"} Jul 27 19:26:34.293204 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:26:34.293097 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-fa6952-openshift-default-6f7dfcbd94-dlfq5" event={"ID":"254c91ed-447c-435f-8573-8b3400da6f2c","Type":"ContainerDied","Data":"70f138e9b41ac2050aba171df05446cbac5d6ae900365c2acb7fd61359215e25"} Jul 27 19:26:34.293204 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:26:34.293125 2579 scope.go:117] "RemoveContainer" containerID="5adad03613a7fc8a823742329a667ed38dca494e35aa6f37ae03e81c1bf6667c" Jul 27 19:26:34.303859 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:26:34.303844 2579 scope.go:117] "RemoveContainer" containerID="bcfcb765a6183f739d2a9b60f9c8eaba2f0eac4ab0c250c87085601b1b5671c9" Jul 27 19:26:34.312775 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:26:34.312755 2579 scope.go:117] "RemoveContainer" containerID="5adad03613a7fc8a823742329a667ed38dca494e35aa6f37ae03e81c1bf6667c" Jul 27 19:26:34.313027 ip-10-0-128-32 kubenswrapper[2579]: E0727 19:26:34.313007 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5adad03613a7fc8a823742329a667ed38dca494e35aa6f37ae03e81c1bf6667c\": container with ID starting with 5adad03613a7fc8a823742329a667ed38dca494e35aa6f37ae03e81c1bf6667c not found: ID does not exist" containerID="5adad03613a7fc8a823742329a667ed38dca494e35aa6f37ae03e81c1bf6667c" Jul 27 19:26:34.313078 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:26:34.313035 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5adad03613a7fc8a823742329a667ed38dca494e35aa6f37ae03e81c1bf6667c"} err="failed to get container status \"5adad03613a7fc8a823742329a667ed38dca494e35aa6f37ae03e81c1bf6667c\": rpc error: code = NotFound desc = could not find container \"5adad03613a7fc8a823742329a667ed38dca494e35aa6f37ae03e81c1bf6667c\": container with ID starting with 5adad03613a7fc8a823742329a667ed38dca494e35aa6f37ae03e81c1bf6667c not found: ID does not exist" Jul 27 19:26:34.313078 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:26:34.313053 2579 scope.go:117] "RemoveContainer" containerID="bcfcb765a6183f739d2a9b60f9c8eaba2f0eac4ab0c250c87085601b1b5671c9" Jul 27 19:26:34.313331 ip-10-0-128-32 kubenswrapper[2579]: E0727 19:26:34.313297 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bcfcb765a6183f739d2a9b60f9c8eaba2f0eac4ab0c250c87085601b1b5671c9\": container with ID starting with bcfcb765a6183f739d2a9b60f9c8eaba2f0eac4ab0c250c87085601b1b5671c9 not found: ID does not exist" containerID="bcfcb765a6183f739d2a9b60f9c8eaba2f0eac4ab0c250c87085601b1b5671c9" Jul 27 19:26:34.313393 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:26:34.313339 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bcfcb765a6183f739d2a9b60f9c8eaba2f0eac4ab0c250c87085601b1b5671c9"} err="failed to get container status \"bcfcb765a6183f739d2a9b60f9c8eaba2f0eac4ab0c250c87085601b1b5671c9\": rpc error: code = NotFound desc = could not find container \"bcfcb765a6183f739d2a9b60f9c8eaba2f0eac4ab0c250c87085601b1b5671c9\": container with ID starting with bcfcb765a6183f739d2a9b60f9c8eaba2f0eac4ab0c250c87085601b1b5671c9 not found: ID does not exist" Jul 27 19:26:34.325793 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:26:34.325757 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-fa6952-openshift-default-6f7dfcbd94-dlfq5"] Jul 27 19:26:34.329717 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:26:34.329692 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-fa6952-openshift-default-6f7dfcbd94-dlfq5"] Jul 27 19:26:35.493559 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:26:35.493524 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="254c91ed-447c-435f-8573-8b3400da6f2c" path="/var/lib/kubelet/pods/254c91ed-447c-435f-8573-8b3400da6f2c/volumes" Jul 27 19:26:52.344910 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:26:52.344865 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:26:52.482998 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:26:52.482955 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:26:52.581677 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:26:52.581629 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:26:52.662264 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:26:52.662153 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:26:53.077851 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:26:53.077793 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:26:58.119202 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:26:58.119165 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:27:34.483354 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:27:34.483247 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:27:36.949732 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:27:36.949691 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:27:43.605771 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:27:43.605728 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:27:43.704152 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:27:43.704117 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:27:43.909182 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:27:43.909100 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-1d1107-openshift-default-5bfc5f8cb6-mcf9c"] Jul 27 19:27:43.909391 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:27:43.909313 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-1d1107-openshift-default-5bfc5f8cb6-mcf9c" podUID="e07be686-8abf-43d8-8d57-ec992ee9cd35" containerName="istio-proxy" containerID="cri-o://2bf86b10219cf48884d9786c1775b4146ecd26542a1f178a6e0092a78c06e1a2" gracePeriod=30 Jul 27 19:27:43.929127 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:27:43.929093 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:27:44.197209 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:27:44.197125 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:27:46.650839 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:27:46.650801 2579 patch_prober.go:28] interesting pod/e2e-ipp-a-1d1107-openshift-default-5bfc5f8cb6-mcf9c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 27 19:27:46.651248 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:27:46.650863 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-1d1107-openshift-default-5bfc5f8cb6-mcf9c" podUID="e07be686-8abf-43d8-8d57-ec992ee9cd35" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 27 19:27:49.175326 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:27:49.175262 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-1d1107-openshift-default-5bfc5f8cb6-mcf9c" Jul 27 19:27:49.275678 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:27:49.275648 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e07be686-8abf-43d8-8d57-ec992ee9cd35-istio-envoy\") pod \"e07be686-8abf-43d8-8d57-ec992ee9cd35\" (UID: \"e07be686-8abf-43d8-8d57-ec992ee9cd35\") " Jul 27 19:27:49.275855 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:27:49.275690 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e07be686-8abf-43d8-8d57-ec992ee9cd35-istiod-ca-cert\") pod \"e07be686-8abf-43d8-8d57-ec992ee9cd35\" (UID: \"e07be686-8abf-43d8-8d57-ec992ee9cd35\") " Jul 27 19:27:49.275855 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:27:49.275730 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e07be686-8abf-43d8-8d57-ec992ee9cd35-workload-certs\") pod \"e07be686-8abf-43d8-8d57-ec992ee9cd35\" (UID: \"e07be686-8abf-43d8-8d57-ec992ee9cd35\") " Jul 27 19:27:49.275855 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:27:49.275772 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e07be686-8abf-43d8-8d57-ec992ee9cd35-istio-token\") pod \"e07be686-8abf-43d8-8d57-ec992ee9cd35\" (UID: \"e07be686-8abf-43d8-8d57-ec992ee9cd35\") " Jul 27 19:27:49.275855 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:27:49.275837 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e07be686-8abf-43d8-8d57-ec992ee9cd35-workload-socket\") pod \"e07be686-8abf-43d8-8d57-ec992ee9cd35\" (UID: \"e07be686-8abf-43d8-8d57-ec992ee9cd35\") " Jul 27 19:27:49.276076 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:27:49.275879 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e07be686-8abf-43d8-8d57-ec992ee9cd35-credential-socket\") pod \"e07be686-8abf-43d8-8d57-ec992ee9cd35\" (UID: \"e07be686-8abf-43d8-8d57-ec992ee9cd35\") " Jul 27 19:27:49.276076 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:27:49.275913 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e07be686-8abf-43d8-8d57-ec992ee9cd35-istio-data\") pod \"e07be686-8abf-43d8-8d57-ec992ee9cd35\" (UID: \"e07be686-8abf-43d8-8d57-ec992ee9cd35\") " Jul 27 19:27:49.276076 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:27:49.275951 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e07be686-8abf-43d8-8d57-ec992ee9cd35-istio-podinfo\") pod \"e07be686-8abf-43d8-8d57-ec992ee9cd35\" (UID: \"e07be686-8abf-43d8-8d57-ec992ee9cd35\") " Jul 27 19:27:49.276076 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:27:49.275980 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8pdnk\" (UniqueName: \"kubernetes.io/projected/e07be686-8abf-43d8-8d57-ec992ee9cd35-kube-api-access-8pdnk\") pod \"e07be686-8abf-43d8-8d57-ec992ee9cd35\" (UID: \"e07be686-8abf-43d8-8d57-ec992ee9cd35\") " Jul 27 19:27:49.276076 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:27:49.275992 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e07be686-8abf-43d8-8d57-ec992ee9cd35-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e07be686-8abf-43d8-8d57-ec992ee9cd35" (UID: "e07be686-8abf-43d8-8d57-ec992ee9cd35"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 19:27:49.276371 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:27:49.276284 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e07be686-8abf-43d8-8d57-ec992ee9cd35-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e07be686-8abf-43d8-8d57-ec992ee9cd35" (UID: "e07be686-8abf-43d8-8d57-ec992ee9cd35"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 19:27:49.276371 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:27:49.276304 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e07be686-8abf-43d8-8d57-ec992ee9cd35-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e07be686-8abf-43d8-8d57-ec992ee9cd35" (UID: "e07be686-8abf-43d8-8d57-ec992ee9cd35"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 19:27:49.276488 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:27:49.276447 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e07be686-8abf-43d8-8d57-ec992ee9cd35-workload-certs\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 19:27:49.276488 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:27:49.276467 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e07be686-8abf-43d8-8d57-ec992ee9cd35-workload-socket\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 19:27:49.276488 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:27:49.276481 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e07be686-8abf-43d8-8d57-ec992ee9cd35-credential-socket\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 19:27:49.276873 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:27:49.276848 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e07be686-8abf-43d8-8d57-ec992ee9cd35-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e07be686-8abf-43d8-8d57-ec992ee9cd35" (UID: "e07be686-8abf-43d8-8d57-ec992ee9cd35"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 19:27:49.277126 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:27:49.277093 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e07be686-8abf-43d8-8d57-ec992ee9cd35-istio-data" (OuterVolumeSpecName: "istio-data") pod "e07be686-8abf-43d8-8d57-ec992ee9cd35" (UID: "e07be686-8abf-43d8-8d57-ec992ee9cd35"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 19:27:49.278191 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:27:49.278157 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e07be686-8abf-43d8-8d57-ec992ee9cd35-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e07be686-8abf-43d8-8d57-ec992ee9cd35" (UID: "e07be686-8abf-43d8-8d57-ec992ee9cd35"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 19:27:49.278396 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:27:49.278312 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e07be686-8abf-43d8-8d57-ec992ee9cd35-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e07be686-8abf-43d8-8d57-ec992ee9cd35" (UID: "e07be686-8abf-43d8-8d57-ec992ee9cd35"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 27 19:27:49.278396 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:27:49.278353 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e07be686-8abf-43d8-8d57-ec992ee9cd35-istio-token" (OuterVolumeSpecName: "istio-token") pod "e07be686-8abf-43d8-8d57-ec992ee9cd35" (UID: "e07be686-8abf-43d8-8d57-ec992ee9cd35"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 19:27:49.278494 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:27:49.278474 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e07be686-8abf-43d8-8d57-ec992ee9cd35-kube-api-access-8pdnk" (OuterVolumeSpecName: "kube-api-access-8pdnk") pod "e07be686-8abf-43d8-8d57-ec992ee9cd35" (UID: "e07be686-8abf-43d8-8d57-ec992ee9cd35"). InnerVolumeSpecName "kube-api-access-8pdnk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 19:27:49.377594 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:27:49.377563 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e07be686-8abf-43d8-8d57-ec992ee9cd35-istio-data\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 19:27:49.377594 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:27:49.377590 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e07be686-8abf-43d8-8d57-ec992ee9cd35-istio-podinfo\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 19:27:49.377594 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:27:49.377599 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8pdnk\" (UniqueName: \"kubernetes.io/projected/e07be686-8abf-43d8-8d57-ec992ee9cd35-kube-api-access-8pdnk\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 19:27:49.377813 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:27:49.377610 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e07be686-8abf-43d8-8d57-ec992ee9cd35-istio-envoy\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 19:27:49.377813 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:27:49.377620 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e07be686-8abf-43d8-8d57-ec992ee9cd35-istiod-ca-cert\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 19:27:49.377813 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:27:49.377628 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e07be686-8abf-43d8-8d57-ec992ee9cd35-istio-token\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 19:27:49.693908 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:27:49.693876 2579 generic.go:356] "Generic (PLEG): container finished" podID="e07be686-8abf-43d8-8d57-ec992ee9cd35" containerID="2bf86b10219cf48884d9786c1775b4146ecd26542a1f178a6e0092a78c06e1a2" exitCode=0 Jul 27 19:27:49.694112 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:27:49.693939 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-1d1107-openshift-default-5bfc5f8cb6-mcf9c" Jul 27 19:27:49.694112 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:27:49.693969 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-1d1107-openshift-default-5bfc5f8cb6-mcf9c" event={"ID":"e07be686-8abf-43d8-8d57-ec992ee9cd35","Type":"ContainerDied","Data":"2bf86b10219cf48884d9786c1775b4146ecd26542a1f178a6e0092a78c06e1a2"} Jul 27 19:27:49.694112 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:27:49.694029 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-1d1107-openshift-default-5bfc5f8cb6-mcf9c" event={"ID":"e07be686-8abf-43d8-8d57-ec992ee9cd35","Type":"ContainerDied","Data":"dd44dab4daa5cd07eeabd18c534e04e2650c7eedc8447ab70dbea7c6a09d32e1"} Jul 27 19:27:49.694112 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:27:49.694052 2579 scope.go:117] "RemoveContainer" containerID="2bf86b10219cf48884d9786c1775b4146ecd26542a1f178a6e0092a78c06e1a2" Jul 27 19:27:49.706095 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:27:49.706075 2579 scope.go:117] "RemoveContainer" containerID="ffeb57794d62f458acf4d533ccc0472e01418617982ae3f05b6d22d9a4f4207f" Jul 27 19:27:49.715341 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:27:49.715314 2579 scope.go:117] "RemoveContainer" containerID="2bf86b10219cf48884d9786c1775b4146ecd26542a1f178a6e0092a78c06e1a2" Jul 27 19:27:49.715670 ip-10-0-128-32 kubenswrapper[2579]: E0727 19:27:49.715651 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bf86b10219cf48884d9786c1775b4146ecd26542a1f178a6e0092a78c06e1a2\": container with ID starting with 2bf86b10219cf48884d9786c1775b4146ecd26542a1f178a6e0092a78c06e1a2 not found: ID does not exist" containerID="2bf86b10219cf48884d9786c1775b4146ecd26542a1f178a6e0092a78c06e1a2" Jul 27 19:27:49.715739 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:27:49.715678 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bf86b10219cf48884d9786c1775b4146ecd26542a1f178a6e0092a78c06e1a2"} err="failed to get container status \"2bf86b10219cf48884d9786c1775b4146ecd26542a1f178a6e0092a78c06e1a2\": rpc error: code = NotFound desc = could not find container \"2bf86b10219cf48884d9786c1775b4146ecd26542a1f178a6e0092a78c06e1a2\": container with ID starting with 2bf86b10219cf48884d9786c1775b4146ecd26542a1f178a6e0092a78c06e1a2 not found: ID does not exist" Jul 27 19:27:49.715739 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:27:49.715698 2579 scope.go:117] "RemoveContainer" containerID="ffeb57794d62f458acf4d533ccc0472e01418617982ae3f05b6d22d9a4f4207f" Jul 27 19:27:49.715988 ip-10-0-128-32 kubenswrapper[2579]: E0727 19:27:49.715969 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ffeb57794d62f458acf4d533ccc0472e01418617982ae3f05b6d22d9a4f4207f\": container with ID starting with ffeb57794d62f458acf4d533ccc0472e01418617982ae3f05b6d22d9a4f4207f not found: ID does not exist" containerID="ffeb57794d62f458acf4d533ccc0472e01418617982ae3f05b6d22d9a4f4207f" Jul 27 19:27:49.716049 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:27:49.715998 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffeb57794d62f458acf4d533ccc0472e01418617982ae3f05b6d22d9a4f4207f"} err="failed to get container status \"ffeb57794d62f458acf4d533ccc0472e01418617982ae3f05b6d22d9a4f4207f\": rpc error: code = NotFound desc = could not find container \"ffeb57794d62f458acf4d533ccc0472e01418617982ae3f05b6d22d9a4f4207f\": container with ID starting with ffeb57794d62f458acf4d533ccc0472e01418617982ae3f05b6d22d9a4f4207f not found: ID does not exist" Jul 27 19:27:49.717946 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:27:49.717925 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-1d1107-openshift-default-5bfc5f8cb6-mcf9c"] Jul 27 19:27:49.721575 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:27:49.721553 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-1d1107-openshift-default-5bfc5f8cb6-mcf9c"] Jul 27 19:27:51.493593 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:27:51.493557 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e07be686-8abf-43d8-8d57-ec992ee9cd35" path="/var/lib/kubelet/pods/e07be686-8abf-43d8-8d57-ec992ee9cd35/volumes" Jul 27 19:27:56.014573 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:27:56.014531 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:27:56.115174 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:27:56.115123 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:27:56.215767 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:27:56.215685 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:27:59.469458 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:27:59.469408 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:27:59.851277 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:27:59.851212 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:28:07.696763 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:28:07.696727 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:28:07.795953 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:28:07.795912 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:28:08.055390 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:28:08.055353 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-0bd532-openshift-default-59595d6b48-s4vm2"] Jul 27 19:28:08.055619 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:28:08.055560 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-0bd532-openshift-default-59595d6b48-s4vm2" podUID="a467cc20-a0ce-4fcf-b369-4c1ee4b3ed85" containerName="istio-proxy" containerID="cri-o://f1a59c256b482a9669a91884d935d216f22e3a4d30767bf8672acd711d3796b3" gracePeriod=30 Jul 27 19:28:08.137751 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:28:08.137709 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:28:08.335615 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:28:08.335520 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:28:13.312580 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:28:13.312552 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-0bd532-openshift-default-59595d6b48-s4vm2" Jul 27 19:28:13.417193 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:28:13.417098 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a467cc20-a0ce-4fcf-b369-4c1ee4b3ed85-workload-socket\") pod \"a467cc20-a0ce-4fcf-b369-4c1ee4b3ed85\" (UID: \"a467cc20-a0ce-4fcf-b369-4c1ee4b3ed85\") " Jul 27 19:28:13.417193 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:28:13.417149 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zftz6\" (UniqueName: \"kubernetes.io/projected/a467cc20-a0ce-4fcf-b369-4c1ee4b3ed85-kube-api-access-zftz6\") pod \"a467cc20-a0ce-4fcf-b369-4c1ee4b3ed85\" (UID: \"a467cc20-a0ce-4fcf-b369-4c1ee4b3ed85\") " Jul 27 19:28:13.417193 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:28:13.417174 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a467cc20-a0ce-4fcf-b369-4c1ee4b3ed85-workload-certs\") pod \"a467cc20-a0ce-4fcf-b369-4c1ee4b3ed85\" (UID: \"a467cc20-a0ce-4fcf-b369-4c1ee4b3ed85\") " Jul 27 19:28:13.417471 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:28:13.417202 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a467cc20-a0ce-4fcf-b369-4c1ee4b3ed85-credential-socket\") pod \"a467cc20-a0ce-4fcf-b369-4c1ee4b3ed85\" (UID: \"a467cc20-a0ce-4fcf-b369-4c1ee4b3ed85\") " Jul 27 19:28:13.417471 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:28:13.417262 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a467cc20-a0ce-4fcf-b369-4c1ee4b3ed85-istio-envoy\") pod \"a467cc20-a0ce-4fcf-b369-4c1ee4b3ed85\" (UID: \"a467cc20-a0ce-4fcf-b369-4c1ee4b3ed85\") " Jul 27 19:28:13.417471 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:28:13.417342 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a467cc20-a0ce-4fcf-b369-4c1ee4b3ed85-istio-token\") pod \"a467cc20-a0ce-4fcf-b369-4c1ee4b3ed85\" (UID: \"a467cc20-a0ce-4fcf-b369-4c1ee4b3ed85\") " Jul 27 19:28:13.417471 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:28:13.417406 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a467cc20-a0ce-4fcf-b369-4c1ee4b3ed85-istio-data\") pod \"a467cc20-a0ce-4fcf-b369-4c1ee4b3ed85\" (UID: \"a467cc20-a0ce-4fcf-b369-4c1ee4b3ed85\") " Jul 27 19:28:13.417471 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:28:13.417425 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a467cc20-a0ce-4fcf-b369-4c1ee4b3ed85-istiod-ca-cert\") pod \"a467cc20-a0ce-4fcf-b369-4c1ee4b3ed85\" (UID: \"a467cc20-a0ce-4fcf-b369-4c1ee4b3ed85\") " Jul 27 19:28:13.417471 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:28:13.417454 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a467cc20-a0ce-4fcf-b369-4c1ee4b3ed85-istio-podinfo\") pod \"a467cc20-a0ce-4fcf-b369-4c1ee4b3ed85\" (UID: \"a467cc20-a0ce-4fcf-b369-4c1ee4b3ed85\") " Jul 27 19:28:13.417471 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:28:13.417452 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a467cc20-a0ce-4fcf-b369-4c1ee4b3ed85-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a467cc20-a0ce-4fcf-b369-4c1ee4b3ed85" (UID: "a467cc20-a0ce-4fcf-b369-4c1ee4b3ed85"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 19:28:13.417859 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:28:13.417558 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a467cc20-a0ce-4fcf-b369-4c1ee4b3ed85-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a467cc20-a0ce-4fcf-b369-4c1ee4b3ed85" (UID: "a467cc20-a0ce-4fcf-b369-4c1ee4b3ed85"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 19:28:13.417859 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:28:13.417570 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a467cc20-a0ce-4fcf-b369-4c1ee4b3ed85-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a467cc20-a0ce-4fcf-b369-4c1ee4b3ed85" (UID: "a467cc20-a0ce-4fcf-b369-4c1ee4b3ed85"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 19:28:13.417859 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:28:13.417815 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a467cc20-a0ce-4fcf-b369-4c1ee4b3ed85-workload-socket\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 19:28:13.417859 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:28:13.417835 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a467cc20-a0ce-4fcf-b369-4c1ee4b3ed85-workload-certs\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 19:28:13.417859 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:28:13.417850 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a467cc20-a0ce-4fcf-b369-4c1ee4b3ed85-credential-socket\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 19:28:13.417859 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:28:13.417831 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a467cc20-a0ce-4fcf-b369-4c1ee4b3ed85-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a467cc20-a0ce-4fcf-b369-4c1ee4b3ed85" (UID: "a467cc20-a0ce-4fcf-b369-4c1ee4b3ed85"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 19:28:13.418370 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:28:13.418338 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a467cc20-a0ce-4fcf-b369-4c1ee4b3ed85-istio-data" (OuterVolumeSpecName: "istio-data") pod "a467cc20-a0ce-4fcf-b369-4c1ee4b3ed85" (UID: "a467cc20-a0ce-4fcf-b369-4c1ee4b3ed85"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 19:28:13.419745 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:28:13.419716 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a467cc20-a0ce-4fcf-b369-4c1ee4b3ed85-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a467cc20-a0ce-4fcf-b369-4c1ee4b3ed85" (UID: "a467cc20-a0ce-4fcf-b369-4c1ee4b3ed85"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 19:28:13.419837 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:28:13.419743 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a467cc20-a0ce-4fcf-b369-4c1ee4b3ed85-istio-token" (OuterVolumeSpecName: "istio-token") pod "a467cc20-a0ce-4fcf-b369-4c1ee4b3ed85" (UID: "a467cc20-a0ce-4fcf-b369-4c1ee4b3ed85"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 19:28:13.420247 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:28:13.420209 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a467cc20-a0ce-4fcf-b369-4c1ee4b3ed85-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a467cc20-a0ce-4fcf-b369-4c1ee4b3ed85" (UID: "a467cc20-a0ce-4fcf-b369-4c1ee4b3ed85"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 27 19:28:13.420247 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:28:13.420217 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a467cc20-a0ce-4fcf-b369-4c1ee4b3ed85-kube-api-access-zftz6" (OuterVolumeSpecName: "kube-api-access-zftz6") pod "a467cc20-a0ce-4fcf-b369-4c1ee4b3ed85" (UID: "a467cc20-a0ce-4fcf-b369-4c1ee4b3ed85"). InnerVolumeSpecName "kube-api-access-zftz6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 19:28:13.519024 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:28:13.518998 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zftz6\" (UniqueName: \"kubernetes.io/projected/a467cc20-a0ce-4fcf-b369-4c1ee4b3ed85-kube-api-access-zftz6\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 19:28:13.519024 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:28:13.519022 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a467cc20-a0ce-4fcf-b369-4c1ee4b3ed85-istio-envoy\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 19:28:13.519191 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:28:13.519031 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a467cc20-a0ce-4fcf-b369-4c1ee4b3ed85-istio-token\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 19:28:13.519191 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:28:13.519039 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a467cc20-a0ce-4fcf-b369-4c1ee4b3ed85-istio-data\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 19:28:13.519191 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:28:13.519047 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a467cc20-a0ce-4fcf-b369-4c1ee4b3ed85-istiod-ca-cert\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 19:28:13.519191 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:28:13.519054 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a467cc20-a0ce-4fcf-b369-4c1ee4b3ed85-istio-podinfo\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 19:28:13.822147 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:28:13.822111 2579 generic.go:356] "Generic (PLEG): container finished" podID="a467cc20-a0ce-4fcf-b369-4c1ee4b3ed85" containerID="f1a59c256b482a9669a91884d935d216f22e3a4d30767bf8672acd711d3796b3" exitCode=0 Jul 27 19:28:13.822341 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:28:13.822163 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-0bd532-openshift-default-59595d6b48-s4vm2" Jul 27 19:28:13.822341 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:28:13.822179 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-0bd532-openshift-default-59595d6b48-s4vm2" event={"ID":"a467cc20-a0ce-4fcf-b369-4c1ee4b3ed85","Type":"ContainerDied","Data":"f1a59c256b482a9669a91884d935d216f22e3a4d30767bf8672acd711d3796b3"} Jul 27 19:28:13.822341 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:28:13.822247 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-0bd532-openshift-default-59595d6b48-s4vm2" event={"ID":"a467cc20-a0ce-4fcf-b369-4c1ee4b3ed85","Type":"ContainerDied","Data":"0d21be373b6f12b7794b528a2f1b83a01f0f570d6dd1bd68489d9bc46082c1b2"} Jul 27 19:28:13.822341 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:28:13.822282 2579 scope.go:117] "RemoveContainer" containerID="f1a59c256b482a9669a91884d935d216f22e3a4d30767bf8672acd711d3796b3" Jul 27 19:28:13.831857 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:28:13.831838 2579 scope.go:117] "RemoveContainer" containerID="9817d3e6b8de940ea71b6fe5234415ac4d062c5a5165d399e23f8716e7d18873" Jul 27 19:28:13.839711 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:28:13.839693 2579 scope.go:117] "RemoveContainer" containerID="f1a59c256b482a9669a91884d935d216f22e3a4d30767bf8672acd711d3796b3" Jul 27 19:28:13.839945 ip-10-0-128-32 kubenswrapper[2579]: E0727 19:28:13.839926 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1a59c256b482a9669a91884d935d216f22e3a4d30767bf8672acd711d3796b3\": container with ID starting with f1a59c256b482a9669a91884d935d216f22e3a4d30767bf8672acd711d3796b3 not found: ID does not exist" containerID="f1a59c256b482a9669a91884d935d216f22e3a4d30767bf8672acd711d3796b3" Jul 27 19:28:13.839998 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:28:13.839953 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1a59c256b482a9669a91884d935d216f22e3a4d30767bf8672acd711d3796b3"} err="failed to get container status \"f1a59c256b482a9669a91884d935d216f22e3a4d30767bf8672acd711d3796b3\": rpc error: code = NotFound desc = could not find container \"f1a59c256b482a9669a91884d935d216f22e3a4d30767bf8672acd711d3796b3\": container with ID starting with f1a59c256b482a9669a91884d935d216f22e3a4d30767bf8672acd711d3796b3 not found: ID does not exist" Jul 27 19:28:13.839998 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:28:13.839969 2579 scope.go:117] "RemoveContainer" containerID="9817d3e6b8de940ea71b6fe5234415ac4d062c5a5165d399e23f8716e7d18873" Jul 27 19:28:13.840219 ip-10-0-128-32 kubenswrapper[2579]: E0727 19:28:13.840199 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9817d3e6b8de940ea71b6fe5234415ac4d062c5a5165d399e23f8716e7d18873\": container with ID starting with 9817d3e6b8de940ea71b6fe5234415ac4d062c5a5165d399e23f8716e7d18873 not found: ID does not exist" containerID="9817d3e6b8de940ea71b6fe5234415ac4d062c5a5165d399e23f8716e7d18873" Jul 27 19:28:13.840287 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:28:13.840255 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9817d3e6b8de940ea71b6fe5234415ac4d062c5a5165d399e23f8716e7d18873"} err="failed to get container status \"9817d3e6b8de940ea71b6fe5234415ac4d062c5a5165d399e23f8716e7d18873\": rpc error: code = NotFound desc = could not find container \"9817d3e6b8de940ea71b6fe5234415ac4d062c5a5165d399e23f8716e7d18873\": container with ID starting with 9817d3e6b8de940ea71b6fe5234415ac4d062c5a5165d399e23f8716e7d18873 not found: ID does not exist" Jul 27 19:28:13.844409 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:28:13.844378 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-0bd532-openshift-default-59595d6b48-s4vm2"] Jul 27 19:28:13.847747 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:28:13.847724 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-0bd532-openshift-default-59595d6b48-s4vm2"] Jul 27 19:28:15.493621 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:28:15.493589 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a467cc20-a0ce-4fcf-b369-4c1ee4b3ed85" path="/var/lib/kubelet/pods/a467cc20-a0ce-4fcf-b369-4c1ee4b3ed85/volumes" Jul 27 19:28:19.786996 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:28:19.786958 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:28:19.907581 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:28:19.907542 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:28:20.011689 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:28:20.011648 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:28:22.459386 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:28:22.459345 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:28:29.166996 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:28:29.166957 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:28:29.417845 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:28:29.417683 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:28:34.261277 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:28:34.261220 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:28:34.385799 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:28:34.385753 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:28:34.503695 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:28:34.503661 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-7f7473-openshift-default-65db98ff7b-dgl96"] Jul 27 19:28:34.503904 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:28:34.503880 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-7f7473-openshift-default-65db98ff7b-dgl96" podUID="2223e056-27a1-44c6-ac23-924e117185fd" containerName="istio-proxy" containerID="cri-o://d60fe785dc599b076e94609faef8399f3473f3c6a233c56c01f854be295af402" gracePeriod=30 Jul 27 19:28:34.510966 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:28:34.510938 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:28:34.817133 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:28:34.817092 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:28:39.769462 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:28:39.769391 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-7f7473-openshift-default-65db98ff7b-dgl96" Jul 27 19:28:39.862867 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:28:39.862840 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2223e056-27a1-44c6-ac23-924e117185fd-workload-certs\") pod \"2223e056-27a1-44c6-ac23-924e117185fd\" (UID: \"2223e056-27a1-44c6-ac23-924e117185fd\") " Jul 27 19:28:39.863056 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:28:39.862886 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2223e056-27a1-44c6-ac23-924e117185fd-istio-envoy\") pod \"2223e056-27a1-44c6-ac23-924e117185fd\" (UID: \"2223e056-27a1-44c6-ac23-924e117185fd\") " Jul 27 19:28:39.863056 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:28:39.862930 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2223e056-27a1-44c6-ac23-924e117185fd-istio-podinfo\") pod \"2223e056-27a1-44c6-ac23-924e117185fd\" (UID: \"2223e056-27a1-44c6-ac23-924e117185fd\") " Jul 27 19:28:39.863056 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:28:39.862969 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2223e056-27a1-44c6-ac23-924e117185fd-workload-socket\") pod \"2223e056-27a1-44c6-ac23-924e117185fd\" (UID: \"2223e056-27a1-44c6-ac23-924e117185fd\") " Jul 27 19:28:39.863056 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:28:39.863007 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7ljsf\" (UniqueName: \"kubernetes.io/projected/2223e056-27a1-44c6-ac23-924e117185fd-kube-api-access-7ljsf\") pod \"2223e056-27a1-44c6-ac23-924e117185fd\" (UID: \"2223e056-27a1-44c6-ac23-924e117185fd\") " Jul 27 19:28:39.863316 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:28:39.863169 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2223e056-27a1-44c6-ac23-924e117185fd-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2223e056-27a1-44c6-ac23-924e117185fd" (UID: "2223e056-27a1-44c6-ac23-924e117185fd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 19:28:39.863316 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:28:39.863216 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2223e056-27a1-44c6-ac23-924e117185fd-istio-token\") pod \"2223e056-27a1-44c6-ac23-924e117185fd\" (UID: \"2223e056-27a1-44c6-ac23-924e117185fd\") " Jul 27 19:28:39.863428 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:28:39.863314 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2223e056-27a1-44c6-ac23-924e117185fd-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2223e056-27a1-44c6-ac23-924e117185fd" (UID: "2223e056-27a1-44c6-ac23-924e117185fd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 19:28:39.863428 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:28:39.863330 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2223e056-27a1-44c6-ac23-924e117185fd-istiod-ca-cert\") pod \"2223e056-27a1-44c6-ac23-924e117185fd\" (UID: \"2223e056-27a1-44c6-ac23-924e117185fd\") " Jul 27 19:28:39.863428 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:28:39.863380 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2223e056-27a1-44c6-ac23-924e117185fd-istio-data\") pod \"2223e056-27a1-44c6-ac23-924e117185fd\" (UID: \"2223e056-27a1-44c6-ac23-924e117185fd\") " Jul 27 19:28:39.863428 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:28:39.863406 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2223e056-27a1-44c6-ac23-924e117185fd-credential-socket\") pod \"2223e056-27a1-44c6-ac23-924e117185fd\" (UID: \"2223e056-27a1-44c6-ac23-924e117185fd\") " Jul 27 19:28:39.863692 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:28:39.863667 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2223e056-27a1-44c6-ac23-924e117185fd-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2223e056-27a1-44c6-ac23-924e117185fd" (UID: "2223e056-27a1-44c6-ac23-924e117185fd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 19:28:39.864006 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:28:39.863909 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2223e056-27a1-44c6-ac23-924e117185fd-workload-socket\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 19:28:39.864006 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:28:39.863971 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2223e056-27a1-44c6-ac23-924e117185fd-istiod-ca-cert\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 19:28:39.864006 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:28:39.863985 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2223e056-27a1-44c6-ac23-924e117185fd-workload-certs\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 19:28:39.864169 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:28:39.864083 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2223e056-27a1-44c6-ac23-924e117185fd-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2223e056-27a1-44c6-ac23-924e117185fd" (UID: "2223e056-27a1-44c6-ac23-924e117185fd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 19:28:39.864741 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:28:39.864714 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2223e056-27a1-44c6-ac23-924e117185fd-istio-data" (OuterVolumeSpecName: "istio-data") pod "2223e056-27a1-44c6-ac23-924e117185fd" (UID: "2223e056-27a1-44c6-ac23-924e117185fd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 19:28:39.865553 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:28:39.865529 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2223e056-27a1-44c6-ac23-924e117185fd-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2223e056-27a1-44c6-ac23-924e117185fd" (UID: "2223e056-27a1-44c6-ac23-924e117185fd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 27 19:28:39.865654 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:28:39.865549 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2223e056-27a1-44c6-ac23-924e117185fd-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2223e056-27a1-44c6-ac23-924e117185fd" (UID: "2223e056-27a1-44c6-ac23-924e117185fd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 19:28:39.865654 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:28:39.865629 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2223e056-27a1-44c6-ac23-924e117185fd-istio-token" (OuterVolumeSpecName: "istio-token") pod "2223e056-27a1-44c6-ac23-924e117185fd" (UID: "2223e056-27a1-44c6-ac23-924e117185fd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 19:28:39.865735 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:28:39.865642 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2223e056-27a1-44c6-ac23-924e117185fd-kube-api-access-7ljsf" (OuterVolumeSpecName: "kube-api-access-7ljsf") pod "2223e056-27a1-44c6-ac23-924e117185fd" (UID: "2223e056-27a1-44c6-ac23-924e117185fd"). InnerVolumeSpecName "kube-api-access-7ljsf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 19:28:39.962327 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:28:39.962291 2579 generic.go:356] "Generic (PLEG): container finished" podID="2223e056-27a1-44c6-ac23-924e117185fd" containerID="d60fe785dc599b076e94609faef8399f3473f3c6a233c56c01f854be295af402" exitCode=0 Jul 27 19:28:39.962496 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:28:39.962347 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-7f7473-openshift-default-65db98ff7b-dgl96" Jul 27 19:28:39.962496 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:28:39.962374 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-7f7473-openshift-default-65db98ff7b-dgl96" event={"ID":"2223e056-27a1-44c6-ac23-924e117185fd","Type":"ContainerDied","Data":"d60fe785dc599b076e94609faef8399f3473f3c6a233c56c01f854be295af402"} Jul 27 19:28:39.962496 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:28:39.962428 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-7f7473-openshift-default-65db98ff7b-dgl96" event={"ID":"2223e056-27a1-44c6-ac23-924e117185fd","Type":"ContainerDied","Data":"bb1d6735b95cbf5bd9a4dac0af9b3f103ae7b273ed15b5c99a9ac35254f0ae31"} Jul 27 19:28:39.962496 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:28:39.962453 2579 scope.go:117] "RemoveContainer" containerID="d60fe785dc599b076e94609faef8399f3473f3c6a233c56c01f854be295af402" Jul 27 19:28:39.964516 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:28:39.964491 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ljsf\" (UniqueName: \"kubernetes.io/projected/2223e056-27a1-44c6-ac23-924e117185fd-kube-api-access-7ljsf\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 19:28:39.964516 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:28:39.964512 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2223e056-27a1-44c6-ac23-924e117185fd-istio-token\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 19:28:39.964624 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:28:39.964521 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2223e056-27a1-44c6-ac23-924e117185fd-istio-data\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 19:28:39.964624 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:28:39.964529 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2223e056-27a1-44c6-ac23-924e117185fd-credential-socket\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 19:28:39.964624 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:28:39.964538 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2223e056-27a1-44c6-ac23-924e117185fd-istio-envoy\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 19:28:39.964624 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:28:39.964547 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2223e056-27a1-44c6-ac23-924e117185fd-istio-podinfo\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 19:28:39.972418 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:28:39.972404 2579 scope.go:117] "RemoveContainer" containerID="058bdf3c55e20e88fca5ec412826e8e288ad5c0f0130ea4ca6d7d7da4b64b4ec" Jul 27 19:28:39.981304 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:28:39.981283 2579 scope.go:117] "RemoveContainer" containerID="d60fe785dc599b076e94609faef8399f3473f3c6a233c56c01f854be295af402" Jul 27 19:28:39.981570 ip-10-0-128-32 kubenswrapper[2579]: E0727 19:28:39.981548 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d60fe785dc599b076e94609faef8399f3473f3c6a233c56c01f854be295af402\": container with ID starting with d60fe785dc599b076e94609faef8399f3473f3c6a233c56c01f854be295af402 not found: ID does not exist" containerID="d60fe785dc599b076e94609faef8399f3473f3c6a233c56c01f854be295af402" Jul 27 19:28:39.981627 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:28:39.981578 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d60fe785dc599b076e94609faef8399f3473f3c6a233c56c01f854be295af402"} err="failed to get container status \"d60fe785dc599b076e94609faef8399f3473f3c6a233c56c01f854be295af402\": rpc error: code = NotFound desc = could not find container \"d60fe785dc599b076e94609faef8399f3473f3c6a233c56c01f854be295af402\": container with ID starting with d60fe785dc599b076e94609faef8399f3473f3c6a233c56c01f854be295af402 not found: ID does not exist" Jul 27 19:28:39.981627 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:28:39.981594 2579 scope.go:117] "RemoveContainer" containerID="058bdf3c55e20e88fca5ec412826e8e288ad5c0f0130ea4ca6d7d7da4b64b4ec" Jul 27 19:28:39.981817 ip-10-0-128-32 kubenswrapper[2579]: E0727 19:28:39.981798 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"058bdf3c55e20e88fca5ec412826e8e288ad5c0f0130ea4ca6d7d7da4b64b4ec\": container with ID starting with 058bdf3c55e20e88fca5ec412826e8e288ad5c0f0130ea4ca6d7d7da4b64b4ec not found: ID does not exist" containerID="058bdf3c55e20e88fca5ec412826e8e288ad5c0f0130ea4ca6d7d7da4b64b4ec" Jul 27 19:28:39.981870 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:28:39.981827 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"058bdf3c55e20e88fca5ec412826e8e288ad5c0f0130ea4ca6d7d7da4b64b4ec"} err="failed to get container status \"058bdf3c55e20e88fca5ec412826e8e288ad5c0f0130ea4ca6d7d7da4b64b4ec\": rpc error: code = NotFound desc = could not find container \"058bdf3c55e20e88fca5ec412826e8e288ad5c0f0130ea4ca6d7d7da4b64b4ec\": container with ID starting with 058bdf3c55e20e88fca5ec412826e8e288ad5c0f0130ea4ca6d7d7da4b64b4ec not found: ID does not exist" Jul 27 19:28:39.991556 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:28:39.991529 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-7f7473-openshift-default-65db98ff7b-dgl96"] Jul 27 19:28:40.001447 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:28:40.001420 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-7f7473-openshift-default-65db98ff7b-dgl96"] Jul 27 19:28:41.493269 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:28:41.493208 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2223e056-27a1-44c6-ac23-924e117185fd" path="/var/lib/kubelet/pods/2223e056-27a1-44c6-ac23-924e117185fd/volumes" Jul 27 19:28:45.758104 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:28:45.758077 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-mv4hb_715b2084-d79e-4cbf-a057-f2595b4e6b9b/console-operator/1.log" Jul 27 19:28:45.772680 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:28:45.772648 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-mv4hb_715b2084-d79e-4cbf-a057-f2595b4e6b9b/console-operator/1.log" Jul 27 19:28:47.863714 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:28:47.863675 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:28:49.379035 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:28:49.378996 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:28:50.180952 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:28:50.180913 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:28:50.689973 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:28:50.689929 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:28:53.555662 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:28:53.555632 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:28:53.639368 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:28:53.639326 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:28:57.912493 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:28:57.912460 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:28:58.168371 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:28:58.168285 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:28:58.278093 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:28:58.278059 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-6710a9-openshift-default-7d4765f448-58wm4"] Jul 27 19:28:58.278325 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:28:58.278292 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-6710a9-openshift-default-7d4765f448-58wm4" podUID="6ea60e2a-42c9-4459-9c2e-e1809921f6c1" containerName="istio-proxy" containerID="cri-o://79f38ce48191b585b088167202ced3261d5067276a9d7b1ec173e275dfcb7189" gracePeriod=30 Jul 27 19:28:58.597372 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:28:58.597336 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:28:58.767283 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:28:58.767249 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:29:03.542530 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:03.542507 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-6710a9-openshift-default-7d4765f448-58wm4" Jul 27 19:29:03.681831 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:03.681735 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ea60e2a-42c9-4459-9c2e-e1809921f6c1-workload-certs\") pod \"6ea60e2a-42c9-4459-9c2e-e1809921f6c1\" (UID: \"6ea60e2a-42c9-4459-9c2e-e1809921f6c1\") " Jul 27 19:29:03.681831 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:03.681769 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ea60e2a-42c9-4459-9c2e-e1809921f6c1-istiod-ca-cert\") pod \"6ea60e2a-42c9-4459-9c2e-e1809921f6c1\" (UID: \"6ea60e2a-42c9-4459-9c2e-e1809921f6c1\") " Jul 27 19:29:03.681831 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:03.681800 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ea60e2a-42c9-4459-9c2e-e1809921f6c1-istio-podinfo\") pod \"6ea60e2a-42c9-4459-9c2e-e1809921f6c1\" (UID: \"6ea60e2a-42c9-4459-9c2e-e1809921f6c1\") " Jul 27 19:29:03.681831 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:03.681828 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ea60e2a-42c9-4459-9c2e-e1809921f6c1-istio-data\") pod \"6ea60e2a-42c9-4459-9c2e-e1809921f6c1\" (UID: \"6ea60e2a-42c9-4459-9c2e-e1809921f6c1\") " Jul 27 19:29:03.682272 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:03.681849 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ea60e2a-42c9-4459-9c2e-e1809921f6c1-workload-socket\") pod \"6ea60e2a-42c9-4459-9c2e-e1809921f6c1\" (UID: \"6ea60e2a-42c9-4459-9c2e-e1809921f6c1\") " Jul 27 19:29:03.682272 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:03.681864 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ea60e2a-42c9-4459-9c2e-e1809921f6c1-credential-socket\") pod \"6ea60e2a-42c9-4459-9c2e-e1809921f6c1\" (UID: \"6ea60e2a-42c9-4459-9c2e-e1809921f6c1\") " Jul 27 19:29:03.682272 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:03.681931 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ea60e2a-42c9-4459-9c2e-e1809921f6c1-istio-token\") pod \"6ea60e2a-42c9-4459-9c2e-e1809921f6c1\" (UID: \"6ea60e2a-42c9-4459-9c2e-e1809921f6c1\") " Jul 27 19:29:03.682272 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:03.681958 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5nkk4\" (UniqueName: \"kubernetes.io/projected/6ea60e2a-42c9-4459-9c2e-e1809921f6c1-kube-api-access-5nkk4\") pod \"6ea60e2a-42c9-4459-9c2e-e1809921f6c1\" (UID: \"6ea60e2a-42c9-4459-9c2e-e1809921f6c1\") " Jul 27 19:29:03.682272 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:03.681992 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ea60e2a-42c9-4459-9c2e-e1809921f6c1-istio-envoy\") pod \"6ea60e2a-42c9-4459-9c2e-e1809921f6c1\" (UID: \"6ea60e2a-42c9-4459-9c2e-e1809921f6c1\") " Jul 27 19:29:03.682272 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:03.682077 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ea60e2a-42c9-4459-9c2e-e1809921f6c1-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6ea60e2a-42c9-4459-9c2e-e1809921f6c1" (UID: "6ea60e2a-42c9-4459-9c2e-e1809921f6c1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 19:29:03.682593 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:03.682275 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ea60e2a-42c9-4459-9c2e-e1809921f6c1-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6ea60e2a-42c9-4459-9c2e-e1809921f6c1" (UID: "6ea60e2a-42c9-4459-9c2e-e1809921f6c1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 19:29:03.682593 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:03.682296 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ea60e2a-42c9-4459-9c2e-e1809921f6c1-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6ea60e2a-42c9-4459-9c2e-e1809921f6c1" (UID: "6ea60e2a-42c9-4459-9c2e-e1809921f6c1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 19:29:03.682593 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:03.682375 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ea60e2a-42c9-4459-9c2e-e1809921f6c1-workload-certs\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 19:29:03.682593 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:03.682395 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ea60e2a-42c9-4459-9c2e-e1809921f6c1-workload-socket\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 19:29:03.682593 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:03.682460 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ea60e2a-42c9-4459-9c2e-e1809921f6c1-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6ea60e2a-42c9-4459-9c2e-e1809921f6c1" (UID: "6ea60e2a-42c9-4459-9c2e-e1809921f6c1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 19:29:03.683014 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:03.682816 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ea60e2a-42c9-4459-9c2e-e1809921f6c1-istio-data" (OuterVolumeSpecName: "istio-data") pod "6ea60e2a-42c9-4459-9c2e-e1809921f6c1" (UID: "6ea60e2a-42c9-4459-9c2e-e1809921f6c1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 19:29:03.684343 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:03.684314 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ea60e2a-42c9-4459-9c2e-e1809921f6c1-istio-token" (OuterVolumeSpecName: "istio-token") pod "6ea60e2a-42c9-4459-9c2e-e1809921f6c1" (UID: "6ea60e2a-42c9-4459-9c2e-e1809921f6c1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 19:29:03.684462 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:03.684437 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ea60e2a-42c9-4459-9c2e-e1809921f6c1-kube-api-access-5nkk4" (OuterVolumeSpecName: "kube-api-access-5nkk4") pod "6ea60e2a-42c9-4459-9c2e-e1809921f6c1" (UID: "6ea60e2a-42c9-4459-9c2e-e1809921f6c1"). InnerVolumeSpecName "kube-api-access-5nkk4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 19:29:03.684715 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:03.684694 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ea60e2a-42c9-4459-9c2e-e1809921f6c1-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6ea60e2a-42c9-4459-9c2e-e1809921f6c1" (UID: "6ea60e2a-42c9-4459-9c2e-e1809921f6c1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 19:29:03.684768 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:03.684714 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6ea60e2a-42c9-4459-9c2e-e1809921f6c1-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6ea60e2a-42c9-4459-9c2e-e1809921f6c1" (UID: "6ea60e2a-42c9-4459-9c2e-e1809921f6c1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 27 19:29:03.783825 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:03.783778 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ea60e2a-42c9-4459-9c2e-e1809921f6c1-istio-token\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 19:29:03.783825 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:03.783820 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5nkk4\" (UniqueName: \"kubernetes.io/projected/6ea60e2a-42c9-4459-9c2e-e1809921f6c1-kube-api-access-5nkk4\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 19:29:03.783825 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:03.783835 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ea60e2a-42c9-4459-9c2e-e1809921f6c1-istio-envoy\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 19:29:03.784060 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:03.783848 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ea60e2a-42c9-4459-9c2e-e1809921f6c1-istiod-ca-cert\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 19:29:03.784060 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:03.783859 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ea60e2a-42c9-4459-9c2e-e1809921f6c1-istio-podinfo\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 19:29:03.784060 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:03.783870 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ea60e2a-42c9-4459-9c2e-e1809921f6c1-istio-data\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 19:29:03.784060 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:03.783880 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ea60e2a-42c9-4459-9c2e-e1809921f6c1-credential-socket\") on node \"ip-10-0-128-32.ec2.internal\" DevicePath \"\"" Jul 27 19:29:04.086751 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:04.086715 2579 generic.go:356] "Generic (PLEG): container finished" podID="6ea60e2a-42c9-4459-9c2e-e1809921f6c1" containerID="79f38ce48191b585b088167202ced3261d5067276a9d7b1ec173e275dfcb7189" exitCode=0 Jul 27 19:29:04.086932 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:04.086760 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-6710a9-openshift-default-7d4765f448-58wm4" event={"ID":"6ea60e2a-42c9-4459-9c2e-e1809921f6c1","Type":"ContainerDied","Data":"79f38ce48191b585b088167202ced3261d5067276a9d7b1ec173e275dfcb7189"} Jul 27 19:29:04.086932 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:04.086767 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-6710a9-openshift-default-7d4765f448-58wm4" Jul 27 19:29:04.086932 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:04.086803 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-6710a9-openshift-default-7d4765f448-58wm4" event={"ID":"6ea60e2a-42c9-4459-9c2e-e1809921f6c1","Type":"ContainerDied","Data":"ec66bd861b078546ac3b35342f894747b61c0eec8f99adba7d4cd25533b5e5c4"} Jul 27 19:29:04.086932 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:04.086824 2579 scope.go:117] "RemoveContainer" containerID="79f38ce48191b585b088167202ced3261d5067276a9d7b1ec173e275dfcb7189" Jul 27 19:29:04.096590 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:04.096573 2579 scope.go:117] "RemoveContainer" containerID="bd7fa69c680cdd8e514320c3c626f47b5680fb71aa59fafd98136758770f6602" Jul 27 19:29:04.105302 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:04.105282 2579 scope.go:117] "RemoveContainer" containerID="79f38ce48191b585b088167202ced3261d5067276a9d7b1ec173e275dfcb7189" Jul 27 19:29:04.105573 ip-10-0-128-32 kubenswrapper[2579]: E0727 19:29:04.105551 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79f38ce48191b585b088167202ced3261d5067276a9d7b1ec173e275dfcb7189\": container with ID starting with 79f38ce48191b585b088167202ced3261d5067276a9d7b1ec173e275dfcb7189 not found: ID does not exist" containerID="79f38ce48191b585b088167202ced3261d5067276a9d7b1ec173e275dfcb7189" Jul 27 19:29:04.105620 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:04.105583 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79f38ce48191b585b088167202ced3261d5067276a9d7b1ec173e275dfcb7189"} err="failed to get container status \"79f38ce48191b585b088167202ced3261d5067276a9d7b1ec173e275dfcb7189\": rpc error: code = NotFound desc = could not find container \"79f38ce48191b585b088167202ced3261d5067276a9d7b1ec173e275dfcb7189\": container with ID starting with 79f38ce48191b585b088167202ced3261d5067276a9d7b1ec173e275dfcb7189 not found: ID does not exist" Jul 27 19:29:04.105620 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:04.105601 2579 scope.go:117] "RemoveContainer" containerID="bd7fa69c680cdd8e514320c3c626f47b5680fb71aa59fafd98136758770f6602" Jul 27 19:29:04.105874 ip-10-0-128-32 kubenswrapper[2579]: E0727 19:29:04.105850 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd7fa69c680cdd8e514320c3c626f47b5680fb71aa59fafd98136758770f6602\": container with ID starting with bd7fa69c680cdd8e514320c3c626f47b5680fb71aa59fafd98136758770f6602 not found: ID does not exist" containerID="bd7fa69c680cdd8e514320c3c626f47b5680fb71aa59fafd98136758770f6602" Jul 27 19:29:04.105933 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:04.105884 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd7fa69c680cdd8e514320c3c626f47b5680fb71aa59fafd98136758770f6602"} err="failed to get container status \"bd7fa69c680cdd8e514320c3c626f47b5680fb71aa59fafd98136758770f6602\": rpc error: code = NotFound desc = could not find container \"bd7fa69c680cdd8e514320c3c626f47b5680fb71aa59fafd98136758770f6602\": container with ID starting with bd7fa69c680cdd8e514320c3c626f47b5680fb71aa59fafd98136758770f6602 not found: ID does not exist" Jul 27 19:29:04.118282 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:04.118252 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-6710a9-openshift-default-7d4765f448-58wm4"] Jul 27 19:29:04.123451 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:04.123429 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-6710a9-openshift-default-7d4765f448-58wm4"] Jul 27 19:29:05.493563 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:05.493531 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ea60e2a-42c9-4459-9c2e-e1809921f6c1" path="/var/lib/kubelet/pods/6ea60e2a-42c9-4459-9c2e-e1809921f6c1/volumes" Jul 27 19:29:05.660572 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:05.660527 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-z4tz5_ebf77b9d-9e5d-47cd-aa09-ea05ff63234a/manager/0.log" Jul 27 19:29:05.798047 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:05.798014 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-w9d4l_52b6da00-246b-4954-adcc-5b482d1c41bc/manager/0.log" Jul 27 19:29:05.930348 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:05.930316 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-77ff6dccdd-sdkdx_281fa175-bbef-4309-bbaa-a3de9bf7e6a4/manager/0.log" Jul 27 19:29:06.207543 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:06.207446 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-sgg44_87423332-edfc-4e36-a11b-56324b7db773/manager/0.log" Jul 27 19:29:06.337086 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:06.337055 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-57d99c84d4-5228k_9f6495eb-fc10-4791-b32a-81e55f8b09c8/copy-manifests/0.log" Jul 27 19:29:06.350286 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:06.350261 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-57d99c84d4-5228k_9f6495eb-fc10-4791-b32a-81e55f8b09c8/manager/0.log" Jul 27 19:29:07.144707 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:07.144678 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-66qk2_31d0c692-c97b-472b-b909-9cccd4109f7f/postgres/0.log" Jul 27 19:29:08.718667 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:08.718633 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-7q5jd_76b982a1-38c1-4626-8e90-042e910fdffc/manager/0.log" Jul 27 19:29:08.848536 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:08.848508 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-5bjl6_e2b018f7-aa0d-40d9-8d51-cb3fcaaca11e/manager/0.log" Jul 27 19:29:08.974608 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:08.974524 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-lgnxn_d4cbf21a-5dad-42c5-93ef-f2fa08e5b320/kuadrant-console-plugin/0.log" Jul 27 19:29:09.112517 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:09.112483 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-mkzvs_04dbb35a-cea6-4fb7-aab6-29150d13090d/registry-server/0.log" Jul 27 19:29:09.251180 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:09.251086 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-tgppf_d2145611-dcfa-4590-8f47-e742fe5beca7/manager/0.log" Jul 27 19:29:09.888901 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:09.888866 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-w6ltt_2f32486f-5cb4-48a9-96e0-6ee08e76933a/istio-proxy/0.log" Jul 27 19:29:10.240920 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:10.240833 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7896769b64-tkbg9_2bfb0e76-8207-48bb-810a-6f865001eeb1/kube-auth-proxy/0.log" Jul 27 19:29:10.374950 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:10.374910 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-j2vmg_429930eb-3d30-450f-8e99-75ab3f80ba28/istio-proxy/1.log" Jul 27 19:29:10.643378 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:10.643346 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-8rfpl_77a26a98-9824-423f-be2e-c316fc57d11b/payload-processing/0.log" Jul 27 19:29:10.791972 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:10.791936 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5c874b7d95-95vlk_2b41bffe-08d5-4b52-ae91-dcfebab44a99/router/0.log" Jul 27 19:29:11.170181 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:11.170148 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-9jcc5_3ebf412c-8d28-4361-9384-574b538c5c8c/main/0.log" Jul 27 19:29:11.298608 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:11.298579 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-rdx2x_a43bc0e3-cb23-49fa-afec-a2b93eea86d7/main/0.log" Jul 27 19:29:11.683906 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:11.683872 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-gmxtx_2370a328-d5e9-4e99-b0b6-b35739ee7b66/main/0.log" Jul 27 19:29:18.174638 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:18.174596 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:29:18.403179 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:18.403133 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" limit=1024 Jul 27 19:29:21.287535 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:21.287501 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-pr8j9_bd3d598d-7458-4407-ba85-49402348da87/global-pull-secret-syncer/0.log" Jul 27 19:29:21.444033 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:21.444000 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-67t26_9c37bc5b-9ef7-4e2b-b555-52de0d9861b8/konnectivity-agent/0.log" Jul 27 19:29:21.593642 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:21.593561 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-32.ec2.internal_8aa0e86aff493755e5519df8f9d22c9f/haproxy/0.log" Jul 27 19:29:26.442479 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:26.442436 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-7q5jd_76b982a1-38c1-4626-8e90-042e910fdffc/manager/0.log" Jul 27 19:29:26.525328 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:26.525298 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-5bjl6_e2b018f7-aa0d-40d9-8d51-cb3fcaaca11e/manager/0.log" Jul 27 19:29:26.633518 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:26.633492 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-lgnxn_d4cbf21a-5dad-42c5-93ef-f2fa08e5b320/kuadrant-console-plugin/0.log" Jul 27 19:29:26.721401 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:26.721307 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-mkzvs_04dbb35a-cea6-4fb7-aab6-29150d13090d/registry-server/0.log" Jul 27 19:29:27.080950 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:27.080915 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-tgppf_d2145611-dcfa-4590-8f47-e742fe5beca7/manager/0.log" Jul 27 19:29:29.560163 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:29.560123 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-548b489cb8-g9cd9_1e0a413d-db6c-4206-928b-6d95b365c39b/cluster-monitoring-operator/0.log" Jul 27 19:29:29.955060 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:29.954967 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-l5cnt_5b877858-8e86-4213-9a5b-10a18e69d669/node-exporter/0.log" Jul 27 19:29:29.988446 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:29.988417 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-l5cnt_5b877858-8e86-4213-9a5b-10a18e69d669/kube-rbac-proxy/0.log" Jul 27 19:29:30.030756 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:30.030733 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-l5cnt_5b877858-8e86-4213-9a5b-10a18e69d669/init-textfile/0.log" Jul 27 19:29:31.175420 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:31.175386 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-pshc7/perf-node-gather-daemonset-tfjkv"] Jul 27 19:29:31.175805 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:31.175791 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2223e056-27a1-44c6-ac23-924e117185fd" containerName="istio-proxy" Jul 27 19:29:31.175805 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:31.175806 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="2223e056-27a1-44c6-ac23-924e117185fd" containerName="istio-proxy" Jul 27 19:29:31.175890 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:31.175816 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e07be686-8abf-43d8-8d57-ec992ee9cd35" containerName="istio-proxy" Jul 27 19:29:31.175890 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:31.175822 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e07be686-8abf-43d8-8d57-ec992ee9cd35" containerName="istio-proxy" Jul 27 19:29:31.175890 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:31.175828 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="254c91ed-447c-435f-8573-8b3400da6f2c" containerName="istio-proxy" Jul 27 19:29:31.175890 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:31.175834 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="254c91ed-447c-435f-8573-8b3400da6f2c" containerName="istio-proxy" Jul 27 19:29:31.175890 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:31.175843 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a467cc20-a0ce-4fcf-b369-4c1ee4b3ed85" containerName="istio-proxy" Jul 27 19:29:31.175890 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:31.175847 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a467cc20-a0ce-4fcf-b369-4c1ee4b3ed85" containerName="istio-proxy" Jul 27 19:29:31.175890 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:31.175854 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2223e056-27a1-44c6-ac23-924e117185fd" containerName="istio-proxy" Jul 27 19:29:31.175890 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:31.175861 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="2223e056-27a1-44c6-ac23-924e117185fd" containerName="istio-proxy" Jul 27 19:29:31.175890 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:31.175871 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ea60e2a-42c9-4459-9c2e-e1809921f6c1" containerName="istio-proxy" Jul 27 19:29:31.175890 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:31.175876 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ea60e2a-42c9-4459-9c2e-e1809921f6c1" containerName="istio-proxy" Jul 27 19:29:31.175890 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:31.175882 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e07be686-8abf-43d8-8d57-ec992ee9cd35" containerName="istio-proxy" Jul 27 19:29:31.175890 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:31.175887 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e07be686-8abf-43d8-8d57-ec992ee9cd35" containerName="istio-proxy" Jul 27 19:29:31.175890 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:31.175893 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a467cc20-a0ce-4fcf-b369-4c1ee4b3ed85" containerName="istio-proxy" Jul 27 19:29:31.176359 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:31.175899 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a467cc20-a0ce-4fcf-b369-4c1ee4b3ed85" containerName="istio-proxy" Jul 27 19:29:31.176359 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:31.175917 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ea60e2a-42c9-4459-9c2e-e1809921f6c1" containerName="istio-proxy" Jul 27 19:29:31.176359 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:31.175922 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ea60e2a-42c9-4459-9c2e-e1809921f6c1" containerName="istio-proxy" Jul 27 19:29:31.176359 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:31.175975 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a467cc20-a0ce-4fcf-b369-4c1ee4b3ed85" containerName="istio-proxy" Jul 27 19:29:31.176359 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:31.175982 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e07be686-8abf-43d8-8d57-ec992ee9cd35" containerName="istio-proxy" Jul 27 19:29:31.176359 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:31.175989 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ea60e2a-42c9-4459-9c2e-e1809921f6c1" containerName="istio-proxy" Jul 27 19:29:31.176359 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:31.175994 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e07be686-8abf-43d8-8d57-ec992ee9cd35" containerName="istio-proxy" Jul 27 19:29:31.176359 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:31.175998 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a467cc20-a0ce-4fcf-b369-4c1ee4b3ed85" containerName="istio-proxy" Jul 27 19:29:31.176359 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:31.176004 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="254c91ed-447c-435f-8573-8b3400da6f2c" containerName="istio-proxy" Jul 27 19:29:31.176359 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:31.176012 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="2223e056-27a1-44c6-ac23-924e117185fd" containerName="istio-proxy" Jul 27 19:29:31.176359 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:31.176016 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="254c91ed-447c-435f-8573-8b3400da6f2c" containerName="istio-proxy" Jul 27 19:29:31.176359 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:31.176021 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="2223e056-27a1-44c6-ac23-924e117185fd" containerName="istio-proxy" Jul 27 19:29:31.179326 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:31.179307 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-pshc7/perf-node-gather-daemonset-tfjkv" Jul 27 19:29:31.181840 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:31.181811 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-pshc7\"/\"kube-root-ca.crt\"" Jul 27 19:29:31.182832 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:31.182811 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-pshc7\"/\"default-dockercfg-55v4q\"" Jul 27 19:29:31.182924 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:31.182811 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-pshc7\"/\"openshift-service-ca.crt\"" Jul 27 19:29:31.191167 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:31.191137 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-pshc7/perf-node-gather-daemonset-tfjkv"] Jul 27 19:29:31.223482 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:31.223452 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/027a2355-0608-4d7d-b11c-c71369214fb9-sys\") pod \"perf-node-gather-daemonset-tfjkv\" (UID: \"027a2355-0608-4d7d-b11c-c71369214fb9\") " pod="openshift-must-gather-pshc7/perf-node-gather-daemonset-tfjkv" Jul 27 19:29:31.223628 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:31.223485 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lh5w\" (UniqueName: \"kubernetes.io/projected/027a2355-0608-4d7d-b11c-c71369214fb9-kube-api-access-7lh5w\") pod \"perf-node-gather-daemonset-tfjkv\" (UID: \"027a2355-0608-4d7d-b11c-c71369214fb9\") " pod="openshift-must-gather-pshc7/perf-node-gather-daemonset-tfjkv" Jul 27 19:29:31.223628 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:31.223572 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/027a2355-0608-4d7d-b11c-c71369214fb9-podres\") pod \"perf-node-gather-daemonset-tfjkv\" (UID: \"027a2355-0608-4d7d-b11c-c71369214fb9\") " pod="openshift-must-gather-pshc7/perf-node-gather-daemonset-tfjkv" Jul 27 19:29:31.223628 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:31.223616 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/027a2355-0608-4d7d-b11c-c71369214fb9-proc\") pod \"perf-node-gather-daemonset-tfjkv\" (UID: \"027a2355-0608-4d7d-b11c-c71369214fb9\") " pod="openshift-must-gather-pshc7/perf-node-gather-daemonset-tfjkv" Jul 27 19:29:31.223752 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:31.223650 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/027a2355-0608-4d7d-b11c-c71369214fb9-lib-modules\") pod \"perf-node-gather-daemonset-tfjkv\" (UID: \"027a2355-0608-4d7d-b11c-c71369214fb9\") " pod="openshift-must-gather-pshc7/perf-node-gather-daemonset-tfjkv" Jul 27 19:29:31.324658 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:31.324626 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/027a2355-0608-4d7d-b11c-c71369214fb9-sys\") pod \"perf-node-gather-daemonset-tfjkv\" (UID: \"027a2355-0608-4d7d-b11c-c71369214fb9\") " pod="openshift-must-gather-pshc7/perf-node-gather-daemonset-tfjkv" Jul 27 19:29:31.324658 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:31.324664 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lh5w\" (UniqueName: \"kubernetes.io/projected/027a2355-0608-4d7d-b11c-c71369214fb9-kube-api-access-7lh5w\") pod \"perf-node-gather-daemonset-tfjkv\" (UID: \"027a2355-0608-4d7d-b11c-c71369214fb9\") " pod="openshift-must-gather-pshc7/perf-node-gather-daemonset-tfjkv" Jul 27 19:29:31.324851 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:31.324703 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/027a2355-0608-4d7d-b11c-c71369214fb9-podres\") pod \"perf-node-gather-daemonset-tfjkv\" (UID: \"027a2355-0608-4d7d-b11c-c71369214fb9\") " pod="openshift-must-gather-pshc7/perf-node-gather-daemonset-tfjkv" Jul 27 19:29:31.324851 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:31.324731 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/027a2355-0608-4d7d-b11c-c71369214fb9-proc\") pod \"perf-node-gather-daemonset-tfjkv\" (UID: \"027a2355-0608-4d7d-b11c-c71369214fb9\") " pod="openshift-must-gather-pshc7/perf-node-gather-daemonset-tfjkv" Jul 27 19:29:31.324851 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:31.324755 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/027a2355-0608-4d7d-b11c-c71369214fb9-sys\") pod \"perf-node-gather-daemonset-tfjkv\" (UID: \"027a2355-0608-4d7d-b11c-c71369214fb9\") " pod="openshift-must-gather-pshc7/perf-node-gather-daemonset-tfjkv" Jul 27 19:29:31.324851 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:31.324773 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/027a2355-0608-4d7d-b11c-c71369214fb9-lib-modules\") pod \"perf-node-gather-daemonset-tfjkv\" (UID: \"027a2355-0608-4d7d-b11c-c71369214fb9\") " pod="openshift-must-gather-pshc7/perf-node-gather-daemonset-tfjkv" Jul 27 19:29:31.324851 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:31.324829 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/027a2355-0608-4d7d-b11c-c71369214fb9-proc\") pod \"perf-node-gather-daemonset-tfjkv\" (UID: \"027a2355-0608-4d7d-b11c-c71369214fb9\") " pod="openshift-must-gather-pshc7/perf-node-gather-daemonset-tfjkv" Jul 27 19:29:31.325026 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:31.324869 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/027a2355-0608-4d7d-b11c-c71369214fb9-podres\") pod \"perf-node-gather-daemonset-tfjkv\" (UID: \"027a2355-0608-4d7d-b11c-c71369214fb9\") " pod="openshift-must-gather-pshc7/perf-node-gather-daemonset-tfjkv" Jul 27 19:29:31.325026 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:31.324906 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/027a2355-0608-4d7d-b11c-c71369214fb9-lib-modules\") pod \"perf-node-gather-daemonset-tfjkv\" (UID: \"027a2355-0608-4d7d-b11c-c71369214fb9\") " pod="openshift-must-gather-pshc7/perf-node-gather-daemonset-tfjkv" Jul 27 19:29:31.337250 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:31.337199 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lh5w\" (UniqueName: \"kubernetes.io/projected/027a2355-0608-4d7d-b11c-c71369214fb9-kube-api-access-7lh5w\") pod \"perf-node-gather-daemonset-tfjkv\" (UID: \"027a2355-0608-4d7d-b11c-c71369214fb9\") " pod="openshift-must-gather-pshc7/perf-node-gather-daemonset-tfjkv" Jul 27 19:29:31.489797 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:31.489706 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-pshc7/perf-node-gather-daemonset-tfjkv" Jul 27 19:29:31.632845 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:31.632818 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-pshc7/perf-node-gather-daemonset-tfjkv"] Jul 27 19:29:31.634557 ip-10-0-128-32 kubenswrapper[2579]: W0727 19:29:31.634524 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod027a2355_0608_4d7d_b11c_c71369214fb9.slice/crio-502c2243c56f193f7b66f2b88978cf1b2e06b35c3604c1a647cf08b432459623 WatchSource:0}: Error finding container 502c2243c56f193f7b66f2b88978cf1b2e06b35c3604c1a647cf08b432459623: Status 404 returned error can't find the container with id 502c2243c56f193f7b66f2b88978cf1b2e06b35c3604c1a647cf08b432459623 Jul 27 19:29:32.232540 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:32.232498 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-pshc7/perf-node-gather-daemonset-tfjkv" event={"ID":"027a2355-0608-4d7d-b11c-c71369214fb9","Type":"ContainerStarted","Data":"89080d03e8df0c40c96427776ea64878e1f13b521fdc115572b86c74d98b1757"} Jul 27 19:29:32.232923 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:32.232557 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-pshc7/perf-node-gather-daemonset-tfjkv" Jul 27 19:29:32.232923 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:32.232574 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-pshc7/perf-node-gather-daemonset-tfjkv" event={"ID":"027a2355-0608-4d7d-b11c-c71369214fb9","Type":"ContainerStarted","Data":"502c2243c56f193f7b66f2b88978cf1b2e06b35c3604c1a647cf08b432459623"} Jul 27 19:29:32.253577 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:32.253524 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-pshc7/perf-node-gather-daemonset-tfjkv" podStartSLOduration=1.253509931 podStartE2EDuration="1.253509931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 19:29:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:29:32.25197162 +0000 UTC m=+3647.313660650" watchObservedRunningTime="2026-07-27 19:29:32.253509931 +0000 UTC m=+3647.315198961" Jul 27 19:29:32.434990 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:32.434953 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-74c9f884b7-9pgxt_bfcc4ba6-b269-49a5-847a-2175dc0881fb/networking-console-plugin/0.log" Jul 27 19:29:33.027180 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:33.027135 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-mv4hb_715b2084-d79e-4cbf-a057-f2595b4e6b9b/console-operator/1.log" Jul 27 19:29:33.036666 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:33.036644 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-mv4hb_715b2084-d79e-4cbf-a057-f2595b4e6b9b/console-operator/2.log" Jul 27 19:29:34.207680 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:34.207645 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-549c6d54cf-79snb_74af2906-b939-4580-b91f-5d42c1805d74/volume-data-source-validator/0.log" Jul 27 19:29:35.394553 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:35.394524 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-pqqm5_bf48a4f7-eedd-41cc-9d53-93229f167d40/dns/0.log" Jul 27 19:29:35.418057 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:35.418030 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-pqqm5_bf48a4f7-eedd-41cc-9d53-93229f167d40/kube-rbac-proxy/0.log" Jul 27 19:29:35.569660 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:35.569627 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-ttprc_52bcf393-1ae7-4102-b762-569badf3538c/dns-node-resolver/0.log" Jul 27 19:29:36.293955 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:36.293920 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-fdsdr_8bbbb432-a482-444d-99d3-f2615e9b99bb/node-ca/0.log" Jul 27 19:29:37.444340 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:37.444308 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-w6ltt_2f32486f-5cb4-48a9-96e0-6ee08e76933a/istio-proxy/0.log" Jul 27 19:29:37.657766 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:37.657732 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7896769b64-tkbg9_2bfb0e76-8207-48bb-810a-6f865001eeb1/kube-auth-proxy/0.log" Jul 27 19:29:37.701589 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:37.701510 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-j2vmg_429930eb-3d30-450f-8e99-75ab3f80ba28/istio-proxy/0.log" Jul 27 19:29:37.751103 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:37.751076 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-j2vmg_429930eb-3d30-450f-8e99-75ab3f80ba28/istio-proxy/1.log" Jul 27 19:29:37.855853 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:37.855824 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-8rfpl_77a26a98-9824-423f-be2e-c316fc57d11b/payload-processing/0.log" Jul 27 19:29:37.885099 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:37.885075 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5c874b7d95-95vlk_2b41bffe-08d5-4b52-ae91-dcfebab44a99/router/0.log" Jul 27 19:29:38.248022 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:38.247994 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-pshc7/perf-node-gather-daemonset-tfjkv" Jul 27 19:29:38.947313 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:38.947278 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-6hbg6_e4fef94e-e2c7-4725-9da9-ddb80b15149d/serve-healthcheck-canary/0.log" Jul 27 19:29:39.672920 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:39.672883 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5885fdb759-mhdxk_b61c6434-d128-4ec0-920d-a00b085ae9c6/insights-operator/0.log" Jul 27 19:29:39.674394 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:39.674361 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5885fdb759-mhdxk_b61c6434-d128-4ec0-920d-a00b085ae9c6/insights-operator/1.log" Jul 27 19:29:39.730349 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:39.730323 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dftr7_8c36e039-1d59-4c42-8162-0f275315c3a2/kube-rbac-proxy/0.log" Jul 27 19:29:39.775756 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:39.775733 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dftr7_8c36e039-1d59-4c42-8162-0f275315c3a2/exporter/0.log" Jul 27 19:29:39.815961 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:39.815940 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dftr7_8c36e039-1d59-4c42-8162-0f275315c3a2/extractor/0.log" Jul 27 19:29:41.950829 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:41.950732 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-w6ltt_2f32486f-5cb4-48a9-96e0-6ee08e76933a/istio-proxy/0.log" Jul 27 19:29:42.137589 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:42.137555 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7896769b64-tkbg9_2bfb0e76-8207-48bb-810a-6f865001eeb1/kube-auth-proxy/0.log" Jul 27 19:29:42.167546 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:42.167515 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-j2vmg_429930eb-3d30-450f-8e99-75ab3f80ba28/istio-proxy/0.log" Jul 27 19:29:42.219325 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:42.219247 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-j2vmg_429930eb-3d30-450f-8e99-75ab3f80ba28/istio-proxy/1.log" Jul 27 19:29:42.285280 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:42.285255 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-8rfpl_77a26a98-9824-423f-be2e-c316fc57d11b/payload-processing/0.log" Jul 27 19:29:42.305953 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:42.305930 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5c874b7d95-95vlk_2b41bffe-08d5-4b52-ae91-dcfebab44a99/router/0.log" Jul 27 19:29:42.949608 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:42.949578 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-z4tz5_ebf77b9d-9e5d-47cd-aa09-ea05ff63234a/manager/0.log" Jul 27 19:29:43.335378 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:43.335347 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-w9d4l_52b6da00-246b-4954-adcc-5b482d1c41bc/manager/0.log" Jul 27 19:29:43.715365 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:43.715285 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-77ff6dccdd-sdkdx_281fa175-bbef-4309-bbaa-a3de9bf7e6a4/manager/0.log" Jul 27 19:29:43.773537 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:43.773510 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-w6ltt_2f32486f-5cb4-48a9-96e0-6ee08e76933a/istio-proxy/0.log" Jul 27 19:29:43.972564 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:43.972464 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-sgg44_87423332-edfc-4e36-a11b-56324b7db773/manager/0.log" Jul 27 19:29:43.977621 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:43.977597 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7896769b64-tkbg9_2bfb0e76-8207-48bb-810a-6f865001eeb1/kube-auth-proxy/0.log" Jul 27 19:29:44.008751 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:44.008721 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-j2vmg_429930eb-3d30-450f-8e99-75ab3f80ba28/istio-proxy/0.log" Jul 27 19:29:44.013686 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:44.013661 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-57d99c84d4-5228k_9f6495eb-fc10-4791-b32a-81e55f8b09c8/manager/0.log" Jul 27 19:29:44.039584 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:44.039553 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-57d99c84d4-5228k_9f6495eb-fc10-4791-b32a-81e55f8b09c8/copy-manifests/0.log" Jul 27 19:29:44.064785 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:44.064760 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-j2vmg_429930eb-3d30-450f-8e99-75ab3f80ba28/istio-proxy/1.log" Jul 27 19:29:44.130782 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:44.130755 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-8rfpl_77a26a98-9824-423f-be2e-c316fc57d11b/payload-processing/0.log" Jul 27 19:29:44.145433 ip-10-0-128-32 kubenswrapper[2579]: I0727 19:29:44.145408 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5c874b7d95-95vlk_2b41bffe-08d5-4b52-ae91-dcfebab44a99/router/0.log"