Jul 21 14:59:26.637511 ip-10-0-135-176 systemd[1]: Starting Kubernetes Kubelet... Jul 21 14:59:26.964094 ip-10-0-135-176 kubenswrapper[2563]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 21 14:59:26.964094 ip-10-0-135-176 kubenswrapper[2563]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 21 14:59:26.964094 ip-10-0-135-176 kubenswrapper[2563]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 21 14:59:26.964094 ip-10-0-135-176 kubenswrapper[2563]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 21 14:59:26.964094 ip-10-0-135-176 kubenswrapper[2563]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 21 14:59:26.964924 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.964834 2563 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 21 14:59:26.968867 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.968850 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 21 14:59:26.968867 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.968867 2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 21 14:59:26.968937 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.968872 2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 21 14:59:26.968937 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.968875 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 21 14:59:26.968937 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.968878 2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 21 14:59:26.968937 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.968881 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 21 14:59:26.968937 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.968885 2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 21 14:59:26.968937 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.968887 2563 feature_gate.go:328] unrecognized feature gate: Example2 Jul 21 14:59:26.968937 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.968890 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 21 14:59:26.968937 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.968892 2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 21 14:59:26.968937 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.968895 2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 21 14:59:26.968937 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.968898 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 21 14:59:26.968937 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.968901 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 21 14:59:26.968937 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.968903 2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 21 14:59:26.968937 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.968906 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 21 14:59:26.968937 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.968910 2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 21 14:59:26.968937 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.968914 2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 21 14:59:26.968937 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.968917 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 21 14:59:26.968937 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.968920 2563 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 21 14:59:26.968937 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.968923 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 21 14:59:26.968937 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.968926 2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 21 14:59:26.968937 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.968928 2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 21 14:59:26.969416 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.968932 2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 21 14:59:26.969416 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.968934 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 21 14:59:26.969416 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.968939 2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 21 14:59:26.969416 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.968942 2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 21 14:59:26.969416 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.968945 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 21 14:59:26.969416 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.968948 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 21 14:59:26.969416 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.968952 2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 21 14:59:26.969416 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.968954 2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 21 14:59:26.969416 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.968957 2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 21 14:59:26.969416 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.968959 2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 21 14:59:26.969416 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.968962 2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 21 14:59:26.969416 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.968964 2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 21 14:59:26.969416 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.968967 2563 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 21 14:59:26.969416 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.968970 2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 21 14:59:26.969416 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.968972 2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 21 14:59:26.969416 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.968975 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 21 14:59:26.969416 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.968977 2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 21 14:59:26.969416 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.968979 2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 21 14:59:26.969416 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.968982 2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 21 14:59:26.969416 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.968985 2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 21 14:59:26.969944 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.968987 2563 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 21 14:59:26.969944 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.968991 2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 21 14:59:26.969944 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.968993 2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 21 14:59:26.969944 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.968996 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 21 14:59:26.969944 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.968998 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 21 14:59:26.969944 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.969000 2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 21 14:59:26.969944 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.969003 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 21 14:59:26.969944 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.969005 2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 21 14:59:26.969944 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.969008 2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 21 14:59:26.969944 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.969010 2563 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 21 14:59:26.969944 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.969012 2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 21 14:59:26.969944 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.969015 2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 21 14:59:26.969944 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.969017 2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 21 14:59:26.969944 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.969021 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 21 14:59:26.969944 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.969023 2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 21 14:59:26.969944 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.969026 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 21 14:59:26.969944 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.969028 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 21 14:59:26.969944 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.969031 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 21 14:59:26.969944 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.969034 2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 21 14:59:26.970437 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.969036 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 21 14:59:26.970437 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.969039 2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 21 14:59:26.970437 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.969041 2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 21 14:59:26.970437 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.969044 2563 feature_gate.go:328] unrecognized feature gate: Example Jul 21 14:59:26.970437 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.969046 2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 21 14:59:26.970437 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.969049 2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 21 14:59:26.970437 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.969052 2563 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 21 14:59:26.970437 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.969055 2563 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 21 14:59:26.970437 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.969057 2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 21 14:59:26.970437 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.969060 2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 21 14:59:26.970437 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.969062 2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 21 14:59:26.970437 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.969064 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 21 14:59:26.970437 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.969068 2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 21 14:59:26.970437 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.969071 2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 21 14:59:26.970437 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.969074 2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 21 14:59:26.970437 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.969077 2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 21 14:59:26.970437 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.969079 2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 21 14:59:26.970437 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.969082 2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 21 14:59:26.970437 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.969084 2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 21 14:59:26.971131 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.969086 2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 21 14:59:26.971131 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.969090 2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 21 14:59:26.971131 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.969092 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 21 14:59:26.971131 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.969095 2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 21 14:59:26.971131 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.969098 2563 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 21 14:59:26.971131 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.969100 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 21 14:59:26.971131 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.969103 2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 21 14:59:26.971131 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.969106 2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 21 14:59:26.971131 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.969109 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 21 14:59:26.971131 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.969111 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 21 14:59:26.971131 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.969114 2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 21 14:59:26.971131 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.969116 2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 21 14:59:26.971131 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.969119 2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 21 14:59:26.971131 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.969121 2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 21 14:59:26.971131 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.969124 2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 21 14:59:26.971131 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.969126 2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 21 14:59:26.971131 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.969128 2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 21 14:59:26.971131 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.969131 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 21 14:59:26.971131 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.969134 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 21 14:59:26.971131 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.969841 2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 21 14:59:26.971745 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.969849 2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 21 14:59:26.971745 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.969853 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 21 14:59:26.971745 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.969856 2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 21 14:59:26.971745 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.969859 2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 21 14:59:26.971745 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.969861 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 21 14:59:26.971745 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.969864 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 21 14:59:26.971745 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.969866 2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 21 14:59:26.971745 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.969869 2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 21 14:59:26.971745 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.969872 2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 21 14:59:26.971745 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.969874 2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 21 14:59:26.971745 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.969876 2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 21 14:59:26.971745 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.969879 2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 21 14:59:26.971745 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.969881 2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 21 14:59:26.971745 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.969884 2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 21 14:59:26.971745 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.969886 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 21 14:59:26.971745 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.969889 2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 21 14:59:26.971745 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.969892 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 21 14:59:26.971745 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.969895 2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 21 14:59:26.971745 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.969898 2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 21 14:59:26.971745 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.969901 2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 21 14:59:26.972273 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.969904 2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 21 14:59:26.972273 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.969906 2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 21 14:59:26.972273 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.969909 2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 21 14:59:26.972273 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.969911 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 21 14:59:26.972273 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.969914 2563 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 21 14:59:26.972273 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.969917 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 21 14:59:26.972273 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.969919 2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 21 14:59:26.972273 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.969921 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 21 14:59:26.972273 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.969924 2563 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 21 14:59:26.972273 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.969926 2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 21 14:59:26.972273 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.969929 2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 21 14:59:26.972273 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.969931 2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 21 14:59:26.972273 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.969934 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 21 14:59:26.972273 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.969936 2563 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 21 14:59:26.972273 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.969945 2563 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 21 14:59:26.972273 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.969948 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 21 14:59:26.972273 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.969951 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 21 14:59:26.972273 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.969954 2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 21 14:59:26.972273 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.969957 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 21 14:59:26.972273 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.969959 2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 21 14:59:26.972808 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.969962 2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 21 14:59:26.972808 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.969964 2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 21 14:59:26.972808 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.969966 2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 21 14:59:26.972808 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.969969 2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 21 14:59:26.972808 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.969972 2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 21 14:59:26.972808 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.969974 2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 21 14:59:26.972808 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.969976 2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 21 14:59:26.972808 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.969979 2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 21 14:59:26.972808 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.969981 2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 21 14:59:26.972808 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.969984 2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 21 14:59:26.972808 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.969987 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 21 14:59:26.972808 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.969989 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 21 14:59:26.972808 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.969992 2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 21 14:59:26.972808 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.969995 2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 21 14:59:26.972808 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.969997 2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 21 14:59:26.972808 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.970000 2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 21 14:59:26.972808 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.970002 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 21 14:59:26.972808 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.970005 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 21 14:59:26.972808 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.970008 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 21 14:59:26.973291 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.970010 2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 21 14:59:26.973291 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.970013 2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 21 14:59:26.973291 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.970017 2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 21 14:59:26.973291 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.970020 2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 21 14:59:26.973291 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.970023 2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 21 14:59:26.973291 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.970026 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 21 14:59:26.973291 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.970029 2563 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 21 14:59:26.973291 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.970032 2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 21 14:59:26.973291 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.970035 2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 21 14:59:26.973291 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.970037 2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 21 14:59:26.973291 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.970040 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 21 14:59:26.973291 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.970043 2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 21 14:59:26.973291 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.970046 2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 21 14:59:26.973291 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.970048 2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 21 14:59:26.973291 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.970051 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 21 14:59:26.973291 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.970054 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 21 14:59:26.973291 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.970057 2563 feature_gate.go:328] unrecognized feature gate: Example Jul 21 14:59:26.973291 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.970059 2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 21 14:59:26.973291 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.970061 2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 21 14:59:26.973858 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.970068 2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 21 14:59:26.973858 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.970071 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 21 14:59:26.973858 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.970073 2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 21 14:59:26.973858 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.970075 2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 21 14:59:26.973858 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.970079 2563 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 21 14:59:26.973858 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.970081 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 21 14:59:26.973858 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.970084 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 21 14:59:26.973858 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.970087 2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 21 14:59:26.973858 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.970089 2563 feature_gate.go:328] unrecognized feature gate: Example2 Jul 21 14:59:26.973858 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.970093 2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 21 14:59:26.973858 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.970096 2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 21 14:59:26.973858 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.970099 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 21 14:59:26.973858 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.970102 2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 21 14:59:26.973858 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.970104 2563 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 21 14:59:26.973858 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.970107 2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 21 14:59:26.973858 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.970109 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 21 14:59:26.973858 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.970112 2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 21 14:59:26.973858 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.970114 2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 21 14:59:26.973858 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.970117 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 21 14:59:26.973858 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.970119 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 21 14:59:26.974389 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.971901 2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 21 14:59:26.974389 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.971913 2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 21 14:59:26.974389 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.971916 2563 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 21 14:59:26.974389 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.971920 2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 21 14:59:26.974389 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.971923 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 21 14:59:26.974389 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.971926 2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 21 14:59:26.974389 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.971929 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 21 14:59:26.974389 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.971932 2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 21 14:59:26.974389 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.971935 2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 21 14:59:26.974389 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.971937 2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 21 14:59:26.974389 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.971940 2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 21 14:59:26.974389 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.971943 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 21 14:59:26.974389 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.971945 2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 21 14:59:26.974389 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.971948 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 21 14:59:26.974389 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.971950 2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 21 14:59:26.974389 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.971953 2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 21 14:59:26.974389 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.971956 2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 21 14:59:26.974389 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.971958 2563 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 21 14:59:26.974389 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.971961 2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 21 14:59:26.974860 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.971964 2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 21 14:59:26.974860 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.971966 2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 21 14:59:26.974860 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.971968 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 21 14:59:26.974860 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.971971 2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 21 14:59:26.974860 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.971973 2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 21 14:59:26.974860 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.971976 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 21 14:59:26.974860 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.971978 2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 21 14:59:26.974860 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.971981 2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 21 14:59:26.974860 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.971984 2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 21 14:59:26.974860 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.971986 2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 21 14:59:26.974860 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.971990 2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 21 14:59:26.974860 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.971994 2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 21 14:59:26.974860 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.971998 2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 21 14:59:26.974860 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.972002 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 21 14:59:26.974860 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.972006 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 21 14:59:26.974860 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.972010 2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 21 14:59:26.974860 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.972013 2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 21 14:59:26.974860 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.972016 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 21 14:59:26.974860 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.972019 2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 21 14:59:26.975343 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.972022 2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 21 14:59:26.975343 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.972025 2563 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 21 14:59:26.975343 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.972027 2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 21 14:59:26.975343 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.972030 2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 21 14:59:26.975343 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.972033 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 21 14:59:26.975343 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.972036 2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 21 14:59:26.975343 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.972039 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 21 14:59:26.975343 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.972042 2563 feature_gate.go:328] unrecognized feature gate: Example2 Jul 21 14:59:26.975343 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.972044 2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 21 14:59:26.975343 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.972047 2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 21 14:59:26.975343 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.972049 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 21 14:59:26.975343 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.972052 2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 21 14:59:26.975343 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.972055 2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 21 14:59:26.975343 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.972057 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 21 14:59:26.975343 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.972060 2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 21 14:59:26.975343 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.972062 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 21 14:59:26.975343 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.972065 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 21 14:59:26.975343 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.972068 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 21 14:59:26.975343 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.972070 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 21 14:59:26.975343 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.972073 2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 21 14:59:26.975831 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.972076 2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 21 14:59:26.975831 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.972078 2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 21 14:59:26.975831 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.972081 2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 21 14:59:26.975831 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.972083 2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 21 14:59:26.975831 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.972085 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 21 14:59:26.975831 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.972088 2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 21 14:59:26.975831 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.972090 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 21 14:59:26.975831 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.972092 2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 21 14:59:26.975831 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.972095 2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 21 14:59:26.975831 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.972098 2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 21 14:59:26.975831 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.972100 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 21 14:59:26.975831 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.972102 2563 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 21 14:59:26.975831 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.972105 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 21 14:59:26.975831 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.972107 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 21 14:59:26.975831 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.972110 2563 feature_gate.go:328] unrecognized feature gate: Example Jul 21 14:59:26.975831 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.972112 2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 21 14:59:26.975831 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.972114 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 21 14:59:26.975831 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.972117 2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 21 14:59:26.975831 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.972119 2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 21 14:59:26.975831 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.972121 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 21 14:59:26.976335 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.972124 2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 21 14:59:26.976335 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.972126 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 21 14:59:26.976335 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.972129 2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 21 14:59:26.976335 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.972131 2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 21 14:59:26.976335 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.972133 2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 21 14:59:26.976335 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.972139 2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 21 14:59:26.976335 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.972142 2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 21 14:59:26.976335 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.972144 2563 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 21 14:59:26.976335 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.972147 2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 21 14:59:26.976335 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.972150 2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 21 14:59:26.976335 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.972153 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 21 14:59:26.976335 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.972173 2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 21 14:59:26.976335 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.972176 2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 21 14:59:26.976335 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.972179 2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 21 14:59:26.976335 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.972182 2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 21 14:59:26.976335 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.972184 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 21 14:59:26.976335 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.972187 2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 21 14:59:26.976335 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.972189 2563 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 21 14:59:26.976335 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.972192 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 21 14:59:26.976799 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.972195 2563 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 21 14:59:26.976799 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.972198 2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 21 14:59:26.976799 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.972280 2563 flags.go:64] FLAG: --address="0.0.0.0" Jul 21 14:59:26.976799 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.972291 2563 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 21 14:59:26.976799 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.972300 2563 flags.go:64] FLAG: --anonymous-auth="true" Jul 21 14:59:26.976799 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.972306 2563 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 21 14:59:26.976799 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.972312 2563 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 21 14:59:26.976799 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.972317 2563 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 21 14:59:26.976799 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.972321 2563 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 21 14:59:26.976799 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.972325 2563 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 21 14:59:26.976799 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.972328 2563 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 21 14:59:26.976799 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.972331 2563 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 21 14:59:26.976799 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.972335 2563 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 21 14:59:26.976799 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.972338 2563 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 21 14:59:26.976799 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.972341 2563 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 21 14:59:26.976799 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.972344 2563 flags.go:64] FLAG: --cgroup-root="" Jul 21 14:59:26.976799 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.972346 2563 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 21 14:59:26.976799 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.972349 2563 flags.go:64] FLAG: --client-ca-file="" Jul 21 14:59:26.976799 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.972352 2563 flags.go:64] FLAG: --cloud-provider="external" Jul 21 14:59:26.976799 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.972356 2563 flags.go:64] FLAG: --cluster-dns="[]" Jul 21 14:59:26.976799 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.972360 2563 flags.go:64] FLAG: --cluster-domain="" Jul 21 14:59:26.976799 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.972364 2563 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 21 14:59:26.976799 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.972367 2563 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 21 14:59:26.976799 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.972370 2563 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 21 14:59:26.977383 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.972374 2563 flags.go:64] FLAG: --container-log-max-files="5" Jul 21 14:59:26.977383 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.972378 2563 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 21 14:59:26.977383 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.972381 2563 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 21 14:59:26.977383 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.972384 2563 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 21 14:59:26.977383 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.972387 2563 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 21 14:59:26.977383 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.972390 2563 flags.go:64] FLAG: --contention-profiling="false" Jul 21 14:59:26.977383 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.972393 2563 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 21 14:59:26.977383 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.972396 2563 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 21 14:59:26.977383 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.972399 2563 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 21 14:59:26.977383 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.972402 2563 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 21 14:59:26.977383 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.972407 2563 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 21 14:59:26.977383 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.972409 2563 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 21 14:59:26.977383 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.972412 2563 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 21 14:59:26.977383 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.972415 2563 flags.go:64] FLAG: --enable-load-reader="false" Jul 21 14:59:26.977383 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.972418 2563 flags.go:64] FLAG: --enable-server="true" Jul 21 14:59:26.977383 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.972424 2563 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 21 14:59:26.977383 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.972430 2563 flags.go:64] FLAG: --event-burst="100" Jul 21 14:59:26.977383 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.972433 2563 flags.go:64] FLAG: --event-qps="50" Jul 21 14:59:26.977383 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.972436 2563 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 21 14:59:26.977383 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.972439 2563 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 21 14:59:26.977383 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.972442 2563 flags.go:64] FLAG: --eviction-hard="" Jul 21 14:59:26.977383 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.972445 2563 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 21 14:59:26.977383 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.972448 2563 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 21 14:59:26.977383 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.972451 2563 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 21 14:59:26.977383 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.972453 2563 flags.go:64] FLAG: --eviction-soft="" Jul 21 14:59:26.977987 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.972457 2563 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 21 14:59:26.977987 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.972459 2563 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 21 14:59:26.977987 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.972462 2563 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 21 14:59:26.977987 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.972467 2563 flags.go:64] FLAG: --experimental-mounter-path="" Jul 21 14:59:26.977987 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.972470 2563 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 21 14:59:26.977987 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.972472 2563 flags.go:64] FLAG: --fail-swap-on="true" Jul 21 14:59:26.977987 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.972475 2563 flags.go:64] FLAG: --feature-gates="" Jul 21 14:59:26.977987 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.972479 2563 flags.go:64] FLAG: --file-check-frequency="20s" Jul 21 14:59:26.977987 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.972482 2563 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 21 14:59:26.977987 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.972485 2563 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 21 14:59:26.977987 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.972488 2563 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 21 14:59:26.977987 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.972491 2563 flags.go:64] FLAG: --healthz-port="10248" Jul 21 14:59:26.977987 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.972494 2563 flags.go:64] FLAG: --help="false" Jul 21 14:59:26.977987 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.972496 2563 flags.go:64] FLAG: --hostname-override="ip-10-0-135-176.ec2.internal" Jul 21 14:59:26.977987 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.972500 2563 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 21 14:59:26.977987 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.972502 2563 flags.go:64] FLAG: --http-check-frequency="20s" Jul 21 14:59:26.977987 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.972505 2563 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 21 14:59:26.977987 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.972508 2563 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 21 14:59:26.977987 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.972511 2563 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 21 14:59:26.977987 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.972514 2563 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 21 14:59:26.977987 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.972517 2563 flags.go:64] FLAG: --image-service-endpoint="" Jul 21 14:59:26.977987 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.972520 2563 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 21 14:59:26.977987 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.972524 2563 flags.go:64] FLAG: --kube-api-burst="100" Jul 21 14:59:26.978673 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.972527 2563 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 21 14:59:26.978673 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.972530 2563 flags.go:64] FLAG: --kube-api-qps="50" Jul 21 14:59:26.978673 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.972532 2563 flags.go:64] FLAG: --kube-reserved="" Jul 21 14:59:26.978673 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.972535 2563 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 21 14:59:26.978673 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.972538 2563 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 21 14:59:26.978673 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.972541 2563 flags.go:64] FLAG: --kubelet-cgroups="" Jul 21 14:59:26.978673 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.972543 2563 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 21 14:59:26.978673 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.972546 2563 flags.go:64] FLAG: --lock-file="" Jul 21 14:59:26.978673 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.972549 2563 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 21 14:59:26.978673 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.972551 2563 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 21 14:59:26.978673 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.972554 2563 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 21 14:59:26.978673 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.972560 2563 flags.go:64] FLAG: --log-json-split-stream="false" Jul 21 14:59:26.978673 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.972563 2563 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 21 14:59:26.978673 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.972566 2563 flags.go:64] FLAG: --log-text-split-stream="false" Jul 21 14:59:26.978673 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.972569 2563 flags.go:64] FLAG: --logging-format="text" Jul 21 14:59:26.978673 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.972572 2563 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 21 14:59:26.978673 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.972575 2563 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 21 14:59:26.978673 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.972578 2563 flags.go:64] FLAG: --manifest-url="" Jul 21 14:59:26.978673 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.972580 2563 flags.go:64] FLAG: --manifest-url-header="" Jul 21 14:59:26.978673 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.972585 2563 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 21 14:59:26.978673 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.972588 2563 flags.go:64] FLAG: --max-open-files="1000000" Jul 21 14:59:26.978673 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.972592 2563 flags.go:64] FLAG: --max-pods="110" Jul 21 14:59:26.978673 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.972595 2563 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 21 14:59:26.978673 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.972598 2563 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 21 14:59:26.978673 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.972601 2563 flags.go:64] FLAG: --memory-manager-policy="None" Jul 21 14:59:26.979315 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.972604 2563 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 21 14:59:26.979315 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.972607 2563 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 21 14:59:26.979315 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.972609 2563 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 21 14:59:26.979315 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.972612 2563 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 21 14:59:26.979315 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.972619 2563 flags.go:64] FLAG: --node-status-max-images="50" Jul 21 14:59:26.979315 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.972622 2563 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 21 14:59:26.979315 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.972626 2563 flags.go:64] FLAG: --oom-score-adj="-999" Jul 21 14:59:26.979315 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.972629 2563 flags.go:64] FLAG: --pod-cidr="" Jul 21 14:59:26.979315 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.972632 2563 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0" Jul 21 14:59:26.979315 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.972638 2563 flags.go:64] FLAG: --pod-manifest-path="" Jul 21 14:59:26.979315 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.972641 2563 flags.go:64] FLAG: --pod-max-pids="-1" Jul 21 14:59:26.979315 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.972644 2563 flags.go:64] FLAG: --pods-per-core="0" Jul 21 14:59:26.979315 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.972646 2563 flags.go:64] FLAG: --port="10250" Jul 21 14:59:26.979315 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.972649 2563 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 21 14:59:26.979315 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.972652 2563 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-09090594da636623f" Jul 21 14:59:26.979315 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.972655 2563 flags.go:64] FLAG: --qos-reserved="" Jul 21 14:59:26.979315 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.972657 2563 flags.go:64] FLAG: --read-only-port="10255" Jul 21 14:59:26.979315 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.972660 2563 flags.go:64] FLAG: --register-node="true" Jul 21 14:59:26.979315 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.972663 2563 flags.go:64] FLAG: --register-with-taints="" Jul 21 14:59:26.979315 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.972666 2563 flags.go:64] FLAG: --registry-burst="10" Jul 21 14:59:26.979315 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.972669 2563 flags.go:64] FLAG: --registry-qps="5" Jul 21 14:59:26.979315 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.972673 2563 flags.go:64] FLAG: --reserved-cpus="" Jul 21 14:59:26.979315 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.972675 2563 flags.go:64] FLAG: --reserved-memory="" Jul 21 14:59:26.979315 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.972679 2563 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 21 14:59:26.979315 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.972682 2563 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 21 14:59:26.979910 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.972685 2563 flags.go:64] FLAG: --rotate-certificates="false" Jul 21 14:59:26.979910 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.972688 2563 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 21 14:59:26.979910 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.972691 2563 flags.go:64] FLAG: --runonce="false" Jul 21 14:59:26.979910 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.972696 2563 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 21 14:59:26.979910 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.972699 2563 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 21 14:59:26.979910 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.972702 2563 flags.go:64] FLAG: --seccomp-default="false" Jul 21 14:59:26.979910 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.972705 2563 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 21 14:59:26.979910 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.972707 2563 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 21 14:59:26.979910 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.972711 2563 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 21 14:59:26.979910 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.972714 2563 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 21 14:59:26.979910 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.972717 2563 flags.go:64] FLAG: --storage-driver-password="root" Jul 21 14:59:26.979910 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.972719 2563 flags.go:64] FLAG: --storage-driver-secure="false" Jul 21 14:59:26.979910 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.972722 2563 flags.go:64] FLAG: --storage-driver-table="stats" Jul 21 14:59:26.979910 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.972727 2563 flags.go:64] FLAG: --storage-driver-user="root" Jul 21 14:59:26.979910 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.972729 2563 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 21 14:59:26.979910 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.972732 2563 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 21 14:59:26.979910 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.972735 2563 flags.go:64] FLAG: --system-cgroups="" Jul 21 14:59:26.979910 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.972738 2563 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 21 14:59:26.979910 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.972743 2563 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 21 14:59:26.979910 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.972746 2563 flags.go:64] FLAG: --tls-cert-file="" Jul 21 14:59:26.979910 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.972749 2563 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 21 14:59:26.979910 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.972753 2563 flags.go:64] FLAG: --tls-min-version="" Jul 21 14:59:26.979910 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.972755 2563 flags.go:64] FLAG: --tls-private-key-file="" Jul 21 14:59:26.979910 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.972758 2563 flags.go:64] FLAG: --topology-manager-policy="none" Jul 21 14:59:26.979910 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.972761 2563 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 21 14:59:26.980532 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.972763 2563 flags.go:64] FLAG: --topology-manager-scope="container" Jul 21 14:59:26.980532 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.972766 2563 flags.go:64] FLAG: --v="2" Jul 21 14:59:26.980532 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.972770 2563 flags.go:64] FLAG: --version="false" Jul 21 14:59:26.980532 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.972774 2563 flags.go:64] FLAG: --vmodule="" Jul 21 14:59:26.980532 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.972779 2563 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 21 14:59:26.980532 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.972782 2563 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 21 14:59:26.980532 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.972911 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 21 14:59:26.980532 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.972915 2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 21 14:59:26.980532 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.972918 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 21 14:59:26.980532 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.972921 2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 21 14:59:26.980532 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.972925 2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 21 14:59:26.980532 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.972929 2563 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 21 14:59:26.980532 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.972931 2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 21 14:59:26.980532 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.972934 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 21 14:59:26.980532 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.972937 2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 21 14:59:26.980532 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.972939 2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 21 14:59:26.980532 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.972942 2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 21 14:59:26.980532 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.972944 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 21 14:59:26.980532 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.972949 2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 21 14:59:26.980532 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.972952 2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 21 14:59:26.980532 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.972956 2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 21 14:59:26.981034 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.972959 2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 21 14:59:26.981034 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.972963 2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 21 14:59:26.981034 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.972966 2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 21 14:59:26.981034 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.972969 2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 21 14:59:26.981034 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.972973 2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 21 14:59:26.981034 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.972975 2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 21 14:59:26.981034 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.972978 2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 21 14:59:26.981034 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.972981 2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 21 14:59:26.981034 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.972984 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 21 14:59:26.981034 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.972987 2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 21 14:59:26.981034 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.972990 2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 21 14:59:26.981034 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.972992 2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 21 14:59:26.981034 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.972995 2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 21 14:59:26.981034 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.972998 2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 21 14:59:26.981034 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.973000 2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 21 14:59:26.981034 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.973003 2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 21 14:59:26.981034 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.973005 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 21 14:59:26.981034 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.973008 2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 21 14:59:26.981034 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.973011 2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 21 14:59:26.981504 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.973013 2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 21 14:59:26.981504 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.973015 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 21 14:59:26.981504 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.973019 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 21 14:59:26.981504 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.973022 2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 21 14:59:26.981504 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.973024 2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 21 14:59:26.981504 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.973026 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 21 14:59:26.981504 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.973029 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 21 14:59:26.981504 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.973031 2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 21 14:59:26.981504 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.973034 2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 21 14:59:26.981504 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.973037 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 21 14:59:26.981504 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.973039 2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 21 14:59:26.981504 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.973042 2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 21 14:59:26.981504 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.973044 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 21 14:59:26.981504 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.973047 2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 21 14:59:26.981504 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.973049 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 21 14:59:26.981504 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.973051 2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 21 14:59:26.981504 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.973054 2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 21 14:59:26.981504 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.973056 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 21 14:59:26.981504 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.973059 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 21 14:59:26.981504 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.973061 2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 21 14:59:26.982032 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.973063 2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 21 14:59:26.982032 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.973066 2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 21 14:59:26.982032 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.973068 2563 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 21 14:59:26.982032 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.973071 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 21 14:59:26.982032 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.973073 2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 21 14:59:26.982032 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.973075 2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 21 14:59:26.982032 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.973078 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 21 14:59:26.982032 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.973081 2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 21 14:59:26.982032 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.973083 2563 feature_gate.go:328] unrecognized feature gate: Example Jul 21 14:59:26.982032 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.973085 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 21 14:59:26.982032 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.973088 2563 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 21 14:59:26.982032 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.973091 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 21 14:59:26.982032 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.973093 2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 21 14:59:26.982032 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.973096 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 21 14:59:26.982032 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.973100 2563 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 21 14:59:26.982032 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.973102 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 21 14:59:26.982032 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.973104 2563 feature_gate.go:328] unrecognized feature gate: Example2 Jul 21 14:59:26.982032 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.973107 2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 21 14:59:26.982032 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.973109 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 21 14:59:26.982032 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.973112 2563 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 21 14:59:26.982032 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.973114 2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 21 14:59:26.982621 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.973117 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 21 14:59:26.982621 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.973119 2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 21 14:59:26.982621 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.973122 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 21 14:59:26.982621 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.973125 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 21 14:59:26.982621 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.973128 2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 21 14:59:26.982621 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.973132 2563 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 21 14:59:26.982621 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.973134 2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 21 14:59:26.982621 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.973137 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 21 14:59:26.982621 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.973139 2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 21 14:59:26.982621 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.973142 2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 21 14:59:26.982621 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.973144 2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 21 14:59:26.982621 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.973147 2563 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 21 14:59:26.982621 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.973149 2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 21 14:59:26.982621 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.973152 2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 21 14:59:26.982621 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.973169 2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 21 14:59:26.982621 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.973172 2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 21 14:59:26.982621 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.973174 2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 21 14:59:26.982621 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.973177 2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 21 14:59:26.982621 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.973179 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 21 14:59:26.982621 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.973182 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 21 14:59:26.983108 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.973184 2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 21 14:59:26.983108 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.973187 2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 21 14:59:26.983108 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.973194 2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 21 14:59:26.983108 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.973196 2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 21 14:59:26.983108 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.973840 2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 21 14:59:26.983108 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.980664 2563 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 21 14:59:26.983108 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.980680 2563 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 21 14:59:26.983108 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.980701 2563 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 21 14:59:26.983108 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.980710 2563 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 21 14:59:26.983108 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.980778 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 21 14:59:26.983108 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.980785 2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 21 14:59:26.983108 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.980788 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 21 14:59:26.983108 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.980791 2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 21 14:59:26.983108 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.980794 2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 21 14:59:26.983475 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.980796 2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 21 14:59:26.983475 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.980799 2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 21 14:59:26.983475 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.980802 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 21 14:59:26.983475 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.980804 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 21 14:59:26.983475 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.980807 2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 21 14:59:26.983475 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.980810 2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 21 14:59:26.983475 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.980813 2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 21 14:59:26.983475 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.980816 2563 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 21 14:59:26.983475 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.980818 2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 21 14:59:26.983475 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.980821 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 21 14:59:26.983475 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.980824 2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 21 14:59:26.983475 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.980826 2563 feature_gate.go:328] unrecognized feature gate: Example Jul 21 14:59:26.983475 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.980829 2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 21 14:59:26.983475 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.980831 2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 21 14:59:26.983475 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.980835 2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 21 14:59:26.983475 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.980839 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 21 14:59:26.983475 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.980842 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 21 14:59:26.983475 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.980845 2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 21 14:59:26.983475 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.980847 2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 21 14:59:26.983475 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.980849 2563 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 21 14:59:26.984013 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.980852 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 21 14:59:26.984013 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.980854 2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 21 14:59:26.984013 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.980857 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 21 14:59:26.984013 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.980859 2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 21 14:59:26.984013 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.980863 2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 21 14:59:26.984013 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.980866 2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 21 14:59:26.984013 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.980869 2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 21 14:59:26.984013 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.980871 2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 21 14:59:26.984013 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.980874 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 21 14:59:26.984013 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.980876 2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 21 14:59:26.984013 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.980879 2563 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 21 14:59:26.984013 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.980881 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 21 14:59:26.984013 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.980884 2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 21 14:59:26.984013 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.980886 2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 21 14:59:26.984013 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.980888 2563 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 21 14:59:26.984013 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.980891 2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 21 14:59:26.984013 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.980893 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 21 14:59:26.984013 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.980896 2563 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 21 14:59:26.984013 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.980898 2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 21 14:59:26.984013 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.980901 2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 21 14:59:26.984519 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.980903 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 21 14:59:26.984519 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.980906 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 21 14:59:26.984519 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.980908 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 21 14:59:26.984519 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.980911 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 21 14:59:26.984519 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.980913 2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 21 14:59:26.984519 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.980916 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 21 14:59:26.984519 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.980919 2563 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 21 14:59:26.984519 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.980921 2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 21 14:59:26.984519 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.980924 2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 21 14:59:26.984519 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.980926 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 21 14:59:26.984519 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.980929 2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 21 14:59:26.984519 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.980931 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 21 14:59:26.984519 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.980934 2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 21 14:59:26.984519 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.980936 2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 21 14:59:26.984519 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.980939 2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 21 14:59:26.984519 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.980941 2563 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 21 14:59:26.984519 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.980944 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 21 14:59:26.984519 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.980947 2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 21 14:59:26.984519 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.980950 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 21 14:59:26.984519 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.980953 2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 21 14:59:26.984990 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.980955 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 21 14:59:26.984990 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.980957 2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 21 14:59:26.984990 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.980960 2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 21 14:59:26.984990 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.980964 2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 21 14:59:26.984990 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.980967 2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 21 14:59:26.984990 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.980970 2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 21 14:59:26.984990 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.980973 2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 21 14:59:26.984990 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.980976 2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 21 14:59:26.984990 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.980979 2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 21 14:59:26.984990 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.980981 2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 21 14:59:26.984990 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.980984 2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 21 14:59:26.984990 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.980987 2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 21 14:59:26.984990 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.980989 2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 21 14:59:26.984990 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.980992 2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 21 14:59:26.984990 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.980994 2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 21 14:59:26.984990 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.980997 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 21 14:59:26.984990 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.981000 2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 21 14:59:26.984990 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.981002 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 21 14:59:26.984990 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.981005 2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 21 14:59:26.985530 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.981008 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 21 14:59:26.985530 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.981010 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 21 14:59:26.985530 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.981013 2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 21 14:59:26.985530 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.981015 2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 21 14:59:26.985530 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.981018 2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 21 14:59:26.985530 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.981021 2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 21 14:59:26.985530 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.981023 2563 feature_gate.go:328] unrecognized feature gate: Example2 Jul 21 14:59:26.985530 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.981026 2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 21 14:59:26.985530 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.981029 2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 21 14:59:26.985530 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.981031 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 21 14:59:26.985530 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.981034 2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 21 14:59:26.985530 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.981038 2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 21 14:59:26.985530 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.981040 2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 21 14:59:26.985530 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.981043 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 21 14:59:26.985530 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.981047 2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 21 14:59:26.985896 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.981052 2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 21 14:59:26.985896 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.981199 2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 21 14:59:26.985896 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.981205 2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 21 14:59:26.985896 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.981208 2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 21 14:59:26.985896 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.981211 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 21 14:59:26.985896 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.981214 2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 21 14:59:26.985896 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.981216 2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 21 14:59:26.985896 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.981219 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 21 14:59:26.985896 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.981221 2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 21 14:59:26.985896 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.981224 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 21 14:59:26.985896 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.981226 2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 21 14:59:26.985896 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.981229 2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 21 14:59:26.985896 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.981231 2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 21 14:59:26.985896 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.981234 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 21 14:59:26.985896 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.981236 2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 21 14:59:26.986363 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.981239 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 21 14:59:26.986363 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.981242 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 21 14:59:26.986363 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.981244 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 21 14:59:26.986363 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.981247 2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 21 14:59:26.986363 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.981249 2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 21 14:59:26.986363 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.981252 2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 21 14:59:26.986363 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.981254 2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 21 14:59:26.986363 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.981257 2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 21 14:59:26.986363 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.981259 2563 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 21 14:59:26.986363 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.981262 2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 21 14:59:26.986363 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.981264 2563 feature_gate.go:328] unrecognized feature gate: Example Jul 21 14:59:26.986363 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.981267 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 21 14:59:26.986363 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.981269 2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 21 14:59:26.986363 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.981272 2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 21 14:59:26.986363 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.981275 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 21 14:59:26.986363 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.981277 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 21 14:59:26.986363 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.981280 2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 21 14:59:26.986363 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.981282 2563 feature_gate.go:328] unrecognized feature gate: Example2 Jul 21 14:59:26.986363 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.981286 2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 21 14:59:26.986843 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.981289 2563 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 21 14:59:26.986843 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.981292 2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 21 14:59:26.986843 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.981295 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 21 14:59:26.986843 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.981297 2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 21 14:59:26.986843 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.981299 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 21 14:59:26.986843 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.981302 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 21 14:59:26.986843 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.981304 2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 21 14:59:26.986843 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.981307 2563 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 21 14:59:26.986843 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.981310 2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 21 14:59:26.986843 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.981313 2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 21 14:59:26.986843 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.981316 2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 21 14:59:26.986843 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.981336 2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 21 14:59:26.986843 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.981340 2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 21 14:59:26.986843 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.981343 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 21 14:59:26.986843 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.981346 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 21 14:59:26.986843 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.981348 2563 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 21 14:59:26.986843 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.981351 2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 21 14:59:26.986843 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.981353 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 21 14:59:26.986843 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.981356 2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 21 14:59:26.986843 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.981358 2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 21 14:59:26.987339 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.981361 2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 21 14:59:26.987339 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.981363 2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 21 14:59:26.987339 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.981366 2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 21 14:59:26.987339 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.981368 2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 21 14:59:26.987339 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.981371 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 21 14:59:26.987339 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.981373 2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 21 14:59:26.987339 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.981376 2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 21 14:59:26.987339 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.981378 2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 21 14:59:26.987339 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.981381 2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 21 14:59:26.987339 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.981384 2563 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 21 14:59:26.987339 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.981386 2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 21 14:59:26.987339 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.981388 2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 21 14:59:26.987339 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.981391 2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 21 14:59:26.987339 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.981393 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 21 14:59:26.987339 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.981396 2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 21 14:59:26.987339 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.981398 2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 21 14:59:26.987339 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.981402 2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 21 14:59:26.987339 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.981404 2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 21 14:59:26.987339 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.981407 2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 21 14:59:26.987339 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.981410 2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 21 14:59:26.987907 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.981413 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 21 14:59:26.987907 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.981416 2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 21 14:59:26.987907 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.981418 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 21 14:59:26.987907 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.981421 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 21 14:59:26.987907 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.981424 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 21 14:59:26.987907 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.981426 2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 21 14:59:26.987907 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.981429 2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 21 14:59:26.987907 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.981431 2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 21 14:59:26.987907 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.981434 2563 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 21 14:59:26.987907 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.981436 2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 21 14:59:26.987907 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.981439 2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 21 14:59:26.987907 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.981441 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 21 14:59:26.987907 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.981444 2563 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 21 14:59:26.987907 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.981446 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 21 14:59:26.987907 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.981448 2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 21 14:59:26.987907 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.981451 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 21 14:59:26.987907 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.981453 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 21 14:59:26.987907 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.981455 2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 21 14:59:26.987907 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.981458 2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 21 14:59:26.987907 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.981460 2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 21 14:59:26.988594 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.981463 2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 21 14:59:26.988594 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.981465 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 21 14:59:26.988594 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.981468 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 21 14:59:26.988594 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.981470 2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 21 14:59:26.988594 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.981473 2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 21 14:59:26.988594 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:26.981475 2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 21 14:59:26.988594 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.981479 2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 21 14:59:26.988594 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.981605 2563 server.go:962] "Client rotation is on, will bootstrap in background" Jul 21 14:59:26.988594 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.983725 2563 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 21 14:59:26.988594 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.985086 2563 server.go:1019] "Starting client certificate rotation" Jul 21 14:59:26.988594 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.985211 2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 21 14:59:26.988594 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.986030 2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 21 14:59:26.988594 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.987329 2563 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 21 14:59:26.990849 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:26.990831 2563 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 21 14:59:27.000258 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:27.000239 2563 log.go:25] "Validated CRI v1 runtime API" Jul 21 14:59:27.002389 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:27.002375 2563 log.go:25] "Validated CRI v1 image API" Jul 21 14:59:27.003391 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:27.003376 2563 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 21 14:59:27.005921 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:27.005902 2563 fs.go:135] Filesystem UUIDs: map[758f488b-b251-4243-ba75-e164d3a75f91:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 ccf9f887-ca47-4487-82a7-42bac28eade4:/dev/nvme0n1p3] Jul 21 14:59:27.005985 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:27.005922 2563 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 21 14:59:27.012170 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:27.012015 2563 manager.go:216] Machine: {Timestamp:2026-07-21 14:59:27.010787399 +0000 UTC m=+0.271739474 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3097790 MemoryCapacity:32812171264 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2e1430c26ab972a63c3a93aa1f36c1 SystemUUID:ec2e1430-c26a-b972-a63c-3a93aa1f36c1 BootID:d3acfeda-c5c0-4af7-af6e-8676671e995a Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406085632 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:86:0c:cc:61:99 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:86:0c:cc:61:99 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:a2:e5:97:d5:b4:42 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812171264 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 21 14:59:27.012170 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:27.012142 2563 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 21 14:59:27.012291 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:27.012242 2563 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 21 14:59:27.013860 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:27.013842 2563 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 21 14:59:27.014967 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:27.014939 2563 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 21 14:59:27.015124 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:27.014971 2563 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-176.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 21 14:59:27.015661 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:27.015650 2563 topology_manager.go:138] "Creating topology manager with none policy" Jul 21 14:59:27.015699 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:27.015666 2563 container_manager_linux.go:309] "Creating device plugin manager" Jul 21 14:59:27.015699 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:27.015677 2563 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 21 14:59:27.016236 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:27.016225 2563 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 21 14:59:27.017410 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:27.017399 2563 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 21 14:59:27.021181 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:27.021170 2563 state_mem.go:36] "Initialized new in-memory state store" Jul 21 14:59:27.021308 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:27.021298 2563 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 21 14:59:27.021348 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:27.021333 2563 kubelet.go:486] "Attempting to sync node with API server" Jul 21 14:59:27.021348 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:27.021344 2563 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 21 14:59:27.021403 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:27.021358 2563 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 21 14:59:27.021403 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:27.021367 2563 kubelet.go:398] "Adding apiserver pod source" Jul 21 14:59:27.021403 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:27.021375 2563 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 21 14:59:27.021874 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:27.021859 2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 21 14:59:27.021939 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:27.021887 2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 21 14:59:27.023643 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:27.023626 2563 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 21 14:59:27.024973 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:27.024954 2563 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 21 14:59:27.025064 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:27.024995 2563 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 21 14:59:27.025576 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:27.025562 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 21 14:59:27.025639 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:27.025584 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 21 14:59:27.025639 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:27.025593 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 21 14:59:27.025639 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:27.025601 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 21 14:59:27.025639 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:27.025613 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 21 14:59:27.025639 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:27.025623 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 21 14:59:27.025639 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:27.025635 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 21 14:59:27.025830 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:27.025643 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 21 14:59:27.025830 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:27.025655 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 21 14:59:27.025830 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:27.025665 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 21 14:59:27.025830 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:27.025677 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 21 14:59:27.025830 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:27.025686 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 21 14:59:27.026374 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:27.026363 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 21 14:59:27.026428 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:27.026376 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 21 14:59:27.029644 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:27.029628 2563 server.go:1268] "Started kubelet" Jul 21 14:59:27.029865 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:27.029751 2563 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 21 14:59:27.030236 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:27.029925 2563 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 21 14:59:27.030236 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:27.029845 2563 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 21 14:59:27.030593 ip-10-0-135-176 systemd[1]: Started Kubernetes Kubelet. Jul 21 14:59:27.031425 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:27.031408 2563 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 21 14:59:27.032659 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:27.032630 2563 server.go:310] "Adding debug handlers to kubelet server" Jul 21 14:59:27.032996 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:27.032971 2563 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-176.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 21 14:59:27.033232 ip-10-0-135-176 kubenswrapper[2563]: E0721 14:59:27.033189 2563 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-135-176.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 21 14:59:27.033303 ip-10-0-135-176 kubenswrapper[2563]: E0721 14:59:27.033206 2563 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 21 14:59:27.035987 ip-10-0-135-176 kubenswrapper[2563]: E0721 14:59:27.035968 2563 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 21 14:59:27.039185 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:27.039170 2563 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 21 14:59:27.039185 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:27.039172 2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 21 14:59:27.039436 ip-10-0-135-176 kubenswrapper[2563]: E0721 14:59:27.039415 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-176.ec2.internal\" not found" Jul 21 14:59:27.039833 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:27.039811 2563 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-npbbb" Jul 21 14:59:27.039901 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:27.039847 2563 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 21 14:59:27.039901 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:27.039863 2563 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 21 14:59:27.039997 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:27.039986 2563 reconstruct.go:97] "Volume reconstruction finished" Jul 21 14:59:27.039997 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:27.039992 2563 reconciler.go:29] "Reconciler: start to sync state" Jul 21 14:59:27.040208 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:27.040008 2563 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 21 14:59:27.040208 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:27.040097 2563 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 21 14:59:27.040208 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:27.040108 2563 factory.go:55] Registering systemd factory Jul 21 14:59:27.040208 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:27.040117 2563 factory.go:223] Registration of the systemd container factory successfully Jul 21 14:59:27.040432 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:27.040392 2563 factory.go:153] Registering CRI-O factory Jul 21 14:59:27.040432 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:27.040406 2563 factory.go:223] Registration of the crio container factory successfully Jul 21 14:59:27.040432 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:27.040433 2563 factory.go:103] Registering Raw factory Jul 21 14:59:27.040582 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:27.040446 2563 manager.go:1196] Started watching for new ooms in manager Jul 21 14:59:27.040965 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:27.040950 2563 manager.go:349] Starting recovery of all containers Jul 21 14:59:27.041212 ip-10-0-135-176 kubenswrapper[2563]: E0721 14:59:27.040433 2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-176.ec2.internal.18c4560d32a2014b default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-176.ec2.internal,UID:ip-10-0-135-176.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-135-176.ec2.internal,},FirstTimestamp:2026-07-21 14:59:27.029600587 +0000 UTC m=+0.290552660,LastTimestamp:2026-07-21 14:59:27.029600587 +0000 UTC m=+0.290552660,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-176.ec2.internal,}" Jul 21 14:59:27.046034 ip-10-0-135-176 kubenswrapper[2563]: E0721 14:59:27.046006 2563 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 21 14:59:27.046034 ip-10-0-135-176 kubenswrapper[2563]: E0721 14:59:27.046025 2563 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-135-176.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 21 14:59:27.047366 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:27.047319 2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-npbbb" Jul 21 14:59:27.048649 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:27.048603 2563 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 21 14:59:27.050689 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:27.050632 2563 manager.go:354] Recovery completed Jul 21 14:59:27.056066 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:27.055962 2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 21 14:59:27.058724 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:27.058707 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-176.ec2.internal" event="NodeHasSufficientMemory" Jul 21 14:59:27.058792 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:27.058737 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-176.ec2.internal" event="NodeHasNoDiskPressure" Jul 21 14:59:27.058792 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:27.058751 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-176.ec2.internal" event="NodeHasSufficientPID" Jul 21 14:59:27.059292 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:27.059277 2563 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 21 14:59:27.059340 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:27.059293 2563 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 21 14:59:27.059340 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:27.059298 2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 21 14:59:27.059340 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:27.059311 2563 state_mem.go:36] "Initialized new in-memory state store" Jul 21 14:59:27.060034 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:27.060021 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-176.ec2.internal" event="NodeHasSufficientMemory" Jul 21 14:59:27.060079 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:27.060046 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-176.ec2.internal" event="NodeHasNoDiskPressure" Jul 21 14:59:27.060079 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:27.060055 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-176.ec2.internal" event="NodeHasSufficientPID" Jul 21 14:59:27.060715 ip-10-0-135-176 kubenswrapper[2563]: E0721 14:59:27.060647 2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-176.ec2.internal.18c4560d345e5b33 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-176.ec2.internal,UID:ip-10-0-135-176.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-135-176.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-135-176.ec2.internal,},FirstTimestamp:2026-07-21 14:59:27.058721587 +0000 UTC m=+0.319673663,LastTimestamp:2026-07-21 14:59:27.058721587 +0000 UTC m=+0.319673663,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-176.ec2.internal,}" Jul 21 14:59:27.061338 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:27.061327 2563 policy_none.go:49] "None policy: Start" Jul 21 14:59:27.061395 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:27.061342 2563 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 21 14:59:27.061395 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:27.061352 2563 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 21 14:59:27.063345 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:27.063334 2563 policy_none.go:47] "Start" Jul 21 14:59:27.102779 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:27.102761 2563 manager.go:337] "Starting Device Plugin manager" Jul 21 14:59:27.102863 ip-10-0-135-176 kubenswrapper[2563]: E0721 14:59:27.102800 2563 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 21 14:59:27.102863 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:27.102812 2563 server.go:87] "Starting device plugin registration server" Jul 21 14:59:27.103079 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:27.103064 2563 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 21 14:59:27.103146 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:27.103083 2563 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 21 14:59:27.103225 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:27.103199 2563 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 21 14:59:27.103287 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:27.103276 2563 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 21 14:59:27.103287 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:27.103287 2563 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 21 14:59:27.103796 ip-10-0-135-176 kubenswrapper[2563]: E0721 14:59:27.103757 2563 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 21 14:59:27.103877 ip-10-0-135-176 kubenswrapper[2563]: E0721 14:59:27.103802 2563 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-176.ec2.internal\" not found" Jul 21 14:59:27.203557 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:27.203532 2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 21 14:59:27.205641 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:27.204414 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-176.ec2.internal" event="NodeHasSufficientMemory" Jul 21 14:59:27.205641 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:27.204442 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-176.ec2.internal" event="NodeHasNoDiskPressure" Jul 21 14:59:27.205641 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:27.204455 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-176.ec2.internal" event="NodeHasSufficientPID" Jul 21 14:59:27.205641 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:27.204480 2563 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-135-176.ec2.internal" Jul 21 14:59:27.208491 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:27.208476 2563 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 21 14:59:27.208540 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:27.208514 2563 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 21 14:59:27.208540 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:27.208535 2563 kubelet.go:2448] "Starting kubelet main sync loop" Jul 21 14:59:27.208600 ip-10-0-135-176 kubenswrapper[2563]: E0721 14:59:27.208569 2563 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 21 14:59:27.210619 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:27.210593 2563 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 21 14:59:27.212874 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:27.212858 2563 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-135-176.ec2.internal" Jul 21 14:59:27.212952 ip-10-0-135-176 kubenswrapper[2563]: E0721 14:59:27.212878 2563 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-176.ec2.internal\": node \"ip-10-0-135-176.ec2.internal\" not found" Jul 21 14:59:27.259837 ip-10-0-135-176 kubenswrapper[2563]: E0721 14:59:27.259769 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-176.ec2.internal\" not found" Jul 21 14:59:27.308789 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:27.308757 2563 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-176.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-176.ec2.internal"] Jul 21 14:59:27.308879 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:27.308851 2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 21 14:59:27.311258 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:27.311240 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-176.ec2.internal" event="NodeHasSufficientMemory" Jul 21 14:59:27.311340 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:27.311277 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-176.ec2.internal" event="NodeHasNoDiskPressure" Jul 21 14:59:27.311340 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:27.311288 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-176.ec2.internal" event="NodeHasSufficientPID" Jul 21 14:59:27.313573 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:27.313560 2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 21 14:59:27.313719 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:27.313704 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-176.ec2.internal" Jul 21 14:59:27.313771 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:27.313739 2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 21 14:59:27.314429 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:27.314410 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-176.ec2.internal" event="NodeHasSufficientMemory" Jul 21 14:59:27.314527 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:27.314434 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-176.ec2.internal" event="NodeHasNoDiskPressure" Jul 21 14:59:27.314527 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:27.314462 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-176.ec2.internal" event="NodeHasSufficientMemory" Jul 21 14:59:27.314527 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:27.314480 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-176.ec2.internal" event="NodeHasSufficientPID" Jul 21 14:59:27.314638 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:27.314481 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-176.ec2.internal" event="NodeHasNoDiskPressure" Jul 21 14:59:27.314638 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:27.314594 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-176.ec2.internal" event="NodeHasSufficientPID" Jul 21 14:59:27.316894 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:27.316874 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-176.ec2.internal" Jul 21 14:59:27.316994 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:27.316901 2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 21 14:59:27.317659 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:27.317644 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-176.ec2.internal" event="NodeHasSufficientMemory" Jul 21 14:59:27.317741 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:27.317672 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-176.ec2.internal" event="NodeHasNoDiskPressure" Jul 21 14:59:27.317741 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:27.317684 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-176.ec2.internal" event="NodeHasSufficientPID" Jul 21 14:59:27.340793 ip-10-0-135-176 kubenswrapper[2563]: E0721 14:59:27.340773 2563 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-176.ec2.internal\" not found" node="ip-10-0-135-176.ec2.internal" Jul 21 14:59:27.345208 ip-10-0-135-176 kubenswrapper[2563]: E0721 14:59:27.345189 2563 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-176.ec2.internal\" not found" node="ip-10-0-135-176.ec2.internal" Jul 21 14:59:27.360806 ip-10-0-135-176 kubenswrapper[2563]: E0721 14:59:27.360783 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-176.ec2.internal\" not found" Jul 21 14:59:27.441713 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:27.441677 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b953891c06139c6cc3d45f0903be0d66-config\") pod \"kube-apiserver-proxy-ip-10-0-135-176.ec2.internal\" (UID: \"b953891c06139c6cc3d45f0903be0d66\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-176.ec2.internal" Jul 21 14:59:27.441713 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:27.441711 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1613301c104ee5ff8466f50a74f62241-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-176.ec2.internal\" (UID: \"1613301c104ee5ff8466f50a74f62241\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-176.ec2.internal" Jul 21 14:59:27.441882 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:27.441731 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1613301c104ee5ff8466f50a74f62241-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-176.ec2.internal\" (UID: \"1613301c104ee5ff8466f50a74f62241\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-176.ec2.internal" Jul 21 14:59:27.460877 ip-10-0-135-176 kubenswrapper[2563]: E0721 14:59:27.460854 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-176.ec2.internal\" not found" Jul 21 14:59:27.542593 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:27.542486 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b953891c06139c6cc3d45f0903be0d66-config\") pod \"kube-apiserver-proxy-ip-10-0-135-176.ec2.internal\" (UID: \"b953891c06139c6cc3d45f0903be0d66\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-176.ec2.internal" Jul 21 14:59:27.542593 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:27.542571 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1613301c104ee5ff8466f50a74f62241-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-176.ec2.internal\" (UID: \"1613301c104ee5ff8466f50a74f62241\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-176.ec2.internal" Jul 21 14:59:27.542593 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:27.542587 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1613301c104ee5ff8466f50a74f62241-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-176.ec2.internal\" (UID: \"1613301c104ee5ff8466f50a74f62241\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-176.ec2.internal" Jul 21 14:59:27.542742 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:27.542619 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1613301c104ee5ff8466f50a74f62241-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-176.ec2.internal\" (UID: \"1613301c104ee5ff8466f50a74f62241\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-176.ec2.internal" Jul 21 14:59:27.542742 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:27.542537 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b953891c06139c6cc3d45f0903be0d66-config\") pod \"kube-apiserver-proxy-ip-10-0-135-176.ec2.internal\" (UID: \"b953891c06139c6cc3d45f0903be0d66\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-176.ec2.internal" Jul 21 14:59:27.542742 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:27.542698 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1613301c104ee5ff8466f50a74f62241-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-176.ec2.internal\" (UID: \"1613301c104ee5ff8466f50a74f62241\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-176.ec2.internal" Jul 21 14:59:27.561644 ip-10-0-135-176 kubenswrapper[2563]: E0721 14:59:27.561608 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-176.ec2.internal\" not found" Jul 21 14:59:27.643090 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:27.643065 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-176.ec2.internal" Jul 21 14:59:27.649208 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:27.649184 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-176.ec2.internal" Jul 21 14:59:27.662188 ip-10-0-135-176 kubenswrapper[2563]: E0721 14:59:27.662147 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-176.ec2.internal\" not found" Jul 21 14:59:27.762682 ip-10-0-135-176 kubenswrapper[2563]: E0721 14:59:27.762640 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-176.ec2.internal\" not found" Jul 21 14:59:27.863220 ip-10-0-135-176 kubenswrapper[2563]: E0721 14:59:27.863191 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-176.ec2.internal\" not found" Jul 21 14:59:27.918551 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:27.918517 2563 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 21 14:59:27.932707 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:27.932685 2563 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 21 14:59:27.939567 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:27.939547 2563 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-176.ec2.internal" Jul 21 14:59:27.950086 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:27.950067 2563 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 21 14:59:27.952119 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:27.952106 2563 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-176.ec2.internal" Jul 21 14:59:27.961497 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:27.961481 2563 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 21 14:59:27.984564 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:27.984525 2563 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 21 14:59:27.985042 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:27.984727 2563 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 21 14:59:27.985042 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:27.984751 2563 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 21 14:59:27.985042 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:27.984751 2563 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 21 14:59:28.021844 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:28.021816 2563 apiserver.go:52] "Watching apiserver" Jul 21 14:59:28.039608 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:28.039584 2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 21 14:59:28.043024 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:28.043007 2563 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 21 14:59:28.043972 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:28.043947 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-bj8nf","openshift-dns/node-resolver-h2kn2","openshift-image-registry/node-ca-9qzhs","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-176.ec2.internal","openshift-multus/multus-8vxjj","openshift-multus/multus-additional-cni-plugins-8h5jp","kube-system/konnectivity-agent-m4pp8","kube-system/kube-apiserver-proxy-ip-10-0-135-176.ec2.internal","openshift-multus/network-metrics-daemon-dh8jm","openshift-network-diagnostics/network-check-target-bzwfd","openshift-network-operator/iptables-alerter-7q2dq","openshift-ovn-kubernetes/ovnkube-node-kn9fn","kube-system/global-pull-secret-syncer-jttxg","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vl2rt"] Jul 21 14:59:28.046439 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:28.046419 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bzwfd" Jul 21 14:59:28.046525 ip-10-0-135-176 kubenswrapper[2563]: E0721 14:59:28.046488 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bzwfd" podUID="0683eb8b-8af7-4056-beed-3bfadd624f56" Jul 21 14:59:28.048953 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:28.048934 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jttxg" Jul 21 14:59:28.049037 ip-10-0-135-176 kubenswrapper[2563]: E0721 14:59:28.048988 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jttxg" podUID="e706cd7a-7458-4f77-b02d-3fd5837d735d" Jul 21 14:59:28.050740 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:28.050704 2563 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 21 14:59:28.053238 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:28.051574 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-9qzhs" Jul 21 14:59:28.053238 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:28.052796 2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-20 14:54:28 +0000 UTC" deadline="2028-03-26 08:49:21.45851593 +0000 UTC" Jul 21 14:59:28.053238 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:28.052872 2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14729h49m53.405648138s" Jul 21 14:59:28.054015 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:28.054000 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vl2rt" Jul 21 14:59:28.054502 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:28.054485 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-5t5n7\"" Jul 21 14:59:28.054773 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:28.054753 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 21 14:59:28.054831 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:28.054796 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 21 14:59:28.054967 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:28.054952 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 21 14:59:28.056206 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:28.056192 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-8vxjj" Jul 21 14:59:28.056543 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:28.056527 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 21 14:59:28.056619 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:28.056544 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 21 14:59:28.056619 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:28.056579 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-qjstg\"" Jul 21 14:59:28.056756 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:28.056740 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 21 14:59:28.058428 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:28.058415 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-8h5jp" Jul 21 14:59:28.058494 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:28.058480 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 21 14:59:28.058842 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:28.058828 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 21 14:59:28.058963 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:28.058949 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 21 14:59:28.059628 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:28.059600 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-2wc5m\"" Jul 21 14:59:28.059688 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:28.059656 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 21 14:59:28.060528 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:28.060515 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dh8jm" Jul 21 14:59:28.060596 ip-10-0-135-176 kubenswrapper[2563]: E0721 14:59:28.060575 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dh8jm" podUID="26de6bb3-eb05-461a-958f-406464dea020" Jul 21 14:59:28.060796 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:28.060774 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 21 14:59:28.061056 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:28.061040 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 21 14:59:28.061136 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:28.061117 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-9g589\"" Jul 21 14:59:28.062817 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:28.062803 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-m4pp8" Jul 21 14:59:28.065134 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:28.065117 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-h2kn2" Jul 21 14:59:28.065584 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:28.065567 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 21 14:59:28.065674 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:28.065615 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-6ftlf\"" Jul 21 14:59:28.065804 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:28.065789 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 21 14:59:28.067397 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:28.067380 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-bj8nf" Jul 21 14:59:28.067786 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:28.067770 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 21 14:59:28.067890 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:28.067868 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-qtm5r\"" Jul 21 14:59:28.068720 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:28.068702 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 21 14:59:28.069828 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:28.069812 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 21 14:59:28.070238 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:28.070223 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-7q2dq" Jul 21 14:59:28.070357 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:28.070340 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 21 14:59:28.070508 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:28.070493 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-29vqq\"" Jul 21 14:59:28.071627 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:28.071610 2563 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-qtlcz" Jul 21 14:59:28.072587 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:28.072567 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-kn9fn" Jul 21 14:59:28.072729 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:28.072575 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 21 14:59:28.072800 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:28.072785 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 21 14:59:28.072883 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:28.072869 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 21 14:59:28.073120 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:28.073106 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-jmq5k\"" Jul 21 14:59:28.074950 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:28.074937 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 21 14:59:28.075001 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:28.074952 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 21 14:59:28.075542 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:28.075527 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 21 14:59:28.077905 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:28.077886 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 21 14:59:28.077989 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:28.077912 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-jxxxb\"" Jul 21 14:59:28.077989 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:28.077959 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 21 14:59:28.078064 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:28.078004 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 21 14:59:28.079209 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:28.079191 2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-qtlcz" Jul 21 14:59:29.406897 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.406882 2563 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 21 14:59:29.406951 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.406930 2563 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 21 14:59:29.411300 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.411277 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8e65c9e9-347b-46b1-a1f5-774fde3b8107-os-release\") pod \"multus-8vxjj\" (UID: \"8e65c9e9-347b-46b1-a1f5-774fde3b8107\") " pod="openshift-multus/multus-8vxjj" Jul 21 14:59:29.411409 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.411304 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8e65c9e9-347b-46b1-a1f5-774fde3b8107-host-run-netns\") pod \"multus-8vxjj\" (UID: \"8e65c9e9-347b-46b1-a1f5-774fde3b8107\") " pod="openshift-multus/multus-8vxjj" Jul 21 14:59:29.411409 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.411319 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/327b1f5e-8e27-4572-8025-5c516e59d481-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8h5jp\" (UID: \"327b1f5e-8e27-4572-8025-5c516e59d481\") " pod="openshift-multus/multus-additional-cni-plugins-8h5jp" Jul 21 14:59:29.411409 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.411353 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/42b48be4-5000-431e-9d8b-b932609faca0-hosts-file\") pod \"node-resolver-h2kn2\" (UID: \"42b48be4-5000-431e-9d8b-b932609faca0\") " pod="openshift-dns/node-resolver-h2kn2" Jul 21 14:59:29.411409 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.411377 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42b48be4-5000-431e-9d8b-b932609faca0-tmp-dir\") pod \"node-resolver-h2kn2\" (UID: \"42b48be4-5000-431e-9d8b-b932609faca0\") " pod="openshift-dns/node-resolver-h2kn2" Jul 21 14:59:29.411538 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.411416 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/02f3d0c6-4116-4134-ba51-1c02c73bef05-host-run-netns\") pod \"ovnkube-node-kn9fn\" (UID: \"02f3d0c6-4116-4134-ba51-1c02c73bef05\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn9fn" Jul 21 14:59:29.411538 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.411456 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/216f52c0-8c33-426f-b2eb-244a5787bdeb-konnectivity-ca\") pod \"konnectivity-agent-m4pp8\" (UID: \"216f52c0-8c33-426f-b2eb-244a5787bdeb\") " pod="kube-system/konnectivity-agent-m4pp8" Jul 21 14:59:29.411538 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.411479 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8e65c9e9-347b-46b1-a1f5-774fde3b8107-system-cni-dir\") pod \"multus-8vxjj\" (UID: \"8e65c9e9-347b-46b1-a1f5-774fde3b8107\") " pod="openshift-multus/multus-8vxjj" Jul 21 14:59:29.411538 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.411495 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8e65c9e9-347b-46b1-a1f5-774fde3b8107-cni-binary-copy\") pod \"multus-8vxjj\" (UID: \"8e65c9e9-347b-46b1-a1f5-774fde3b8107\") " pod="openshift-multus/multus-8vxjj" Jul 21 14:59:29.411538 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.411509 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26de6bb3-eb05-461a-958f-406464dea020-metrics-certs\") pod \"network-metrics-daemon-dh8jm\" (UID: \"26de6bb3-eb05-461a-958f-406464dea020\") " pod="openshift-multus/network-metrics-daemon-dh8jm" Jul 21 14:59:29.411538 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.411523 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/02f3d0c6-4116-4134-ba51-1c02c73bef05-systemd-units\") pod \"ovnkube-node-kn9fn\" (UID: \"02f3d0c6-4116-4134-ba51-1c02c73bef05\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn9fn" Jul 21 14:59:29.411699 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.411541 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/02f3d0c6-4116-4134-ba51-1c02c73bef05-host-run-ovn-kubernetes\") pod \"ovnkube-node-kn9fn\" (UID: \"02f3d0c6-4116-4134-ba51-1c02c73bef05\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn9fn" Jul 21 14:59:29.411699 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.411561 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a6348ec1-f9c5-487a-9cfe-abd129eadfe0-etc-systemd\") pod \"tuned-bj8nf\" (UID: \"a6348ec1-f9c5-487a-9cfe-abd129eadfe0\") " pod="openshift-cluster-node-tuning-operator/tuned-bj8nf" Jul 21 14:59:29.411699 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.411576 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4f638c31-bce1-4b95-97a3-410069ef52e9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vl2rt\" (UID: \"4f638c31-bce1-4b95-97a3-410069ef52e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vl2rt" Jul 21 14:59:29.411699 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.411603 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8e65c9e9-347b-46b1-a1f5-774fde3b8107-host-run-multus-certs\") pod \"multus-8vxjj\" (UID: \"8e65c9e9-347b-46b1-a1f5-774fde3b8107\") " pod="openshift-multus/multus-8vxjj" Jul 21 14:59:29.411699 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.411623 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8e65c9e9-347b-46b1-a1f5-774fde3b8107-etc-kubernetes\") pod \"multus-8vxjj\" (UID: \"8e65c9e9-347b-46b1-a1f5-774fde3b8107\") " pod="openshift-multus/multus-8vxjj" Jul 21 14:59:29.411699 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.411636 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/02f3d0c6-4116-4134-ba51-1c02c73bef05-run-ovn\") pod \"ovnkube-node-kn9fn\" (UID: \"02f3d0c6-4116-4134-ba51-1c02c73bef05\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn9fn" Jul 21 14:59:29.411699 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.411655 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e706cd7a-7458-4f77-b02d-3fd5837d735d-dbus\") pod \"global-pull-secret-syncer-jttxg\" (UID: \"e706cd7a-7458-4f77-b02d-3fd5837d735d\") " pod="kube-system/global-pull-secret-syncer-jttxg" Jul 21 14:59:29.411699 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.411670 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4f638c31-bce1-4b95-97a3-410069ef52e9-socket-dir\") pod \"aws-ebs-csi-driver-node-vl2rt\" (UID: \"4f638c31-bce1-4b95-97a3-410069ef52e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vl2rt" Jul 21 14:59:29.411699 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.411687 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8e65c9e9-347b-46b1-a1f5-774fde3b8107-host-var-lib-kubelet\") pod \"multus-8vxjj\" (UID: \"8e65c9e9-347b-46b1-a1f5-774fde3b8107\") " pod="openshift-multus/multus-8vxjj" Jul 21 14:59:29.411949 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.411714 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8e65c9e9-347b-46b1-a1f5-774fde3b8107-multus-cni-dir\") pod \"multus-8vxjj\" (UID: \"8e65c9e9-347b-46b1-a1f5-774fde3b8107\") " pod="openshift-multus/multus-8vxjj" Jul 21 14:59:29.411949 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.411747 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8e65c9e9-347b-46b1-a1f5-774fde3b8107-multus-socket-dir-parent\") pod \"multus-8vxjj\" (UID: \"8e65c9e9-347b-46b1-a1f5-774fde3b8107\") " pod="openshift-multus/multus-8vxjj" Jul 21 14:59:29.411949 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.411765 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8e65c9e9-347b-46b1-a1f5-774fde3b8107-host-var-lib-cni-multus\") pod \"multus-8vxjj\" (UID: \"8e65c9e9-347b-46b1-a1f5-774fde3b8107\") " pod="openshift-multus/multus-8vxjj" Jul 21 14:59:29.411949 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.411794 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8e65c9e9-347b-46b1-a1f5-774fde3b8107-multus-daemon-config\") pod \"multus-8vxjj\" (UID: \"8e65c9e9-347b-46b1-a1f5-774fde3b8107\") " pod="openshift-multus/multus-8vxjj" Jul 21 14:59:29.411949 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.411808 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/02f3d0c6-4116-4134-ba51-1c02c73bef05-etc-openvswitch\") pod \"ovnkube-node-kn9fn\" (UID: \"02f3d0c6-4116-4134-ba51-1c02c73bef05\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn9fn" Jul 21 14:59:29.411949 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.411822 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4d94d908-6ab5-4463-a878-0b340bb80365-serviceca\") pod \"node-ca-9qzhs\" (UID: \"4d94d908-6ab5-4463-a878-0b340bb80365\") " pod="openshift-image-registry/node-ca-9qzhs" Jul 21 14:59:29.411949 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.411836 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a6348ec1-f9c5-487a-9cfe-abd129eadfe0-etc-kubernetes\") pod \"tuned-bj8nf\" (UID: \"a6348ec1-f9c5-487a-9cfe-abd129eadfe0\") " pod="openshift-cluster-node-tuning-operator/tuned-bj8nf" Jul 21 14:59:29.411949 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.411856 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r785r\" (UniqueName: \"kubernetes.io/projected/327b1f5e-8e27-4572-8025-5c516e59d481-kube-api-access-r785r\") pod \"multus-additional-cni-plugins-8h5jp\" (UID: \"327b1f5e-8e27-4572-8025-5c516e59d481\") " pod="openshift-multus/multus-additional-cni-plugins-8h5jp" Jul 21 14:59:29.411949 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.411881 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/02f3d0c6-4116-4134-ba51-1c02c73bef05-env-overrides\") pod \"ovnkube-node-kn9fn\" (UID: \"02f3d0c6-4116-4134-ba51-1c02c73bef05\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn9fn" Jul 21 14:59:29.411949 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.411902 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a6348ec1-f9c5-487a-9cfe-abd129eadfe0-lib-modules\") pod \"tuned-bj8nf\" (UID: \"a6348ec1-f9c5-487a-9cfe-abd129eadfe0\") " pod="openshift-cluster-node-tuning-operator/tuned-bj8nf" Jul 21 14:59:29.411949 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.411935 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e706cd7a-7458-4f77-b02d-3fd5837d735d-kubelet-config\") pod \"global-pull-secret-syncer-jttxg\" (UID: \"e706cd7a-7458-4f77-b02d-3fd5837d735d\") " pod="kube-system/global-pull-secret-syncer-jttxg" Jul 21 14:59:29.412236 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.411956 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4f638c31-bce1-4b95-97a3-410069ef52e9-etc-selinux\") pod \"aws-ebs-csi-driver-node-vl2rt\" (UID: \"4f638c31-bce1-4b95-97a3-410069ef52e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vl2rt" Jul 21 14:59:29.412236 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.411985 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8e65c9e9-347b-46b1-a1f5-774fde3b8107-multus-conf-dir\") pod \"multus-8vxjj\" (UID: \"8e65c9e9-347b-46b1-a1f5-774fde3b8107\") " pod="openshift-multus/multus-8vxjj" Jul 21 14:59:29.412236 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.412001 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4v77\" (UniqueName: \"kubernetes.io/projected/8e65c9e9-347b-46b1-a1f5-774fde3b8107-kube-api-access-d4v77\") pod \"multus-8vxjj\" (UID: \"8e65c9e9-347b-46b1-a1f5-774fde3b8107\") " pod="openshift-multus/multus-8vxjj" Jul 21 14:59:29.412236 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.412016 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/02f3d0c6-4116-4134-ba51-1c02c73bef05-host-kubelet\") pod \"ovnkube-node-kn9fn\" (UID: \"02f3d0c6-4116-4134-ba51-1c02c73bef05\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn9fn" Jul 21 14:59:29.412236 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.412029 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/216f52c0-8c33-426f-b2eb-244a5787bdeb-agent-certs\") pod \"konnectivity-agent-m4pp8\" (UID: \"216f52c0-8c33-426f-b2eb-244a5787bdeb\") " pod="kube-system/konnectivity-agent-m4pp8" Jul 21 14:59:29.412236 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.412045 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a6348ec1-f9c5-487a-9cfe-abd129eadfe0-etc-sysconfig\") pod \"tuned-bj8nf\" (UID: \"a6348ec1-f9c5-487a-9cfe-abd129eadfe0\") " pod="openshift-cluster-node-tuning-operator/tuned-bj8nf" Jul 21 14:59:29.412236 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.412058 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a6348ec1-f9c5-487a-9cfe-abd129eadfe0-var-lib-kubelet\") pod \"tuned-bj8nf\" (UID: \"a6348ec1-f9c5-487a-9cfe-abd129eadfe0\") " pod="openshift-cluster-node-tuning-operator/tuned-bj8nf" Jul 21 14:59:29.412236 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.412077 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a6348ec1-f9c5-487a-9cfe-abd129eadfe0-etc-tuned\") pod \"tuned-bj8nf\" (UID: \"a6348ec1-f9c5-487a-9cfe-abd129eadfe0\") " pod="openshift-cluster-node-tuning-operator/tuned-bj8nf" Jul 21 14:59:29.412236 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.412098 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8e65c9e9-347b-46b1-a1f5-774fde3b8107-host-var-lib-cni-bin\") pod \"multus-8vxjj\" (UID: \"8e65c9e9-347b-46b1-a1f5-774fde3b8107\") " pod="openshift-multus/multus-8vxjj" Jul 21 14:59:29.412236 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.412124 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/327b1f5e-8e27-4572-8025-5c516e59d481-cni-binary-copy\") pod \"multus-additional-cni-plugins-8h5jp\" (UID: \"327b1f5e-8e27-4572-8025-5c516e59d481\") " pod="openshift-multus/multus-additional-cni-plugins-8h5jp" Jul 21 14:59:29.412236 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.412146 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/02f3d0c6-4116-4134-ba51-1c02c73bef05-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kn9fn\" (UID: \"02f3d0c6-4116-4134-ba51-1c02c73bef05\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn9fn" Jul 21 14:59:29.412236 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.412170 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a6348ec1-f9c5-487a-9cfe-abd129eadfe0-sys\") pod \"tuned-bj8nf\" (UID: \"a6348ec1-f9c5-487a-9cfe-abd129eadfe0\") " pod="openshift-cluster-node-tuning-operator/tuned-bj8nf" Jul 21 14:59:29.412236 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.412184 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a6348ec1-f9c5-487a-9cfe-abd129eadfe0-host\") pod \"tuned-bj8nf\" (UID: \"a6348ec1-f9c5-487a-9cfe-abd129eadfe0\") " pod="openshift-cluster-node-tuning-operator/tuned-bj8nf" Jul 21 14:59:29.412236 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.412197 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/02f3d0c6-4116-4134-ba51-1c02c73bef05-ovnkube-config\") pod \"ovnkube-node-kn9fn\" (UID: \"02f3d0c6-4116-4134-ba51-1c02c73bef05\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn9fn" Jul 21 14:59:29.412236 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.412214 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a6348ec1-f9c5-487a-9cfe-abd129eadfe0-etc-sysctl-d\") pod \"tuned-bj8nf\" (UID: \"a6348ec1-f9c5-487a-9cfe-abd129eadfe0\") " pod="openshift-cluster-node-tuning-operator/tuned-bj8nf" Jul 21 14:59:29.412236 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.412237 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8bf74430-9a26-4473-ace9-3d6034a94cf1-host-slash\") pod \"iptables-alerter-7q2dq\" (UID: \"8bf74430-9a26-4473-ace9-3d6034a94cf1\") " pod="openshift-network-operator/iptables-alerter-7q2dq" Jul 21 14:59:29.412699 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.412259 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8e65c9e9-347b-46b1-a1f5-774fde3b8107-cnibin\") pod \"multus-8vxjj\" (UID: \"8e65c9e9-347b-46b1-a1f5-774fde3b8107\") " pod="openshift-multus/multus-8vxjj" Jul 21 14:59:29.412699 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.412282 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/327b1f5e-8e27-4572-8025-5c516e59d481-cnibin\") pod \"multus-additional-cni-plugins-8h5jp\" (UID: \"327b1f5e-8e27-4572-8025-5c516e59d481\") " pod="openshift-multus/multus-additional-cni-plugins-8h5jp" Jul 21 14:59:29.412699 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.412300 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/02f3d0c6-4116-4134-ba51-1c02c73bef05-var-lib-openvswitch\") pod \"ovnkube-node-kn9fn\" (UID: \"02f3d0c6-4116-4134-ba51-1c02c73bef05\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn9fn" Jul 21 14:59:29.412699 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.412316 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/02f3d0c6-4116-4134-ba51-1c02c73bef05-node-log\") pod \"ovnkube-node-kn9fn\" (UID: \"02f3d0c6-4116-4134-ba51-1c02c73bef05\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn9fn" Jul 21 14:59:29.412699 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.412344 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4d94d908-6ab5-4463-a878-0b340bb80365-host\") pod \"node-ca-9qzhs\" (UID: \"4d94d908-6ab5-4463-a878-0b340bb80365\") " pod="openshift-image-registry/node-ca-9qzhs" Jul 21 14:59:29.412699 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.412366 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a6348ec1-f9c5-487a-9cfe-abd129eadfe0-etc-sysctl-conf\") pod \"tuned-bj8nf\" (UID: \"a6348ec1-f9c5-487a-9cfe-abd129eadfe0\") " pod="openshift-cluster-node-tuning-operator/tuned-bj8nf" Jul 21 14:59:29.412699 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.412394 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86c9j\" (UniqueName: \"kubernetes.io/projected/a6348ec1-f9c5-487a-9cfe-abd129eadfe0-kube-api-access-86c9j\") pod \"tuned-bj8nf\" (UID: \"a6348ec1-f9c5-487a-9cfe-abd129eadfe0\") " pod="openshift-cluster-node-tuning-operator/tuned-bj8nf" Jul 21 14:59:29.412699 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.412414 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8d8n\" (UniqueName: \"kubernetes.io/projected/8bf74430-9a26-4473-ace9-3d6034a94cf1-kube-api-access-v8d8n\") pod \"iptables-alerter-7q2dq\" (UID: \"8bf74430-9a26-4473-ace9-3d6034a94cf1\") " pod="openshift-network-operator/iptables-alerter-7q2dq" Jul 21 14:59:29.412699 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.412442 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/327b1f5e-8e27-4572-8025-5c516e59d481-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8h5jp\" (UID: \"327b1f5e-8e27-4572-8025-5c516e59d481\") " pod="openshift-multus/multus-additional-cni-plugins-8h5jp" Jul 21 14:59:29.412699 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.412477 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/02f3d0c6-4116-4134-ba51-1c02c73bef05-host-slash\") pod \"ovnkube-node-kn9fn\" (UID: \"02f3d0c6-4116-4134-ba51-1c02c73bef05\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn9fn" Jul 21 14:59:29.412699 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.412501 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e706cd7a-7458-4f77-b02d-3fd5837d735d-original-pull-secret\") pod \"global-pull-secret-syncer-jttxg\" (UID: \"e706cd7a-7458-4f77-b02d-3fd5837d735d\") " pod="kube-system/global-pull-secret-syncer-jttxg" Jul 21 14:59:29.412699 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.412516 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4f638c31-bce1-4b95-97a3-410069ef52e9-registration-dir\") pod \"aws-ebs-csi-driver-node-vl2rt\" (UID: \"4f638c31-bce1-4b95-97a3-410069ef52e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vl2rt" Jul 21 14:59:29.412699 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.412536 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4f638c31-bce1-4b95-97a3-410069ef52e9-device-dir\") pod \"aws-ebs-csi-driver-node-vl2rt\" (UID: \"4f638c31-bce1-4b95-97a3-410069ef52e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vl2rt" Jul 21 14:59:29.412699 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.412564 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/327b1f5e-8e27-4572-8025-5c516e59d481-os-release\") pod \"multus-additional-cni-plugins-8h5jp\" (UID: \"327b1f5e-8e27-4572-8025-5c516e59d481\") " pod="openshift-multus/multus-additional-cni-plugins-8h5jp" Jul 21 14:59:29.412699 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.412593 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lm8nt\" (UniqueName: \"kubernetes.io/projected/26de6bb3-eb05-461a-958f-406464dea020-kube-api-access-lm8nt\") pod \"network-metrics-daemon-dh8jm\" (UID: \"26de6bb3-eb05-461a-958f-406464dea020\") " pod="openshift-multus/network-metrics-daemon-dh8jm" Jul 21 14:59:29.412699 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.412607 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/02f3d0c6-4116-4134-ba51-1c02c73bef05-host-cni-netd\") pod \"ovnkube-node-kn9fn\" (UID: \"02f3d0c6-4116-4134-ba51-1c02c73bef05\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn9fn" Jul 21 14:59:29.413198 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.412622 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/02f3d0c6-4116-4134-ba51-1c02c73bef05-ovn-node-metrics-cert\") pod \"ovnkube-node-kn9fn\" (UID: \"02f3d0c6-4116-4134-ba51-1c02c73bef05\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn9fn" Jul 21 14:59:29.413198 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.412641 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a6348ec1-f9c5-487a-9cfe-abd129eadfe0-tmp\") pod \"tuned-bj8nf\" (UID: \"a6348ec1-f9c5-487a-9cfe-abd129eadfe0\") " pod="openshift-cluster-node-tuning-operator/tuned-bj8nf" Jul 21 14:59:29.413198 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.412664 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8bf74430-9a26-4473-ace9-3d6034a94cf1-iptables-alerter-script\") pod \"iptables-alerter-7q2dq\" (UID: \"8bf74430-9a26-4473-ace9-3d6034a94cf1\") " pod="openshift-network-operator/iptables-alerter-7q2dq" Jul 21 14:59:29.413198 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.412700 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4f638c31-bce1-4b95-97a3-410069ef52e9-sys-fs\") pod \"aws-ebs-csi-driver-node-vl2rt\" (UID: \"4f638c31-bce1-4b95-97a3-410069ef52e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vl2rt" Jul 21 14:59:29.413198 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.412716 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8e65c9e9-347b-46b1-a1f5-774fde3b8107-hostroot\") pod \"multus-8vxjj\" (UID: \"8e65c9e9-347b-46b1-a1f5-774fde3b8107\") " pod="openshift-multus/multus-8vxjj" Jul 21 14:59:29.413198 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.412737 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvwvn\" (UniqueName: \"kubernetes.io/projected/42b48be4-5000-431e-9d8b-b932609faca0-kube-api-access-lvwvn\") pod \"node-resolver-h2kn2\" (UID: \"42b48be4-5000-431e-9d8b-b932609faca0\") " pod="openshift-dns/node-resolver-h2kn2" Jul 21 14:59:29.413198 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.412775 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/02f3d0c6-4116-4134-ba51-1c02c73bef05-run-systemd\") pod \"ovnkube-node-kn9fn\" (UID: \"02f3d0c6-4116-4134-ba51-1c02c73bef05\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn9fn" Jul 21 14:59:29.413198 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.412815 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/02f3d0c6-4116-4134-ba51-1c02c73bef05-log-socket\") pod \"ovnkube-node-kn9fn\" (UID: \"02f3d0c6-4116-4134-ba51-1c02c73bef05\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn9fn" Jul 21 14:59:29.413198 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.412837 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/02f3d0c6-4116-4134-ba51-1c02c73bef05-host-cni-bin\") pod \"ovnkube-node-kn9fn\" (UID: \"02f3d0c6-4116-4134-ba51-1c02c73bef05\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn9fn" Jul 21 14:59:29.413198 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.412851 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dh6gh\" (UniqueName: \"kubernetes.io/projected/0683eb8b-8af7-4056-beed-3bfadd624f56-kube-api-access-dh6gh\") pod \"network-check-target-bzwfd\" (UID: \"0683eb8b-8af7-4056-beed-3bfadd624f56\") " pod="openshift-network-diagnostics/network-check-target-bzwfd" Jul 21 14:59:29.413198 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.412874 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a6348ec1-f9c5-487a-9cfe-abd129eadfe0-run\") pod \"tuned-bj8nf\" (UID: \"a6348ec1-f9c5-487a-9cfe-abd129eadfe0\") " pod="openshift-cluster-node-tuning-operator/tuned-bj8nf" Jul 21 14:59:29.413198 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.412890 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/327b1f5e-8e27-4572-8025-5c516e59d481-system-cni-dir\") pod \"multus-additional-cni-plugins-8h5jp\" (UID: \"327b1f5e-8e27-4572-8025-5c516e59d481\") " pod="openshift-multus/multus-additional-cni-plugins-8h5jp" Jul 21 14:59:29.413198 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.412906 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/02f3d0c6-4116-4134-ba51-1c02c73bef05-run-openvswitch\") pod \"ovnkube-node-kn9fn\" (UID: \"02f3d0c6-4116-4134-ba51-1c02c73bef05\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn9fn" Jul 21 14:59:29.413198 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.412938 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkwxv\" (UniqueName: \"kubernetes.io/projected/4d94d908-6ab5-4463-a878-0b340bb80365-kube-api-access-qkwxv\") pod \"node-ca-9qzhs\" (UID: \"4d94d908-6ab5-4463-a878-0b340bb80365\") " pod="openshift-image-registry/node-ca-9qzhs" Jul 21 14:59:29.413198 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.412961 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a6348ec1-f9c5-487a-9cfe-abd129eadfe0-etc-modprobe-d\") pod \"tuned-bj8nf\" (UID: \"a6348ec1-f9c5-487a-9cfe-abd129eadfe0\") " pod="openshift-cluster-node-tuning-operator/tuned-bj8nf" Jul 21 14:59:29.413198 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.412984 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/327b1f5e-8e27-4572-8025-5c516e59d481-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8h5jp\" (UID: \"327b1f5e-8e27-4572-8025-5c516e59d481\") " pod="openshift-multus/multus-additional-cni-plugins-8h5jp" Jul 21 14:59:29.413695 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.413003 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/02f3d0c6-4116-4134-ba51-1c02c73bef05-ovnkube-script-lib\") pod \"ovnkube-node-kn9fn\" (UID: \"02f3d0c6-4116-4134-ba51-1c02c73bef05\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn9fn" Jul 21 14:59:29.413695 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.413027 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sm77n\" (UniqueName: \"kubernetes.io/projected/02f3d0c6-4116-4134-ba51-1c02c73bef05-kube-api-access-sm77n\") pod \"ovnkube-node-kn9fn\" (UID: \"02f3d0c6-4116-4134-ba51-1c02c73bef05\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn9fn" Jul 21 14:59:29.413695 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.413052 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86qgd\" (UniqueName: \"kubernetes.io/projected/4f638c31-bce1-4b95-97a3-410069ef52e9-kube-api-access-86qgd\") pod \"aws-ebs-csi-driver-node-vl2rt\" (UID: \"4f638c31-bce1-4b95-97a3-410069ef52e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vl2rt" Jul 21 14:59:29.413695 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.413069 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8e65c9e9-347b-46b1-a1f5-774fde3b8107-host-run-k8s-cni-cncf-io\") pod \"multus-8vxjj\" (UID: \"8e65c9e9-347b-46b1-a1f5-774fde3b8107\") " pod="openshift-multus/multus-8vxjj" Jul 21 14:59:29.514197 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.514171 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8e65c9e9-347b-46b1-a1f5-774fde3b8107-cnibin\") pod \"multus-8vxjj\" (UID: \"8e65c9e9-347b-46b1-a1f5-774fde3b8107\") " pod="openshift-multus/multus-8vxjj" Jul 21 14:59:29.514197 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.514202 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/327b1f5e-8e27-4572-8025-5c516e59d481-cnibin\") pod \"multus-additional-cni-plugins-8h5jp\" (UID: \"327b1f5e-8e27-4572-8025-5c516e59d481\") " pod="openshift-multus/multus-additional-cni-plugins-8h5jp" Jul 21 14:59:29.514455 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.514218 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/02f3d0c6-4116-4134-ba51-1c02c73bef05-var-lib-openvswitch\") pod \"ovnkube-node-kn9fn\" (UID: \"02f3d0c6-4116-4134-ba51-1c02c73bef05\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn9fn" Jul 21 14:59:29.514455 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.514232 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/02f3d0c6-4116-4134-ba51-1c02c73bef05-node-log\") pod \"ovnkube-node-kn9fn\" (UID: \"02f3d0c6-4116-4134-ba51-1c02c73bef05\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn9fn" Jul 21 14:59:29.514455 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.514258 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4d94d908-6ab5-4463-a878-0b340bb80365-host\") pod \"node-ca-9qzhs\" (UID: \"4d94d908-6ab5-4463-a878-0b340bb80365\") " pod="openshift-image-registry/node-ca-9qzhs" Jul 21 14:59:29.514455 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.514280 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a6348ec1-f9c5-487a-9cfe-abd129eadfe0-etc-sysctl-conf\") pod \"tuned-bj8nf\" (UID: \"a6348ec1-f9c5-487a-9cfe-abd129eadfe0\") " pod="openshift-cluster-node-tuning-operator/tuned-bj8nf" Jul 21 14:59:29.514455 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.514298 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/327b1f5e-8e27-4572-8025-5c516e59d481-cnibin\") pod \"multus-additional-cni-plugins-8h5jp\" (UID: \"327b1f5e-8e27-4572-8025-5c516e59d481\") " pod="openshift-multus/multus-additional-cni-plugins-8h5jp" Jul 21 14:59:29.514455 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.514305 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/02f3d0c6-4116-4134-ba51-1c02c73bef05-node-log\") pod \"ovnkube-node-kn9fn\" (UID: \"02f3d0c6-4116-4134-ba51-1c02c73bef05\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn9fn" Jul 21 14:59:29.514455 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.514298 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/02f3d0c6-4116-4134-ba51-1c02c73bef05-var-lib-openvswitch\") pod \"ovnkube-node-kn9fn\" (UID: \"02f3d0c6-4116-4134-ba51-1c02c73bef05\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn9fn" Jul 21 14:59:29.514455 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.514302 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86c9j\" (UniqueName: \"kubernetes.io/projected/a6348ec1-f9c5-487a-9cfe-abd129eadfe0-kube-api-access-86c9j\") pod \"tuned-bj8nf\" (UID: \"a6348ec1-f9c5-487a-9cfe-abd129eadfe0\") " pod="openshift-cluster-node-tuning-operator/tuned-bj8nf" Jul 21 14:59:29.514455 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.514341 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4d94d908-6ab5-4463-a878-0b340bb80365-host\") pod \"node-ca-9qzhs\" (UID: \"4d94d908-6ab5-4463-a878-0b340bb80365\") " pod="openshift-image-registry/node-ca-9qzhs" Jul 21 14:59:29.514455 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.514420 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8d8n\" (UniqueName: \"kubernetes.io/projected/8bf74430-9a26-4473-ace9-3d6034a94cf1-kube-api-access-v8d8n\") pod \"iptables-alerter-7q2dq\" (UID: \"8bf74430-9a26-4473-ace9-3d6034a94cf1\") " pod="openshift-network-operator/iptables-alerter-7q2dq" Jul 21 14:59:29.514455 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.514455 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/327b1f5e-8e27-4572-8025-5c516e59d481-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8h5jp\" (UID: \"327b1f5e-8e27-4572-8025-5c516e59d481\") " pod="openshift-multus/multus-additional-cni-plugins-8h5jp" Jul 21 14:59:29.514965 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.514297 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8e65c9e9-347b-46b1-a1f5-774fde3b8107-cnibin\") pod \"multus-8vxjj\" (UID: \"8e65c9e9-347b-46b1-a1f5-774fde3b8107\") " pod="openshift-multus/multus-8vxjj" Jul 21 14:59:29.514965 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.514484 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/02f3d0c6-4116-4134-ba51-1c02c73bef05-host-slash\") pod \"ovnkube-node-kn9fn\" (UID: \"02f3d0c6-4116-4134-ba51-1c02c73bef05\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn9fn" Jul 21 14:59:29.514965 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.514475 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a6348ec1-f9c5-487a-9cfe-abd129eadfe0-etc-sysctl-conf\") pod \"tuned-bj8nf\" (UID: \"a6348ec1-f9c5-487a-9cfe-abd129eadfe0\") " pod="openshift-cluster-node-tuning-operator/tuned-bj8nf" Jul 21 14:59:29.514965 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.514526 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/02f3d0c6-4116-4134-ba51-1c02c73bef05-host-slash\") pod \"ovnkube-node-kn9fn\" (UID: \"02f3d0c6-4116-4134-ba51-1c02c73bef05\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn9fn" Jul 21 14:59:29.514965 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.514528 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e706cd7a-7458-4f77-b02d-3fd5837d735d-original-pull-secret\") pod \"global-pull-secret-syncer-jttxg\" (UID: \"e706cd7a-7458-4f77-b02d-3fd5837d735d\") " pod="kube-system/global-pull-secret-syncer-jttxg" Jul 21 14:59:29.514965 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.514559 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4f638c31-bce1-4b95-97a3-410069ef52e9-registration-dir\") pod \"aws-ebs-csi-driver-node-vl2rt\" (UID: \"4f638c31-bce1-4b95-97a3-410069ef52e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vl2rt" Jul 21 14:59:29.514965 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.514583 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4f638c31-bce1-4b95-97a3-410069ef52e9-device-dir\") pod \"aws-ebs-csi-driver-node-vl2rt\" (UID: \"4f638c31-bce1-4b95-97a3-410069ef52e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vl2rt" Jul 21 14:59:29.514965 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.514607 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/327b1f5e-8e27-4572-8025-5c516e59d481-os-release\") pod \"multus-additional-cni-plugins-8h5jp\" (UID: \"327b1f5e-8e27-4572-8025-5c516e59d481\") " pod="openshift-multus/multus-additional-cni-plugins-8h5jp" Jul 21 14:59:29.514965 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.514628 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4f638c31-bce1-4b95-97a3-410069ef52e9-registration-dir\") pod \"aws-ebs-csi-driver-node-vl2rt\" (UID: \"4f638c31-bce1-4b95-97a3-410069ef52e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vl2rt" Jul 21 14:59:29.514965 ip-10-0-135-176 kubenswrapper[2563]: E0721 14:59:29.514638 2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 21 14:59:29.514965 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.514633 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lm8nt\" (UniqueName: \"kubernetes.io/projected/26de6bb3-eb05-461a-958f-406464dea020-kube-api-access-lm8nt\") pod \"network-metrics-daemon-dh8jm\" (UID: \"26de6bb3-eb05-461a-958f-406464dea020\") " pod="openshift-multus/network-metrics-daemon-dh8jm" Jul 21 14:59:29.514965 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.514666 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/02f3d0c6-4116-4134-ba51-1c02c73bef05-host-cni-netd\") pod \"ovnkube-node-kn9fn\" (UID: \"02f3d0c6-4116-4134-ba51-1c02c73bef05\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn9fn" Jul 21 14:59:29.514965 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.514674 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4f638c31-bce1-4b95-97a3-410069ef52e9-device-dir\") pod \"aws-ebs-csi-driver-node-vl2rt\" (UID: \"4f638c31-bce1-4b95-97a3-410069ef52e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vl2rt" Jul 21 14:59:29.514965 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.514696 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/02f3d0c6-4116-4134-ba51-1c02c73bef05-ovn-node-metrics-cert\") pod \"ovnkube-node-kn9fn\" (UID: \"02f3d0c6-4116-4134-ba51-1c02c73bef05\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn9fn" Jul 21 14:59:29.514965 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.514756 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/327b1f5e-8e27-4572-8025-5c516e59d481-os-release\") pod \"multus-additional-cni-plugins-8h5jp\" (UID: \"327b1f5e-8e27-4572-8025-5c516e59d481\") " pod="openshift-multus/multus-additional-cni-plugins-8h5jp" Jul 21 14:59:29.514965 ip-10-0-135-176 kubenswrapper[2563]: E0721 14:59:29.514767 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e706cd7a-7458-4f77-b02d-3fd5837d735d-original-pull-secret podName:e706cd7a-7458-4f77-b02d-3fd5837d735d nodeName:}" failed. No retries permitted until 2026-07-21 14:59:30.014737644 +0000 UTC m=+2.010523039 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e706cd7a-7458-4f77-b02d-3fd5837d735d-original-pull-secret") pod "global-pull-secret-syncer-jttxg" (UID: "e706cd7a-7458-4f77-b02d-3fd5837d735d") : object "kube-system"/"original-pull-secret" not registered Jul 21 14:59:29.514965 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.514794 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a6348ec1-f9c5-487a-9cfe-abd129eadfe0-tmp\") pod \"tuned-bj8nf\" (UID: \"a6348ec1-f9c5-487a-9cfe-abd129eadfe0\") " pod="openshift-cluster-node-tuning-operator/tuned-bj8nf" Jul 21 14:59:29.515798 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.514802 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/02f3d0c6-4116-4134-ba51-1c02c73bef05-host-cni-netd\") pod \"ovnkube-node-kn9fn\" (UID: \"02f3d0c6-4116-4134-ba51-1c02c73bef05\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn9fn" Jul 21 14:59:29.515798 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.514826 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8bf74430-9a26-4473-ace9-3d6034a94cf1-iptables-alerter-script\") pod \"iptables-alerter-7q2dq\" (UID: \"8bf74430-9a26-4473-ace9-3d6034a94cf1\") " pod="openshift-network-operator/iptables-alerter-7q2dq" Jul 21 14:59:29.515798 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.514856 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4f638c31-bce1-4b95-97a3-410069ef52e9-sys-fs\") pod \"aws-ebs-csi-driver-node-vl2rt\" (UID: \"4f638c31-bce1-4b95-97a3-410069ef52e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vl2rt" Jul 21 14:59:29.515798 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.514884 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8e65c9e9-347b-46b1-a1f5-774fde3b8107-hostroot\") pod \"multus-8vxjj\" (UID: \"8e65c9e9-347b-46b1-a1f5-774fde3b8107\") " pod="openshift-multus/multus-8vxjj" Jul 21 14:59:29.515798 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.514915 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvwvn\" (UniqueName: \"kubernetes.io/projected/42b48be4-5000-431e-9d8b-b932609faca0-kube-api-access-lvwvn\") pod \"node-resolver-h2kn2\" (UID: \"42b48be4-5000-431e-9d8b-b932609faca0\") " pod="openshift-dns/node-resolver-h2kn2" Jul 21 14:59:29.515798 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.514940 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/02f3d0c6-4116-4134-ba51-1c02c73bef05-run-systemd\") pod \"ovnkube-node-kn9fn\" (UID: \"02f3d0c6-4116-4134-ba51-1c02c73bef05\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn9fn" Jul 21 14:59:29.515798 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.514963 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/02f3d0c6-4116-4134-ba51-1c02c73bef05-log-socket\") pod \"ovnkube-node-kn9fn\" (UID: \"02f3d0c6-4116-4134-ba51-1c02c73bef05\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn9fn" Jul 21 14:59:29.515798 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.514986 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/02f3d0c6-4116-4134-ba51-1c02c73bef05-host-cni-bin\") pod \"ovnkube-node-kn9fn\" (UID: \"02f3d0c6-4116-4134-ba51-1c02c73bef05\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn9fn" Jul 21 14:59:29.515798 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.515012 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dh6gh\" (UniqueName: \"kubernetes.io/projected/0683eb8b-8af7-4056-beed-3bfadd624f56-kube-api-access-dh6gh\") pod \"network-check-target-bzwfd\" (UID: \"0683eb8b-8af7-4056-beed-3bfadd624f56\") " pod="openshift-network-diagnostics/network-check-target-bzwfd" Jul 21 14:59:29.515798 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.515034 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a6348ec1-f9c5-487a-9cfe-abd129eadfe0-run\") pod \"tuned-bj8nf\" (UID: \"a6348ec1-f9c5-487a-9cfe-abd129eadfe0\") " pod="openshift-cluster-node-tuning-operator/tuned-bj8nf" Jul 21 14:59:29.515798 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.515060 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/327b1f5e-8e27-4572-8025-5c516e59d481-system-cni-dir\") pod \"multus-additional-cni-plugins-8h5jp\" (UID: \"327b1f5e-8e27-4572-8025-5c516e59d481\") " pod="openshift-multus/multus-additional-cni-plugins-8h5jp" Jul 21 14:59:29.515798 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.515131 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/02f3d0c6-4116-4134-ba51-1c02c73bef05-run-openvswitch\") pod \"ovnkube-node-kn9fn\" (UID: \"02f3d0c6-4116-4134-ba51-1c02c73bef05\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn9fn" Jul 21 14:59:29.515798 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.515155 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkwxv\" (UniqueName: \"kubernetes.io/projected/4d94d908-6ab5-4463-a878-0b340bb80365-kube-api-access-qkwxv\") pod \"node-ca-9qzhs\" (UID: \"4d94d908-6ab5-4463-a878-0b340bb80365\") " pod="openshift-image-registry/node-ca-9qzhs" Jul 21 14:59:29.515798 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.515180 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a6348ec1-f9c5-487a-9cfe-abd129eadfe0-etc-modprobe-d\") pod \"tuned-bj8nf\" (UID: \"a6348ec1-f9c5-487a-9cfe-abd129eadfe0\") " pod="openshift-cluster-node-tuning-operator/tuned-bj8nf" Jul 21 14:59:29.515798 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.515208 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/327b1f5e-8e27-4572-8025-5c516e59d481-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8h5jp\" (UID: \"327b1f5e-8e27-4572-8025-5c516e59d481\") " pod="openshift-multus/multus-additional-cni-plugins-8h5jp" Jul 21 14:59:29.515798 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.515211 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8e65c9e9-347b-46b1-a1f5-774fde3b8107-hostroot\") pod \"multus-8vxjj\" (UID: \"8e65c9e9-347b-46b1-a1f5-774fde3b8107\") " pod="openshift-multus/multus-8vxjj" Jul 21 14:59:29.515798 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.515238 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/327b1f5e-8e27-4572-8025-5c516e59d481-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8h5jp\" (UID: \"327b1f5e-8e27-4572-8025-5c516e59d481\") " pod="openshift-multus/multus-additional-cni-plugins-8h5jp" Jul 21 14:59:29.516629 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.515277 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/02f3d0c6-4116-4134-ba51-1c02c73bef05-log-socket\") pod \"ovnkube-node-kn9fn\" (UID: \"02f3d0c6-4116-4134-ba51-1c02c73bef05\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn9fn" Jul 21 14:59:29.516629 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.515313 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/02f3d0c6-4116-4134-ba51-1c02c73bef05-host-cni-bin\") pod \"ovnkube-node-kn9fn\" (UID: \"02f3d0c6-4116-4134-ba51-1c02c73bef05\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn9fn" Jul 21 14:59:29.516629 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.516077 2563 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 21 14:59:29.516629 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.516134 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/02f3d0c6-4116-4134-ba51-1c02c73bef05-run-systemd\") pod \"ovnkube-node-kn9fn\" (UID: \"02f3d0c6-4116-4134-ba51-1c02c73bef05\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn9fn" Jul 21 14:59:29.516629 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.516182 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/02f3d0c6-4116-4134-ba51-1c02c73bef05-ovnkube-script-lib\") pod \"ovnkube-node-kn9fn\" (UID: \"02f3d0c6-4116-4134-ba51-1c02c73bef05\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn9fn" Jul 21 14:59:29.516629 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.516222 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sm77n\" (UniqueName: \"kubernetes.io/projected/02f3d0c6-4116-4134-ba51-1c02c73bef05-kube-api-access-sm77n\") pod \"ovnkube-node-kn9fn\" (UID: \"02f3d0c6-4116-4134-ba51-1c02c73bef05\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn9fn" Jul 21 14:59:29.516629 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.516257 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86qgd\" (UniqueName: \"kubernetes.io/projected/4f638c31-bce1-4b95-97a3-410069ef52e9-kube-api-access-86qgd\") pod \"aws-ebs-csi-driver-node-vl2rt\" (UID: \"4f638c31-bce1-4b95-97a3-410069ef52e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vl2rt" Jul 21 14:59:29.516629 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.516266 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8bf74430-9a26-4473-ace9-3d6034a94cf1-iptables-alerter-script\") pod \"iptables-alerter-7q2dq\" (UID: \"8bf74430-9a26-4473-ace9-3d6034a94cf1\") " pod="openshift-network-operator/iptables-alerter-7q2dq" Jul 21 14:59:29.516629 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.516282 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a6348ec1-f9c5-487a-9cfe-abd129eadfe0-etc-modprobe-d\") pod \"tuned-bj8nf\" (UID: \"a6348ec1-f9c5-487a-9cfe-abd129eadfe0\") " pod="openshift-cluster-node-tuning-operator/tuned-bj8nf" Jul 21 14:59:29.516629 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.516288 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8e65c9e9-347b-46b1-a1f5-774fde3b8107-host-run-k8s-cni-cncf-io\") pod \"multus-8vxjj\" (UID: \"8e65c9e9-347b-46b1-a1f5-774fde3b8107\") " pod="openshift-multus/multus-8vxjj" Jul 21 14:59:29.516629 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.516349 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4f638c31-bce1-4b95-97a3-410069ef52e9-sys-fs\") pod \"aws-ebs-csi-driver-node-vl2rt\" (UID: \"4f638c31-bce1-4b95-97a3-410069ef52e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vl2rt" Jul 21 14:59:29.516629 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.516568 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8e65c9e9-347b-46b1-a1f5-774fde3b8107-os-release\") pod \"multus-8vxjj\" (UID: \"8e65c9e9-347b-46b1-a1f5-774fde3b8107\") " pod="openshift-multus/multus-8vxjj" Jul 21 14:59:29.516629 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.516577 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a6348ec1-f9c5-487a-9cfe-abd129eadfe0-run\") pod \"tuned-bj8nf\" (UID: \"a6348ec1-f9c5-487a-9cfe-abd129eadfe0\") " pod="openshift-cluster-node-tuning-operator/tuned-bj8nf" Jul 21 14:59:29.516629 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.516628 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8e65c9e9-347b-46b1-a1f5-774fde3b8107-host-run-k8s-cni-cncf-io\") pod \"multus-8vxjj\" (UID: \"8e65c9e9-347b-46b1-a1f5-774fde3b8107\") " pod="openshift-multus/multus-8vxjj" Jul 21 14:59:29.517308 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.516648 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8e65c9e9-347b-46b1-a1f5-774fde3b8107-os-release\") pod \"multus-8vxjj\" (UID: \"8e65c9e9-347b-46b1-a1f5-774fde3b8107\") " pod="openshift-multus/multus-8vxjj" Jul 21 14:59:29.517308 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.516664 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/02f3d0c6-4116-4134-ba51-1c02c73bef05-run-openvswitch\") pod \"ovnkube-node-kn9fn\" (UID: \"02f3d0c6-4116-4134-ba51-1c02c73bef05\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn9fn" Jul 21 14:59:29.517308 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.516712 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/327b1f5e-8e27-4572-8025-5c516e59d481-system-cni-dir\") pod \"multus-additional-cni-plugins-8h5jp\" (UID: \"327b1f5e-8e27-4572-8025-5c516e59d481\") " pod="openshift-multus/multus-additional-cni-plugins-8h5jp" Jul 21 14:59:29.517308 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.516838 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8e65c9e9-347b-46b1-a1f5-774fde3b8107-host-run-netns\") pod \"multus-8vxjj\" (UID: \"8e65c9e9-347b-46b1-a1f5-774fde3b8107\") " pod="openshift-multus/multus-8vxjj" Jul 21 14:59:29.517308 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.516884 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/327b1f5e-8e27-4572-8025-5c516e59d481-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8h5jp\" (UID: \"327b1f5e-8e27-4572-8025-5c516e59d481\") " pod="openshift-multus/multus-additional-cni-plugins-8h5jp" Jul 21 14:59:29.517308 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.516957 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8e65c9e9-347b-46b1-a1f5-774fde3b8107-host-run-netns\") pod \"multus-8vxjj\" (UID: \"8e65c9e9-347b-46b1-a1f5-774fde3b8107\") " pod="openshift-multus/multus-8vxjj" Jul 21 14:59:29.517308 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.516940 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/327b1f5e-8e27-4572-8025-5c516e59d481-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8h5jp\" (UID: \"327b1f5e-8e27-4572-8025-5c516e59d481\") " pod="openshift-multus/multus-additional-cni-plugins-8h5jp" Jul 21 14:59:29.517308 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.516996 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/42b48be4-5000-431e-9d8b-b932609faca0-hosts-file\") pod \"node-resolver-h2kn2\" (UID: \"42b48be4-5000-431e-9d8b-b932609faca0\") " pod="openshift-dns/node-resolver-h2kn2" Jul 21 14:59:29.517308 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.517043 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42b48be4-5000-431e-9d8b-b932609faca0-tmp-dir\") pod \"node-resolver-h2kn2\" (UID: \"42b48be4-5000-431e-9d8b-b932609faca0\") " pod="openshift-dns/node-resolver-h2kn2" Jul 21 14:59:29.517308 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.517078 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/02f3d0c6-4116-4134-ba51-1c02c73bef05-host-run-netns\") pod \"ovnkube-node-kn9fn\" (UID: \"02f3d0c6-4116-4134-ba51-1c02c73bef05\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn9fn" Jul 21 14:59:29.517308 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.517122 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/327b1f5e-8e27-4572-8025-5c516e59d481-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8h5jp\" (UID: \"327b1f5e-8e27-4572-8025-5c516e59d481\") " pod="openshift-multus/multus-additional-cni-plugins-8h5jp" Jul 21 14:59:29.517308 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.517142 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/216f52c0-8c33-426f-b2eb-244a5787bdeb-konnectivity-ca\") pod \"konnectivity-agent-m4pp8\" (UID: \"216f52c0-8c33-426f-b2eb-244a5787bdeb\") " pod="kube-system/konnectivity-agent-m4pp8" Jul 21 14:59:29.517308 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.517181 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/42b48be4-5000-431e-9d8b-b932609faca0-hosts-file\") pod \"node-resolver-h2kn2\" (UID: \"42b48be4-5000-431e-9d8b-b932609faca0\") " pod="openshift-dns/node-resolver-h2kn2" Jul 21 14:59:29.517308 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.517186 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/02f3d0c6-4116-4134-ba51-1c02c73bef05-ovnkube-script-lib\") pod \"ovnkube-node-kn9fn\" (UID: \"02f3d0c6-4116-4134-ba51-1c02c73bef05\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn9fn" Jul 21 14:59:29.517308 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.517172 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8e65c9e9-347b-46b1-a1f5-774fde3b8107-system-cni-dir\") pod \"multus-8vxjj\" (UID: \"8e65c9e9-347b-46b1-a1f5-774fde3b8107\") " pod="openshift-multus/multus-8vxjj" Jul 21 14:59:29.517308 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.517254 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/02f3d0c6-4116-4134-ba51-1c02c73bef05-host-run-netns\") pod \"ovnkube-node-kn9fn\" (UID: \"02f3d0c6-4116-4134-ba51-1c02c73bef05\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn9fn" Jul 21 14:59:29.518046 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.517418 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8e65c9e9-347b-46b1-a1f5-774fde3b8107-cni-binary-copy\") pod \"multus-8vxjj\" (UID: \"8e65c9e9-347b-46b1-a1f5-774fde3b8107\") " pod="openshift-multus/multus-8vxjj" Jul 21 14:59:29.518046 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.517499 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26de6bb3-eb05-461a-958f-406464dea020-metrics-certs\") pod \"network-metrics-daemon-dh8jm\" (UID: \"26de6bb3-eb05-461a-958f-406464dea020\") " pod="openshift-multus/network-metrics-daemon-dh8jm" Jul 21 14:59:29.518046 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.517523 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8e65c9e9-347b-46b1-a1f5-774fde3b8107-system-cni-dir\") pod \"multus-8vxjj\" (UID: \"8e65c9e9-347b-46b1-a1f5-774fde3b8107\") " pod="openshift-multus/multus-8vxjj" Jul 21 14:59:29.518046 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.517571 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/02f3d0c6-4116-4134-ba51-1c02c73bef05-systemd-units\") pod \"ovnkube-node-kn9fn\" (UID: \"02f3d0c6-4116-4134-ba51-1c02c73bef05\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn9fn" Jul 21 14:59:29.518046 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.517622 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/02f3d0c6-4116-4134-ba51-1c02c73bef05-host-run-ovn-kubernetes\") pod \"ovnkube-node-kn9fn\" (UID: \"02f3d0c6-4116-4134-ba51-1c02c73bef05\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn9fn" Jul 21 14:59:29.518046 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.517816 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a6348ec1-f9c5-487a-9cfe-abd129eadfe0-etc-systemd\") pod \"tuned-bj8nf\" (UID: \"a6348ec1-f9c5-487a-9cfe-abd129eadfe0\") " pod="openshift-cluster-node-tuning-operator/tuned-bj8nf" Jul 21 14:59:29.518046 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.517847 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4f638c31-bce1-4b95-97a3-410069ef52e9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vl2rt\" (UID: \"4f638c31-bce1-4b95-97a3-410069ef52e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vl2rt" Jul 21 14:59:29.518046 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.517883 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8e65c9e9-347b-46b1-a1f5-774fde3b8107-host-run-multus-certs\") pod \"multus-8vxjj\" (UID: \"8e65c9e9-347b-46b1-a1f5-774fde3b8107\") " pod="openshift-multus/multus-8vxjj" Jul 21 14:59:29.518046 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.517915 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8e65c9e9-347b-46b1-a1f5-774fde3b8107-etc-kubernetes\") pod \"multus-8vxjj\" (UID: \"8e65c9e9-347b-46b1-a1f5-774fde3b8107\") " pod="openshift-multus/multus-8vxjj" Jul 21 14:59:29.518497 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.518079 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/216f52c0-8c33-426f-b2eb-244a5787bdeb-konnectivity-ca\") pod \"konnectivity-agent-m4pp8\" (UID: \"216f52c0-8c33-426f-b2eb-244a5787bdeb\") " pod="kube-system/konnectivity-agent-m4pp8" Jul 21 14:59:29.518497 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.518102 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8e65c9e9-347b-46b1-a1f5-774fde3b8107-cni-binary-copy\") pod \"multus-8vxjj\" (UID: \"8e65c9e9-347b-46b1-a1f5-774fde3b8107\") " pod="openshift-multus/multus-8vxjj" Jul 21 14:59:29.518497 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.518168 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/02f3d0c6-4116-4134-ba51-1c02c73bef05-systemd-units\") pod \"ovnkube-node-kn9fn\" (UID: \"02f3d0c6-4116-4134-ba51-1c02c73bef05\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn9fn" Jul 21 14:59:29.518497 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.518162 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/02f3d0c6-4116-4134-ba51-1c02c73bef05-run-ovn\") pod \"ovnkube-node-kn9fn\" (UID: \"02f3d0c6-4116-4134-ba51-1c02c73bef05\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn9fn" Jul 21 14:59:29.518497 ip-10-0-135-176 kubenswrapper[2563]: E0721 14:59:29.518182 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 14:59:29.518497 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.518214 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8e65c9e9-347b-46b1-a1f5-774fde3b8107-etc-kubernetes\") pod \"multus-8vxjj\" (UID: \"8e65c9e9-347b-46b1-a1f5-774fde3b8107\") " pod="openshift-multus/multus-8vxjj" Jul 21 14:59:29.518497 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.518237 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e706cd7a-7458-4f77-b02d-3fd5837d735d-dbus\") pod \"global-pull-secret-syncer-jttxg\" (UID: \"e706cd7a-7458-4f77-b02d-3fd5837d735d\") " pod="kube-system/global-pull-secret-syncer-jttxg" Jul 21 14:59:29.518497 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.518265 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/02f3d0c6-4116-4134-ba51-1c02c73bef05-host-run-ovn-kubernetes\") pod \"ovnkube-node-kn9fn\" (UID: \"02f3d0c6-4116-4134-ba51-1c02c73bef05\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn9fn" Jul 21 14:59:29.518497 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.518279 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4f638c31-bce1-4b95-97a3-410069ef52e9-socket-dir\") pod \"aws-ebs-csi-driver-node-vl2rt\" (UID: \"4f638c31-bce1-4b95-97a3-410069ef52e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vl2rt" Jul 21 14:59:29.518497 ip-10-0-135-176 kubenswrapper[2563]: E0721 14:59:29.518385 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26de6bb3-eb05-461a-958f-406464dea020-metrics-certs podName:26de6bb3-eb05-461a-958f-406464dea020 nodeName:}" failed. No retries permitted until 2026-07-21 14:59:30.018362654 +0000 UTC m=+2.014148067 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/26de6bb3-eb05-461a-958f-406464dea020-metrics-certs") pod "network-metrics-daemon-dh8jm" (UID: "26de6bb3-eb05-461a-958f-406464dea020") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 14:59:29.518497 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.518371 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8e65c9e9-347b-46b1-a1f5-774fde3b8107-host-run-multus-certs\") pod \"multus-8vxjj\" (UID: \"8e65c9e9-347b-46b1-a1f5-774fde3b8107\") " pod="openshift-multus/multus-8vxjj" Jul 21 14:59:29.518497 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.518427 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a6348ec1-f9c5-487a-9cfe-abd129eadfe0-etc-systemd\") pod \"tuned-bj8nf\" (UID: \"a6348ec1-f9c5-487a-9cfe-abd129eadfe0\") " pod="openshift-cluster-node-tuning-operator/tuned-bj8nf" Jul 21 14:59:29.518497 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.518430 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/02f3d0c6-4116-4134-ba51-1c02c73bef05-run-ovn\") pod \"ovnkube-node-kn9fn\" (UID: \"02f3d0c6-4116-4134-ba51-1c02c73bef05\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn9fn" Jul 21 14:59:29.518497 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.518463 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8e65c9e9-347b-46b1-a1f5-774fde3b8107-host-var-lib-kubelet\") pod \"multus-8vxjj\" (UID: \"8e65c9e9-347b-46b1-a1f5-774fde3b8107\") " pod="openshift-multus/multus-8vxjj" Jul 21 14:59:29.519102 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.518525 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8e65c9e9-347b-46b1-a1f5-774fde3b8107-multus-cni-dir\") pod \"multus-8vxjj\" (UID: \"8e65c9e9-347b-46b1-a1f5-774fde3b8107\") " pod="openshift-multus/multus-8vxjj" Jul 21 14:59:29.519102 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.518540 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e706cd7a-7458-4f77-b02d-3fd5837d735d-dbus\") pod \"global-pull-secret-syncer-jttxg\" (UID: \"e706cd7a-7458-4f77-b02d-3fd5837d735d\") " pod="kube-system/global-pull-secret-syncer-jttxg" Jul 21 14:59:29.519102 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.518542 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4f638c31-bce1-4b95-97a3-410069ef52e9-socket-dir\") pod \"aws-ebs-csi-driver-node-vl2rt\" (UID: \"4f638c31-bce1-4b95-97a3-410069ef52e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vl2rt" Jul 21 14:59:29.519102 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.518559 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8e65c9e9-347b-46b1-a1f5-774fde3b8107-multus-socket-dir-parent\") pod \"multus-8vxjj\" (UID: \"8e65c9e9-347b-46b1-a1f5-774fde3b8107\") " pod="openshift-multus/multus-8vxjj" Jul 21 14:59:29.519102 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.518588 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4f638c31-bce1-4b95-97a3-410069ef52e9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vl2rt\" (UID: \"4f638c31-bce1-4b95-97a3-410069ef52e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vl2rt" Jul 21 14:59:29.519102 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.518639 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8e65c9e9-347b-46b1-a1f5-774fde3b8107-host-var-lib-cni-multus\") pod \"multus-8vxjj\" (UID: \"8e65c9e9-347b-46b1-a1f5-774fde3b8107\") " pod="openshift-multus/multus-8vxjj" Jul 21 14:59:29.519102 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.518672 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8e65c9e9-347b-46b1-a1f5-774fde3b8107-multus-daemon-config\") pod \"multus-8vxjj\" (UID: \"8e65c9e9-347b-46b1-a1f5-774fde3b8107\") " pod="openshift-multus/multus-8vxjj" Jul 21 14:59:29.519102 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.518640 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8e65c9e9-347b-46b1-a1f5-774fde3b8107-host-var-lib-kubelet\") pod \"multus-8vxjj\" (UID: \"8e65c9e9-347b-46b1-a1f5-774fde3b8107\") " pod="openshift-multus/multus-8vxjj" Jul 21 14:59:29.519102 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.518695 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8e65c9e9-347b-46b1-a1f5-774fde3b8107-multus-cni-dir\") pod \"multus-8vxjj\" (UID: \"8e65c9e9-347b-46b1-a1f5-774fde3b8107\") " pod="openshift-multus/multus-8vxjj" Jul 21 14:59:29.519102 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.518741 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/02f3d0c6-4116-4134-ba51-1c02c73bef05-etc-openvswitch\") pod \"ovnkube-node-kn9fn\" (UID: \"02f3d0c6-4116-4134-ba51-1c02c73bef05\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn9fn" Jul 21 14:59:29.519102 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.518777 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4d94d908-6ab5-4463-a878-0b340bb80365-serviceca\") pod \"node-ca-9qzhs\" (UID: \"4d94d908-6ab5-4463-a878-0b340bb80365\") " pod="openshift-image-registry/node-ca-9qzhs" Jul 21 14:59:29.519102 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.518849 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a6348ec1-f9c5-487a-9cfe-abd129eadfe0-etc-kubernetes\") pod \"tuned-bj8nf\" (UID: \"a6348ec1-f9c5-487a-9cfe-abd129eadfe0\") " pod="openshift-cluster-node-tuning-operator/tuned-bj8nf" Jul 21 14:59:29.519102 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.518876 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42b48be4-5000-431e-9d8b-b932609faca0-tmp-dir\") pod \"node-resolver-h2kn2\" (UID: \"42b48be4-5000-431e-9d8b-b932609faca0\") " pod="openshift-dns/node-resolver-h2kn2" Jul 21 14:59:29.519102 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.518886 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r785r\" (UniqueName: \"kubernetes.io/projected/327b1f5e-8e27-4572-8025-5c516e59d481-kube-api-access-r785r\") pod \"multus-additional-cni-plugins-8h5jp\" (UID: \"327b1f5e-8e27-4572-8025-5c516e59d481\") " pod="openshift-multus/multus-additional-cni-plugins-8h5jp" Jul 21 14:59:29.519102 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.518955 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/02f3d0c6-4116-4134-ba51-1c02c73bef05-env-overrides\") pod \"ovnkube-node-kn9fn\" (UID: \"02f3d0c6-4116-4134-ba51-1c02c73bef05\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn9fn" Jul 21 14:59:29.519102 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.518992 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a6348ec1-f9c5-487a-9cfe-abd129eadfe0-lib-modules\") pod \"tuned-bj8nf\" (UID: \"a6348ec1-f9c5-487a-9cfe-abd129eadfe0\") " pod="openshift-cluster-node-tuning-operator/tuned-bj8nf" Jul 21 14:59:29.519102 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.519026 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e706cd7a-7458-4f77-b02d-3fd5837d735d-kubelet-config\") pod \"global-pull-secret-syncer-jttxg\" (UID: \"e706cd7a-7458-4f77-b02d-3fd5837d735d\") " pod="kube-system/global-pull-secret-syncer-jttxg" Jul 21 14:59:29.519865 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.519079 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4f638c31-bce1-4b95-97a3-410069ef52e9-etc-selinux\") pod \"aws-ebs-csi-driver-node-vl2rt\" (UID: \"4f638c31-bce1-4b95-97a3-410069ef52e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vl2rt" Jul 21 14:59:29.519865 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.519119 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8e65c9e9-347b-46b1-a1f5-774fde3b8107-multus-conf-dir\") pod \"multus-8vxjj\" (UID: \"8e65c9e9-347b-46b1-a1f5-774fde3b8107\") " pod="openshift-multus/multus-8vxjj" Jul 21 14:59:29.519865 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.519239 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4v77\" (UniqueName: \"kubernetes.io/projected/8e65c9e9-347b-46b1-a1f5-774fde3b8107-kube-api-access-d4v77\") pod \"multus-8vxjj\" (UID: \"8e65c9e9-347b-46b1-a1f5-774fde3b8107\") " pod="openshift-multus/multus-8vxjj" Jul 21 14:59:29.519865 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.519283 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/02f3d0c6-4116-4134-ba51-1c02c73bef05-host-kubelet\") pod \"ovnkube-node-kn9fn\" (UID: \"02f3d0c6-4116-4134-ba51-1c02c73bef05\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn9fn" Jul 21 14:59:29.519865 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.519307 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4d94d908-6ab5-4463-a878-0b340bb80365-serviceca\") pod \"node-ca-9qzhs\" (UID: \"4d94d908-6ab5-4463-a878-0b340bb80365\") " pod="openshift-image-registry/node-ca-9qzhs" Jul 21 14:59:29.519865 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.519465 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4f638c31-bce1-4b95-97a3-410069ef52e9-etc-selinux\") pod \"aws-ebs-csi-driver-node-vl2rt\" (UID: \"4f638c31-bce1-4b95-97a3-410069ef52e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vl2rt" Jul 21 14:59:29.519865 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.519598 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/216f52c0-8c33-426f-b2eb-244a5787bdeb-agent-certs\") pod \"konnectivity-agent-m4pp8\" (UID: \"216f52c0-8c33-426f-b2eb-244a5787bdeb\") " pod="kube-system/konnectivity-agent-m4pp8" Jul 21 14:59:29.519865 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.519633 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a6348ec1-f9c5-487a-9cfe-abd129eadfe0-etc-sysconfig\") pod \"tuned-bj8nf\" (UID: \"a6348ec1-f9c5-487a-9cfe-abd129eadfe0\") " pod="openshift-cluster-node-tuning-operator/tuned-bj8nf" Jul 21 14:59:29.519865 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.519668 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a6348ec1-f9c5-487a-9cfe-abd129eadfe0-var-lib-kubelet\") pod \"tuned-bj8nf\" (UID: \"a6348ec1-f9c5-487a-9cfe-abd129eadfe0\") " pod="openshift-cluster-node-tuning-operator/tuned-bj8nf" Jul 21 14:59:29.519865 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.519701 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a6348ec1-f9c5-487a-9cfe-abd129eadfe0-etc-tuned\") pod \"tuned-bj8nf\" (UID: \"a6348ec1-f9c5-487a-9cfe-abd129eadfe0\") " pod="openshift-cluster-node-tuning-operator/tuned-bj8nf" Jul 21 14:59:29.519865 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.519735 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8e65c9e9-347b-46b1-a1f5-774fde3b8107-host-var-lib-cni-bin\") pod \"multus-8vxjj\" (UID: \"8e65c9e9-347b-46b1-a1f5-774fde3b8107\") " pod="openshift-multus/multus-8vxjj" Jul 21 14:59:29.519865 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.519744 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a6348ec1-f9c5-487a-9cfe-abd129eadfe0-tmp\") pod \"tuned-bj8nf\" (UID: \"a6348ec1-f9c5-487a-9cfe-abd129eadfe0\") " pod="openshift-cluster-node-tuning-operator/tuned-bj8nf" Jul 21 14:59:29.519865 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.519807 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/02f3d0c6-4116-4134-ba51-1c02c73bef05-host-kubelet\") pod \"ovnkube-node-kn9fn\" (UID: \"02f3d0c6-4116-4134-ba51-1c02c73bef05\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn9fn" Jul 21 14:59:29.519865 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.519850 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/02f3d0c6-4116-4134-ba51-1c02c73bef05-etc-openvswitch\") pod \"ovnkube-node-kn9fn\" (UID: \"02f3d0c6-4116-4134-ba51-1c02c73bef05\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn9fn" Jul 21 14:59:29.519865 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.519856 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/327b1f5e-8e27-4572-8025-5c516e59d481-cni-binary-copy\") pod \"multus-additional-cni-plugins-8h5jp\" (UID: \"327b1f5e-8e27-4572-8025-5c516e59d481\") " pod="openshift-multus/multus-additional-cni-plugins-8h5jp" Jul 21 14:59:29.519865 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.519848 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/02f3d0c6-4116-4134-ba51-1c02c73bef05-env-overrides\") pod \"ovnkube-node-kn9fn\" (UID: \"02f3d0c6-4116-4134-ba51-1c02c73bef05\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn9fn" Jul 21 14:59:29.520645 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.519896 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/02f3d0c6-4116-4134-ba51-1c02c73bef05-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kn9fn\" (UID: \"02f3d0c6-4116-4134-ba51-1c02c73bef05\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn9fn" Jul 21 14:59:29.520645 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.519922 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e706cd7a-7458-4f77-b02d-3fd5837d735d-kubelet-config\") pod \"global-pull-secret-syncer-jttxg\" (UID: \"e706cd7a-7458-4f77-b02d-3fd5837d735d\") " pod="kube-system/global-pull-secret-syncer-jttxg" Jul 21 14:59:29.520645 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.519927 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a6348ec1-f9c5-487a-9cfe-abd129eadfe0-sys\") pod \"tuned-bj8nf\" (UID: \"a6348ec1-f9c5-487a-9cfe-abd129eadfe0\") " pod="openshift-cluster-node-tuning-operator/tuned-bj8nf" Jul 21 14:59:29.520645 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.519951 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a6348ec1-f9c5-487a-9cfe-abd129eadfe0-var-lib-kubelet\") pod \"tuned-bj8nf\" (UID: \"a6348ec1-f9c5-487a-9cfe-abd129eadfe0\") " pod="openshift-cluster-node-tuning-operator/tuned-bj8nf" Jul 21 14:59:29.520645 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.519958 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a6348ec1-f9c5-487a-9cfe-abd129eadfe0-host\") pod \"tuned-bj8nf\" (UID: \"a6348ec1-f9c5-487a-9cfe-abd129eadfe0\") " pod="openshift-cluster-node-tuning-operator/tuned-bj8nf" Jul 21 14:59:29.520645 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.519992 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/02f3d0c6-4116-4134-ba51-1c02c73bef05-ovnkube-config\") pod \"ovnkube-node-kn9fn\" (UID: \"02f3d0c6-4116-4134-ba51-1c02c73bef05\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn9fn" Jul 21 14:59:29.520645 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.520020 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a6348ec1-f9c5-487a-9cfe-abd129eadfe0-lib-modules\") pod \"tuned-bj8nf\" (UID: \"a6348ec1-f9c5-487a-9cfe-abd129eadfe0\") " pod="openshift-cluster-node-tuning-operator/tuned-bj8nf" Jul 21 14:59:29.520645 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.520028 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8e65c9e9-347b-46b1-a1f5-774fde3b8107-host-var-lib-cni-bin\") pod \"multus-8vxjj\" (UID: \"8e65c9e9-347b-46b1-a1f5-774fde3b8107\") " pod="openshift-multus/multus-8vxjj" Jul 21 14:59:29.520645 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.519899 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/02f3d0c6-4116-4134-ba51-1c02c73bef05-ovn-node-metrics-cert\") pod \"ovnkube-node-kn9fn\" (UID: \"02f3d0c6-4116-4134-ba51-1c02c73bef05\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn9fn" Jul 21 14:59:29.520645 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.520028 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a6348ec1-f9c5-487a-9cfe-abd129eadfe0-etc-sysctl-d\") pod \"tuned-bj8nf\" (UID: \"a6348ec1-f9c5-487a-9cfe-abd129eadfe0\") " pod="openshift-cluster-node-tuning-operator/tuned-bj8nf" Jul 21 14:59:29.520645 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.520108 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a6348ec1-f9c5-487a-9cfe-abd129eadfe0-etc-kubernetes\") pod \"tuned-bj8nf\" (UID: \"a6348ec1-f9c5-487a-9cfe-abd129eadfe0\") " pod="openshift-cluster-node-tuning-operator/tuned-bj8nf" Jul 21 14:59:29.520645 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.520118 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a6348ec1-f9c5-487a-9cfe-abd129eadfe0-sys\") pod \"tuned-bj8nf\" (UID: \"a6348ec1-f9c5-487a-9cfe-abd129eadfe0\") " pod="openshift-cluster-node-tuning-operator/tuned-bj8nf" Jul 21 14:59:29.520645 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.520126 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a6348ec1-f9c5-487a-9cfe-abd129eadfe0-etc-sysconfig\") pod \"tuned-bj8nf\" (UID: \"a6348ec1-f9c5-487a-9cfe-abd129eadfe0\") " pod="openshift-cluster-node-tuning-operator/tuned-bj8nf" Jul 21 14:59:29.520645 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.520154 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8bf74430-9a26-4473-ace9-3d6034a94cf1-host-slash\") pod \"iptables-alerter-7q2dq\" (UID: \"8bf74430-9a26-4473-ace9-3d6034a94cf1\") " pod="openshift-network-operator/iptables-alerter-7q2dq" Jul 21 14:59:29.520645 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.520170 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/02f3d0c6-4116-4134-ba51-1c02c73bef05-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kn9fn\" (UID: \"02f3d0c6-4116-4134-ba51-1c02c73bef05\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn9fn" Jul 21 14:59:29.520645 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.520210 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a6348ec1-f9c5-487a-9cfe-abd129eadfe0-etc-sysctl-d\") pod \"tuned-bj8nf\" (UID: \"a6348ec1-f9c5-487a-9cfe-abd129eadfe0\") " pod="openshift-cluster-node-tuning-operator/tuned-bj8nf" Jul 21 14:59:29.520645 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.520266 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8bf74430-9a26-4473-ace9-3d6034a94cf1-host-slash\") pod \"iptables-alerter-7q2dq\" (UID: \"8bf74430-9a26-4473-ace9-3d6034a94cf1\") " pod="openshift-network-operator/iptables-alerter-7q2dq" Jul 21 14:59:29.521437 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.520319 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8e65c9e9-347b-46b1-a1f5-774fde3b8107-multus-daemon-config\") pod \"multus-8vxjj\" (UID: \"8e65c9e9-347b-46b1-a1f5-774fde3b8107\") " pod="openshift-multus/multus-8vxjj" Jul 21 14:59:29.521437 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.520403 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8e65c9e9-347b-46b1-a1f5-774fde3b8107-host-var-lib-cni-multus\") pod \"multus-8vxjj\" (UID: \"8e65c9e9-347b-46b1-a1f5-774fde3b8107\") " pod="openshift-multus/multus-8vxjj" Jul 21 14:59:29.521437 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.520461 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8e65c9e9-347b-46b1-a1f5-774fde3b8107-multus-conf-dir\") pod \"multus-8vxjj\" (UID: \"8e65c9e9-347b-46b1-a1f5-774fde3b8107\") " pod="openshift-multus/multus-8vxjj" Jul 21 14:59:29.521437 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.520502 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a6348ec1-f9c5-487a-9cfe-abd129eadfe0-host\") pod \"tuned-bj8nf\" (UID: \"a6348ec1-f9c5-487a-9cfe-abd129eadfe0\") " pod="openshift-cluster-node-tuning-operator/tuned-bj8nf" Jul 21 14:59:29.521437 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.521241 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/02f3d0c6-4116-4134-ba51-1c02c73bef05-ovnkube-config\") pod \"ovnkube-node-kn9fn\" (UID: \"02f3d0c6-4116-4134-ba51-1c02c73bef05\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn9fn" Jul 21 14:59:29.521437 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.521388 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8e65c9e9-347b-46b1-a1f5-774fde3b8107-multus-socket-dir-parent\") pod \"multus-8vxjj\" (UID: \"8e65c9e9-347b-46b1-a1f5-774fde3b8107\") " pod="openshift-multus/multus-8vxjj" Jul 21 14:59:29.521952 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.521896 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/327b1f5e-8e27-4572-8025-5c516e59d481-cni-binary-copy\") pod \"multus-additional-cni-plugins-8h5jp\" (UID: \"327b1f5e-8e27-4572-8025-5c516e59d481\") " pod="openshift-multus/multus-additional-cni-plugins-8h5jp" Jul 21 14:59:29.523395 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.523373 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/216f52c0-8c33-426f-b2eb-244a5787bdeb-agent-certs\") pod \"konnectivity-agent-m4pp8\" (UID: \"216f52c0-8c33-426f-b2eb-244a5787bdeb\") " pod="kube-system/konnectivity-agent-m4pp8" Jul 21 14:59:29.524407 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.524381 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a6348ec1-f9c5-487a-9cfe-abd129eadfe0-etc-tuned\") pod \"tuned-bj8nf\" (UID: \"a6348ec1-f9c5-487a-9cfe-abd129eadfe0\") " pod="openshift-cluster-node-tuning-operator/tuned-bj8nf" Jul 21 14:59:29.524861 ip-10-0-135-176 kubenswrapper[2563]: E0721 14:59:29.524845 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 21 14:59:29.524861 ip-10-0-135-176 kubenswrapper[2563]: E0721 14:59:29.524860 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 21 14:59:29.524998 ip-10-0-135-176 kubenswrapper[2563]: E0721 14:59:29.524869 2563 projected.go:196] Error preparing data for projected volume kube-api-access-dh6gh for pod openshift-network-diagnostics/network-check-target-bzwfd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 14:59:29.524998 ip-10-0-135-176 kubenswrapper[2563]: E0721 14:59:29.524926 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0683eb8b-8af7-4056-beed-3bfadd624f56-kube-api-access-dh6gh podName:0683eb8b-8af7-4056-beed-3bfadd624f56 nodeName:}" failed. No retries permitted until 2026-07-21 14:59:30.024911249 +0000 UTC m=+2.020696657 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-dh6gh" (UniqueName: "kubernetes.io/projected/0683eb8b-8af7-4056-beed-3bfadd624f56-kube-api-access-dh6gh") pod "network-check-target-bzwfd" (UID: "0683eb8b-8af7-4056-beed-3bfadd624f56") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 14:59:29.526101 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.526076 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-86c9j\" (UniqueName: \"kubernetes.io/projected/a6348ec1-f9c5-487a-9cfe-abd129eadfe0-kube-api-access-86c9j\") pod \"tuned-bj8nf\" (UID: \"a6348ec1-f9c5-487a-9cfe-abd129eadfe0\") " pod="openshift-cluster-node-tuning-operator/tuned-bj8nf" Jul 21 14:59:29.527684 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.527662 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sm77n\" (UniqueName: \"kubernetes.io/projected/02f3d0c6-4116-4134-ba51-1c02c73bef05-kube-api-access-sm77n\") pod \"ovnkube-node-kn9fn\" (UID: \"02f3d0c6-4116-4134-ba51-1c02c73bef05\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn9fn" Jul 21 14:59:29.528189 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.527918 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-86qgd\" (UniqueName: \"kubernetes.io/projected/4f638c31-bce1-4b95-97a3-410069ef52e9-kube-api-access-86qgd\") pod \"aws-ebs-csi-driver-node-vl2rt\" (UID: \"4f638c31-bce1-4b95-97a3-410069ef52e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vl2rt" Jul 21 14:59:29.528189 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.528155 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvwvn\" (UniqueName: \"kubernetes.io/projected/42b48be4-5000-431e-9d8b-b932609faca0-kube-api-access-lvwvn\") pod \"node-resolver-h2kn2\" (UID: \"42b48be4-5000-431e-9d8b-b932609faca0\") " pod="openshift-dns/node-resolver-h2kn2" Jul 21 14:59:29.528614 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.528590 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lm8nt\" (UniqueName: \"kubernetes.io/projected/26de6bb3-eb05-461a-958f-406464dea020-kube-api-access-lm8nt\") pod \"network-metrics-daemon-dh8jm\" (UID: \"26de6bb3-eb05-461a-958f-406464dea020\") " pod="openshift-multus/network-metrics-daemon-dh8jm" Jul 21 14:59:29.528721 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.528698 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8d8n\" (UniqueName: \"kubernetes.io/projected/8bf74430-9a26-4473-ace9-3d6034a94cf1-kube-api-access-v8d8n\") pod \"iptables-alerter-7q2dq\" (UID: \"8bf74430-9a26-4473-ace9-3d6034a94cf1\") " pod="openshift-network-operator/iptables-alerter-7q2dq" Jul 21 14:59:29.529630 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.529606 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4v77\" (UniqueName: \"kubernetes.io/projected/8e65c9e9-347b-46b1-a1f5-774fde3b8107-kube-api-access-d4v77\") pod \"multus-8vxjj\" (UID: \"8e65c9e9-347b-46b1-a1f5-774fde3b8107\") " pod="openshift-multus/multus-8vxjj" Jul 21 14:59:29.529756 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.529734 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkwxv\" (UniqueName: \"kubernetes.io/projected/4d94d908-6ab5-4463-a878-0b340bb80365-kube-api-access-qkwxv\") pod \"node-ca-9qzhs\" (UID: \"4d94d908-6ab5-4463-a878-0b340bb80365\") " pod="openshift-image-registry/node-ca-9qzhs" Jul 21 14:59:29.532542 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.532517 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r785r\" (UniqueName: \"kubernetes.io/projected/327b1f5e-8e27-4572-8025-5c516e59d481-kube-api-access-r785r\") pod \"multus-additional-cni-plugins-8h5jp\" (UID: \"327b1f5e-8e27-4572-8025-5c516e59d481\") " pod="openshift-multus/multus-additional-cni-plugins-8h5jp" Jul 21 14:59:29.575211 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:29.575180 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb953891c06139c6cc3d45f0903be0d66.slice/crio-4d173c84b2cde6f050a7b08cc579dd98c4f650747b94b8bc162017e87c408ce9 WatchSource:0}: Error finding container 4d173c84b2cde6f050a7b08cc579dd98c4f650747b94b8bc162017e87c408ce9: Status 404 returned error can't find the container with id 4d173c84b2cde6f050a7b08cc579dd98c4f650747b94b8bc162017e87c408ce9 Jul 21 14:59:29.578475 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.578457 2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 21 14:59:29.646881 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.646817 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-9qzhs" Jul 21 14:59:29.656795 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:29.656770 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d94d908_6ab5_4463_a878_0b340bb80365.slice/crio-7fc6d82652200c3218d69755066de8e783f503acbd96ae1992a7f9fb8eac2fae WatchSource:0}: Error finding container 7fc6d82652200c3218d69755066de8e783f503acbd96ae1992a7f9fb8eac2fae: Status 404 returned error can't find the container with id 7fc6d82652200c3218d69755066de8e783f503acbd96ae1992a7f9fb8eac2fae Jul 21 14:59:29.671266 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.671246 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vl2rt" Jul 21 14:59:29.679236 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:29.679215 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f638c31_bce1_4b95_97a3_410069ef52e9.slice/crio-92a0d782007fc3e1a1334c52bd2040f76a49c9c4411e0d619b5b352b78acf090 WatchSource:0}: Error finding container 92a0d782007fc3e1a1334c52bd2040f76a49c9c4411e0d619b5b352b78acf090: Status 404 returned error can't find the container with id 92a0d782007fc3e1a1334c52bd2040f76a49c9c4411e0d619b5b352b78acf090 Jul 21 14:59:29.689546 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.689530 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-8vxjj" Jul 21 14:59:29.697089 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:29.697069 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e65c9e9_347b_46b1_a1f5_774fde3b8107.slice/crio-9de9f8a2dcd9915e9317c08a87c40b39031c08fe34fa0d0ad9907ad75b366a75 WatchSource:0}: Error finding container 9de9f8a2dcd9915e9317c08a87c40b39031c08fe34fa0d0ad9907ad75b366a75: Status 404 returned error can't find the container with id 9de9f8a2dcd9915e9317c08a87c40b39031c08fe34fa0d0ad9907ad75b366a75 Jul 21 14:59:29.703395 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.703376 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-8h5jp" Jul 21 14:59:29.711788 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:29.711762 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod327b1f5e_8e27_4572_8025_5c516e59d481.slice/crio-0a3e0b64e0cfc85472f24646a251cae3019d34c0c5c0e97a320f07d0c1d9c950 WatchSource:0}: Error finding container 0a3e0b64e0cfc85472f24646a251cae3019d34c0c5c0e97a320f07d0c1d9c950: Status 404 returned error can't find the container with id 0a3e0b64e0cfc85472f24646a251cae3019d34c0c5c0e97a320f07d0c1d9c950 Jul 21 14:59:29.726212 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.726185 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-m4pp8" Jul 21 14:59:29.733928 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:29.733908 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod216f52c0_8c33_426f_b2eb_244a5787bdeb.slice/crio-76be6b91b038bcfe1d094612ebe1476d4ec57b490a9dfc698c8ddacb266452a9 WatchSource:0}: Error finding container 76be6b91b038bcfe1d094612ebe1476d4ec57b490a9dfc698c8ddacb266452a9: Status 404 returned error can't find the container with id 76be6b91b038bcfe1d094612ebe1476d4ec57b490a9dfc698c8ddacb266452a9 Jul 21 14:59:29.741703 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.741688 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-h2kn2" Jul 21 14:59:29.746352 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.746312 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-bj8nf" Jul 21 14:59:29.749347 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:29.749317 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42b48be4_5000_431e_9d8b_b932609faca0.slice/crio-61d78d33718c1e86581e9a3478a5600548dcefe1fbce0effdc96484621f9617c WatchSource:0}: Error finding container 61d78d33718c1e86581e9a3478a5600548dcefe1fbce0effdc96484621f9617c: Status 404 returned error can't find the container with id 61d78d33718c1e86581e9a3478a5600548dcefe1fbce0effdc96484621f9617c Jul 21 14:59:29.755779 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:29.755754 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6348ec1_f9c5_487a_9cfe_abd129eadfe0.slice/crio-0ec909e0e22b354a2d04c078ef4db7050f4ed25554d30cd281d705b484a3770c WatchSource:0}: Error finding container 0ec909e0e22b354a2d04c078ef4db7050f4ed25554d30cd281d705b484a3770c: Status 404 returned error can't find the container with id 0ec909e0e22b354a2d04c078ef4db7050f4ed25554d30cd281d705b484a3770c Jul 21 14:59:29.779895 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.779879 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-7q2dq" Jul 21 14:59:29.785619 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:29.785601 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-kn9fn" Jul 21 14:59:29.788463 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:29.788440 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bf74430_9a26_4473_ace9_3d6034a94cf1.slice/crio-3adfd0bb8d3ff16810af083da6dbf9e75c985febdd902ac2dfd41d5d2d5bd7bc WatchSource:0}: Error finding container 3adfd0bb8d3ff16810af083da6dbf9e75c985febdd902ac2dfd41d5d2d5bd7bc: Status 404 returned error can't find the container with id 3adfd0bb8d3ff16810af083da6dbf9e75c985febdd902ac2dfd41d5d2d5bd7bc Jul 21 14:59:29.793570 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:29.793549 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02f3d0c6_4116_4134_ba51_1c02c73bef05.slice/crio-6eedf6e2551d9100fa037a50eeca5ee78552fd9ccee6db6cc75430fee82850ac WatchSource:0}: Error finding container 6eedf6e2551d9100fa037a50eeca5ee78552fd9ccee6db6cc75430fee82850ac: Status 404 returned error can't find the container with id 6eedf6e2551d9100fa037a50eeca5ee78552fd9ccee6db6cc75430fee82850ac Jul 21 14:59:30.022921 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:30.022828 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26de6bb3-eb05-461a-958f-406464dea020-metrics-certs\") pod \"network-metrics-daemon-dh8jm\" (UID: \"26de6bb3-eb05-461a-958f-406464dea020\") " pod="openshift-multus/network-metrics-daemon-dh8jm" Jul 21 14:59:30.022921 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:30.022892 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e706cd7a-7458-4f77-b02d-3fd5837d735d-original-pull-secret\") pod \"global-pull-secret-syncer-jttxg\" (UID: \"e706cd7a-7458-4f77-b02d-3fd5837d735d\") " pod="kube-system/global-pull-secret-syncer-jttxg" Jul 21 14:59:30.023141 ip-10-0-135-176 kubenswrapper[2563]: E0721 14:59:30.022983 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 14:59:30.023141 ip-10-0-135-176 kubenswrapper[2563]: E0721 14:59:30.023013 2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 21 14:59:30.023141 ip-10-0-135-176 kubenswrapper[2563]: E0721 14:59:30.023054 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26de6bb3-eb05-461a-958f-406464dea020-metrics-certs podName:26de6bb3-eb05-461a-958f-406464dea020 nodeName:}" failed. No retries permitted until 2026-07-21 14:59:31.023035626 +0000 UTC m=+3.018821026 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/26de6bb3-eb05-461a-958f-406464dea020-metrics-certs") pod "network-metrics-daemon-dh8jm" (UID: "26de6bb3-eb05-461a-958f-406464dea020") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 14:59:30.023141 ip-10-0-135-176 kubenswrapper[2563]: E0721 14:59:30.023068 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e706cd7a-7458-4f77-b02d-3fd5837d735d-original-pull-secret podName:e706cd7a-7458-4f77-b02d-3fd5837d735d nodeName:}" failed. No retries permitted until 2026-07-21 14:59:31.02306115 +0000 UTC m=+3.018846545 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e706cd7a-7458-4f77-b02d-3fd5837d735d-original-pull-secret") pod "global-pull-secret-syncer-jttxg" (UID: "e706cd7a-7458-4f77-b02d-3fd5837d735d") : object "kube-system"/"original-pull-secret" not registered Jul 21 14:59:30.123655 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:30.123621 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dh6gh\" (UniqueName: \"kubernetes.io/projected/0683eb8b-8af7-4056-beed-3bfadd624f56-kube-api-access-dh6gh\") pod \"network-check-target-bzwfd\" (UID: \"0683eb8b-8af7-4056-beed-3bfadd624f56\") " pod="openshift-network-diagnostics/network-check-target-bzwfd" Jul 21 14:59:30.123823 ip-10-0-135-176 kubenswrapper[2563]: E0721 14:59:30.123785 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 21 14:59:30.123823 ip-10-0-135-176 kubenswrapper[2563]: E0721 14:59:30.123802 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 21 14:59:30.123823 ip-10-0-135-176 kubenswrapper[2563]: E0721 14:59:30.123811 2563 projected.go:196] Error preparing data for projected volume kube-api-access-dh6gh for pod openshift-network-diagnostics/network-check-target-bzwfd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 14:59:30.123990 ip-10-0-135-176 kubenswrapper[2563]: E0721 14:59:30.123866 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0683eb8b-8af7-4056-beed-3bfadd624f56-kube-api-access-dh6gh podName:0683eb8b-8af7-4056-beed-3bfadd624f56 nodeName:}" failed. No retries permitted until 2026-07-21 14:59:31.1238521 +0000 UTC m=+3.119637495 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-dh6gh" (UniqueName: "kubernetes.io/projected/0683eb8b-8af7-4056-beed-3bfadd624f56-kube-api-access-dh6gh") pod "network-check-target-bzwfd" (UID: "0683eb8b-8af7-4056-beed-3bfadd624f56") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 14:59:30.347223 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:30.347153 2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-20 14:54:29 +0000 UTC" deadline="2028-01-14 15:19:39.496084918 +0000 UTC" Jul 21 14:59:30.347223 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:30.347220 2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13008h20m9.148868424s" Jul 21 14:59:30.490303 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:30.490212 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-h2kn2" event={"ID":"42b48be4-5000-431e-9d8b-b932609faca0","Type":"ContainerStarted","Data":"61d78d33718c1e86581e9a3478a5600548dcefe1fbce0effdc96484621f9617c"} Jul 21 14:59:30.507744 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:30.507661 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8h5jp" event={"ID":"327b1f5e-8e27-4572-8025-5c516e59d481","Type":"ContainerStarted","Data":"0a3e0b64e0cfc85472f24646a251cae3019d34c0c5c0e97a320f07d0c1d9c950"} Jul 21 14:59:30.514855 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:30.514805 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vl2rt" event={"ID":"4f638c31-bce1-4b95-97a3-410069ef52e9","Type":"ContainerStarted","Data":"92a0d782007fc3e1a1334c52bd2040f76a49c9c4411e0d619b5b352b78acf090"} Jul 21 14:59:30.524171 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:30.524131 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-9qzhs" event={"ID":"4d94d908-6ab5-4463-a878-0b340bb80365","Type":"ContainerStarted","Data":"7fc6d82652200c3218d69755066de8e783f503acbd96ae1992a7f9fb8eac2fae"} Jul 21 14:59:30.554665 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:30.554617 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-176.ec2.internal" event={"ID":"b953891c06139c6cc3d45f0903be0d66","Type":"ContainerStarted","Data":"4d173c84b2cde6f050a7b08cc579dd98c4f650747b94b8bc162017e87c408ce9"} Jul 21 14:59:30.567784 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:30.567735 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kn9fn" event={"ID":"02f3d0c6-4116-4134-ba51-1c02c73bef05","Type":"ContainerStarted","Data":"6eedf6e2551d9100fa037a50eeca5ee78552fd9ccee6db6cc75430fee82850ac"} Jul 21 14:59:30.574943 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:30.574883 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-7q2dq" event={"ID":"8bf74430-9a26-4473-ace9-3d6034a94cf1","Type":"ContainerStarted","Data":"3adfd0bb8d3ff16810af083da6dbf9e75c985febdd902ac2dfd41d5d2d5bd7bc"} Jul 21 14:59:30.579999 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:30.579969 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-bj8nf" event={"ID":"a6348ec1-f9c5-487a-9cfe-abd129eadfe0","Type":"ContainerStarted","Data":"0ec909e0e22b354a2d04c078ef4db7050f4ed25554d30cd281d705b484a3770c"} Jul 21 14:59:30.593770 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:30.593735 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-m4pp8" event={"ID":"216f52c0-8c33-426f-b2eb-244a5787bdeb","Type":"ContainerStarted","Data":"76be6b91b038bcfe1d094612ebe1476d4ec57b490a9dfc698c8ddacb266452a9"} Jul 21 14:59:30.606741 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:30.606665 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-8vxjj" event={"ID":"8e65c9e9-347b-46b1-a1f5-774fde3b8107","Type":"ContainerStarted","Data":"9de9f8a2dcd9915e9317c08a87c40b39031c08fe34fa0d0ad9907ad75b366a75"} Jul 21 14:59:30.648560 ip-10-0-135-176 kubenswrapper[2563]: W0721 14:59:30.648506 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1613301c104ee5ff8466f50a74f62241.slice/crio-1849da5b1a0b0821a2f971d8c33abaa47f713a500238ab301322f736c6720b9e WatchSource:0}: Error finding container 1849da5b1a0b0821a2f971d8c33abaa47f713a500238ab301322f736c6720b9e: Status 404 returned error can't find the container with id 1849da5b1a0b0821a2f971d8c33abaa47f713a500238ab301322f736c6720b9e Jul 21 14:59:30.779288 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:30.779259 2563 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 21 14:59:31.007191 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:31.006898 2563 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 21 14:59:31.032820 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:31.032007 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e706cd7a-7458-4f77-b02d-3fd5837d735d-original-pull-secret\") pod \"global-pull-secret-syncer-jttxg\" (UID: \"e706cd7a-7458-4f77-b02d-3fd5837d735d\") " pod="kube-system/global-pull-secret-syncer-jttxg" Jul 21 14:59:31.032820 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:31.032080 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26de6bb3-eb05-461a-958f-406464dea020-metrics-certs\") pod \"network-metrics-daemon-dh8jm\" (UID: \"26de6bb3-eb05-461a-958f-406464dea020\") " pod="openshift-multus/network-metrics-daemon-dh8jm" Jul 21 14:59:31.032820 ip-10-0-135-176 kubenswrapper[2563]: E0721 14:59:31.032227 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 14:59:31.032820 ip-10-0-135-176 kubenswrapper[2563]: E0721 14:59:31.032288 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26de6bb3-eb05-461a-958f-406464dea020-metrics-certs podName:26de6bb3-eb05-461a-958f-406464dea020 nodeName:}" failed. No retries permitted until 2026-07-21 14:59:33.032270102 +0000 UTC m=+5.028055510 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/26de6bb3-eb05-461a-958f-406464dea020-metrics-certs") pod "network-metrics-daemon-dh8jm" (UID: "26de6bb3-eb05-461a-958f-406464dea020") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 14:59:31.032820 ip-10-0-135-176 kubenswrapper[2563]: E0721 14:59:31.032700 2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 21 14:59:31.032820 ip-10-0-135-176 kubenswrapper[2563]: E0721 14:59:31.032764 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e706cd7a-7458-4f77-b02d-3fd5837d735d-original-pull-secret podName:e706cd7a-7458-4f77-b02d-3fd5837d735d nodeName:}" failed. No retries permitted until 2026-07-21 14:59:33.032748098 +0000 UTC m=+5.028533496 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e706cd7a-7458-4f77-b02d-3fd5837d735d-original-pull-secret") pod "global-pull-secret-syncer-jttxg" (UID: "e706cd7a-7458-4f77-b02d-3fd5837d735d") : object "kube-system"/"original-pull-secret" not registered Jul 21 14:59:31.133437 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:31.132722 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dh6gh\" (UniqueName: \"kubernetes.io/projected/0683eb8b-8af7-4056-beed-3bfadd624f56-kube-api-access-dh6gh\") pod \"network-check-target-bzwfd\" (UID: \"0683eb8b-8af7-4056-beed-3bfadd624f56\") " pod="openshift-network-diagnostics/network-check-target-bzwfd" Jul 21 14:59:31.133437 ip-10-0-135-176 kubenswrapper[2563]: E0721 14:59:31.132945 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 21 14:59:31.133437 ip-10-0-135-176 kubenswrapper[2563]: E0721 14:59:31.132969 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 21 14:59:31.133437 ip-10-0-135-176 kubenswrapper[2563]: E0721 14:59:31.132982 2563 projected.go:196] Error preparing data for projected volume kube-api-access-dh6gh for pod openshift-network-diagnostics/network-check-target-bzwfd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 14:59:31.133437 ip-10-0-135-176 kubenswrapper[2563]: E0721 14:59:31.133041 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0683eb8b-8af7-4056-beed-3bfadd624f56-kube-api-access-dh6gh podName:0683eb8b-8af7-4056-beed-3bfadd624f56 nodeName:}" failed. No retries permitted until 2026-07-21 14:59:33.133023889 +0000 UTC m=+5.128809288 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-dh6gh" (UniqueName: "kubernetes.io/projected/0683eb8b-8af7-4056-beed-3bfadd624f56-kube-api-access-dh6gh") pod "network-check-target-bzwfd" (UID: "0683eb8b-8af7-4056-beed-3bfadd624f56") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 14:59:31.348357 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:31.348297 2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-20 14:54:29 +0000 UTC" deadline="2028-02-08 23:35:05.10256958 +0000 UTC" Jul 21 14:59:31.348357 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:31.348355 2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13616h35m33.754219034s" Jul 21 14:59:31.474733 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:31.474691 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dh8jm" Jul 21 14:59:31.474896 ip-10-0-135-176 kubenswrapper[2563]: E0721 14:59:31.474830 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dh8jm" podUID="26de6bb3-eb05-461a-958f-406464dea020" Jul 21 14:59:31.475276 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:31.475254 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bzwfd" Jul 21 14:59:31.475419 ip-10-0-135-176 kubenswrapper[2563]: E0721 14:59:31.475367 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bzwfd" podUID="0683eb8b-8af7-4056-beed-3bfadd624f56" Jul 21 14:59:31.475489 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:31.475430 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jttxg" Jul 21 14:59:31.475537 ip-10-0-135-176 kubenswrapper[2563]: E0721 14:59:31.475506 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jttxg" podUID="e706cd7a-7458-4f77-b02d-3fd5837d735d" Jul 21 14:59:31.634608 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:31.634562 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-176.ec2.internal" event={"ID":"1613301c104ee5ff8466f50a74f62241","Type":"ContainerStarted","Data":"1849da5b1a0b0821a2f971d8c33abaa47f713a500238ab301322f736c6720b9e"} Jul 21 14:59:31.970454 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:31.970183 2563 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 21 14:59:33.050169 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:33.049925 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e706cd7a-7458-4f77-b02d-3fd5837d735d-original-pull-secret\") pod \"global-pull-secret-syncer-jttxg\" (UID: \"e706cd7a-7458-4f77-b02d-3fd5837d735d\") " pod="kube-system/global-pull-secret-syncer-jttxg" Jul 21 14:59:33.050169 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:33.050008 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26de6bb3-eb05-461a-958f-406464dea020-metrics-certs\") pod \"network-metrics-daemon-dh8jm\" (UID: \"26de6bb3-eb05-461a-958f-406464dea020\") " pod="openshift-multus/network-metrics-daemon-dh8jm" Jul 21 14:59:33.050169 ip-10-0-135-176 kubenswrapper[2563]: E0721 14:59:33.050069 2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 21 14:59:33.050169 ip-10-0-135-176 kubenswrapper[2563]: E0721 14:59:33.050104 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 14:59:33.050169 ip-10-0-135-176 kubenswrapper[2563]: E0721 14:59:33.050139 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e706cd7a-7458-4f77-b02d-3fd5837d735d-original-pull-secret podName:e706cd7a-7458-4f77-b02d-3fd5837d735d nodeName:}" failed. No retries permitted until 2026-07-21 14:59:37.050119796 +0000 UTC m=+9.045905212 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e706cd7a-7458-4f77-b02d-3fd5837d735d-original-pull-secret") pod "global-pull-secret-syncer-jttxg" (UID: "e706cd7a-7458-4f77-b02d-3fd5837d735d") : object "kube-system"/"original-pull-secret" not registered Jul 21 14:59:33.050169 ip-10-0-135-176 kubenswrapper[2563]: E0721 14:59:33.050160 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26de6bb3-eb05-461a-958f-406464dea020-metrics-certs podName:26de6bb3-eb05-461a-958f-406464dea020 nodeName:}" failed. No retries permitted until 2026-07-21 14:59:37.050148815 +0000 UTC m=+9.045934210 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/26de6bb3-eb05-461a-958f-406464dea020-metrics-certs") pod "network-metrics-daemon-dh8jm" (UID: "26de6bb3-eb05-461a-958f-406464dea020") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 14:59:33.151291 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:33.151198 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dh6gh\" (UniqueName: \"kubernetes.io/projected/0683eb8b-8af7-4056-beed-3bfadd624f56-kube-api-access-dh6gh\") pod \"network-check-target-bzwfd\" (UID: \"0683eb8b-8af7-4056-beed-3bfadd624f56\") " pod="openshift-network-diagnostics/network-check-target-bzwfd" Jul 21 14:59:33.151489 ip-10-0-135-176 kubenswrapper[2563]: E0721 14:59:33.151464 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 21 14:59:33.151489 ip-10-0-135-176 kubenswrapper[2563]: E0721 14:59:33.151486 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 21 14:59:33.151592 ip-10-0-135-176 kubenswrapper[2563]: E0721 14:59:33.151498 2563 projected.go:196] Error preparing data for projected volume kube-api-access-dh6gh for pod openshift-network-diagnostics/network-check-target-bzwfd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 14:59:33.151592 ip-10-0-135-176 kubenswrapper[2563]: E0721 14:59:33.151555 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0683eb8b-8af7-4056-beed-3bfadd624f56-kube-api-access-dh6gh podName:0683eb8b-8af7-4056-beed-3bfadd624f56 nodeName:}" failed. No retries permitted until 2026-07-21 14:59:37.151536267 +0000 UTC m=+9.147321679 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-dh6gh" (UniqueName: "kubernetes.io/projected/0683eb8b-8af7-4056-beed-3bfadd624f56-kube-api-access-dh6gh") pod "network-check-target-bzwfd" (UID: "0683eb8b-8af7-4056-beed-3bfadd624f56") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 14:59:33.474282 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:33.474248 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dh8jm" Jul 21 14:59:33.474480 ip-10-0-135-176 kubenswrapper[2563]: E0721 14:59:33.474407 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dh8jm" podUID="26de6bb3-eb05-461a-958f-406464dea020" Jul 21 14:59:33.474819 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:33.474799 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bzwfd" Jul 21 14:59:33.474932 ip-10-0-135-176 kubenswrapper[2563]: E0721 14:59:33.474893 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bzwfd" podUID="0683eb8b-8af7-4056-beed-3bfadd624f56" Jul 21 14:59:33.474985 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:33.474956 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jttxg" Jul 21 14:59:33.475038 ip-10-0-135-176 kubenswrapper[2563]: E0721 14:59:33.475025 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jttxg" podUID="e706cd7a-7458-4f77-b02d-3fd5837d735d" Jul 21 14:59:35.474379 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:35.474343 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dh8jm" Jul 21 14:59:35.475242 ip-10-0-135-176 kubenswrapper[2563]: E0721 14:59:35.474504 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dh8jm" podUID="26de6bb3-eb05-461a-958f-406464dea020" Jul 21 14:59:35.475242 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:35.474904 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bzwfd" Jul 21 14:59:35.475242 ip-10-0-135-176 kubenswrapper[2563]: E0721 14:59:35.474990 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bzwfd" podUID="0683eb8b-8af7-4056-beed-3bfadd624f56" Jul 21 14:59:35.475242 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:35.475033 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jttxg" Jul 21 14:59:35.475242 ip-10-0-135-176 kubenswrapper[2563]: E0721 14:59:35.475083 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jttxg" podUID="e706cd7a-7458-4f77-b02d-3fd5837d735d" Jul 21 14:59:37.086898 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:37.086061 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e706cd7a-7458-4f77-b02d-3fd5837d735d-original-pull-secret\") pod \"global-pull-secret-syncer-jttxg\" (UID: \"e706cd7a-7458-4f77-b02d-3fd5837d735d\") " pod="kube-system/global-pull-secret-syncer-jttxg" Jul 21 14:59:37.086898 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:37.086154 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26de6bb3-eb05-461a-958f-406464dea020-metrics-certs\") pod \"network-metrics-daemon-dh8jm\" (UID: \"26de6bb3-eb05-461a-958f-406464dea020\") " pod="openshift-multus/network-metrics-daemon-dh8jm" Jul 21 14:59:37.086898 ip-10-0-135-176 kubenswrapper[2563]: E0721 14:59:37.086292 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 14:59:37.086898 ip-10-0-135-176 kubenswrapper[2563]: E0721 14:59:37.086379 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26de6bb3-eb05-461a-958f-406464dea020-metrics-certs podName:26de6bb3-eb05-461a-958f-406464dea020 nodeName:}" failed. No retries permitted until 2026-07-21 14:59:45.086357808 +0000 UTC m=+17.082143206 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/26de6bb3-eb05-461a-958f-406464dea020-metrics-certs") pod "network-metrics-daemon-dh8jm" (UID: "26de6bb3-eb05-461a-958f-406464dea020") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 14:59:37.086898 ip-10-0-135-176 kubenswrapper[2563]: E0721 14:59:37.086809 2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 21 14:59:37.086898 ip-10-0-135-176 kubenswrapper[2563]: E0721 14:59:37.086860 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e706cd7a-7458-4f77-b02d-3fd5837d735d-original-pull-secret podName:e706cd7a-7458-4f77-b02d-3fd5837d735d nodeName:}" failed. No retries permitted until 2026-07-21 14:59:45.086844792 +0000 UTC m=+17.082630191 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e706cd7a-7458-4f77-b02d-3fd5837d735d-original-pull-secret") pod "global-pull-secret-syncer-jttxg" (UID: "e706cd7a-7458-4f77-b02d-3fd5837d735d") : object "kube-system"/"original-pull-secret" not registered Jul 21 14:59:37.186812 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:37.186771 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dh6gh\" (UniqueName: \"kubernetes.io/projected/0683eb8b-8af7-4056-beed-3bfadd624f56-kube-api-access-dh6gh\") pod \"network-check-target-bzwfd\" (UID: \"0683eb8b-8af7-4056-beed-3bfadd624f56\") " pod="openshift-network-diagnostics/network-check-target-bzwfd" Jul 21 14:59:37.186975 ip-10-0-135-176 kubenswrapper[2563]: E0721 14:59:37.186935 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 21 14:59:37.186975 ip-10-0-135-176 kubenswrapper[2563]: E0721 14:59:37.186952 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 21 14:59:37.186975 ip-10-0-135-176 kubenswrapper[2563]: E0721 14:59:37.186961 2563 projected.go:196] Error preparing data for projected volume kube-api-access-dh6gh for pod openshift-network-diagnostics/network-check-target-bzwfd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 14:59:37.187147 ip-10-0-135-176 kubenswrapper[2563]: E0721 14:59:37.187026 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0683eb8b-8af7-4056-beed-3bfadd624f56-kube-api-access-dh6gh podName:0683eb8b-8af7-4056-beed-3bfadd624f56 nodeName:}" failed. No retries permitted until 2026-07-21 14:59:45.18700562 +0000 UTC m=+17.182791040 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-dh6gh" (UniqueName: "kubernetes.io/projected/0683eb8b-8af7-4056-beed-3bfadd624f56-kube-api-access-dh6gh") pod "network-check-target-bzwfd" (UID: "0683eb8b-8af7-4056-beed-3bfadd624f56") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 14:59:37.474259 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:37.474169 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bzwfd" Jul 21 14:59:37.474446 ip-10-0-135-176 kubenswrapper[2563]: E0721 14:59:37.474295 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bzwfd" podUID="0683eb8b-8af7-4056-beed-3bfadd624f56" Jul 21 14:59:37.474513 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:37.474466 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dh8jm" Jul 21 14:59:37.474729 ip-10-0-135-176 kubenswrapper[2563]: E0721 14:59:37.474579 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dh8jm" podUID="26de6bb3-eb05-461a-958f-406464dea020" Jul 21 14:59:37.474729 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:37.474647 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jttxg" Jul 21 14:59:37.474729 ip-10-0-135-176 kubenswrapper[2563]: E0721 14:59:37.474701 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jttxg" podUID="e706cd7a-7458-4f77-b02d-3fd5837d735d" Jul 21 14:59:39.474669 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:39.474592 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dh8jm" Jul 21 14:59:39.474669 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:39.474631 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jttxg" Jul 21 14:59:39.475170 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:39.474596 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bzwfd" Jul 21 14:59:39.475170 ip-10-0-135-176 kubenswrapper[2563]: E0721 14:59:39.474742 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dh8jm" podUID="26de6bb3-eb05-461a-958f-406464dea020" Jul 21 14:59:39.475272 ip-10-0-135-176 kubenswrapper[2563]: E0721 14:59:39.475171 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jttxg" podUID="e706cd7a-7458-4f77-b02d-3fd5837d735d" Jul 21 14:59:39.475272 ip-10-0-135-176 kubenswrapper[2563]: E0721 14:59:39.475259 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bzwfd" podUID="0683eb8b-8af7-4056-beed-3bfadd624f56" Jul 21 14:59:41.474275 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:41.474086 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bzwfd" Jul 21 14:59:41.474664 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:41.474089 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jttxg" Jul 21 14:59:41.474664 ip-10-0-135-176 kubenswrapper[2563]: E0721 14:59:41.474360 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bzwfd" podUID="0683eb8b-8af7-4056-beed-3bfadd624f56" Jul 21 14:59:41.474664 ip-10-0-135-176 kubenswrapper[2563]: E0721 14:59:41.474433 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jttxg" podUID="e706cd7a-7458-4f77-b02d-3fd5837d735d" Jul 21 14:59:41.474664 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:41.474123 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dh8jm" Jul 21 14:59:41.474664 ip-10-0-135-176 kubenswrapper[2563]: E0721 14:59:41.474526 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dh8jm" podUID="26de6bb3-eb05-461a-958f-406464dea020" Jul 21 14:59:43.474268 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:43.474232 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dh8jm" Jul 21 14:59:43.474268 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:43.474261 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jttxg" Jul 21 14:59:43.474740 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:43.474248 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bzwfd" Jul 21 14:59:43.474740 ip-10-0-135-176 kubenswrapper[2563]: E0721 14:59:43.474405 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dh8jm" podUID="26de6bb3-eb05-461a-958f-406464dea020" Jul 21 14:59:43.474740 ip-10-0-135-176 kubenswrapper[2563]: E0721 14:59:43.474487 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jttxg" podUID="e706cd7a-7458-4f77-b02d-3fd5837d735d" Jul 21 14:59:43.474740 ip-10-0-135-176 kubenswrapper[2563]: E0721 14:59:43.474575 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bzwfd" podUID="0683eb8b-8af7-4056-beed-3bfadd624f56" Jul 21 14:59:45.149876 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:45.149840 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26de6bb3-eb05-461a-958f-406464dea020-metrics-certs\") pod \"network-metrics-daemon-dh8jm\" (UID: \"26de6bb3-eb05-461a-958f-406464dea020\") " pod="openshift-multus/network-metrics-daemon-dh8jm" Jul 21 14:59:45.150382 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:45.149915 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e706cd7a-7458-4f77-b02d-3fd5837d735d-original-pull-secret\") pod \"global-pull-secret-syncer-jttxg\" (UID: \"e706cd7a-7458-4f77-b02d-3fd5837d735d\") " pod="kube-system/global-pull-secret-syncer-jttxg" Jul 21 14:59:45.150382 ip-10-0-135-176 kubenswrapper[2563]: E0721 14:59:45.150006 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 14:59:45.150382 ip-10-0-135-176 kubenswrapper[2563]: E0721 14:59:45.150051 2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 21 14:59:45.150382 ip-10-0-135-176 kubenswrapper[2563]: E0721 14:59:45.150081 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26de6bb3-eb05-461a-958f-406464dea020-metrics-certs podName:26de6bb3-eb05-461a-958f-406464dea020 nodeName:}" failed. No retries permitted until 2026-07-21 15:00:01.150058958 +0000 UTC m=+33.145844353 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/26de6bb3-eb05-461a-958f-406464dea020-metrics-certs") pod "network-metrics-daemon-dh8jm" (UID: "26de6bb3-eb05-461a-958f-406464dea020") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 14:59:45.150382 ip-10-0-135-176 kubenswrapper[2563]: E0721 14:59:45.150120 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e706cd7a-7458-4f77-b02d-3fd5837d735d-original-pull-secret podName:e706cd7a-7458-4f77-b02d-3fd5837d735d nodeName:}" failed. No retries permitted until 2026-07-21 15:00:01.150100752 +0000 UTC m=+33.145886168 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e706cd7a-7458-4f77-b02d-3fd5837d735d-original-pull-secret") pod "global-pull-secret-syncer-jttxg" (UID: "e706cd7a-7458-4f77-b02d-3fd5837d735d") : object "kube-system"/"original-pull-secret" not registered Jul 21 14:59:45.250406 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:45.250368 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dh6gh\" (UniqueName: \"kubernetes.io/projected/0683eb8b-8af7-4056-beed-3bfadd624f56-kube-api-access-dh6gh\") pod \"network-check-target-bzwfd\" (UID: \"0683eb8b-8af7-4056-beed-3bfadd624f56\") " pod="openshift-network-diagnostics/network-check-target-bzwfd" Jul 21 14:59:45.250623 ip-10-0-135-176 kubenswrapper[2563]: E0721 14:59:45.250594 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 21 14:59:45.250623 ip-10-0-135-176 kubenswrapper[2563]: E0721 14:59:45.250625 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 21 14:59:45.250817 ip-10-0-135-176 kubenswrapper[2563]: E0721 14:59:45.250640 2563 projected.go:196] Error preparing data for projected volume kube-api-access-dh6gh for pod openshift-network-diagnostics/network-check-target-bzwfd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 14:59:45.250817 ip-10-0-135-176 kubenswrapper[2563]: E0721 14:59:45.250714 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0683eb8b-8af7-4056-beed-3bfadd624f56-kube-api-access-dh6gh podName:0683eb8b-8af7-4056-beed-3bfadd624f56 nodeName:}" failed. No retries permitted until 2026-07-21 15:00:01.250696176 +0000 UTC m=+33.246481574 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-dh6gh" (UniqueName: "kubernetes.io/projected/0683eb8b-8af7-4056-beed-3bfadd624f56-kube-api-access-dh6gh") pod "network-check-target-bzwfd" (UID: "0683eb8b-8af7-4056-beed-3bfadd624f56") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 14:59:45.474305 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:45.474219 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bzwfd" Jul 21 14:59:45.474499 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:45.474350 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jttxg" Jul 21 14:59:45.474499 ip-10-0-135-176 kubenswrapper[2563]: E0721 14:59:45.474360 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bzwfd" podUID="0683eb8b-8af7-4056-beed-3bfadd624f56" Jul 21 14:59:45.474499 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:45.474385 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dh8jm" Jul 21 14:59:45.474499 ip-10-0-135-176 kubenswrapper[2563]: E0721 14:59:45.474474 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jttxg" podUID="e706cd7a-7458-4f77-b02d-3fd5837d735d" Jul 21 14:59:45.474688 ip-10-0-135-176 kubenswrapper[2563]: E0721 14:59:45.474572 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dh8jm" podUID="26de6bb3-eb05-461a-958f-406464dea020" Jul 21 14:59:47.474312 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:47.474271 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bzwfd" Jul 21 14:59:47.474791 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:47.474274 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jttxg" Jul 21 14:59:47.474791 ip-10-0-135-176 kubenswrapper[2563]: E0721 14:59:47.474407 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bzwfd" podUID="0683eb8b-8af7-4056-beed-3bfadd624f56" Jul 21 14:59:47.474791 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:47.474277 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dh8jm" Jul 21 14:59:47.474791 ip-10-0-135-176 kubenswrapper[2563]: E0721 14:59:47.474516 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jttxg" podUID="e706cd7a-7458-4f77-b02d-3fd5837d735d" Jul 21 14:59:47.474791 ip-10-0-135-176 kubenswrapper[2563]: E0721 14:59:47.474590 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dh8jm" podUID="26de6bb3-eb05-461a-958f-406464dea020" Jul 21 14:59:49.474952 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:49.474908 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bzwfd" Jul 21 14:59:49.474952 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:49.474944 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dh8jm" Jul 21 14:59:49.474952 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:49.474955 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jttxg" Jul 21 14:59:49.475462 ip-10-0-135-176 kubenswrapper[2563]: E0721 14:59:49.475039 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bzwfd" podUID="0683eb8b-8af7-4056-beed-3bfadd624f56" Jul 21 14:59:49.475462 ip-10-0-135-176 kubenswrapper[2563]: E0721 14:59:49.475128 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dh8jm" podUID="26de6bb3-eb05-461a-958f-406464dea020" Jul 21 14:59:49.475462 ip-10-0-135-176 kubenswrapper[2563]: E0721 14:59:49.475213 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jttxg" podUID="e706cd7a-7458-4f77-b02d-3fd5837d735d" Jul 21 14:59:50.668223 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:50.668184 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-bj8nf" event={"ID":"a6348ec1-f9c5-487a-9cfe-abd129eadfe0","Type":"ContainerStarted","Data":"459b9022a7b45ad23d6a77d987fdbd5bf24a9fd57bbda461ec36135fb980c460"} Jul 21 14:59:50.669686 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:50.669662 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-8vxjj" event={"ID":"8e65c9e9-347b-46b1-a1f5-774fde3b8107","Type":"ContainerStarted","Data":"405c662b01aa6ef37f97c78c9f7c647266c6e2616f59213915ec39e7d68b375c"} Jul 21 14:59:50.671984 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:50.671960 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-176.ec2.internal" event={"ID":"b953891c06139c6cc3d45f0903be0d66","Type":"ContainerStarted","Data":"a0cf83479cf9575c9cab84666490a87fcd2e8d7088c6064fd8ee1ab0d62cb6d9"} Jul 21 14:59:50.690444 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:50.690353 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-bj8nf" podStartSLOduration=2.103552995 podStartE2EDuration="22.690318634s" totalImagesPullingTime="20.586765639s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 14:59:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 14:59:50.689979194 +0000 UTC m=+22.685764610" watchObservedRunningTime="2026-07-21 14:59:50.690318634 +0000 UTC m=+22.686104081" Jul 21 14:59:50.736628 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:50.736586 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-176.ec2.internal" podStartSLOduration=21.736565928 podStartE2EDuration="21.736565928s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 14:59:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 14:59:50.707942293 +0000 UTC m=+22.703727711" watchObservedRunningTime="2026-07-21 14:59:50.736565928 +0000 UTC m=+22.732351343" Jul 21 14:59:50.737155 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:50.737122 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-8vxjj" podStartSLOduration=2.082297685 podStartE2EDuration="22.737113058s" totalImagesPullingTime="20.654815373s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 14:59:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 14:59:50.736919746 +0000 UTC m=+22.732705165" watchObservedRunningTime="2026-07-21 14:59:50.737113058 +0000 UTC m=+22.732898475" Jul 21 14:59:51.475021 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:51.474853 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jttxg" Jul 21 14:59:51.475201 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:51.474911 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bzwfd" Jul 21 14:59:51.475201 ip-10-0-135-176 kubenswrapper[2563]: E0721 14:59:51.475141 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jttxg" podUID="e706cd7a-7458-4f77-b02d-3fd5837d735d" Jul 21 14:59:51.475201 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:51.474911 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dh8jm" Jul 21 14:59:51.475385 ip-10-0-135-176 kubenswrapper[2563]: E0721 14:59:51.475198 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bzwfd" podUID="0683eb8b-8af7-4056-beed-3bfadd624f56" Jul 21 14:59:51.475385 ip-10-0-135-176 kubenswrapper[2563]: E0721 14:59:51.475258 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dh8jm" podUID="26de6bb3-eb05-461a-958f-406464dea020" Jul 21 14:59:51.675144 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:51.675106 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-h2kn2" event={"ID":"42b48be4-5000-431e-9d8b-b932609faca0","Type":"ContainerStarted","Data":"8373c6f038b6da62de54e1fd3219281e2054ce6eef1c8bbfa953afd58f6c7909"} Jul 21 14:59:51.676675 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:51.676648 2563 generic.go:356] "Generic (PLEG): container finished" podID="327b1f5e-8e27-4572-8025-5c516e59d481" containerID="03eba2e7f3345836c3ce453d36371685aaff3a99e91b11277c6db7323053db25" exitCode=0 Jul 21 14:59:51.676784 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:51.676727 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8h5jp" event={"ID":"327b1f5e-8e27-4572-8025-5c516e59d481","Type":"ContainerDied","Data":"03eba2e7f3345836c3ce453d36371685aaff3a99e91b11277c6db7323053db25"} Jul 21 14:59:51.678103 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:51.678070 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vl2rt" event={"ID":"4f638c31-bce1-4b95-97a3-410069ef52e9","Type":"ContainerStarted","Data":"21f22bca9b6c992157d9d84654f9b75e703637148223e98e8e87701d9b5dbabe"} Jul 21 14:59:51.679685 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:51.679659 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-9qzhs" event={"ID":"4d94d908-6ab5-4463-a878-0b340bb80365","Type":"ContainerStarted","Data":"52475629c8212dfc1b75779f9e3e029f80c5ed54480e30b93bb9fa6bdd6d1dcc"} Jul 21 14:59:51.681214 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:51.681192 2563 generic.go:356] "Generic (PLEG): container finished" podID="1613301c104ee5ff8466f50a74f62241" containerID="ba3c370b1d989e13f2811ba5e35a83a4e3ca00d4259f0ddad7b6bec92dce050c" exitCode=0 Jul 21 14:59:51.681310 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:51.681263 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-176.ec2.internal" event={"ID":"1613301c104ee5ff8466f50a74f62241","Type":"ContainerDied","Data":"ba3c370b1d989e13f2811ba5e35a83a4e3ca00d4259f0ddad7b6bec92dce050c"} Jul 21 14:59:51.684192 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:51.684170 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kn9fn_02f3d0c6-4116-4134-ba51-1c02c73bef05/ovn-acl-logging/0.log" Jul 21 14:59:51.684567 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:51.684546 2563 generic.go:356] "Generic (PLEG): container finished" podID="02f3d0c6-4116-4134-ba51-1c02c73bef05" containerID="94f408538005b1934f7e2d623c812015bfbe3db8f2a8769f3ff4fb8134a812f9" exitCode=1 Jul 21 14:59:51.684665 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:51.684612 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kn9fn" event={"ID":"02f3d0c6-4116-4134-ba51-1c02c73bef05","Type":"ContainerStarted","Data":"2cac59dcd3dab84fac7cc11c690764a0c3e971f063d9dee6fb0cb25fd2827d0d"} Jul 21 14:59:51.684715 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:51.684678 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kn9fn" event={"ID":"02f3d0c6-4116-4134-ba51-1c02c73bef05","Type":"ContainerStarted","Data":"d2d536e906f47185713d8cae3522944c330eebb94f511405ad6de678c4568857"} Jul 21 14:59:51.684763 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:51.684722 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kn9fn" event={"ID":"02f3d0c6-4116-4134-ba51-1c02c73bef05","Type":"ContainerStarted","Data":"d77311f0ea0e78f1dd077b23b38e78338e7b2e2164336dba21fafe9ef5a92958"} Jul 21 14:59:51.684801 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:51.684767 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kn9fn" event={"ID":"02f3d0c6-4116-4134-ba51-1c02c73bef05","Type":"ContainerStarted","Data":"82d41ee1a89aa091237c61a9ed08b3f2b575763d95ff1942438bfea35cf78c1d"} Jul 21 14:59:51.684801 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:51.684797 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kn9fn" event={"ID":"02f3d0c6-4116-4134-ba51-1c02c73bef05","Type":"ContainerDied","Data":"94f408538005b1934f7e2d623c812015bfbe3db8f2a8769f3ff4fb8134a812f9"} Jul 21 14:59:51.684877 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:51.684825 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kn9fn" event={"ID":"02f3d0c6-4116-4134-ba51-1c02c73bef05","Type":"ContainerStarted","Data":"10dca9263a4641f34d5c07d8a850531283e42b1314d54b91238dca79c49abe95"} Jul 21 14:59:51.686066 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:51.686043 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-m4pp8" event={"ID":"216f52c0-8c33-426f-b2eb-244a5787bdeb","Type":"ContainerStarted","Data":"c955d132f6331e51df587690b90102e87ef3e037644551c2df74a3ff16c4644e"} Jul 21 14:59:51.691472 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:51.691434 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-h2kn2" podStartSLOduration=3.121257007 podStartE2EDuration="23.691422616s" totalImagesPullingTime="20.570165609s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 14:59:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 14:59:51.690819387 +0000 UTC m=+23.686604804" watchObservedRunningTime="2026-07-21 14:59:51.691422616 +0000 UTC m=+23.687208033" Jul 21 14:59:51.705128 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:51.705088 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-m4pp8" podStartSLOduration=3.133627875 podStartE2EDuration="23.705073573s" totalImagesPullingTime="20.571445698s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 14:59:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 14:59:51.704694517 +0000 UTC m=+23.700479934" watchObservedRunningTime="2026-07-21 14:59:51.705073573 +0000 UTC m=+23.700858989" Jul 21 14:59:51.718854 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:51.718811 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-9qzhs" podStartSLOduration=3.069323045 podStartE2EDuration="23.718795352s" totalImagesPullingTime="20.649472307s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 14:59:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 14:59:51.718639703 +0000 UTC m=+23.714425121" watchObservedRunningTime="2026-07-21 14:59:51.718795352 +0000 UTC m=+23.714580769" Jul 21 14:59:52.689345 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:52.689097 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-7q2dq" event={"ID":"8bf74430-9a26-4473-ace9-3d6034a94cf1","Type":"ContainerStarted","Data":"d8b98a38c61802cbefb4c881fa3d8401f41b2f5ff1e0c66c4d5e83aea7ccaed3"} Jul 21 14:59:52.705342 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:52.705285 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-7q2dq" podStartSLOduration=4.17262483 podStartE2EDuration="24.705271335s" totalImagesPullingTime="20.532646505s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 14:59:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 14:59:52.705154781 +0000 UTC m=+24.700940197" watchObservedRunningTime="2026-07-21 14:59:52.705271335 +0000 UTC m=+24.701056752" Jul 21 14:59:53.474781 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:53.474741 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dh8jm" Jul 21 14:59:53.474781 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:53.474783 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bzwfd" Jul 21 14:59:53.475088 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:53.474754 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jttxg" Jul 21 14:59:53.475088 ip-10-0-135-176 kubenswrapper[2563]: E0721 14:59:53.474859 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dh8jm" podUID="26de6bb3-eb05-461a-958f-406464dea020" Jul 21 14:59:53.475088 ip-10-0-135-176 kubenswrapper[2563]: E0721 14:59:53.474926 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jttxg" podUID="e706cd7a-7458-4f77-b02d-3fd5837d735d" Jul 21 14:59:53.475088 ip-10-0-135-176 kubenswrapper[2563]: E0721 14:59:53.474995 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bzwfd" podUID="0683eb8b-8af7-4056-beed-3bfadd624f56" Jul 21 14:59:53.693562 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:53.693531 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-176.ec2.internal" event={"ID":"1613301c104ee5ff8466f50a74f62241","Type":"ContainerStarted","Data":"6e94da161334a159db1440a77449230ce93d5299201b188121c53d012b158311"} Jul 21 14:59:53.696987 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:53.696967 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kn9fn_02f3d0c6-4116-4134-ba51-1c02c73bef05/ovn-acl-logging/0.log" Jul 21 14:59:53.709980 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:53.709856 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-176.ec2.internal" podStartSLOduration=24.709841568 podStartE2EDuration="24.709841568s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 14:59:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 14:59:53.709708481 +0000 UTC m=+25.705493899" watchObservedRunningTime="2026-07-21 14:59:53.709841568 +0000 UTC m=+25.705626984" Jul 21 14:59:54.648129 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:54.648099 2563 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 21 14:59:54.702952 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:54.702930 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kn9fn_02f3d0c6-4116-4134-ba51-1c02c73bef05/ovn-acl-logging/0.log" Jul 21 14:59:54.703393 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:54.703363 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kn9fn" event={"ID":"02f3d0c6-4116-4134-ba51-1c02c73bef05","Type":"ContainerStarted","Data":"287928bffbb31ae0130d804bff5ffb5034bae8f6ab0def9caffb10c4f961822d"} Jul 21 14:59:54.704851 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:54.704827 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vl2rt" event={"ID":"4f638c31-bce1-4b95-97a3-410069ef52e9","Type":"ContainerStarted","Data":"2d821131edd9c920bcb217439c50470e091c7b1141eca10594e105e5381a6cd7"} Jul 21 14:59:55.387344 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:55.387214 2563 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-21T14:59:54.648123765Z","UUID":"b4be91d1-5452-401e-9e8b-d8981d397795","Handler":null,"Name":"","Endpoint":""} Jul 21 14:59:55.388971 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:55.388952 2563 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 21 14:59:55.389105 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:55.389000 2563 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 21 14:59:55.474199 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:55.474154 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jttxg" Jul 21 14:59:55.474382 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:55.474154 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bzwfd" Jul 21 14:59:55.474382 ip-10-0-135-176 kubenswrapper[2563]: E0721 14:59:55.474289 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jttxg" podUID="e706cd7a-7458-4f77-b02d-3fd5837d735d" Jul 21 14:59:55.474382 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:55.474159 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dh8jm" Jul 21 14:59:55.474382 ip-10-0-135-176 kubenswrapper[2563]: E0721 14:59:55.474363 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bzwfd" podUID="0683eb8b-8af7-4056-beed-3bfadd624f56" Jul 21 14:59:55.474558 ip-10-0-135-176 kubenswrapper[2563]: E0721 14:59:55.474464 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dh8jm" podUID="26de6bb3-eb05-461a-958f-406464dea020" Jul 21 14:59:55.886698 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:55.886665 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-m4pp8" Jul 21 14:59:55.887201 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:55.887162 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-m4pp8" Jul 21 14:59:56.712141 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:56.711910 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8h5jp" event={"ID":"327b1f5e-8e27-4572-8025-5c516e59d481","Type":"ContainerStarted","Data":"eda70d56cc6591b0919a2c853ac52474ee69193758f4be7015d36a8d43502615"} Jul 21 14:59:56.715783 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:56.715761 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kn9fn_02f3d0c6-4116-4134-ba51-1c02c73bef05/ovn-acl-logging/0.log" Jul 21 14:59:56.716356 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:56.716302 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kn9fn" event={"ID":"02f3d0c6-4116-4134-ba51-1c02c73bef05","Type":"ContainerStarted","Data":"4837a1b5d904afb3028e5392fcec088d4d116bbf4334187a82e6dd2b057070f9"} Jul 21 14:59:56.716811 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:56.716791 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-m4pp8" Jul 21 14:59:56.716947 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:56.716931 2563 scope.go:117] "RemoveContainer" containerID="94f408538005b1934f7e2d623c812015bfbe3db8f2a8769f3ff4fb8134a812f9" Jul 21 14:59:56.717084 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:56.717066 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-m4pp8" Jul 21 14:59:57.474925 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:57.474900 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dh8jm" Jul 21 14:59:57.475718 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:57.474900 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jttxg" Jul 21 14:59:57.475718 ip-10-0-135-176 kubenswrapper[2563]: E0721 14:59:57.475017 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dh8jm" podUID="26de6bb3-eb05-461a-958f-406464dea020" Jul 21 14:59:57.475718 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:57.474900 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bzwfd" Jul 21 14:59:57.475718 ip-10-0-135-176 kubenswrapper[2563]: E0721 14:59:57.475089 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jttxg" podUID="e706cd7a-7458-4f77-b02d-3fd5837d735d" Jul 21 14:59:57.475718 ip-10-0-135-176 kubenswrapper[2563]: E0721 14:59:57.475150 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bzwfd" podUID="0683eb8b-8af7-4056-beed-3bfadd624f56" Jul 21 14:59:57.721778 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:57.721699 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kn9fn_02f3d0c6-4116-4134-ba51-1c02c73bef05/ovn-acl-logging/0.log" Jul 21 14:59:57.722136 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:57.722110 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kn9fn" event={"ID":"02f3d0c6-4116-4134-ba51-1c02c73bef05","Type":"ContainerStarted","Data":"7a3970bfb87b16bd92b54feec7eda074edaded21f630f823171972310b73bfc0"} Jul 21 14:59:57.722403 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:57.722383 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kn9fn" Jul 21 14:59:57.722537 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:57.722433 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kn9fn" Jul 21 14:59:57.722537 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:57.722514 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kn9fn" Jul 21 14:59:57.724129 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:57.724103 2563 generic.go:356] "Generic (PLEG): container finished" podID="327b1f5e-8e27-4572-8025-5c516e59d481" containerID="eda70d56cc6591b0919a2c853ac52474ee69193758f4be7015d36a8d43502615" exitCode=0 Jul 21 14:59:57.724231 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:57.724192 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8h5jp" event={"ID":"327b1f5e-8e27-4572-8025-5c516e59d481","Type":"ContainerDied","Data":"eda70d56cc6591b0919a2c853ac52474ee69193758f4be7015d36a8d43502615"} Jul 21 14:59:57.726447 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:57.726422 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vl2rt" event={"ID":"4f638c31-bce1-4b95-97a3-410069ef52e9","Type":"ContainerStarted","Data":"20bc3494f6d98b2007ecc976fbce8f5b3492aa11defb824687012a407851c7a7"} Jul 21 14:59:57.739805 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:57.739781 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kn9fn" Jul 21 14:59:57.740369 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:57.740347 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kn9fn" Jul 21 14:59:57.757032 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:57.756985 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-kn9fn" podStartSLOduration=8.818573311 podStartE2EDuration="29.756971475s" totalImagesPullingTime="20.938398164s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 14:59:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 14:59:57.756449493 +0000 UTC m=+29.752234909" watchObservedRunningTime="2026-07-21 14:59:57.756971475 +0000 UTC m=+29.752756892" Jul 21 14:59:57.830828 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:57.830775 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vl2rt" podStartSLOduration=3.411664429 podStartE2EDuration="29.830760622s" totalImagesPullingTime="26.419096193s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 14:59:28 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 14:59:57.830238432 +0000 UTC m=+29.826023851" watchObservedRunningTime="2026-07-21 14:59:57.830760622 +0000 UTC m=+29.826546039" Jul 21 14:59:58.434302 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:58.434060 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-jttxg"] Jul 21 14:59:58.434565 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:58.434437 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jttxg" Jul 21 14:59:58.434645 ip-10-0-135-176 kubenswrapper[2563]: E0721 14:59:58.434563 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jttxg" podUID="e706cd7a-7458-4f77-b02d-3fd5837d735d" Jul 21 14:59:58.438868 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:58.438844 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-bzwfd"] Jul 21 14:59:58.439000 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:58.438963 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bzwfd" Jul 21 14:59:58.439069 ip-10-0-135-176 kubenswrapper[2563]: E0721 14:59:58.439049 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bzwfd" podUID="0683eb8b-8af7-4056-beed-3bfadd624f56" Jul 21 14:59:58.440549 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:58.440529 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-dh8jm"] Jul 21 14:59:58.440690 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:58.440621 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dh8jm" Jul 21 14:59:58.440755 ip-10-0-135-176 kubenswrapper[2563]: E0721 14:59:58.440718 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dh8jm" podUID="26de6bb3-eb05-461a-958f-406464dea020" Jul 21 14:59:59.474458 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:59.474426 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bzwfd" Jul 21 14:59:59.474886 ip-10-0-135-176 kubenswrapper[2563]: E0721 14:59:59.474530 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bzwfd" podUID="0683eb8b-8af7-4056-beed-3bfadd624f56" Jul 21 14:59:59.733198 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:59.733104 2563 generic.go:356] "Generic (PLEG): container finished" podID="327b1f5e-8e27-4572-8025-5c516e59d481" containerID="fb027fdfa77cf82cf3e23b92e8faecce6064c8a3008d4092815e7baa21b2135e" exitCode=0 Jul 21 14:59:59.733384 ip-10-0-135-176 kubenswrapper[2563]: I0721 14:59:59.733190 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8h5jp" event={"ID":"327b1f5e-8e27-4572-8025-5c516e59d481","Type":"ContainerDied","Data":"fb027fdfa77cf82cf3e23b92e8faecce6064c8a3008d4092815e7baa21b2135e"} Jul 21 15:00:00.474285 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:00:00.474247 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jttxg" Jul 21 15:00:00.474285 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:00:00.474277 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dh8jm" Jul 21 15:00:00.474522 ip-10-0-135-176 kubenswrapper[2563]: E0721 15:00:00.474401 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jttxg" podUID="e706cd7a-7458-4f77-b02d-3fd5837d735d" Jul 21 15:00:00.474876 ip-10-0-135-176 kubenswrapper[2563]: E0721 15:00:00.474531 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dh8jm" podUID="26de6bb3-eb05-461a-958f-406464dea020" Jul 21 15:00:01.165656 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:00:01.165625 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26de6bb3-eb05-461a-958f-406464dea020-metrics-certs\") pod \"network-metrics-daemon-dh8jm\" (UID: \"26de6bb3-eb05-461a-958f-406464dea020\") " pod="openshift-multus/network-metrics-daemon-dh8jm" Jul 21 15:00:01.165823 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:00:01.165675 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e706cd7a-7458-4f77-b02d-3fd5837d735d-original-pull-secret\") pod \"global-pull-secret-syncer-jttxg\" (UID: \"e706cd7a-7458-4f77-b02d-3fd5837d735d\") " pod="kube-system/global-pull-secret-syncer-jttxg" Jul 21 15:00:01.165823 ip-10-0-135-176 kubenswrapper[2563]: E0721 15:00:01.165766 2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 21 15:00:01.165823 ip-10-0-135-176 kubenswrapper[2563]: E0721 15:00:01.165769 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 15:00:01.165823 ip-10-0-135-176 kubenswrapper[2563]: E0721 15:00:01.165817 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e706cd7a-7458-4f77-b02d-3fd5837d735d-original-pull-secret podName:e706cd7a-7458-4f77-b02d-3fd5837d735d nodeName:}" failed. No retries permitted until 2026-07-21 15:00:33.165802003 +0000 UTC m=+65.161587398 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e706cd7a-7458-4f77-b02d-3fd5837d735d-original-pull-secret") pod "global-pull-secret-syncer-jttxg" (UID: "e706cd7a-7458-4f77-b02d-3fd5837d735d") : object "kube-system"/"original-pull-secret" not registered Jul 21 15:00:01.165957 ip-10-0-135-176 kubenswrapper[2563]: E0721 15:00:01.165830 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26de6bb3-eb05-461a-958f-406464dea020-metrics-certs podName:26de6bb3-eb05-461a-958f-406464dea020 nodeName:}" failed. No retries permitted until 2026-07-21 15:00:33.165823603 +0000 UTC m=+65.161608998 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/26de6bb3-eb05-461a-958f-406464dea020-metrics-certs") pod "network-metrics-daemon-dh8jm" (UID: "26de6bb3-eb05-461a-958f-406464dea020") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 15:00:01.266921 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:00:01.266887 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dh6gh\" (UniqueName: \"kubernetes.io/projected/0683eb8b-8af7-4056-beed-3bfadd624f56-kube-api-access-dh6gh\") pod \"network-check-target-bzwfd\" (UID: \"0683eb8b-8af7-4056-beed-3bfadd624f56\") " pod="openshift-network-diagnostics/network-check-target-bzwfd" Jul 21 15:00:01.267095 ip-10-0-135-176 kubenswrapper[2563]: E0721 15:00:01.267002 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 21 15:00:01.267095 ip-10-0-135-176 kubenswrapper[2563]: E0721 15:00:01.267017 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 21 15:00:01.267095 ip-10-0-135-176 kubenswrapper[2563]: E0721 15:00:01.267026 2563 projected.go:196] Error preparing data for projected volume kube-api-access-dh6gh for pod openshift-network-diagnostics/network-check-target-bzwfd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 15:00:01.267095 ip-10-0-135-176 kubenswrapper[2563]: E0721 15:00:01.267074 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0683eb8b-8af7-4056-beed-3bfadd624f56-kube-api-access-dh6gh podName:0683eb8b-8af7-4056-beed-3bfadd624f56 nodeName:}" failed. No retries permitted until 2026-07-21 15:00:33.267060782 +0000 UTC m=+65.262846177 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-dh6gh" (UniqueName: "kubernetes.io/projected/0683eb8b-8af7-4056-beed-3bfadd624f56-kube-api-access-dh6gh") pod "network-check-target-bzwfd" (UID: "0683eb8b-8af7-4056-beed-3bfadd624f56") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 15:00:01.474520 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:00:01.474442 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bzwfd" Jul 21 15:00:01.474878 ip-10-0-135-176 kubenswrapper[2563]: E0721 15:00:01.474543 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bzwfd" podUID="0683eb8b-8af7-4056-beed-3bfadd624f56" Jul 21 15:00:02.475001 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:00:02.474961 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jttxg" Jul 21 15:00:02.475001 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:00:02.475002 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dh8jm" Jul 21 15:00:02.475421 ip-10-0-135-176 kubenswrapper[2563]: E0721 15:00:02.475090 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jttxg" podUID="e706cd7a-7458-4f77-b02d-3fd5837d735d" Jul 21 15:00:02.475421 ip-10-0-135-176 kubenswrapper[2563]: E0721 15:00:02.475208 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dh8jm" podUID="26de6bb3-eb05-461a-958f-406464dea020" Jul 21 15:00:02.741988 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:00:02.741899 2563 generic.go:356] "Generic (PLEG): container finished" podID="327b1f5e-8e27-4572-8025-5c516e59d481" containerID="75bec9037055620ee7ea3e0fce1f88e0c7d9e08df8d96eafe89b8a6851fe6208" exitCode=0 Jul 21 15:00:02.741988 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:00:02.741958 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8h5jp" event={"ID":"327b1f5e-8e27-4572-8025-5c516e59d481","Type":"ContainerDied","Data":"75bec9037055620ee7ea3e0fce1f88e0c7d9e08df8d96eafe89b8a6851fe6208"} Jul 21 15:00:03.474545 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:00:03.474356 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bzwfd" Jul 21 15:00:03.474695 ip-10-0-135-176 kubenswrapper[2563]: E0721 15:00:03.474627 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bzwfd" podUID="0683eb8b-8af7-4056-beed-3bfadd624f56" Jul 21 15:00:04.370814 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:00:04.370733 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-176.ec2.internal" event="NodeReady" Jul 21 15:00:04.371217 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:00:04.370919 2563 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 21 15:00:04.419434 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:00:04.419402 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-dpzts"] Jul 21 15:00:04.423137 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:00:04.423119 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-dpzts" Jul 21 15:00:04.426099 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:00:04.426070 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 21 15:00:04.426224 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:00:04.426132 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 21 15:00:04.426224 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:00:04.426141 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-bhwsr\"" Jul 21 15:00:04.435227 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:00:04.434318 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-dpzts"] Jul 21 15:00:04.436177 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:00:04.436152 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-vnx89"] Jul 21 15:00:04.439861 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:00:04.439842 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-vnx89" Jul 21 15:00:04.443195 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:00:04.443021 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 21 15:00:04.443195 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:00:04.443065 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 21 15:00:04.443195 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:00:04.443023 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 21 15:00:04.443195 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:00:04.443154 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-frd55\"" Jul 21 15:00:04.448440 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:00:04.448415 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-vnx89"] Jul 21 15:00:04.474161 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:00:04.474134 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jttxg" Jul 21 15:00:04.474316 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:00:04.474139 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dh8jm" Jul 21 15:00:04.477499 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:00:04.477480 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 21 15:00:04.477626 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:00:04.477512 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 21 15:00:04.477626 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:00:04.477577 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-x2gv2\"" Jul 21 15:00:04.586903 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:00:04.586866 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/28687086-f5b5-4889-afdb-8145abba8b1a-config-volume\") pod \"dns-default-dpzts\" (UID: \"28687086-f5b5-4889-afdb-8145abba8b1a\") " pod="openshift-dns/dns-default-dpzts" Jul 21 15:00:04.587089 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:00:04.586968 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmrj8\" (UniqueName: \"kubernetes.io/projected/9d1e2d09-f1a3-4fee-a2c3-86647059afe5-kube-api-access-vmrj8\") pod \"ingress-canary-vnx89\" (UID: \"9d1e2d09-f1a3-4fee-a2c3-86647059afe5\") " pod="openshift-ingress-canary/ingress-canary-vnx89" Jul 21 15:00:04.587089 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:00:04.587015 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28687086-f5b5-4889-afdb-8145abba8b1a-tmp-dir\") pod \"dns-default-dpzts\" (UID: \"28687086-f5b5-4889-afdb-8145abba8b1a\") " pod="openshift-dns/dns-default-dpzts" Jul 21 15:00:04.587208 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:00:04.587096 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/28687086-f5b5-4889-afdb-8145abba8b1a-metrics-tls\") pod \"dns-default-dpzts\" (UID: \"28687086-f5b5-4889-afdb-8145abba8b1a\") " pod="openshift-dns/dns-default-dpzts" Jul 21 15:00:04.587208 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:00:04.587126 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9d1e2d09-f1a3-4fee-a2c3-86647059afe5-cert\") pod \"ingress-canary-vnx89\" (UID: \"9d1e2d09-f1a3-4fee-a2c3-86647059afe5\") " pod="openshift-ingress-canary/ingress-canary-vnx89" Jul 21 15:00:04.587208 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:00:04.587161 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8f2k5\" (UniqueName: \"kubernetes.io/projected/28687086-f5b5-4889-afdb-8145abba8b1a-kube-api-access-8f2k5\") pod \"dns-default-dpzts\" (UID: \"28687086-f5b5-4889-afdb-8145abba8b1a\") " pod="openshift-dns/dns-default-dpzts" Jul 21 15:00:04.687876 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:00:04.687722 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28687086-f5b5-4889-afdb-8145abba8b1a-tmp-dir\") pod \"dns-default-dpzts\" (UID: \"28687086-f5b5-4889-afdb-8145abba8b1a\") " pod="openshift-dns/dns-default-dpzts" Jul 21 15:00:04.687876 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:00:04.687788 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/28687086-f5b5-4889-afdb-8145abba8b1a-metrics-tls\") pod \"dns-default-dpzts\" (UID: \"28687086-f5b5-4889-afdb-8145abba8b1a\") " pod="openshift-dns/dns-default-dpzts" Jul 21 15:00:04.687876 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:00:04.687815 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9d1e2d09-f1a3-4fee-a2c3-86647059afe5-cert\") pod \"ingress-canary-vnx89\" (UID: \"9d1e2d09-f1a3-4fee-a2c3-86647059afe5\") " pod="openshift-ingress-canary/ingress-canary-vnx89" Jul 21 15:00:04.687876 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:00:04.687848 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8f2k5\" (UniqueName: \"kubernetes.io/projected/28687086-f5b5-4889-afdb-8145abba8b1a-kube-api-access-8f2k5\") pod \"dns-default-dpzts\" (UID: \"28687086-f5b5-4889-afdb-8145abba8b1a\") " pod="openshift-dns/dns-default-dpzts" Jul 21 15:00:04.688390 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:00:04.687882 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/28687086-f5b5-4889-afdb-8145abba8b1a-config-volume\") pod \"dns-default-dpzts\" (UID: \"28687086-f5b5-4889-afdb-8145abba8b1a\") " pod="openshift-dns/dns-default-dpzts" Jul 21 15:00:04.688390 ip-10-0-135-176 kubenswrapper[2563]: E0721 15:00:04.687913 2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 21 15:00:04.688390 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:00:04.687926 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmrj8\" (UniqueName: \"kubernetes.io/projected/9d1e2d09-f1a3-4fee-a2c3-86647059afe5-kube-api-access-vmrj8\") pod \"ingress-canary-vnx89\" (UID: \"9d1e2d09-f1a3-4fee-a2c3-86647059afe5\") " pod="openshift-ingress-canary/ingress-canary-vnx89" Jul 21 15:00:04.688390 ip-10-0-135-176 kubenswrapper[2563]: E0721 15:00:04.687987 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/28687086-f5b5-4889-afdb-8145abba8b1a-metrics-tls podName:28687086-f5b5-4889-afdb-8145abba8b1a nodeName:}" failed. No retries permitted until 2026-07-21 15:00:05.187963222 +0000 UTC m=+37.183748617 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/28687086-f5b5-4889-afdb-8145abba8b1a-metrics-tls") pod "dns-default-dpzts" (UID: "28687086-f5b5-4889-afdb-8145abba8b1a") : secret "dns-default-metrics-tls" not found Jul 21 15:00:04.688390 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:00:04.688154 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28687086-f5b5-4889-afdb-8145abba8b1a-tmp-dir\") pod \"dns-default-dpzts\" (UID: \"28687086-f5b5-4889-afdb-8145abba8b1a\") " pod="openshift-dns/dns-default-dpzts" Jul 21 15:00:04.688727 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:00:04.688619 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/28687086-f5b5-4889-afdb-8145abba8b1a-config-volume\") pod \"dns-default-dpzts\" (UID: \"28687086-f5b5-4889-afdb-8145abba8b1a\") " pod="openshift-dns/dns-default-dpzts" Jul 21 15:00:04.692790 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:00:04.692770 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9d1e2d09-f1a3-4fee-a2c3-86647059afe5-cert\") pod \"ingress-canary-vnx89\" (UID: \"9d1e2d09-f1a3-4fee-a2c3-86647059afe5\") " pod="openshift-ingress-canary/ingress-canary-vnx89" Jul 21 15:00:04.698205 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:00:04.698176 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8f2k5\" (UniqueName: \"kubernetes.io/projected/28687086-f5b5-4889-afdb-8145abba8b1a-kube-api-access-8f2k5\") pod \"dns-default-dpzts\" (UID: \"28687086-f5b5-4889-afdb-8145abba8b1a\") " pod="openshift-dns/dns-default-dpzts" Jul 21 15:00:04.698346 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:00:04.698306 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmrj8\" (UniqueName: \"kubernetes.io/projected/9d1e2d09-f1a3-4fee-a2c3-86647059afe5-kube-api-access-vmrj8\") pod \"ingress-canary-vnx89\" (UID: \"9d1e2d09-f1a3-4fee-a2c3-86647059afe5\") " pod="openshift-ingress-canary/ingress-canary-vnx89" Jul 21 15:00:04.750357 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:00:04.750307 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-vnx89" Jul 21 15:00:04.918429 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:00:04.918365 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-vnx89"] Jul 21 15:00:04.924583 ip-10-0-135-176 kubenswrapper[2563]: W0721 15:00:04.924554 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d1e2d09_f1a3_4fee_a2c3_86647059afe5.slice/crio-bdf9d0a3f75ec578150ad9e3a1f7490d4037b5f26174ce474713a0293bcd9720 WatchSource:0}: Error finding container bdf9d0a3f75ec578150ad9e3a1f7490d4037b5f26174ce474713a0293bcd9720: Status 404 returned error can't find the container with id bdf9d0a3f75ec578150ad9e3a1f7490d4037b5f26174ce474713a0293bcd9720 Jul 21 15:00:05.192864 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:00:05.192825 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/28687086-f5b5-4889-afdb-8145abba8b1a-metrics-tls\") pod \"dns-default-dpzts\" (UID: \"28687086-f5b5-4889-afdb-8145abba8b1a\") " pod="openshift-dns/dns-default-dpzts" Jul 21 15:00:05.193062 ip-10-0-135-176 kubenswrapper[2563]: E0721 15:00:05.193033 2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 21 15:00:05.193143 ip-10-0-135-176 kubenswrapper[2563]: E0721 15:00:05.193129 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/28687086-f5b5-4889-afdb-8145abba8b1a-metrics-tls podName:28687086-f5b5-4889-afdb-8145abba8b1a nodeName:}" failed. No retries permitted until 2026-07-21 15:00:06.193106903 +0000 UTC m=+38.188892300 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/28687086-f5b5-4889-afdb-8145abba8b1a-metrics-tls") pod "dns-default-dpzts" (UID: "28687086-f5b5-4889-afdb-8145abba8b1a") : secret "dns-default-metrics-tls" not found Jul 21 15:00:05.474119 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:00:05.474041 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bzwfd" Jul 21 15:00:05.477490 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:00:05.477467 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 21 15:00:05.478615 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:00:05.478599 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 21 15:00:05.478673 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:00:05.478635 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-nplsp\"" Jul 21 15:00:05.750247 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:00:05.750152 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-vnx89" event={"ID":"9d1e2d09-f1a3-4fee-a2c3-86647059afe5","Type":"ContainerStarted","Data":"bdf9d0a3f75ec578150ad9e3a1f7490d4037b5f26174ce474713a0293bcd9720"} Jul 21 15:00:06.200242 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:00:06.200208 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/28687086-f5b5-4889-afdb-8145abba8b1a-metrics-tls\") pod \"dns-default-dpzts\" (UID: \"28687086-f5b5-4889-afdb-8145abba8b1a\") " pod="openshift-dns/dns-default-dpzts" Jul 21 15:00:06.200437 ip-10-0-135-176 kubenswrapper[2563]: E0721 15:00:06.200375 2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 21 15:00:06.200507 ip-10-0-135-176 kubenswrapper[2563]: E0721 15:00:06.200444 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/28687086-f5b5-4889-afdb-8145abba8b1a-metrics-tls podName:28687086-f5b5-4889-afdb-8145abba8b1a nodeName:}" failed. No retries permitted until 2026-07-21 15:00:08.200425346 +0000 UTC m=+40.196210744 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/28687086-f5b5-4889-afdb-8145abba8b1a-metrics-tls") pod "dns-default-dpzts" (UID: "28687086-f5b5-4889-afdb-8145abba8b1a") : secret "dns-default-metrics-tls" not found Jul 21 15:00:08.214724 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:00:08.214526 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/28687086-f5b5-4889-afdb-8145abba8b1a-metrics-tls\") pod \"dns-default-dpzts\" (UID: \"28687086-f5b5-4889-afdb-8145abba8b1a\") " pod="openshift-dns/dns-default-dpzts" Jul 21 15:00:08.215192 ip-10-0-135-176 kubenswrapper[2563]: E0721 15:00:08.214684 2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 21 15:00:08.215192 ip-10-0-135-176 kubenswrapper[2563]: E0721 15:00:08.214836 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/28687086-f5b5-4889-afdb-8145abba8b1a-metrics-tls podName:28687086-f5b5-4889-afdb-8145abba8b1a nodeName:}" failed. No retries permitted until 2026-07-21 15:00:12.214819625 +0000 UTC m=+44.210605023 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/28687086-f5b5-4889-afdb-8145abba8b1a-metrics-tls") pod "dns-default-dpzts" (UID: "28687086-f5b5-4889-afdb-8145abba8b1a") : secret "dns-default-metrics-tls" not found Jul 21 15:00:09.763077 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:00:09.762977 2563 generic.go:356] "Generic (PLEG): container finished" podID="327b1f5e-8e27-4572-8025-5c516e59d481" containerID="d3437cde0a40c94a73034996fb0e5f931beabb7bd6f59b4cd0696d285d4b8273" exitCode=0 Jul 21 15:00:09.763077 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:00:09.763052 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8h5jp" event={"ID":"327b1f5e-8e27-4572-8025-5c516e59d481","Type":"ContainerDied","Data":"d3437cde0a40c94a73034996fb0e5f931beabb7bd6f59b4cd0696d285d4b8273"} Jul 21 15:00:09.764600 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:00:09.764579 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-vnx89" event={"ID":"9d1e2d09-f1a3-4fee-a2c3-86647059afe5","Type":"ContainerStarted","Data":"be6b33cd6cf690025de421fca329b39f82bb7bcb260992fc4d4be7295212182f"} Jul 21 15:00:09.805354 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:00:09.805293 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-vnx89" podStartSLOduration=1.3452717889999999 podStartE2EDuration="5.805277711s" totalImagesPullingTime="4.460005922s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 15:00:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 15:00:09.805158898 +0000 UTC m=+41.800944315" watchObservedRunningTime="2026-07-21 15:00:09.805277711 +0000 UTC m=+41.801063128" Jul 21 15:00:10.769685 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:00:10.769646 2563 generic.go:356] "Generic (PLEG): container finished" podID="327b1f5e-8e27-4572-8025-5c516e59d481" containerID="bd8f324b0affcbfd017a1ef85995bba91404aa76715ceeda67871a53a8bed3f3" exitCode=0 Jul 21 15:00:10.770047 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:00:10.769730 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8h5jp" event={"ID":"327b1f5e-8e27-4572-8025-5c516e59d481","Type":"ContainerDied","Data":"bd8f324b0affcbfd017a1ef85995bba91404aa76715ceeda67871a53a8bed3f3"} Jul 21 15:00:11.775768 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:00:11.775732 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8h5jp" event={"ID":"327b1f5e-8e27-4572-8025-5c516e59d481","Type":"ContainerStarted","Data":"0b87d23de43c7e45ffa78f8365de5272a8b5cb87e0a6614137901aeedec8996c"} Jul 21 15:00:11.803138 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:00:11.803078 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-8h5jp" podStartSLOduration=7.904587853 podStartE2EDuration="43.803059258s" totalImagesPullingTime="35.898471405s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 14:59:28 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 15:00:11.801165415 +0000 UTC m=+43.796950848" watchObservedRunningTime="2026-07-21 15:00:11.803059258 +0000 UTC m=+43.798844678" Jul 21 15:00:12.246207 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:00:12.246172 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/28687086-f5b5-4889-afdb-8145abba8b1a-metrics-tls\") pod \"dns-default-dpzts\" (UID: \"28687086-f5b5-4889-afdb-8145abba8b1a\") " pod="openshift-dns/dns-default-dpzts" Jul 21 15:00:12.246389 ip-10-0-135-176 kubenswrapper[2563]: E0721 15:00:12.246275 2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 21 15:00:12.246389 ip-10-0-135-176 kubenswrapper[2563]: E0721 15:00:12.246348 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/28687086-f5b5-4889-afdb-8145abba8b1a-metrics-tls podName:28687086-f5b5-4889-afdb-8145abba8b1a nodeName:}" failed. No retries permitted until 2026-07-21 15:00:20.246318689 +0000 UTC m=+52.242104084 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/28687086-f5b5-4889-afdb-8145abba8b1a-metrics-tls") pod "dns-default-dpzts" (UID: "28687086-f5b5-4889-afdb-8145abba8b1a") : secret "dns-default-metrics-tls" not found Jul 21 15:00:20.301342 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:00:20.301288 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/28687086-f5b5-4889-afdb-8145abba8b1a-metrics-tls\") pod \"dns-default-dpzts\" (UID: \"28687086-f5b5-4889-afdb-8145abba8b1a\") " pod="openshift-dns/dns-default-dpzts" Jul 21 15:00:20.301825 ip-10-0-135-176 kubenswrapper[2563]: E0721 15:00:20.301455 2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 21 15:00:20.301825 ip-10-0-135-176 kubenswrapper[2563]: E0721 15:00:20.301525 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/28687086-f5b5-4889-afdb-8145abba8b1a-metrics-tls podName:28687086-f5b5-4889-afdb-8145abba8b1a nodeName:}" failed. No retries permitted until 2026-07-21 15:00:36.301506003 +0000 UTC m=+68.297291409 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/28687086-f5b5-4889-afdb-8145abba8b1a-metrics-tls") pod "dns-default-dpzts" (UID: "28687086-f5b5-4889-afdb-8145abba8b1a") : secret "dns-default-metrics-tls" not found Jul 21 15:00:29.744647 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:00:29.744611 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kn9fn" Jul 21 15:00:33.191693 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:00:33.191646 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e706cd7a-7458-4f77-b02d-3fd5837d735d-original-pull-secret\") pod \"global-pull-secret-syncer-jttxg\" (UID: \"e706cd7a-7458-4f77-b02d-3fd5837d735d\") " pod="kube-system/global-pull-secret-syncer-jttxg" Jul 21 15:00:33.192143 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:00:33.191712 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26de6bb3-eb05-461a-958f-406464dea020-metrics-certs\") pod \"network-metrics-daemon-dh8jm\" (UID: \"26de6bb3-eb05-461a-958f-406464dea020\") " pod="openshift-multus/network-metrics-daemon-dh8jm" Jul 21 15:00:33.195208 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:00:33.195192 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 21 15:00:33.195251 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:00:33.195221 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 21 15:00:33.202726 ip-10-0-135-176 kubenswrapper[2563]: E0721 15:00:33.202703 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 21 15:00:33.202781 ip-10-0-135-176 kubenswrapper[2563]: E0721 15:00:33.202767 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26de6bb3-eb05-461a-958f-406464dea020-metrics-certs podName:26de6bb3-eb05-461a-958f-406464dea020 nodeName:}" failed. No retries permitted until 2026-07-21 15:01:37.202748265 +0000 UTC m=+129.198533660 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/26de6bb3-eb05-461a-958f-406464dea020-metrics-certs") pod "network-metrics-daemon-dh8jm" (UID: "26de6bb3-eb05-461a-958f-406464dea020") : secret "metrics-daemon-secret" not found Jul 21 15:00:33.205567 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:00:33.205549 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e706cd7a-7458-4f77-b02d-3fd5837d735d-original-pull-secret\") pod \"global-pull-secret-syncer-jttxg\" (UID: \"e706cd7a-7458-4f77-b02d-3fd5837d735d\") " pod="kube-system/global-pull-secret-syncer-jttxg" Jul 21 15:00:33.285658 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:00:33.285626 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jttxg" Jul 21 15:00:33.292861 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:00:33.292836 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dh6gh\" (UniqueName: \"kubernetes.io/projected/0683eb8b-8af7-4056-beed-3bfadd624f56-kube-api-access-dh6gh\") pod \"network-check-target-bzwfd\" (UID: \"0683eb8b-8af7-4056-beed-3bfadd624f56\") " pod="openshift-network-diagnostics/network-check-target-bzwfd" Jul 21 15:00:33.296124 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:00:33.296099 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 21 15:00:33.306357 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:00:33.306314 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 21 15:00:33.316093 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:00:33.316074 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dh6gh\" (UniqueName: \"kubernetes.io/projected/0683eb8b-8af7-4056-beed-3bfadd624f56-kube-api-access-dh6gh\") pod \"network-check-target-bzwfd\" (UID: \"0683eb8b-8af7-4056-beed-3bfadd624f56\") " pod="openshift-network-diagnostics/network-check-target-bzwfd" Jul 21 15:00:33.387625 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:00:33.387596 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-nplsp\"" Jul 21 15:00:33.395281 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:00:33.395254 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bzwfd" Jul 21 15:00:33.408135 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:00:33.407960 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-jttxg"] Jul 21 15:00:33.411692 ip-10-0-135-176 kubenswrapper[2563]: W0721 15:00:33.411663 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode706cd7a_7458_4f77_b02d_3fd5837d735d.slice/crio-08fb9f9cabea8315b5daa6216d94690ea2ed7cf5aff64e64227da7d42912a59b WatchSource:0}: Error finding container 08fb9f9cabea8315b5daa6216d94690ea2ed7cf5aff64e64227da7d42912a59b: Status 404 returned error can't find the container with id 08fb9f9cabea8315b5daa6216d94690ea2ed7cf5aff64e64227da7d42912a59b Jul 21 15:00:33.519386 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:00:33.519357 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-bzwfd"] Jul 21 15:00:33.522541 ip-10-0-135-176 kubenswrapper[2563]: W0721 15:00:33.522509 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0683eb8b_8af7_4056_beed_3bfadd624f56.slice/crio-8c166311bde9aacd1f1fdccb045dfe1eea6279f5dd1f562183d56e49612772a9 WatchSource:0}: Error finding container 8c166311bde9aacd1f1fdccb045dfe1eea6279f5dd1f562183d56e49612772a9: Status 404 returned error can't find the container with id 8c166311bde9aacd1f1fdccb045dfe1eea6279f5dd1f562183d56e49612772a9 Jul 21 15:00:33.826437 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:00:33.826401 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-jttxg" event={"ID":"e706cd7a-7458-4f77-b02d-3fd5837d735d","Type":"ContainerStarted","Data":"08fb9f9cabea8315b5daa6216d94690ea2ed7cf5aff64e64227da7d42912a59b"} Jul 21 15:00:33.827501 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:00:33.827479 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-bzwfd" event={"ID":"0683eb8b-8af7-4056-beed-3bfadd624f56","Type":"ContainerStarted","Data":"8c166311bde9aacd1f1fdccb045dfe1eea6279f5dd1f562183d56e49612772a9"} Jul 21 15:00:36.313631 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:00:36.313591 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/28687086-f5b5-4889-afdb-8145abba8b1a-metrics-tls\") pod \"dns-default-dpzts\" (UID: \"28687086-f5b5-4889-afdb-8145abba8b1a\") " pod="openshift-dns/dns-default-dpzts" Jul 21 15:00:36.314195 ip-10-0-135-176 kubenswrapper[2563]: E0721 15:00:36.313721 2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 21 15:00:36.314195 ip-10-0-135-176 kubenswrapper[2563]: E0721 15:00:36.313781 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/28687086-f5b5-4889-afdb-8145abba8b1a-metrics-tls podName:28687086-f5b5-4889-afdb-8145abba8b1a nodeName:}" failed. No retries permitted until 2026-07-21 15:01:08.313764155 +0000 UTC m=+100.309549550 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/28687086-f5b5-4889-afdb-8145abba8b1a-metrics-tls") pod "dns-default-dpzts" (UID: "28687086-f5b5-4889-afdb-8145abba8b1a") : secret "dns-default-metrics-tls" not found Jul 21 15:00:38.843289 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:00:38.843253 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-bzwfd" event={"ID":"0683eb8b-8af7-4056-beed-3bfadd624f56","Type":"ContainerStarted","Data":"ab6decf557e8daffcc9228eaa0c999721f7ae54b3479cc6c6f9aaa1e3f76226f"} Jul 21 15:00:38.843750 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:00:38.843368 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-bzwfd" Jul 21 15:00:38.844742 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:00:38.844716 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-jttxg" event={"ID":"e706cd7a-7458-4f77-b02d-3fd5837d735d","Type":"ContainerStarted","Data":"496f7b4ddeb72fa2b8539d7ae6c463164c59743550ee120c46e5ac58c246dd88"} Jul 21 15:00:38.861031 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:00:38.860973 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-bzwfd" podStartSLOduration=66.190801605 podStartE2EDuration="1m10.86095721s" totalImagesPullingTime="4.670155605s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 14:59:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 15:00:38.860502019 +0000 UTC m=+70.856287438" watchObservedRunningTime="2026-07-21 15:00:38.86095721 +0000 UTC m=+70.856742646" Jul 21 15:00:38.879409 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:00:38.879365 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-jttxg" podStartSLOduration=66.094044326 podStartE2EDuration="1m10.879349939s" totalImagesPullingTime="4.785305613s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 14:59:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 15:00:38.878817356 +0000 UTC m=+70.874602788" watchObservedRunningTime="2026-07-21 15:00:38.879349939 +0000 UTC m=+70.875135346" Jul 21 15:01:08.317363 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:01:08.317250 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/28687086-f5b5-4889-afdb-8145abba8b1a-metrics-tls\") pod \"dns-default-dpzts\" (UID: \"28687086-f5b5-4889-afdb-8145abba8b1a\") " pod="openshift-dns/dns-default-dpzts" Jul 21 15:01:08.317775 ip-10-0-135-176 kubenswrapper[2563]: E0721 15:01:08.317392 2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 21 15:01:08.317775 ip-10-0-135-176 kubenswrapper[2563]: E0721 15:01:08.317445 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/28687086-f5b5-4889-afdb-8145abba8b1a-metrics-tls podName:28687086-f5b5-4889-afdb-8145abba8b1a nodeName:}" failed. No retries permitted until 2026-07-21 15:02:12.317429351 +0000 UTC m=+164.313214748 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/28687086-f5b5-4889-afdb-8145abba8b1a-metrics-tls") pod "dns-default-dpzts" (UID: "28687086-f5b5-4889-afdb-8145abba8b1a") : secret "dns-default-metrics-tls" not found Jul 21 15:01:09.848920 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:01:09.848893 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-bzwfd" Jul 21 15:01:37.209004 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:01:37.208956 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26de6bb3-eb05-461a-958f-406464dea020-metrics-certs\") pod \"network-metrics-daemon-dh8jm\" (UID: \"26de6bb3-eb05-461a-958f-406464dea020\") " pod="openshift-multus/network-metrics-daemon-dh8jm" Jul 21 15:01:37.209425 ip-10-0-135-176 kubenswrapper[2563]: E0721 15:01:37.209108 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 21 15:01:37.209425 ip-10-0-135-176 kubenswrapper[2563]: E0721 15:01:37.209180 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26de6bb3-eb05-461a-958f-406464dea020-metrics-certs podName:26de6bb3-eb05-461a-958f-406464dea020 nodeName:}" failed. No retries permitted until 2026-07-21 15:03:39.209161363 +0000 UTC m=+251.204946758 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/26de6bb3-eb05-461a-958f-406464dea020-metrics-certs") pod "network-metrics-daemon-dh8jm" (UID: "26de6bb3-eb05-461a-958f-406464dea020") : secret "metrics-daemon-secret" not found Jul 21 15:02:07.435600 ip-10-0-135-176 kubenswrapper[2563]: E0721 15:02:07.435544 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-dpzts" podUID="28687086-f5b5-4889-afdb-8145abba8b1a" Jul 21 15:02:07.491140 ip-10-0-135-176 kubenswrapper[2563]: E0721 15:02:07.491101 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-dh8jm" podUID="26de6bb3-eb05-461a-958f-406464dea020" Jul 21 15:02:08.052530 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:02:08.052500 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-dpzts" Jul 21 15:02:12.342804 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:02:12.342768 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/28687086-f5b5-4889-afdb-8145abba8b1a-metrics-tls\") pod \"dns-default-dpzts\" (UID: \"28687086-f5b5-4889-afdb-8145abba8b1a\") " pod="openshift-dns/dns-default-dpzts" Jul 21 15:02:12.343185 ip-10-0-135-176 kubenswrapper[2563]: E0721 15:02:12.342924 2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 21 15:02:12.343185 ip-10-0-135-176 kubenswrapper[2563]: E0721 15:02:12.342993 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/28687086-f5b5-4889-afdb-8145abba8b1a-metrics-tls podName:28687086-f5b5-4889-afdb-8145abba8b1a nodeName:}" failed. No retries permitted until 2026-07-21 15:04:14.342975438 +0000 UTC m=+286.338760836 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/28687086-f5b5-4889-afdb-8145abba8b1a-metrics-tls") pod "dns-default-dpzts" (UID: "28687086-f5b5-4889-afdb-8145abba8b1a") : secret "dns-default-metrics-tls" not found Jul 21 15:02:22.474142 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:02:22.474062 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dh8jm" Jul 21 15:03:30.332852 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:30.332819 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-jd987"] Jul 21 15:03:30.335699 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:30.335672 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-jd987" Jul 21 15:03:30.336978 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:30.336956 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-76cf8966f8-6mtbk"] Jul 21 15:03:30.339512 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:30.339495 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-6mtbk" Jul 21 15:03:30.343243 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:30.343216 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-6c2j9\"" Jul 21 15:03:30.343407 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:30.343388 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 21 15:03:30.343476 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:30.343458 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 21 15:03:30.343723 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:30.343700 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 21 15:03:30.344149 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:30.344132 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-bfv6n"] Jul 21 15:03:30.344840 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:30.344824 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 21 15:03:30.344905 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:30.344856 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 21 15:03:30.346247 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:30.346230 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 21 15:03:30.346366 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:30.346287 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-rpp5g\"" Jul 21 15:03:30.347004 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:30.346988 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-bfv6n" Jul 21 15:03:30.352648 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:30.352630 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 21 15:03:30.352724 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:30.352630 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 21 15:03:30.352724 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:30.352634 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 21 15:03:30.353416 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:30.353398 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-jsj4r\"" Jul 21 15:03:30.353510 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:30.353454 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 21 15:03:30.373000 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:30.370768 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-jd987"] Jul 21 15:03:30.373000 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:30.370867 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-76cf8966f8-6mtbk"] Jul 21 15:03:30.373634 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:30.373612 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/53f7dd9f-230e-4c5c-bc17-bd14332b1923-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-bfv6n\" (UID: \"53f7dd9f-230e-4c5c-bc17-bd14332b1923\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-bfv6n" Jul 21 15:03:30.373687 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:30.373650 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jct5g\" (UniqueName: \"kubernetes.io/projected/feec54ec-7a62-4ad4-94bf-f28792b7dd4f-kube-api-access-jct5g\") pod \"volume-data-source-validator-78cf4c6f6-jd987\" (UID: \"feec54ec-7a62-4ad4-94bf-f28792b7dd4f\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-jd987" Jul 21 15:03:30.373687 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:30.373668 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5jkw\" (UniqueName: \"kubernetes.io/projected/53f7dd9f-230e-4c5c-bc17-bd14332b1923-kube-api-access-f5jkw\") pod \"kube-storage-version-migrator-operator-7658d9b749-bfv6n\" (UID: \"53f7dd9f-230e-4c5c-bc17-bd14332b1923\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-bfv6n" Jul 21 15:03:30.373779 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:30.373688 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0f91a500-cd56-4cc9-8f38-d60a5e8a0562-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-6mtbk\" (UID: \"0f91a500-cd56-4cc9-8f38-d60a5e8a0562\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-6mtbk" Jul 21 15:03:30.373779 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:30.373726 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2r5sf\" (UniqueName: \"kubernetes.io/projected/0f91a500-cd56-4cc9-8f38-d60a5e8a0562-kube-api-access-2r5sf\") pod \"cluster-monitoring-operator-76cf8966f8-6mtbk\" (UID: \"0f91a500-cd56-4cc9-8f38-d60a5e8a0562\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-6mtbk" Jul 21 15:03:30.373779 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:30.373773 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/53f7dd9f-230e-4c5c-bc17-bd14332b1923-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-bfv6n\" (UID: \"53f7dd9f-230e-4c5c-bc17-bd14332b1923\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-bfv6n" Jul 21 15:03:30.373899 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:30.373793 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/0f91a500-cd56-4cc9-8f38-d60a5e8a0562-telemetry-config\") pod \"cluster-monitoring-operator-76cf8966f8-6mtbk\" (UID: \"0f91a500-cd56-4cc9-8f38-d60a5e8a0562\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-6mtbk" Jul 21 15:03:30.387073 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:30.387045 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-bfv6n"] Jul 21 15:03:30.432638 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:30.432603 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-fd497b998-txmbc"] Jul 21 15:03:30.435524 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:30.435506 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7bc5b6bf85-djs2t"] Jul 21 15:03:30.435651 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:30.435631 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-fd497b998-txmbc" Jul 21 15:03:30.438140 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:30.438123 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-djs2t" Jul 21 15:03:30.438489 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:30.438472 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 21 15:03:30.438639 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:30.438488 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 21 15:03:30.438768 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:30.438752 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 21 15:03:30.438839 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:30.438774 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 21 15:03:30.438839 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:30.438781 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 21 15:03:30.438942 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:30.438901 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-6xz6w\"" Jul 21 15:03:30.439018 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:30.439003 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 21 15:03:30.440881 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:30.440866 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-cmgwc\"" Jul 21 15:03:30.451261 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:30.451177 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7bc5b6bf85-djs2t"] Jul 21 15:03:30.452160 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:30.452139 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-fd497b998-txmbc"] Jul 21 15:03:30.475123 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:30.475094 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/5cec8b20-be0d-4f3d-a308-6d6056a96db7-default-certificate\") pod \"router-default-fd497b998-txmbc\" (UID: \"5cec8b20-be0d-4f3d-a308-6d6056a96db7\") " pod="openshift-ingress/router-default-fd497b998-txmbc" Jul 21 15:03:30.475123 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:30.475125 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5cec8b20-be0d-4f3d-a308-6d6056a96db7-metrics-certs\") pod \"router-default-fd497b998-txmbc\" (UID: \"5cec8b20-be0d-4f3d-a308-6d6056a96db7\") " pod="openshift-ingress/router-default-fd497b998-txmbc" Jul 21 15:03:30.475389 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:30.475148 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/53f7dd9f-230e-4c5c-bc17-bd14332b1923-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-bfv6n\" (UID: \"53f7dd9f-230e-4c5c-bc17-bd14332b1923\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-bfv6n" Jul 21 15:03:30.475389 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:30.475166 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/0f91a500-cd56-4cc9-8f38-d60a5e8a0562-telemetry-config\") pod \"cluster-monitoring-operator-76cf8966f8-6mtbk\" (UID: \"0f91a500-cd56-4cc9-8f38-d60a5e8a0562\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-6mtbk" Jul 21 15:03:30.475389 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:30.475181 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/5cec8b20-be0d-4f3d-a308-6d6056a96db7-stats-auth\") pod \"router-default-fd497b998-txmbc\" (UID: \"5cec8b20-be0d-4f3d-a308-6d6056a96db7\") " pod="openshift-ingress/router-default-fd497b998-txmbc" Jul 21 15:03:30.475389 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:30.475274 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/53f7dd9f-230e-4c5c-bc17-bd14332b1923-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-bfv6n\" (UID: \"53f7dd9f-230e-4c5c-bc17-bd14332b1923\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-bfv6n" Jul 21 15:03:30.475389 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:30.475310 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jct5g\" (UniqueName: \"kubernetes.io/projected/feec54ec-7a62-4ad4-94bf-f28792b7dd4f-kube-api-access-jct5g\") pod \"volume-data-source-validator-78cf4c6f6-jd987\" (UID: \"feec54ec-7a62-4ad4-94bf-f28792b7dd4f\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-jd987" Jul 21 15:03:30.475389 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:30.475353 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5jkw\" (UniqueName: \"kubernetes.io/projected/53f7dd9f-230e-4c5c-bc17-bd14332b1923-kube-api-access-f5jkw\") pod \"kube-storage-version-migrator-operator-7658d9b749-bfv6n\" (UID: \"53f7dd9f-230e-4c5c-bc17-bd14332b1923\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-bfv6n" Jul 21 15:03:30.475389 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:30.475389 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0f91a500-cd56-4cc9-8f38-d60a5e8a0562-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-6mtbk\" (UID: \"0f91a500-cd56-4cc9-8f38-d60a5e8a0562\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-6mtbk" Jul 21 15:03:30.475778 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:30.475418 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpbxt\" (UniqueName: \"kubernetes.io/projected/5cec8b20-be0d-4f3d-a308-6d6056a96db7-kube-api-access-kpbxt\") pod \"router-default-fd497b998-txmbc\" (UID: \"5cec8b20-be0d-4f3d-a308-6d6056a96db7\") " pod="openshift-ingress/router-default-fd497b998-txmbc" Jul 21 15:03:30.475778 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:30.475443 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5cec8b20-be0d-4f3d-a308-6d6056a96db7-service-ca-bundle\") pod \"router-default-fd497b998-txmbc\" (UID: \"5cec8b20-be0d-4f3d-a308-6d6056a96db7\") " pod="openshift-ingress/router-default-fd497b998-txmbc" Jul 21 15:03:30.475778 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:30.475481 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2r5sf\" (UniqueName: \"kubernetes.io/projected/0f91a500-cd56-4cc9-8f38-d60a5e8a0562-kube-api-access-2r5sf\") pod \"cluster-monitoring-operator-76cf8966f8-6mtbk\" (UID: \"0f91a500-cd56-4cc9-8f38-d60a5e8a0562\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-6mtbk" Jul 21 15:03:30.475778 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:30.475506 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcvsx\" (UniqueName: \"kubernetes.io/projected/c1fb87fc-ef05-446d-bc6c-6059d487f2fe-kube-api-access-fcvsx\") pod \"network-check-source-7bc5b6bf85-djs2t\" (UID: \"c1fb87fc-ef05-446d-bc6c-6059d487f2fe\") " pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-djs2t" Jul 21 15:03:30.475778 ip-10-0-135-176 kubenswrapper[2563]: E0721 15:03:30.475563 2563 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 21 15:03:30.475778 ip-10-0-135-176 kubenswrapper[2563]: E0721 15:03:30.475632 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f91a500-cd56-4cc9-8f38-d60a5e8a0562-cluster-monitoring-operator-tls podName:0f91a500-cd56-4cc9-8f38-d60a5e8a0562 nodeName:}" failed. No retries permitted until 2026-07-21 15:03:30.975613814 +0000 UTC m=+242.971399209 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/0f91a500-cd56-4cc9-8f38-d60a5e8a0562-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-6mtbk" (UID: "0f91a500-cd56-4cc9-8f38-d60a5e8a0562") : secret "cluster-monitoring-operator-tls" not found Jul 21 15:03:30.476061 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:30.475863 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/53f7dd9f-230e-4c5c-bc17-bd14332b1923-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-bfv6n\" (UID: \"53f7dd9f-230e-4c5c-bc17-bd14332b1923\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-bfv6n" Jul 21 15:03:30.476061 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:30.475989 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/0f91a500-cd56-4cc9-8f38-d60a5e8a0562-telemetry-config\") pod \"cluster-monitoring-operator-76cf8966f8-6mtbk\" (UID: \"0f91a500-cd56-4cc9-8f38-d60a5e8a0562\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-6mtbk" Jul 21 15:03:30.477918 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:30.477901 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/53f7dd9f-230e-4c5c-bc17-bd14332b1923-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-bfv6n\" (UID: \"53f7dd9f-230e-4c5c-bc17-bd14332b1923\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-bfv6n" Jul 21 15:03:30.505825 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:30.505797 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5jkw\" (UniqueName: \"kubernetes.io/projected/53f7dd9f-230e-4c5c-bc17-bd14332b1923-kube-api-access-f5jkw\") pod \"kube-storage-version-migrator-operator-7658d9b749-bfv6n\" (UID: \"53f7dd9f-230e-4c5c-bc17-bd14332b1923\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-bfv6n" Jul 21 15:03:30.506140 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:30.506117 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jct5g\" (UniqueName: \"kubernetes.io/projected/feec54ec-7a62-4ad4-94bf-f28792b7dd4f-kube-api-access-jct5g\") pod \"volume-data-source-validator-78cf4c6f6-jd987\" (UID: \"feec54ec-7a62-4ad4-94bf-f28792b7dd4f\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-jd987" Jul 21 15:03:30.506189 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:30.506117 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2r5sf\" (UniqueName: \"kubernetes.io/projected/0f91a500-cd56-4cc9-8f38-d60a5e8a0562-kube-api-access-2r5sf\") pod \"cluster-monitoring-operator-76cf8966f8-6mtbk\" (UID: \"0f91a500-cd56-4cc9-8f38-d60a5e8a0562\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-6mtbk" Jul 21 15:03:30.575872 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:30.575844 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpbxt\" (UniqueName: \"kubernetes.io/projected/5cec8b20-be0d-4f3d-a308-6d6056a96db7-kube-api-access-kpbxt\") pod \"router-default-fd497b998-txmbc\" (UID: \"5cec8b20-be0d-4f3d-a308-6d6056a96db7\") " pod="openshift-ingress/router-default-fd497b998-txmbc" Jul 21 15:03:30.575872 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:30.575877 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5cec8b20-be0d-4f3d-a308-6d6056a96db7-service-ca-bundle\") pod \"router-default-fd497b998-txmbc\" (UID: \"5cec8b20-be0d-4f3d-a308-6d6056a96db7\") " pod="openshift-ingress/router-default-fd497b998-txmbc" Jul 21 15:03:30.576093 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:30.575909 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcvsx\" (UniqueName: \"kubernetes.io/projected/c1fb87fc-ef05-446d-bc6c-6059d487f2fe-kube-api-access-fcvsx\") pod \"network-check-source-7bc5b6bf85-djs2t\" (UID: \"c1fb87fc-ef05-446d-bc6c-6059d487f2fe\") " pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-djs2t" Jul 21 15:03:30.576093 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:30.575942 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/5cec8b20-be0d-4f3d-a308-6d6056a96db7-default-certificate\") pod \"router-default-fd497b998-txmbc\" (UID: \"5cec8b20-be0d-4f3d-a308-6d6056a96db7\") " pod="openshift-ingress/router-default-fd497b998-txmbc" Jul 21 15:03:30.576093 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:30.575965 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5cec8b20-be0d-4f3d-a308-6d6056a96db7-metrics-certs\") pod \"router-default-fd497b998-txmbc\" (UID: \"5cec8b20-be0d-4f3d-a308-6d6056a96db7\") " pod="openshift-ingress/router-default-fd497b998-txmbc" Jul 21 15:03:30.576093 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:30.575989 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/5cec8b20-be0d-4f3d-a308-6d6056a96db7-stats-auth\") pod \"router-default-fd497b998-txmbc\" (UID: \"5cec8b20-be0d-4f3d-a308-6d6056a96db7\") " pod="openshift-ingress/router-default-fd497b998-txmbc" Jul 21 15:03:30.576093 ip-10-0-135-176 kubenswrapper[2563]: E0721 15:03:30.576052 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5cec8b20-be0d-4f3d-a308-6d6056a96db7-service-ca-bundle podName:5cec8b20-be0d-4f3d-a308-6d6056a96db7 nodeName:}" failed. No retries permitted until 2026-07-21 15:03:31.076029941 +0000 UTC m=+243.071815363 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5cec8b20-be0d-4f3d-a308-6d6056a96db7-service-ca-bundle") pod "router-default-fd497b998-txmbc" (UID: "5cec8b20-be0d-4f3d-a308-6d6056a96db7") : configmap references non-existent config key: service-ca.crt Jul 21 15:03:30.576093 ip-10-0-135-176 kubenswrapper[2563]: E0721 15:03:30.576085 2563 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 21 15:03:30.576415 ip-10-0-135-176 kubenswrapper[2563]: E0721 15:03:30.576143 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5cec8b20-be0d-4f3d-a308-6d6056a96db7-metrics-certs podName:5cec8b20-be0d-4f3d-a308-6d6056a96db7 nodeName:}" failed. No retries permitted until 2026-07-21 15:03:31.076130403 +0000 UTC m=+243.071915814 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5cec8b20-be0d-4f3d-a308-6d6056a96db7-metrics-certs") pod "router-default-fd497b998-txmbc" (UID: "5cec8b20-be0d-4f3d-a308-6d6056a96db7") : secret "router-metrics-certs-default" not found Jul 21 15:03:30.578529 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:30.578505 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/5cec8b20-be0d-4f3d-a308-6d6056a96db7-stats-auth\") pod \"router-default-fd497b998-txmbc\" (UID: \"5cec8b20-be0d-4f3d-a308-6d6056a96db7\") " pod="openshift-ingress/router-default-fd497b998-txmbc" Jul 21 15:03:30.578632 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:30.578556 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/5cec8b20-be0d-4f3d-a308-6d6056a96db7-default-certificate\") pod \"router-default-fd497b998-txmbc\" (UID: \"5cec8b20-be0d-4f3d-a308-6d6056a96db7\") " pod="openshift-ingress/router-default-fd497b998-txmbc" Jul 21 15:03:30.585203 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:30.585150 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcvsx\" (UniqueName: \"kubernetes.io/projected/c1fb87fc-ef05-446d-bc6c-6059d487f2fe-kube-api-access-fcvsx\") pod \"network-check-source-7bc5b6bf85-djs2t\" (UID: \"c1fb87fc-ef05-446d-bc6c-6059d487f2fe\") " pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-djs2t" Jul 21 15:03:30.585203 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:30.585170 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpbxt\" (UniqueName: \"kubernetes.io/projected/5cec8b20-be0d-4f3d-a308-6d6056a96db7-kube-api-access-kpbxt\") pod \"router-default-fd497b998-txmbc\" (UID: \"5cec8b20-be0d-4f3d-a308-6d6056a96db7\") " pod="openshift-ingress/router-default-fd497b998-txmbc" Jul 21 15:03:30.646430 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:30.646379 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-jd987" Jul 21 15:03:30.655876 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:30.655846 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-bfv6n" Jul 21 15:03:30.750006 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:30.749981 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-djs2t" Jul 21 15:03:30.778728 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:30.778634 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-jd987"] Jul 21 15:03:30.781377 ip-10-0-135-176 kubenswrapper[2563]: W0721 15:03:30.781353 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfeec54ec_7a62_4ad4_94bf_f28792b7dd4f.slice/crio-f82c42678f55882a629a35e4eae06e0266891901a6ce2a2887ebaa1d832a675e WatchSource:0}: Error finding container f82c42678f55882a629a35e4eae06e0266891901a6ce2a2887ebaa1d832a675e: Status 404 returned error can't find the container with id f82c42678f55882a629a35e4eae06e0266891901a6ce2a2887ebaa1d832a675e Jul 21 15:03:30.796551 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:30.796526 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-bfv6n"] Jul 21 15:03:30.800153 ip-10-0-135-176 kubenswrapper[2563]: W0721 15:03:30.800127 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53f7dd9f_230e_4c5c_bc17_bd14332b1923.slice/crio-9553c418a7d16e0ca1b20d2bc34a5db3b4822e1a3a15294e4efd9a41423a9684 WatchSource:0}: Error finding container 9553c418a7d16e0ca1b20d2bc34a5db3b4822e1a3a15294e4efd9a41423a9684: Status 404 returned error can't find the container with id 9553c418a7d16e0ca1b20d2bc34a5db3b4822e1a3a15294e4efd9a41423a9684 Jul 21 15:03:30.876206 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:30.876125 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7bc5b6bf85-djs2t"] Jul 21 15:03:30.879103 ip-10-0-135-176 kubenswrapper[2563]: W0721 15:03:30.879072 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1fb87fc_ef05_446d_bc6c_6059d487f2fe.slice/crio-ff906f5b9afa31cffa7fa775a1720a79c171648b4fda48d9bbc124ac7894b501 WatchSource:0}: Error finding container ff906f5b9afa31cffa7fa775a1720a79c171648b4fda48d9bbc124ac7894b501: Status 404 returned error can't find the container with id ff906f5b9afa31cffa7fa775a1720a79c171648b4fda48d9bbc124ac7894b501 Jul 21 15:03:30.979416 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:30.979380 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0f91a500-cd56-4cc9-8f38-d60a5e8a0562-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-6mtbk\" (UID: \"0f91a500-cd56-4cc9-8f38-d60a5e8a0562\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-6mtbk" Jul 21 15:03:30.979584 ip-10-0-135-176 kubenswrapper[2563]: E0721 15:03:30.979507 2563 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 21 15:03:30.979584 ip-10-0-135-176 kubenswrapper[2563]: E0721 15:03:30.979568 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f91a500-cd56-4cc9-8f38-d60a5e8a0562-cluster-monitoring-operator-tls podName:0f91a500-cd56-4cc9-8f38-d60a5e8a0562 nodeName:}" failed. No retries permitted until 2026-07-21 15:03:31.979552417 +0000 UTC m=+243.975337814 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/0f91a500-cd56-4cc9-8f38-d60a5e8a0562-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-6mtbk" (UID: "0f91a500-cd56-4cc9-8f38-d60a5e8a0562") : secret "cluster-monitoring-operator-tls" not found Jul 21 15:03:31.080551 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:31.080513 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5cec8b20-be0d-4f3d-a308-6d6056a96db7-service-ca-bundle\") pod \"router-default-fd497b998-txmbc\" (UID: \"5cec8b20-be0d-4f3d-a308-6d6056a96db7\") " pod="openshift-ingress/router-default-fd497b998-txmbc" Jul 21 15:03:31.080737 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:31.080568 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5cec8b20-be0d-4f3d-a308-6d6056a96db7-metrics-certs\") pod \"router-default-fd497b998-txmbc\" (UID: \"5cec8b20-be0d-4f3d-a308-6d6056a96db7\") " pod="openshift-ingress/router-default-fd497b998-txmbc" Jul 21 15:03:31.080737 ip-10-0-135-176 kubenswrapper[2563]: E0721 15:03:31.080696 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5cec8b20-be0d-4f3d-a308-6d6056a96db7-service-ca-bundle podName:5cec8b20-be0d-4f3d-a308-6d6056a96db7 nodeName:}" failed. No retries permitted until 2026-07-21 15:03:32.080677685 +0000 UTC m=+244.076463082 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5cec8b20-be0d-4f3d-a308-6d6056a96db7-service-ca-bundle") pod "router-default-fd497b998-txmbc" (UID: "5cec8b20-be0d-4f3d-a308-6d6056a96db7") : configmap references non-existent config key: service-ca.crt Jul 21 15:03:31.080737 ip-10-0-135-176 kubenswrapper[2563]: E0721 15:03:31.080696 2563 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 21 15:03:31.080905 ip-10-0-135-176 kubenswrapper[2563]: E0721 15:03:31.080741 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5cec8b20-be0d-4f3d-a308-6d6056a96db7-metrics-certs podName:5cec8b20-be0d-4f3d-a308-6d6056a96db7 nodeName:}" failed. No retries permitted until 2026-07-21 15:03:32.080734338 +0000 UTC m=+244.076519732 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5cec8b20-be0d-4f3d-a308-6d6056a96db7-metrics-certs") pod "router-default-fd497b998-txmbc" (UID: "5cec8b20-be0d-4f3d-a308-6d6056a96db7") : secret "router-metrics-certs-default" not found Jul 21 15:03:31.247419 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:31.247310 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-jd987" event={"ID":"feec54ec-7a62-4ad4-94bf-f28792b7dd4f","Type":"ContainerStarted","Data":"f82c42678f55882a629a35e4eae06e0266891901a6ce2a2887ebaa1d832a675e"} Jul 21 15:03:31.248699 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:31.248665 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-djs2t" event={"ID":"c1fb87fc-ef05-446d-bc6c-6059d487f2fe","Type":"ContainerStarted","Data":"2ec6417a663a8e784eb5597483790093f68f91a570416b3cf7bd9672010f70ab"} Jul 21 15:03:31.248838 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:31.248717 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-djs2t" event={"ID":"c1fb87fc-ef05-446d-bc6c-6059d487f2fe","Type":"ContainerStarted","Data":"ff906f5b9afa31cffa7fa775a1720a79c171648b4fda48d9bbc124ac7894b501"} Jul 21 15:03:31.249675 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:31.249645 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-bfv6n" event={"ID":"53f7dd9f-230e-4c5c-bc17-bd14332b1923","Type":"ContainerStarted","Data":"9553c418a7d16e0ca1b20d2bc34a5db3b4822e1a3a15294e4efd9a41423a9684"} Jul 21 15:03:31.266749 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:31.266691 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-djs2t" podStartSLOduration=1.266671213 podStartE2EDuration="1.266671213s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 15:03:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 15:03:31.265675349 +0000 UTC m=+243.261460791" watchObservedRunningTime="2026-07-21 15:03:31.266671213 +0000 UTC m=+243.262456634" Jul 21 15:03:31.989428 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:31.989361 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0f91a500-cd56-4cc9-8f38-d60a5e8a0562-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-6mtbk\" (UID: \"0f91a500-cd56-4cc9-8f38-d60a5e8a0562\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-6mtbk" Jul 21 15:03:31.989791 ip-10-0-135-176 kubenswrapper[2563]: E0721 15:03:31.989531 2563 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 21 15:03:31.989791 ip-10-0-135-176 kubenswrapper[2563]: E0721 15:03:31.989589 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f91a500-cd56-4cc9-8f38-d60a5e8a0562-cluster-monitoring-operator-tls podName:0f91a500-cd56-4cc9-8f38-d60a5e8a0562 nodeName:}" failed. No retries permitted until 2026-07-21 15:03:33.989573486 +0000 UTC m=+245.985358880 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/0f91a500-cd56-4cc9-8f38-d60a5e8a0562-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-6mtbk" (UID: "0f91a500-cd56-4cc9-8f38-d60a5e8a0562") : secret "cluster-monitoring-operator-tls" not found Jul 21 15:03:32.090345 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:32.090293 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5cec8b20-be0d-4f3d-a308-6d6056a96db7-service-ca-bundle\") pod \"router-default-fd497b998-txmbc\" (UID: \"5cec8b20-be0d-4f3d-a308-6d6056a96db7\") " pod="openshift-ingress/router-default-fd497b998-txmbc" Jul 21 15:03:32.090524 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:32.090378 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5cec8b20-be0d-4f3d-a308-6d6056a96db7-metrics-certs\") pod \"router-default-fd497b998-txmbc\" (UID: \"5cec8b20-be0d-4f3d-a308-6d6056a96db7\") " pod="openshift-ingress/router-default-fd497b998-txmbc" Jul 21 15:03:32.090587 ip-10-0-135-176 kubenswrapper[2563]: E0721 15:03:32.090576 2563 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 21 15:03:32.090659 ip-10-0-135-176 kubenswrapper[2563]: E0721 15:03:32.090639 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5cec8b20-be0d-4f3d-a308-6d6056a96db7-metrics-certs podName:5cec8b20-be0d-4f3d-a308-6d6056a96db7 nodeName:}" failed. No retries permitted until 2026-07-21 15:03:34.090620033 +0000 UTC m=+246.086405431 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5cec8b20-be0d-4f3d-a308-6d6056a96db7-metrics-certs") pod "router-default-fd497b998-txmbc" (UID: "5cec8b20-be0d-4f3d-a308-6d6056a96db7") : secret "router-metrics-certs-default" not found Jul 21 15:03:32.090803 ip-10-0-135-176 kubenswrapper[2563]: E0721 15:03:32.090771 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5cec8b20-be0d-4f3d-a308-6d6056a96db7-service-ca-bundle podName:5cec8b20-be0d-4f3d-a308-6d6056a96db7 nodeName:}" failed. No retries permitted until 2026-07-21 15:03:34.090750317 +0000 UTC m=+246.086535724 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5cec8b20-be0d-4f3d-a308-6d6056a96db7-service-ca-bundle") pod "router-default-fd497b998-txmbc" (UID: "5cec8b20-be0d-4f3d-a308-6d6056a96db7") : configmap references non-existent config key: service-ca.crt Jul 21 15:03:32.254976 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:32.254888 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-jd987" event={"ID":"feec54ec-7a62-4ad4-94bf-f28792b7dd4f","Type":"ContainerStarted","Data":"bad5c9edb52f8f0602b3b14466c907139712565c85c743d878f8561ae12baa43"} Jul 21 15:03:32.273227 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:32.273174 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-jd987" podStartSLOduration=1.123296075 podStartE2EDuration="2.273153277s" totalImagesPullingTime="1.149857202s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 15:03:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 15:03:32.271345829 +0000 UTC m=+244.267131270" watchObservedRunningTime="2026-07-21 15:03:32.273153277 +0000 UTC m=+244.268938694" Jul 21 15:03:34.006311 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:34.006267 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0f91a500-cd56-4cc9-8f38-d60a5e8a0562-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-6mtbk\" (UID: \"0f91a500-cd56-4cc9-8f38-d60a5e8a0562\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-6mtbk" Jul 21 15:03:34.006736 ip-10-0-135-176 kubenswrapper[2563]: E0721 15:03:34.006464 2563 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 21 15:03:34.006736 ip-10-0-135-176 kubenswrapper[2563]: E0721 15:03:34.006549 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f91a500-cd56-4cc9-8f38-d60a5e8a0562-cluster-monitoring-operator-tls podName:0f91a500-cd56-4cc9-8f38-d60a5e8a0562 nodeName:}" failed. No retries permitted until 2026-07-21 15:03:38.006526611 +0000 UTC m=+250.002312009 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/0f91a500-cd56-4cc9-8f38-d60a5e8a0562-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-6mtbk" (UID: "0f91a500-cd56-4cc9-8f38-d60a5e8a0562") : secret "cluster-monitoring-operator-tls" not found Jul 21 15:03:34.107029 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:34.106990 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5cec8b20-be0d-4f3d-a308-6d6056a96db7-service-ca-bundle\") pod \"router-default-fd497b998-txmbc\" (UID: \"5cec8b20-be0d-4f3d-a308-6d6056a96db7\") " pod="openshift-ingress/router-default-fd497b998-txmbc" Jul 21 15:03:34.107192 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:34.107039 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5cec8b20-be0d-4f3d-a308-6d6056a96db7-metrics-certs\") pod \"router-default-fd497b998-txmbc\" (UID: \"5cec8b20-be0d-4f3d-a308-6d6056a96db7\") " pod="openshift-ingress/router-default-fd497b998-txmbc" Jul 21 15:03:34.107192 ip-10-0-135-176 kubenswrapper[2563]: E0721 15:03:34.107151 2563 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 21 15:03:34.107262 ip-10-0-135-176 kubenswrapper[2563]: E0721 15:03:34.107195 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5cec8b20-be0d-4f3d-a308-6d6056a96db7-metrics-certs podName:5cec8b20-be0d-4f3d-a308-6d6056a96db7 nodeName:}" failed. No retries permitted until 2026-07-21 15:03:38.107182055 +0000 UTC m=+250.102967450 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5cec8b20-be0d-4f3d-a308-6d6056a96db7-metrics-certs") pod "router-default-fd497b998-txmbc" (UID: "5cec8b20-be0d-4f3d-a308-6d6056a96db7") : secret "router-metrics-certs-default" not found Jul 21 15:03:34.107262 ip-10-0-135-176 kubenswrapper[2563]: E0721 15:03:34.107207 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5cec8b20-be0d-4f3d-a308-6d6056a96db7-service-ca-bundle podName:5cec8b20-be0d-4f3d-a308-6d6056a96db7 nodeName:}" failed. No retries permitted until 2026-07-21 15:03:38.107201112 +0000 UTC m=+250.102986507 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5cec8b20-be0d-4f3d-a308-6d6056a96db7-service-ca-bundle") pod "router-default-fd497b998-txmbc" (UID: "5cec8b20-be0d-4f3d-a308-6d6056a96db7") : configmap references non-existent config key: service-ca.crt Jul 21 15:03:34.265301 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:34.265214 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-bfv6n" event={"ID":"53f7dd9f-230e-4c5c-bc17-bd14332b1923","Type":"ContainerStarted","Data":"663588196fab0b13cc95f8ed768135df3a69cda11d66636a77c3a89ac4a30d32"} Jul 21 15:03:34.284922 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:34.284876 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-bfv6n" podStartSLOduration=1.602883062 podStartE2EDuration="4.284861799s" totalImagesPullingTime="2.681978737s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 15:03:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 15:03:34.283085547 +0000 UTC m=+246.278870963" watchObservedRunningTime="2026-07-21 15:03:34.284861799 +0000 UTC m=+246.280647253" Jul 21 15:03:34.755843 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:34.755812 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-p45k9"] Jul 21 15:03:34.759180 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:34.759159 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-p45k9" Jul 21 15:03:34.761981 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:34.761959 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Jul 21 15:03:34.762101 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:34.762051 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Jul 21 15:03:34.762260 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:34.762247 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-qzdmw\"" Jul 21 15:03:34.769086 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:34.769063 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-p45k9"] Jul 21 15:03:34.813123 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:34.813087 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nr4bz\" (UniqueName: \"kubernetes.io/projected/5d344c8c-6b9f-44f4-a10a-d75f0d798087-kube-api-access-nr4bz\") pod \"migrator-7cc9c7ffd7-p45k9\" (UID: \"5d344c8c-6b9f-44f4-a10a-d75f0d798087\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-p45k9" Jul 21 15:03:34.913762 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:34.913724 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nr4bz\" (UniqueName: \"kubernetes.io/projected/5d344c8c-6b9f-44f4-a10a-d75f0d798087-kube-api-access-nr4bz\") pod \"migrator-7cc9c7ffd7-p45k9\" (UID: \"5d344c8c-6b9f-44f4-a10a-d75f0d798087\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-p45k9" Jul 21 15:03:34.922267 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:34.922240 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nr4bz\" (UniqueName: \"kubernetes.io/projected/5d344c8c-6b9f-44f4-a10a-d75f0d798087-kube-api-access-nr4bz\") pod \"migrator-7cc9c7ffd7-p45k9\" (UID: \"5d344c8c-6b9f-44f4-a10a-d75f0d798087\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-p45k9" Jul 21 15:03:35.068009 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:35.067918 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-p45k9" Jul 21 15:03:35.193268 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:35.193239 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-p45k9"] Jul 21 15:03:35.197906 ip-10-0-135-176 kubenswrapper[2563]: W0721 15:03:35.197881 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d344c8c_6b9f_44f4_a10a_d75f0d798087.slice/crio-fd77f1a79d23a6b97f71f28ca536e44ed377b430e453bb455f52c17862b97dd6 WatchSource:0}: Error finding container fd77f1a79d23a6b97f71f28ca536e44ed377b430e453bb455f52c17862b97dd6: Status 404 returned error can't find the container with id fd77f1a79d23a6b97f71f28ca536e44ed377b430e453bb455f52c17862b97dd6 Jul 21 15:03:35.269075 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:35.269040 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-p45k9" event={"ID":"5d344c8c-6b9f-44f4-a10a-d75f0d798087","Type":"ContainerStarted","Data":"fd77f1a79d23a6b97f71f28ca536e44ed377b430e453bb455f52c17862b97dd6"} Jul 21 15:03:36.690134 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:36.690084 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-gvst7"] Jul 21 15:03:36.693511 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:36.693487 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-gvst7" Jul 21 15:03:36.696936 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:36.696911 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 21 15:03:36.696936 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:36.696941 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 21 15:03:36.698041 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:36.698007 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 21 15:03:36.698041 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:36.698035 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 21 15:03:36.698240 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:36.698124 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-zxxxn\"" Jul 21 15:03:36.708246 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:36.708226 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-gvst7"] Jul 21 15:03:36.727515 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:36.727408 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8864333e-37a1-4558-ae06-751ef7d6a898-crio-socket\") pod \"insights-runtime-extractor-gvst7\" (UID: \"8864333e-37a1-4558-ae06-751ef7d6a898\") " pod="openshift-insights/insights-runtime-extractor-gvst7" Jul 21 15:03:36.727515 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:36.727449 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8864333e-37a1-4558-ae06-751ef7d6a898-data-volume\") pod \"insights-runtime-extractor-gvst7\" (UID: \"8864333e-37a1-4558-ae06-751ef7d6a898\") " pod="openshift-insights/insights-runtime-extractor-gvst7" Jul 21 15:03:36.727515 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:36.727492 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8864333e-37a1-4558-ae06-751ef7d6a898-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gvst7\" (UID: \"8864333e-37a1-4558-ae06-751ef7d6a898\") " pod="openshift-insights/insights-runtime-extractor-gvst7" Jul 21 15:03:36.727515 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:36.727515 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8864333e-37a1-4558-ae06-751ef7d6a898-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gvst7\" (UID: \"8864333e-37a1-4558-ae06-751ef7d6a898\") " pod="openshift-insights/insights-runtime-extractor-gvst7" Jul 21 15:03:36.727830 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:36.727605 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w86z9\" (UniqueName: \"kubernetes.io/projected/8864333e-37a1-4558-ae06-751ef7d6a898-kube-api-access-w86z9\") pod \"insights-runtime-extractor-gvst7\" (UID: \"8864333e-37a1-4558-ae06-751ef7d6a898\") " pod="openshift-insights/insights-runtime-extractor-gvst7" Jul 21 15:03:36.828198 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:36.828153 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8864333e-37a1-4558-ae06-751ef7d6a898-crio-socket\") pod \"insights-runtime-extractor-gvst7\" (UID: \"8864333e-37a1-4558-ae06-751ef7d6a898\") " pod="openshift-insights/insights-runtime-extractor-gvst7" Jul 21 15:03:36.828198 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:36.828200 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8864333e-37a1-4558-ae06-751ef7d6a898-data-volume\") pod \"insights-runtime-extractor-gvst7\" (UID: \"8864333e-37a1-4558-ae06-751ef7d6a898\") " pod="openshift-insights/insights-runtime-extractor-gvst7" Jul 21 15:03:36.828492 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:36.828245 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8864333e-37a1-4558-ae06-751ef7d6a898-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gvst7\" (UID: \"8864333e-37a1-4558-ae06-751ef7d6a898\") " pod="openshift-insights/insights-runtime-extractor-gvst7" Jul 21 15:03:36.828492 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:36.828270 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8864333e-37a1-4558-ae06-751ef7d6a898-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gvst7\" (UID: \"8864333e-37a1-4558-ae06-751ef7d6a898\") " pod="openshift-insights/insights-runtime-extractor-gvst7" Jul 21 15:03:36.828492 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:36.828307 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8864333e-37a1-4558-ae06-751ef7d6a898-crio-socket\") pod \"insights-runtime-extractor-gvst7\" (UID: \"8864333e-37a1-4558-ae06-751ef7d6a898\") " pod="openshift-insights/insights-runtime-extractor-gvst7" Jul 21 15:03:36.828492 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:36.828380 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w86z9\" (UniqueName: \"kubernetes.io/projected/8864333e-37a1-4558-ae06-751ef7d6a898-kube-api-access-w86z9\") pod \"insights-runtime-extractor-gvst7\" (UID: \"8864333e-37a1-4558-ae06-751ef7d6a898\") " pod="openshift-insights/insights-runtime-extractor-gvst7" Jul 21 15:03:36.828492 ip-10-0-135-176 kubenswrapper[2563]: E0721 15:03:36.828401 2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 21 15:03:36.828492 ip-10-0-135-176 kubenswrapper[2563]: E0721 15:03:36.828429 2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 21 15:03:36.828492 ip-10-0-135-176 kubenswrapper[2563]: E0721 15:03:36.828464 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8864333e-37a1-4558-ae06-751ef7d6a898-kube-rbac-proxy-cm podName:8864333e-37a1-4558-ae06-751ef7d6a898 nodeName:}" failed. No retries permitted until 2026-07-21 15:03:37.328444294 +0000 UTC m=+249.324229706 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8864333e-37a1-4558-ae06-751ef7d6a898-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gvst7" (UID: "8864333e-37a1-4558-ae06-751ef7d6a898") : configmap "kube-rbac-proxy" not found Jul 21 15:03:36.828492 ip-10-0-135-176 kubenswrapper[2563]: E0721 15:03:36.828484 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8864333e-37a1-4558-ae06-751ef7d6a898-insights-runtime-extractor-tls podName:8864333e-37a1-4558-ae06-751ef7d6a898 nodeName:}" failed. No retries permitted until 2026-07-21 15:03:37.328470621 +0000 UTC m=+249.324256016 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8864333e-37a1-4558-ae06-751ef7d6a898-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gvst7" (UID: "8864333e-37a1-4558-ae06-751ef7d6a898") : secret "insights-runtime-extractor-tls" not found Jul 21 15:03:36.828888 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:36.828646 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8864333e-37a1-4558-ae06-751ef7d6a898-data-volume\") pod \"insights-runtime-extractor-gvst7\" (UID: \"8864333e-37a1-4558-ae06-751ef7d6a898\") " pod="openshift-insights/insights-runtime-extractor-gvst7" Jul 21 15:03:36.836888 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:36.836865 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w86z9\" (UniqueName: \"kubernetes.io/projected/8864333e-37a1-4558-ae06-751ef7d6a898-kube-api-access-w86z9\") pod \"insights-runtime-extractor-gvst7\" (UID: \"8864333e-37a1-4558-ae06-751ef7d6a898\") " pod="openshift-insights/insights-runtime-extractor-gvst7" Jul 21 15:03:37.276515 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:37.276426 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-p45k9" event={"ID":"5d344c8c-6b9f-44f4-a10a-d75f0d798087","Type":"ContainerStarted","Data":"299124029c6cca0405e0536d98807ee442564a4186ed953acde5bc46b5ec449d"} Jul 21 15:03:37.276515 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:37.276505 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-p45k9" event={"ID":"5d344c8c-6b9f-44f4-a10a-d75f0d798087","Type":"ContainerStarted","Data":"1f20154ee35df17bf4ea7d94b61887a89d17603aa17b6470b6a93a1d1cf09ec4"} Jul 21 15:03:37.296563 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:37.296515 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-p45k9" podStartSLOduration=1.6060646840000001 podStartE2EDuration="3.296500078s" totalImagesPullingTime="1.690435394s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 15:03:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 15:03:37.29632082 +0000 UTC m=+249.292106237" watchObservedRunningTime="2026-07-21 15:03:37.296500078 +0000 UTC m=+249.292285494" Jul 21 15:03:37.331901 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:37.331861 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8864333e-37a1-4558-ae06-751ef7d6a898-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gvst7\" (UID: \"8864333e-37a1-4558-ae06-751ef7d6a898\") " pod="openshift-insights/insights-runtime-extractor-gvst7" Jul 21 15:03:37.332055 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:37.331904 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8864333e-37a1-4558-ae06-751ef7d6a898-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gvst7\" (UID: \"8864333e-37a1-4558-ae06-751ef7d6a898\") " pod="openshift-insights/insights-runtime-extractor-gvst7" Jul 21 15:03:37.332055 ip-10-0-135-176 kubenswrapper[2563]: E0721 15:03:37.331982 2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 21 15:03:37.332165 ip-10-0-135-176 kubenswrapper[2563]: E0721 15:03:37.332056 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8864333e-37a1-4558-ae06-751ef7d6a898-kube-rbac-proxy-cm podName:8864333e-37a1-4558-ae06-751ef7d6a898 nodeName:}" failed. No retries permitted until 2026-07-21 15:03:38.332036105 +0000 UTC m=+250.327821500 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8864333e-37a1-4558-ae06-751ef7d6a898-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gvst7" (UID: "8864333e-37a1-4558-ae06-751ef7d6a898") : configmap "kube-rbac-proxy" not found Jul 21 15:03:37.332165 ip-10-0-135-176 kubenswrapper[2563]: E0721 15:03:37.332084 2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 21 15:03:37.332165 ip-10-0-135-176 kubenswrapper[2563]: E0721 15:03:37.332132 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8864333e-37a1-4558-ae06-751ef7d6a898-insights-runtime-extractor-tls podName:8864333e-37a1-4558-ae06-751ef7d6a898 nodeName:}" failed. No retries permitted until 2026-07-21 15:03:38.33212039 +0000 UTC m=+250.327905787 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8864333e-37a1-4558-ae06-751ef7d6a898-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gvst7" (UID: "8864333e-37a1-4558-ae06-751ef7d6a898") : secret "insights-runtime-extractor-tls" not found Jul 21 15:03:38.036553 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:38.036515 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0f91a500-cd56-4cc9-8f38-d60a5e8a0562-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-6mtbk\" (UID: \"0f91a500-cd56-4cc9-8f38-d60a5e8a0562\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-6mtbk" Jul 21 15:03:38.036938 ip-10-0-135-176 kubenswrapper[2563]: E0721 15:03:38.036668 2563 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 21 15:03:38.036938 ip-10-0-135-176 kubenswrapper[2563]: E0721 15:03:38.036743 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f91a500-cd56-4cc9-8f38-d60a5e8a0562-cluster-monitoring-operator-tls podName:0f91a500-cd56-4cc9-8f38-d60a5e8a0562 nodeName:}" failed. No retries permitted until 2026-07-21 15:03:46.036723956 +0000 UTC m=+258.032509354 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/0f91a500-cd56-4cc9-8f38-d60a5e8a0562-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-6mtbk" (UID: "0f91a500-cd56-4cc9-8f38-d60a5e8a0562") : secret "cluster-monitoring-operator-tls" not found Jul 21 15:03:38.137439 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:38.137401 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5cec8b20-be0d-4f3d-a308-6d6056a96db7-service-ca-bundle\") pod \"router-default-fd497b998-txmbc\" (UID: \"5cec8b20-be0d-4f3d-a308-6d6056a96db7\") " pod="openshift-ingress/router-default-fd497b998-txmbc" Jul 21 15:03:38.137628 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:38.137475 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5cec8b20-be0d-4f3d-a308-6d6056a96db7-metrics-certs\") pod \"router-default-fd497b998-txmbc\" (UID: \"5cec8b20-be0d-4f3d-a308-6d6056a96db7\") " pod="openshift-ingress/router-default-fd497b998-txmbc" Jul 21 15:03:38.137628 ip-10-0-135-176 kubenswrapper[2563]: E0721 15:03:38.137583 2563 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 21 15:03:38.137628 ip-10-0-135-176 kubenswrapper[2563]: E0721 15:03:38.137585 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5cec8b20-be0d-4f3d-a308-6d6056a96db7-service-ca-bundle podName:5cec8b20-be0d-4f3d-a308-6d6056a96db7 nodeName:}" failed. No retries permitted until 2026-07-21 15:03:46.137565931 +0000 UTC m=+258.133351339 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5cec8b20-be0d-4f3d-a308-6d6056a96db7-service-ca-bundle") pod "router-default-fd497b998-txmbc" (UID: "5cec8b20-be0d-4f3d-a308-6d6056a96db7") : configmap references non-existent config key: service-ca.crt Jul 21 15:03:38.137765 ip-10-0-135-176 kubenswrapper[2563]: E0721 15:03:38.137648 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5cec8b20-be0d-4f3d-a308-6d6056a96db7-metrics-certs podName:5cec8b20-be0d-4f3d-a308-6d6056a96db7 nodeName:}" failed. No retries permitted until 2026-07-21 15:03:46.13763055 +0000 UTC m=+258.133415950 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5cec8b20-be0d-4f3d-a308-6d6056a96db7-metrics-certs") pod "router-default-fd497b998-txmbc" (UID: "5cec8b20-be0d-4f3d-a308-6d6056a96db7") : secret "router-metrics-certs-default" not found Jul 21 15:03:38.339381 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:38.339345 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8864333e-37a1-4558-ae06-751ef7d6a898-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gvst7\" (UID: \"8864333e-37a1-4558-ae06-751ef7d6a898\") " pod="openshift-insights/insights-runtime-extractor-gvst7" Jul 21 15:03:38.339568 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:38.339390 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8864333e-37a1-4558-ae06-751ef7d6a898-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gvst7\" (UID: \"8864333e-37a1-4558-ae06-751ef7d6a898\") " pod="openshift-insights/insights-runtime-extractor-gvst7" Jul 21 15:03:38.339568 ip-10-0-135-176 kubenswrapper[2563]: E0721 15:03:38.339425 2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 21 15:03:38.339568 ip-10-0-135-176 kubenswrapper[2563]: E0721 15:03:38.339489 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8864333e-37a1-4558-ae06-751ef7d6a898-kube-rbac-proxy-cm podName:8864333e-37a1-4558-ae06-751ef7d6a898 nodeName:}" failed. No retries permitted until 2026-07-21 15:03:40.33947369 +0000 UTC m=+252.335259084 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8864333e-37a1-4558-ae06-751ef7d6a898-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gvst7" (UID: "8864333e-37a1-4558-ae06-751ef7d6a898") : configmap "kube-rbac-proxy" not found Jul 21 15:03:38.339568 ip-10-0-135-176 kubenswrapper[2563]: E0721 15:03:38.339523 2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 21 15:03:38.339734 ip-10-0-135-176 kubenswrapper[2563]: E0721 15:03:38.339582 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8864333e-37a1-4558-ae06-751ef7d6a898-insights-runtime-extractor-tls podName:8864333e-37a1-4558-ae06-751ef7d6a898 nodeName:}" failed. No retries permitted until 2026-07-21 15:03:40.339565374 +0000 UTC m=+252.335350769 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8864333e-37a1-4558-ae06-751ef7d6a898-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gvst7" (UID: "8864333e-37a1-4558-ae06-751ef7d6a898") : secret "insights-runtime-extractor-tls" not found Jul 21 15:03:39.246708 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:39.246671 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26de6bb3-eb05-461a-958f-406464dea020-metrics-certs\") pod \"network-metrics-daemon-dh8jm\" (UID: \"26de6bb3-eb05-461a-958f-406464dea020\") " pod="openshift-multus/network-metrics-daemon-dh8jm" Jul 21 15:03:39.247074 ip-10-0-135-176 kubenswrapper[2563]: E0721 15:03:39.246822 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 21 15:03:39.247074 ip-10-0-135-176 kubenswrapper[2563]: E0721 15:03:39.246890 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26de6bb3-eb05-461a-958f-406464dea020-metrics-certs podName:26de6bb3-eb05-461a-958f-406464dea020 nodeName:}" failed. No retries permitted until 2026-07-21 15:05:41.246874874 +0000 UTC m=+373.242660269 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/26de6bb3-eb05-461a-958f-406464dea020-metrics-certs") pod "network-metrics-daemon-dh8jm" (UID: "26de6bb3-eb05-461a-958f-406464dea020") : secret "metrics-daemon-secret" not found Jul 21 15:03:39.274596 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:39.274567 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-h2kn2_42b48be4-5000-431e-9d8b-b932609faca0/dns-node-resolver/0.log" Jul 21 15:03:40.356205 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:40.356170 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8864333e-37a1-4558-ae06-751ef7d6a898-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gvst7\" (UID: \"8864333e-37a1-4558-ae06-751ef7d6a898\") " pod="openshift-insights/insights-runtime-extractor-gvst7" Jul 21 15:03:40.356205 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:40.356205 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8864333e-37a1-4558-ae06-751ef7d6a898-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gvst7\" (UID: \"8864333e-37a1-4558-ae06-751ef7d6a898\") " pod="openshift-insights/insights-runtime-extractor-gvst7" Jul 21 15:03:40.356646 ip-10-0-135-176 kubenswrapper[2563]: E0721 15:03:40.356294 2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 21 15:03:40.356646 ip-10-0-135-176 kubenswrapper[2563]: E0721 15:03:40.356375 2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 21 15:03:40.356646 ip-10-0-135-176 kubenswrapper[2563]: E0721 15:03:40.356395 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8864333e-37a1-4558-ae06-751ef7d6a898-kube-rbac-proxy-cm podName:8864333e-37a1-4558-ae06-751ef7d6a898 nodeName:}" failed. No retries permitted until 2026-07-21 15:03:44.356371115 +0000 UTC m=+256.352156510 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8864333e-37a1-4558-ae06-751ef7d6a898-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gvst7" (UID: "8864333e-37a1-4558-ae06-751ef7d6a898") : configmap "kube-rbac-proxy" not found Jul 21 15:03:40.356646 ip-10-0-135-176 kubenswrapper[2563]: E0721 15:03:40.356416 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8864333e-37a1-4558-ae06-751ef7d6a898-insights-runtime-extractor-tls podName:8864333e-37a1-4558-ae06-751ef7d6a898 nodeName:}" failed. No retries permitted until 2026-07-21 15:03:44.356404915 +0000 UTC m=+256.352190311 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8864333e-37a1-4558-ae06-751ef7d6a898-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gvst7" (UID: "8864333e-37a1-4558-ae06-751ef7d6a898") : secret "insights-runtime-extractor-tls" not found Jul 21 15:03:40.674826 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:40.674749 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-9qzhs_4d94d908-6ab5-4463-a878-0b340bb80365/node-ca/0.log" Jul 21 15:03:41.475362 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:41.475310 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-vnx89_9d1e2d09-f1a3-4fee-a2c3-86647059afe5/serve-healthcheck-canary/0.log" Jul 21 15:03:41.675477 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:41.675444 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7cc9c7ffd7-p45k9_5d344c8c-6b9f-44f4-a10a-d75f0d798087/migrator/0.log" Jul 21 15:03:41.875233 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:41.875204 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7cc9c7ffd7-p45k9_5d344c8c-6b9f-44f4-a10a-d75f0d798087/graceful-termination/0.log" Jul 21 15:03:42.076633 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:42.076592 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7658d9b749-bfv6n_53f7dd9f-230e-4c5c-bc17-bd14332b1923/kube-storage-version-migrator-operator/0.log" Jul 21 15:03:44.393029 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:44.392976 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8864333e-37a1-4558-ae06-751ef7d6a898-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gvst7\" (UID: \"8864333e-37a1-4558-ae06-751ef7d6a898\") " pod="openshift-insights/insights-runtime-extractor-gvst7" Jul 21 15:03:44.393029 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:44.393026 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8864333e-37a1-4558-ae06-751ef7d6a898-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gvst7\" (UID: \"8864333e-37a1-4558-ae06-751ef7d6a898\") " pod="openshift-insights/insights-runtime-extractor-gvst7" Jul 21 15:03:44.393543 ip-10-0-135-176 kubenswrapper[2563]: E0721 15:03:44.393106 2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 21 15:03:44.393543 ip-10-0-135-176 kubenswrapper[2563]: E0721 15:03:44.393173 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8864333e-37a1-4558-ae06-751ef7d6a898-kube-rbac-proxy-cm podName:8864333e-37a1-4558-ae06-751ef7d6a898 nodeName:}" failed. No retries permitted until 2026-07-21 15:03:52.393157741 +0000 UTC m=+264.388943135 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8864333e-37a1-4558-ae06-751ef7d6a898-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gvst7" (UID: "8864333e-37a1-4558-ae06-751ef7d6a898") : configmap "kube-rbac-proxy" not found Jul 21 15:03:44.393543 ip-10-0-135-176 kubenswrapper[2563]: E0721 15:03:44.393222 2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 21 15:03:44.393543 ip-10-0-135-176 kubenswrapper[2563]: E0721 15:03:44.393290 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8864333e-37a1-4558-ae06-751ef7d6a898-insights-runtime-extractor-tls podName:8864333e-37a1-4558-ae06-751ef7d6a898 nodeName:}" failed. No retries permitted until 2026-07-21 15:03:52.393271627 +0000 UTC m=+264.389057039 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8864333e-37a1-4558-ae06-751ef7d6a898-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gvst7" (UID: "8864333e-37a1-4558-ae06-751ef7d6a898") : secret "insights-runtime-extractor-tls" not found Jul 21 15:03:46.109289 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:46.109250 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0f91a500-cd56-4cc9-8f38-d60a5e8a0562-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-6mtbk\" (UID: \"0f91a500-cd56-4cc9-8f38-d60a5e8a0562\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-6mtbk" Jul 21 15:03:46.111834 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:46.111815 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0f91a500-cd56-4cc9-8f38-d60a5e8a0562-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-6mtbk\" (UID: \"0f91a500-cd56-4cc9-8f38-d60a5e8a0562\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-6mtbk" Jul 21 15:03:46.209697 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:46.209653 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5cec8b20-be0d-4f3d-a308-6d6056a96db7-service-ca-bundle\") pod \"router-default-fd497b998-txmbc\" (UID: \"5cec8b20-be0d-4f3d-a308-6d6056a96db7\") " pod="openshift-ingress/router-default-fd497b998-txmbc" Jul 21 15:03:46.209868 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:46.209727 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5cec8b20-be0d-4f3d-a308-6d6056a96db7-metrics-certs\") pod \"router-default-fd497b998-txmbc\" (UID: \"5cec8b20-be0d-4f3d-a308-6d6056a96db7\") " pod="openshift-ingress/router-default-fd497b998-txmbc" Jul 21 15:03:46.210269 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:46.210248 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5cec8b20-be0d-4f3d-a308-6d6056a96db7-service-ca-bundle\") pod \"router-default-fd497b998-txmbc\" (UID: \"5cec8b20-be0d-4f3d-a308-6d6056a96db7\") " pod="openshift-ingress/router-default-fd497b998-txmbc" Jul 21 15:03:46.212131 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:46.212111 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5cec8b20-be0d-4f3d-a308-6d6056a96db7-metrics-certs\") pod \"router-default-fd497b998-txmbc\" (UID: \"5cec8b20-be0d-4f3d-a308-6d6056a96db7\") " pod="openshift-ingress/router-default-fd497b998-txmbc" Jul 21 15:03:46.252108 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:46.252071 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-6mtbk" Jul 21 15:03:46.345785 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:46.345743 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-fd497b998-txmbc" Jul 21 15:03:46.377047 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:46.377017 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-76cf8966f8-6mtbk"] Jul 21 15:03:46.379438 ip-10-0-135-176 kubenswrapper[2563]: W0721 15:03:46.379407 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f91a500_cd56_4cc9_8f38_d60a5e8a0562.slice/crio-0fa122d267bd13f40cc58725cdc35671825307f6dd892b6fe90be3e068d5f274 WatchSource:0}: Error finding container 0fa122d267bd13f40cc58725cdc35671825307f6dd892b6fe90be3e068d5f274: Status 404 returned error can't find the container with id 0fa122d267bd13f40cc58725cdc35671825307f6dd892b6fe90be3e068d5f274 Jul 21 15:03:46.477877 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:46.477859 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-fd497b998-txmbc"] Jul 21 15:03:46.479777 ip-10-0-135-176 kubenswrapper[2563]: W0721 15:03:46.479750 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cec8b20_be0d_4f3d_a308_6d6056a96db7.slice/crio-df47232060b9a472870b01cd6c6a04ee133215f7ff2685366268ad42168f213b WatchSource:0}: Error finding container df47232060b9a472870b01cd6c6a04ee133215f7ff2685366268ad42168f213b: Status 404 returned error can't find the container with id df47232060b9a472870b01cd6c6a04ee133215f7ff2685366268ad42168f213b Jul 21 15:03:47.308313 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:47.308277 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-6mtbk" event={"ID":"0f91a500-cd56-4cc9-8f38-d60a5e8a0562","Type":"ContainerStarted","Data":"0fa122d267bd13f40cc58725cdc35671825307f6dd892b6fe90be3e068d5f274"} Jul 21 15:03:47.309672 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:47.309652 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-fd497b998-txmbc" event={"ID":"5cec8b20-be0d-4f3d-a308-6d6056a96db7","Type":"ContainerStarted","Data":"866d7df5b04ecbf2885e791d5d8841c70ce0eea81d1ed39af100a41aa910bd70"} Jul 21 15:03:47.309779 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:47.309689 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-fd497b998-txmbc" event={"ID":"5cec8b20-be0d-4f3d-a308-6d6056a96db7","Type":"ContainerStarted","Data":"df47232060b9a472870b01cd6c6a04ee133215f7ff2685366268ad42168f213b"} Jul 21 15:03:47.332378 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:47.332311 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-fd497b998-txmbc" podStartSLOduration=17.332297369 podStartE2EDuration="17.332297369s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 15:03:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 15:03:47.330440164 +0000 UTC m=+259.326225581" watchObservedRunningTime="2026-07-21 15:03:47.332297369 +0000 UTC m=+259.328082786" Jul 21 15:03:47.345907 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:47.345882 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-fd497b998-txmbc" Jul 21 15:03:47.348536 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:47.348516 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-fd497b998-txmbc" Jul 21 15:03:48.313795 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:48.313710 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-fd497b998-txmbc" Jul 21 15:03:48.315220 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:48.315198 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-fd497b998-txmbc" Jul 21 15:03:49.317855 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:49.317775 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-6mtbk" event={"ID":"0f91a500-cd56-4cc9-8f38-d60a5e8a0562","Type":"ContainerStarted","Data":"351aad71c7a8135c16dac758fdfc335b160b21f6b6cea3991cd26f0a72f8da5f"} Jul 21 15:03:49.335026 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:49.334978 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-6mtbk" podStartSLOduration=17.220255656 podStartE2EDuration="19.33496366s" totalImagesPullingTime="2.114708004s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 15:03:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 15:03:49.334663558 +0000 UTC m=+261.330448975" watchObservedRunningTime="2026-07-21 15:03:49.33496366 +0000 UTC m=+261.330749099" Jul 21 15:03:52.456194 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:52.456133 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8864333e-37a1-4558-ae06-751ef7d6a898-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gvst7\" (UID: \"8864333e-37a1-4558-ae06-751ef7d6a898\") " pod="openshift-insights/insights-runtime-extractor-gvst7" Jul 21 15:03:52.456194 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:52.456193 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8864333e-37a1-4558-ae06-751ef7d6a898-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gvst7\" (UID: \"8864333e-37a1-4558-ae06-751ef7d6a898\") " pod="openshift-insights/insights-runtime-extractor-gvst7" Jul 21 15:03:52.456848 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:52.456819 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8864333e-37a1-4558-ae06-751ef7d6a898-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gvst7\" (UID: \"8864333e-37a1-4558-ae06-751ef7d6a898\") " pod="openshift-insights/insights-runtime-extractor-gvst7" Jul 21 15:03:52.458621 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:52.458603 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8864333e-37a1-4558-ae06-751ef7d6a898-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gvst7\" (UID: \"8864333e-37a1-4558-ae06-751ef7d6a898\") " pod="openshift-insights/insights-runtime-extractor-gvst7" Jul 21 15:03:52.603680 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:52.603630 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-gvst7" Jul 21 15:03:52.734301 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:52.734274 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-gvst7"] Jul 21 15:03:52.736839 ip-10-0-135-176 kubenswrapper[2563]: W0721 15:03:52.736813 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8864333e_37a1_4558_ae06_751ef7d6a898.slice/crio-5b63036e87a01a44d3cdce4ada08b3dbd02ed694953d078290fa97e66c205e76 WatchSource:0}: Error finding container 5b63036e87a01a44d3cdce4ada08b3dbd02ed694953d078290fa97e66c205e76: Status 404 returned error can't find the container with id 5b63036e87a01a44d3cdce4ada08b3dbd02ed694953d078290fa97e66c205e76 Jul 21 15:03:53.331511 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:53.331475 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gvst7" event={"ID":"8864333e-37a1-4558-ae06-751ef7d6a898","Type":"ContainerStarted","Data":"5b63036e87a01a44d3cdce4ada08b3dbd02ed694953d078290fa97e66c205e76"} Jul 21 15:03:58.351128 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:03:58.351026 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gvst7" event={"ID":"8864333e-37a1-4558-ae06-751ef7d6a898","Type":"ContainerStarted","Data":"3f8ac9ea49fd506e49659abb35acd84564b36da125e591c2b1706d1fef0c7079"} Jul 21 15:04:02.366258 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:02.366215 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gvst7" event={"ID":"8864333e-37a1-4558-ae06-751ef7d6a898","Type":"ContainerStarted","Data":"2d6a4bf6ec4ccde58f67e1f8ba940d62f531df2ad8b6c3d989b39276c4069f2d"} Jul 21 15:04:05.377983 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:05.377941 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gvst7" event={"ID":"8864333e-37a1-4558-ae06-751ef7d6a898","Type":"ContainerStarted","Data":"6369627a989cc2e1370f6eee0b7f3b60ab4a863d9d7ea68ffcc536e02c67f877"} Jul 21 15:04:05.401441 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:05.401384 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-gvst7" podStartSLOduration=17.010171066 podStartE2EDuration="29.401368489s" totalImagesPullingTime="12.391197423s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 15:03:36 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 15:04:05.39916654 +0000 UTC m=+277.394951983" watchObservedRunningTime="2026-07-21 15:04:05.401368489 +0000 UTC m=+277.397153905" Jul 21 15:04:10.919604 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:10.919570 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-d5d6c8b9-z5c2j"] Jul 21 15:04:10.924021 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:10.924003 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d5d6c8b9-z5c2j" Jul 21 15:04:10.928596 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:10.928558 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 21 15:04:10.928596 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:10.928593 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 21 15:04:10.928799 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:10.928605 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 21 15:04:10.928799 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:10.928627 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 21 15:04:10.928799 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:10.928646 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 21 15:04:10.928799 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:10.928593 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 21 15:04:10.928799 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:10.928567 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-q9zfc\"" Jul 21 15:04:10.929018 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:10.928807 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 21 15:04:10.934545 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:10.934521 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 21 15:04:10.935023 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:10.935003 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-d5d6c8b9-z5c2j"] Jul 21 15:04:11.053737 ip-10-0-135-176 kubenswrapper[2563]: E0721 15:04:11.053686 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-dpzts" podUID="28687086-f5b5-4889-afdb-8145abba8b1a" Jul 21 15:04:11.115110 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:11.115060 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msqlm\" (UniqueName: \"kubernetes.io/projected/a8dde5b2-9cc3-4592-ae2d-3f558c970e8a-kube-api-access-msqlm\") pod \"console-d5d6c8b9-z5c2j\" (UID: \"a8dde5b2-9cc3-4592-ae2d-3f558c970e8a\") " pod="openshift-console/console-d5d6c8b9-z5c2j" Jul 21 15:04:11.115271 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:11.115150 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a8dde5b2-9cc3-4592-ae2d-3f558c970e8a-console-serving-cert\") pod \"console-d5d6c8b9-z5c2j\" (UID: \"a8dde5b2-9cc3-4592-ae2d-3f558c970e8a\") " pod="openshift-console/console-d5d6c8b9-z5c2j" Jul 21 15:04:11.115271 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:11.115207 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a8dde5b2-9cc3-4592-ae2d-3f558c970e8a-service-ca\") pod \"console-d5d6c8b9-z5c2j\" (UID: \"a8dde5b2-9cc3-4592-ae2d-3f558c970e8a\") " pod="openshift-console/console-d5d6c8b9-z5c2j" Jul 21 15:04:11.115271 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:11.115255 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a8dde5b2-9cc3-4592-ae2d-3f558c970e8a-oauth-serving-cert\") pod \"console-d5d6c8b9-z5c2j\" (UID: \"a8dde5b2-9cc3-4592-ae2d-3f558c970e8a\") " pod="openshift-console/console-d5d6c8b9-z5c2j" Jul 21 15:04:11.115420 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:11.115307 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a8dde5b2-9cc3-4592-ae2d-3f558c970e8a-console-oauth-config\") pod \"console-d5d6c8b9-z5c2j\" (UID: \"a8dde5b2-9cc3-4592-ae2d-3f558c970e8a\") " pod="openshift-console/console-d5d6c8b9-z5c2j" Jul 21 15:04:11.115420 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:11.115378 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a8dde5b2-9cc3-4592-ae2d-3f558c970e8a-console-config\") pod \"console-d5d6c8b9-z5c2j\" (UID: \"a8dde5b2-9cc3-4592-ae2d-3f558c970e8a\") " pod="openshift-console/console-d5d6c8b9-z5c2j" Jul 21 15:04:11.115420 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:11.115396 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a8dde5b2-9cc3-4592-ae2d-3f558c970e8a-trusted-ca-bundle\") pod \"console-d5d6c8b9-z5c2j\" (UID: \"a8dde5b2-9cc3-4592-ae2d-3f558c970e8a\") " pod="openshift-console/console-d5d6c8b9-z5c2j" Jul 21 15:04:11.216526 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:11.216418 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a8dde5b2-9cc3-4592-ae2d-3f558c970e8a-console-serving-cert\") pod \"console-d5d6c8b9-z5c2j\" (UID: \"a8dde5b2-9cc3-4592-ae2d-3f558c970e8a\") " pod="openshift-console/console-d5d6c8b9-z5c2j" Jul 21 15:04:11.216526 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:11.216483 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a8dde5b2-9cc3-4592-ae2d-3f558c970e8a-service-ca\") pod \"console-d5d6c8b9-z5c2j\" (UID: \"a8dde5b2-9cc3-4592-ae2d-3f558c970e8a\") " pod="openshift-console/console-d5d6c8b9-z5c2j" Jul 21 15:04:11.216526 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:11.216529 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a8dde5b2-9cc3-4592-ae2d-3f558c970e8a-oauth-serving-cert\") pod \"console-d5d6c8b9-z5c2j\" (UID: \"a8dde5b2-9cc3-4592-ae2d-3f558c970e8a\") " pod="openshift-console/console-d5d6c8b9-z5c2j" Jul 21 15:04:11.216757 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:11.216572 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a8dde5b2-9cc3-4592-ae2d-3f558c970e8a-console-oauth-config\") pod \"console-d5d6c8b9-z5c2j\" (UID: \"a8dde5b2-9cc3-4592-ae2d-3f558c970e8a\") " pod="openshift-console/console-d5d6c8b9-z5c2j" Jul 21 15:04:11.216757 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:11.216682 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a8dde5b2-9cc3-4592-ae2d-3f558c970e8a-console-config\") pod \"console-d5d6c8b9-z5c2j\" (UID: \"a8dde5b2-9cc3-4592-ae2d-3f558c970e8a\") " pod="openshift-console/console-d5d6c8b9-z5c2j" Jul 21 15:04:11.216757 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:11.216740 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a8dde5b2-9cc3-4592-ae2d-3f558c970e8a-trusted-ca-bundle\") pod \"console-d5d6c8b9-z5c2j\" (UID: \"a8dde5b2-9cc3-4592-ae2d-3f558c970e8a\") " pod="openshift-console/console-d5d6c8b9-z5c2j" Jul 21 15:04:11.216902 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:11.216772 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msqlm\" (UniqueName: \"kubernetes.io/projected/a8dde5b2-9cc3-4592-ae2d-3f558c970e8a-kube-api-access-msqlm\") pod \"console-d5d6c8b9-z5c2j\" (UID: \"a8dde5b2-9cc3-4592-ae2d-3f558c970e8a\") " pod="openshift-console/console-d5d6c8b9-z5c2j" Jul 21 15:04:11.217469 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:11.217444 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a8dde5b2-9cc3-4592-ae2d-3f558c970e8a-oauth-serving-cert\") pod \"console-d5d6c8b9-z5c2j\" (UID: \"a8dde5b2-9cc3-4592-ae2d-3f558c970e8a\") " pod="openshift-console/console-d5d6c8b9-z5c2j" Jul 21 15:04:11.217603 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:11.217442 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a8dde5b2-9cc3-4592-ae2d-3f558c970e8a-service-ca\") pod \"console-d5d6c8b9-z5c2j\" (UID: \"a8dde5b2-9cc3-4592-ae2d-3f558c970e8a\") " pod="openshift-console/console-d5d6c8b9-z5c2j" Jul 21 15:04:11.217698 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:11.217675 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a8dde5b2-9cc3-4592-ae2d-3f558c970e8a-console-config\") pod \"console-d5d6c8b9-z5c2j\" (UID: \"a8dde5b2-9cc3-4592-ae2d-3f558c970e8a\") " pod="openshift-console/console-d5d6c8b9-z5c2j" Jul 21 15:04:11.217771 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:11.217748 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a8dde5b2-9cc3-4592-ae2d-3f558c970e8a-trusted-ca-bundle\") pod \"console-d5d6c8b9-z5c2j\" (UID: \"a8dde5b2-9cc3-4592-ae2d-3f558c970e8a\") " pod="openshift-console/console-d5d6c8b9-z5c2j" Jul 21 15:04:11.219037 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:11.219016 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a8dde5b2-9cc3-4592-ae2d-3f558c970e8a-console-oauth-config\") pod \"console-d5d6c8b9-z5c2j\" (UID: \"a8dde5b2-9cc3-4592-ae2d-3f558c970e8a\") " pod="openshift-console/console-d5d6c8b9-z5c2j" Jul 21 15:04:11.219111 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:11.219083 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a8dde5b2-9cc3-4592-ae2d-3f558c970e8a-console-serving-cert\") pod \"console-d5d6c8b9-z5c2j\" (UID: \"a8dde5b2-9cc3-4592-ae2d-3f558c970e8a\") " pod="openshift-console/console-d5d6c8b9-z5c2j" Jul 21 15:04:11.225690 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:11.225670 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-msqlm\" (UniqueName: \"kubernetes.io/projected/a8dde5b2-9cc3-4592-ae2d-3f558c970e8a-kube-api-access-msqlm\") pod \"console-d5d6c8b9-z5c2j\" (UID: \"a8dde5b2-9cc3-4592-ae2d-3f558c970e8a\") " pod="openshift-console/console-d5d6c8b9-z5c2j" Jul 21 15:04:11.233453 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:11.233434 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d5d6c8b9-z5c2j" Jul 21 15:04:11.357006 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:11.356972 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-d5d6c8b9-z5c2j"] Jul 21 15:04:11.359616 ip-10-0-135-176 kubenswrapper[2563]: W0721 15:04:11.359590 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8dde5b2_9cc3_4592_ae2d_3f558c970e8a.slice/crio-f60b1cf8ec1c9438e174282dfba03c805b07d5e068f5f591960eec46668c8d32 WatchSource:0}: Error finding container f60b1cf8ec1c9438e174282dfba03c805b07d5e068f5f591960eec46668c8d32: Status 404 returned error can't find the container with id f60b1cf8ec1c9438e174282dfba03c805b07d5e068f5f591960eec46668c8d32 Jul 21 15:04:11.397677 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:11.397649 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-dpzts" Jul 21 15:04:11.397677 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:11.397668 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d5d6c8b9-z5c2j" event={"ID":"a8dde5b2-9cc3-4592-ae2d-3f558c970e8a","Type":"ContainerStarted","Data":"f60b1cf8ec1c9438e174282dfba03c805b07d5e068f5f591960eec46668c8d32"} Jul 21 15:04:12.444273 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:12.444236 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-49gbt"] Jul 21 15:04:12.447867 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:12.447846 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-49gbt" Jul 21 15:04:12.451465 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:12.451437 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 21 15:04:12.451619 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:12.451596 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 21 15:04:12.451820 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:12.451801 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-5j684\"" Jul 21 15:04:12.452852 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:12.452835 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 21 15:04:12.452942 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:12.452899 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 21 15:04:12.628946 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:12.628916 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqgkf\" (UniqueName: \"kubernetes.io/projected/db2b1697-e840-48cf-bd68-ac5c48dca546-kube-api-access-sqgkf\") pod \"node-exporter-49gbt\" (UID: \"db2b1697-e840-48cf-bd68-ac5c48dca546\") " pod="openshift-monitoring/node-exporter-49gbt" Jul 21 15:04:12.629124 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:12.628967 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/db2b1697-e840-48cf-bd68-ac5c48dca546-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-49gbt\" (UID: \"db2b1697-e840-48cf-bd68-ac5c48dca546\") " pod="openshift-monitoring/node-exporter-49gbt" Jul 21 15:04:12.629124 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:12.629008 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/db2b1697-e840-48cf-bd68-ac5c48dca546-node-exporter-wtmp\") pod \"node-exporter-49gbt\" (UID: \"db2b1697-e840-48cf-bd68-ac5c48dca546\") " pod="openshift-monitoring/node-exporter-49gbt" Jul 21 15:04:12.629124 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:12.629081 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/db2b1697-e840-48cf-bd68-ac5c48dca546-node-exporter-accelerators-collector-config\") pod \"node-exporter-49gbt\" (UID: \"db2b1697-e840-48cf-bd68-ac5c48dca546\") " pod="openshift-monitoring/node-exporter-49gbt" Jul 21 15:04:12.629270 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:12.629134 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/db2b1697-e840-48cf-bd68-ac5c48dca546-node-exporter-tls\") pod \"node-exporter-49gbt\" (UID: \"db2b1697-e840-48cf-bd68-ac5c48dca546\") " pod="openshift-monitoring/node-exporter-49gbt" Jul 21 15:04:12.629270 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:12.629166 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/db2b1697-e840-48cf-bd68-ac5c48dca546-root\") pod \"node-exporter-49gbt\" (UID: \"db2b1697-e840-48cf-bd68-ac5c48dca546\") " pod="openshift-monitoring/node-exporter-49gbt" Jul 21 15:04:12.629270 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:12.629193 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/db2b1697-e840-48cf-bd68-ac5c48dca546-node-exporter-textfile\") pod \"node-exporter-49gbt\" (UID: \"db2b1697-e840-48cf-bd68-ac5c48dca546\") " pod="openshift-monitoring/node-exporter-49gbt" Jul 21 15:04:12.629414 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:12.629307 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/db2b1697-e840-48cf-bd68-ac5c48dca546-metrics-client-ca\") pod \"node-exporter-49gbt\" (UID: \"db2b1697-e840-48cf-bd68-ac5c48dca546\") " pod="openshift-monitoring/node-exporter-49gbt" Jul 21 15:04:12.629414 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:12.629371 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/db2b1697-e840-48cf-bd68-ac5c48dca546-sys\") pod \"node-exporter-49gbt\" (UID: \"db2b1697-e840-48cf-bd68-ac5c48dca546\") " pod="openshift-monitoring/node-exporter-49gbt" Jul 21 15:04:12.729880 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:12.729799 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqgkf\" (UniqueName: \"kubernetes.io/projected/db2b1697-e840-48cf-bd68-ac5c48dca546-kube-api-access-sqgkf\") pod \"node-exporter-49gbt\" (UID: \"db2b1697-e840-48cf-bd68-ac5c48dca546\") " pod="openshift-monitoring/node-exporter-49gbt" Jul 21 15:04:12.729880 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:12.729857 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/db2b1697-e840-48cf-bd68-ac5c48dca546-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-49gbt\" (UID: \"db2b1697-e840-48cf-bd68-ac5c48dca546\") " pod="openshift-monitoring/node-exporter-49gbt" Jul 21 15:04:12.730099 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:12.729909 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/db2b1697-e840-48cf-bd68-ac5c48dca546-node-exporter-wtmp\") pod \"node-exporter-49gbt\" (UID: \"db2b1697-e840-48cf-bd68-ac5c48dca546\") " pod="openshift-monitoring/node-exporter-49gbt" Jul 21 15:04:12.730099 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:12.729942 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/db2b1697-e840-48cf-bd68-ac5c48dca546-node-exporter-accelerators-collector-config\") pod \"node-exporter-49gbt\" (UID: \"db2b1697-e840-48cf-bd68-ac5c48dca546\") " pod="openshift-monitoring/node-exporter-49gbt" Jul 21 15:04:12.730099 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:12.729978 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/db2b1697-e840-48cf-bd68-ac5c48dca546-node-exporter-tls\") pod \"node-exporter-49gbt\" (UID: \"db2b1697-e840-48cf-bd68-ac5c48dca546\") " pod="openshift-monitoring/node-exporter-49gbt" Jul 21 15:04:12.730099 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:12.730009 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/db2b1697-e840-48cf-bd68-ac5c48dca546-root\") pod \"node-exporter-49gbt\" (UID: \"db2b1697-e840-48cf-bd68-ac5c48dca546\") " pod="openshift-monitoring/node-exporter-49gbt" Jul 21 15:04:12.730099 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:12.730034 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/db2b1697-e840-48cf-bd68-ac5c48dca546-node-exporter-textfile\") pod \"node-exporter-49gbt\" (UID: \"db2b1697-e840-48cf-bd68-ac5c48dca546\") " pod="openshift-monitoring/node-exporter-49gbt" Jul 21 15:04:12.730099 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:12.730099 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/db2b1697-e840-48cf-bd68-ac5c48dca546-metrics-client-ca\") pod \"node-exporter-49gbt\" (UID: \"db2b1697-e840-48cf-bd68-ac5c48dca546\") " pod="openshift-monitoring/node-exporter-49gbt" Jul 21 15:04:12.730403 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:12.730124 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/db2b1697-e840-48cf-bd68-ac5c48dca546-sys\") pod \"node-exporter-49gbt\" (UID: \"db2b1697-e840-48cf-bd68-ac5c48dca546\") " pod="openshift-monitoring/node-exporter-49gbt" Jul 21 15:04:12.730403 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:12.730220 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/db2b1697-e840-48cf-bd68-ac5c48dca546-root\") pod \"node-exporter-49gbt\" (UID: \"db2b1697-e840-48cf-bd68-ac5c48dca546\") " pod="openshift-monitoring/node-exporter-49gbt" Jul 21 15:04:12.730403 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:12.730382 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/db2b1697-e840-48cf-bd68-ac5c48dca546-node-exporter-wtmp\") pod \"node-exporter-49gbt\" (UID: \"db2b1697-e840-48cf-bd68-ac5c48dca546\") " pod="openshift-monitoring/node-exporter-49gbt" Jul 21 15:04:12.730552 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:12.730427 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/db2b1697-e840-48cf-bd68-ac5c48dca546-sys\") pod \"node-exporter-49gbt\" (UID: \"db2b1697-e840-48cf-bd68-ac5c48dca546\") " pod="openshift-monitoring/node-exporter-49gbt" Jul 21 15:04:12.730552 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:12.730454 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/db2b1697-e840-48cf-bd68-ac5c48dca546-node-exporter-textfile\") pod \"node-exporter-49gbt\" (UID: \"db2b1697-e840-48cf-bd68-ac5c48dca546\") " pod="openshift-monitoring/node-exporter-49gbt" Jul 21 15:04:12.730701 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:12.730679 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/db2b1697-e840-48cf-bd68-ac5c48dca546-node-exporter-accelerators-collector-config\") pod \"node-exporter-49gbt\" (UID: \"db2b1697-e840-48cf-bd68-ac5c48dca546\") " pod="openshift-monitoring/node-exporter-49gbt" Jul 21 15:04:12.731010 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:12.730932 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/db2b1697-e840-48cf-bd68-ac5c48dca546-metrics-client-ca\") pod \"node-exporter-49gbt\" (UID: \"db2b1697-e840-48cf-bd68-ac5c48dca546\") " pod="openshift-monitoring/node-exporter-49gbt" Jul 21 15:04:12.732957 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:12.732905 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/db2b1697-e840-48cf-bd68-ac5c48dca546-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-49gbt\" (UID: \"db2b1697-e840-48cf-bd68-ac5c48dca546\") " pod="openshift-monitoring/node-exporter-49gbt" Jul 21 15:04:12.733369 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:12.733351 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/db2b1697-e840-48cf-bd68-ac5c48dca546-node-exporter-tls\") pod \"node-exporter-49gbt\" (UID: \"db2b1697-e840-48cf-bd68-ac5c48dca546\") " pod="openshift-monitoring/node-exporter-49gbt" Jul 21 15:04:12.747157 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:12.747133 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqgkf\" (UniqueName: \"kubernetes.io/projected/db2b1697-e840-48cf-bd68-ac5c48dca546-kube-api-access-sqgkf\") pod \"node-exporter-49gbt\" (UID: \"db2b1697-e840-48cf-bd68-ac5c48dca546\") " pod="openshift-monitoring/node-exporter-49gbt" Jul 21 15:04:12.759165 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:12.759143 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-49gbt" Jul 21 15:04:12.771464 ip-10-0-135-176 kubenswrapper[2563]: W0721 15:04:12.771438 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb2b1697_e840_48cf_bd68_ac5c48dca546.slice/crio-01856b3585cb75b5d7b4651f9666f13c20e87842f0c44b93b21a2e83a96a9168 WatchSource:0}: Error finding container 01856b3585cb75b5d7b4651f9666f13c20e87842f0c44b93b21a2e83a96a9168: Status 404 returned error can't find the container with id 01856b3585cb75b5d7b4651f9666f13c20e87842f0c44b93b21a2e83a96a9168 Jul 21 15:04:13.405608 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:13.405566 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-49gbt" event={"ID":"db2b1697-e840-48cf-bd68-ac5c48dca546","Type":"ContainerStarted","Data":"01856b3585cb75b5d7b4651f9666f13c20e87842f0c44b93b21a2e83a96a9168"} Jul 21 15:04:14.344726 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:14.344684 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/28687086-f5b5-4889-afdb-8145abba8b1a-metrics-tls\") pod \"dns-default-dpzts\" (UID: \"28687086-f5b5-4889-afdb-8145abba8b1a\") " pod="openshift-dns/dns-default-dpzts" Jul 21 15:04:14.347435 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:14.347408 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/28687086-f5b5-4889-afdb-8145abba8b1a-metrics-tls\") pod \"dns-default-dpzts\" (UID: \"28687086-f5b5-4889-afdb-8145abba8b1a\") " pod="openshift-dns/dns-default-dpzts" Jul 21 15:04:14.401570 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:14.401479 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-bhwsr\"" Jul 21 15:04:14.408607 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:14.408563 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-dpzts" Jul 21 15:04:14.411452 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:14.411419 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d5d6c8b9-z5c2j" event={"ID":"a8dde5b2-9cc3-4592-ae2d-3f558c970e8a","Type":"ContainerStarted","Data":"815d9586f3778cfd1f4ebf9c89dd0458d4393af1eaa9ec22e06143be9fa0d920"} Jul 21 15:04:14.433923 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:14.433863 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-d5d6c8b9-z5c2j" podStartSLOduration=1.694413312 podStartE2EDuration="4.433846064s" totalImagesPullingTime="2.739432752s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 15:04:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 15:04:14.43230343 +0000 UTC m=+286.428088862" watchObservedRunningTime="2026-07-21 15:04:14.433846064 +0000 UTC m=+286.429631532" Jul 21 15:04:14.551898 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:14.551865 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-dpzts"] Jul 21 15:04:14.659454 ip-10-0-135-176 kubenswrapper[2563]: W0721 15:04:14.659383 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28687086_f5b5_4889_afdb_8145abba8b1a.slice/crio-50df5aa14635dd71eb77e10849df24ac0a2f6c94eff9c4446b95c904393650e1 WatchSource:0}: Error finding container 50df5aa14635dd71eb77e10849df24ac0a2f6c94eff9c4446b95c904393650e1: Status 404 returned error can't find the container with id 50df5aa14635dd71eb77e10849df24ac0a2f6c94eff9c4446b95c904393650e1 Jul 21 15:04:15.390979 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:15.390952 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-756c494bbd-kwhhc"] Jul 21 15:04:15.394384 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:15.394366 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-756c494bbd-kwhhc" Jul 21 15:04:15.398571 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:15.398550 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 21 15:04:15.398807 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:15.398791 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-bpbtgs416dei2\"" Jul 21 15:04:15.398888 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:15.398866 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 21 15:04:15.398980 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:15.398965 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 21 15:04:15.399032 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:15.399009 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 21 15:04:15.400005 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:15.399984 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-5n97h\"" Jul 21 15:04:15.400096 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:15.400008 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 21 15:04:15.408644 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:15.408620 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-756c494bbd-kwhhc"] Jul 21 15:04:15.416240 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:15.416212 2563 generic.go:356] "Generic (PLEG): container finished" podID="db2b1697-e840-48cf-bd68-ac5c48dca546" containerID="f676e7c7106da11e34ade408aba8eee609f1127552aea097d03751528672b67b" exitCode=0 Jul 21 15:04:15.416372 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:15.416249 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-49gbt" event={"ID":"db2b1697-e840-48cf-bd68-ac5c48dca546","Type":"ContainerDied","Data":"f676e7c7106da11e34ade408aba8eee609f1127552aea097d03751528672b67b"} Jul 21 15:04:15.417484 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:15.417459 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dpzts" event={"ID":"28687086-f5b5-4889-afdb-8145abba8b1a","Type":"ContainerStarted","Data":"50df5aa14635dd71eb77e10849df24ac0a2f6c94eff9c4446b95c904393650e1"} Jul 21 15:04:15.455414 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:15.455389 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/74f05f06-466c-4a88-bb75-58634a88d2d5-secret-thanos-querier-tls\") pod \"thanos-querier-756c494bbd-kwhhc\" (UID: \"74f05f06-466c-4a88-bb75-58634a88d2d5\") " pod="openshift-monitoring/thanos-querier-756c494bbd-kwhhc" Jul 21 15:04:15.455553 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:15.455423 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/74f05f06-466c-4a88-bb75-58634a88d2d5-secret-grpc-tls\") pod \"thanos-querier-756c494bbd-kwhhc\" (UID: \"74f05f06-466c-4a88-bb75-58634a88d2d5\") " pod="openshift-monitoring/thanos-querier-756c494bbd-kwhhc" Jul 21 15:04:15.455553 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:15.455525 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/74f05f06-466c-4a88-bb75-58634a88d2d5-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-756c494bbd-kwhhc\" (UID: \"74f05f06-466c-4a88-bb75-58634a88d2d5\") " pod="openshift-monitoring/thanos-querier-756c494bbd-kwhhc" Jul 21 15:04:15.455708 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:15.455685 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/74f05f06-466c-4a88-bb75-58634a88d2d5-metrics-client-ca\") pod \"thanos-querier-756c494bbd-kwhhc\" (UID: \"74f05f06-466c-4a88-bb75-58634a88d2d5\") " pod="openshift-monitoring/thanos-querier-756c494bbd-kwhhc" Jul 21 15:04:15.455775 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:15.455755 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/74f05f06-466c-4a88-bb75-58634a88d2d5-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-756c494bbd-kwhhc\" (UID: \"74f05f06-466c-4a88-bb75-58634a88d2d5\") " pod="openshift-monitoring/thanos-querier-756c494bbd-kwhhc" Jul 21 15:04:15.455831 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:15.455798 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/74f05f06-466c-4a88-bb75-58634a88d2d5-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-756c494bbd-kwhhc\" (UID: \"74f05f06-466c-4a88-bb75-58634a88d2d5\") " pod="openshift-monitoring/thanos-querier-756c494bbd-kwhhc" Jul 21 15:04:15.455831 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:15.455822 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24lqw\" (UniqueName: \"kubernetes.io/projected/74f05f06-466c-4a88-bb75-58634a88d2d5-kube-api-access-24lqw\") pod \"thanos-querier-756c494bbd-kwhhc\" (UID: \"74f05f06-466c-4a88-bb75-58634a88d2d5\") " pod="openshift-monitoring/thanos-querier-756c494bbd-kwhhc" Jul 21 15:04:15.455924 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:15.455857 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/74f05f06-466c-4a88-bb75-58634a88d2d5-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-756c494bbd-kwhhc\" (UID: \"74f05f06-466c-4a88-bb75-58634a88d2d5\") " pod="openshift-monitoring/thanos-querier-756c494bbd-kwhhc" Jul 21 15:04:15.556951 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:15.556923 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/74f05f06-466c-4a88-bb75-58634a88d2d5-secret-grpc-tls\") pod \"thanos-querier-756c494bbd-kwhhc\" (UID: \"74f05f06-466c-4a88-bb75-58634a88d2d5\") " pod="openshift-monitoring/thanos-querier-756c494bbd-kwhhc" Jul 21 15:04:15.557078 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:15.556959 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/74f05f06-466c-4a88-bb75-58634a88d2d5-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-756c494bbd-kwhhc\" (UID: \"74f05f06-466c-4a88-bb75-58634a88d2d5\") " pod="openshift-monitoring/thanos-querier-756c494bbd-kwhhc" Jul 21 15:04:15.557078 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:15.557001 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/74f05f06-466c-4a88-bb75-58634a88d2d5-metrics-client-ca\") pod \"thanos-querier-756c494bbd-kwhhc\" (UID: \"74f05f06-466c-4a88-bb75-58634a88d2d5\") " pod="openshift-monitoring/thanos-querier-756c494bbd-kwhhc" Jul 21 15:04:15.557078 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:15.557022 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/74f05f06-466c-4a88-bb75-58634a88d2d5-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-756c494bbd-kwhhc\" (UID: \"74f05f06-466c-4a88-bb75-58634a88d2d5\") " pod="openshift-monitoring/thanos-querier-756c494bbd-kwhhc" Jul 21 15:04:15.557078 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:15.557047 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/74f05f06-466c-4a88-bb75-58634a88d2d5-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-756c494bbd-kwhhc\" (UID: \"74f05f06-466c-4a88-bb75-58634a88d2d5\") " pod="openshift-monitoring/thanos-querier-756c494bbd-kwhhc" Jul 21 15:04:15.557078 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:15.557067 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24lqw\" (UniqueName: \"kubernetes.io/projected/74f05f06-466c-4a88-bb75-58634a88d2d5-kube-api-access-24lqw\") pod \"thanos-querier-756c494bbd-kwhhc\" (UID: \"74f05f06-466c-4a88-bb75-58634a88d2d5\") " pod="openshift-monitoring/thanos-querier-756c494bbd-kwhhc" Jul 21 15:04:15.557252 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:15.557095 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/74f05f06-466c-4a88-bb75-58634a88d2d5-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-756c494bbd-kwhhc\" (UID: \"74f05f06-466c-4a88-bb75-58634a88d2d5\") " pod="openshift-monitoring/thanos-querier-756c494bbd-kwhhc" Jul 21 15:04:15.557252 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:15.557132 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/74f05f06-466c-4a88-bb75-58634a88d2d5-secret-thanos-querier-tls\") pod \"thanos-querier-756c494bbd-kwhhc\" (UID: \"74f05f06-466c-4a88-bb75-58634a88d2d5\") " pod="openshift-monitoring/thanos-querier-756c494bbd-kwhhc" Jul 21 15:04:15.557959 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:15.557933 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/74f05f06-466c-4a88-bb75-58634a88d2d5-metrics-client-ca\") pod \"thanos-querier-756c494bbd-kwhhc\" (UID: \"74f05f06-466c-4a88-bb75-58634a88d2d5\") " pod="openshift-monitoring/thanos-querier-756c494bbd-kwhhc" Jul 21 15:04:15.559644 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:15.559600 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/74f05f06-466c-4a88-bb75-58634a88d2d5-secret-grpc-tls\") pod \"thanos-querier-756c494bbd-kwhhc\" (UID: \"74f05f06-466c-4a88-bb75-58634a88d2d5\") " pod="openshift-monitoring/thanos-querier-756c494bbd-kwhhc" Jul 21 15:04:15.560142 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:15.560124 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/74f05f06-466c-4a88-bb75-58634a88d2d5-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-756c494bbd-kwhhc\" (UID: \"74f05f06-466c-4a88-bb75-58634a88d2d5\") " pod="openshift-monitoring/thanos-querier-756c494bbd-kwhhc" Jul 21 15:04:15.560364 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:15.560345 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/74f05f06-466c-4a88-bb75-58634a88d2d5-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-756c494bbd-kwhhc\" (UID: \"74f05f06-466c-4a88-bb75-58634a88d2d5\") " pod="openshift-monitoring/thanos-querier-756c494bbd-kwhhc" Jul 21 15:04:15.560584 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:15.560565 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/74f05f06-466c-4a88-bb75-58634a88d2d5-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-756c494bbd-kwhhc\" (UID: \"74f05f06-466c-4a88-bb75-58634a88d2d5\") " pod="openshift-monitoring/thanos-querier-756c494bbd-kwhhc" Jul 21 15:04:15.561439 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:15.561416 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/74f05f06-466c-4a88-bb75-58634a88d2d5-secret-thanos-querier-tls\") pod \"thanos-querier-756c494bbd-kwhhc\" (UID: \"74f05f06-466c-4a88-bb75-58634a88d2d5\") " pod="openshift-monitoring/thanos-querier-756c494bbd-kwhhc" Jul 21 15:04:15.562096 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:15.562073 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/74f05f06-466c-4a88-bb75-58634a88d2d5-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-756c494bbd-kwhhc\" (UID: \"74f05f06-466c-4a88-bb75-58634a88d2d5\") " pod="openshift-monitoring/thanos-querier-756c494bbd-kwhhc" Jul 21 15:04:15.566005 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:15.565985 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-24lqw\" (UniqueName: \"kubernetes.io/projected/74f05f06-466c-4a88-bb75-58634a88d2d5-kube-api-access-24lqw\") pod \"thanos-querier-756c494bbd-kwhhc\" (UID: \"74f05f06-466c-4a88-bb75-58634a88d2d5\") " pod="openshift-monitoring/thanos-querier-756c494bbd-kwhhc" Jul 21 15:04:15.703203 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:15.703118 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-756c494bbd-kwhhc" Jul 21 15:04:15.861922 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:15.858561 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-756c494bbd-kwhhc"] Jul 21 15:04:16.422606 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:16.422548 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-756c494bbd-kwhhc" event={"ID":"74f05f06-466c-4a88-bb75-58634a88d2d5","Type":"ContainerStarted","Data":"3ffa7d9936a5ea6e7719341f1e959c6bc064ffa1e32837069d944a39e9d3959c"} Jul 21 15:04:16.425271 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:16.425244 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-49gbt" event={"ID":"db2b1697-e840-48cf-bd68-ac5c48dca546","Type":"ContainerStarted","Data":"662d66b0a6887c4fca2413066083d5d1bb9cbcefde1a85c1c42a9fe95c3abcdc"} Jul 21 15:04:16.425406 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:16.425295 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-49gbt" event={"ID":"db2b1697-e840-48cf-bd68-ac5c48dca546","Type":"ContainerStarted","Data":"dfd17d7806ef6ec2930e9fe65a566e95d194e301fd7a82426c0fc17682d1c15f"} Jul 21 15:04:16.448924 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:16.448872 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-49gbt" podStartSLOduration=2.449378265 podStartE2EDuration="4.448850416s" totalImagesPullingTime="1.929842275s" totalInitContainerRuntime="69.629876ms" isStatefulPod=true podCreationTimestamp="2026-07-21 15:04:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 15:04:16.446532171 +0000 UTC m=+288.442317588" watchObservedRunningTime="2026-07-21 15:04:16.448850416 +0000 UTC m=+288.444635833" Jul 21 15:04:17.431410 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:17.431359 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dpzts" event={"ID":"28687086-f5b5-4889-afdb-8145abba8b1a","Type":"ContainerStarted","Data":"49435a10f732048798628d9b8776803304fcc6077992571ca8a63dcd2e125e66"} Jul 21 15:04:17.431842 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:17.431438 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dpzts" event={"ID":"28687086-f5b5-4889-afdb-8145abba8b1a","Type":"ContainerStarted","Data":"d6c30217198e35a26a4c819f1f55ee8ff8c947f629a4a2e3a7221d64608dabad"} Jul 21 15:04:17.431842 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:17.431650 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-dpzts" Jul 21 15:04:17.453937 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:17.453885 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-dpzts" podStartSLOduration=251.193966778 podStartE2EDuration="4m13.453864588s" totalImagesPullingTime="2.25989781s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 15:00:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 15:04:17.45166672 +0000 UTC m=+289.447452148" watchObservedRunningTime="2026-07-21 15:04:17.453864588 +0000 UTC m=+289.449650005" Jul 21 15:04:18.437179 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:18.437143 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-756c494bbd-kwhhc" event={"ID":"74f05f06-466c-4a88-bb75-58634a88d2d5","Type":"ContainerStarted","Data":"53363e71d528877abe136655c1d6b350dea6d85311b9517d5ea683db2222cb47"} Jul 21 15:04:18.437754 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:18.437213 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-756c494bbd-kwhhc" event={"ID":"74f05f06-466c-4a88-bb75-58634a88d2d5","Type":"ContainerStarted","Data":"e689f7779ab40095bfaba48f6f19136768c20045668a89a7ce1a763694868a7c"} Jul 21 15:04:18.437754 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:18.437240 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-756c494bbd-kwhhc" event={"ID":"74f05f06-466c-4a88-bb75-58634a88d2d5","Type":"ContainerStarted","Data":"31cc757e8e50e6a6a133e35e6af67cb357fa66a3caa980c5d9cc8d3a55a5b74f"} Jul 21 15:04:19.444392 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:19.444358 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-756c494bbd-kwhhc" event={"ID":"74f05f06-466c-4a88-bb75-58634a88d2d5","Type":"ContainerStarted","Data":"1a6d99d0661fb4ac491027987e707a8b8c1f48a18ffcb68405b1083ba4a6fadd"} Jul 21 15:04:19.444848 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:19.444465 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-756c494bbd-kwhhc" event={"ID":"74f05f06-466c-4a88-bb75-58634a88d2d5","Type":"ContainerStarted","Data":"2459a5a21edf53722e216f9be8a951ea3271bf142cf26c0620332a1bcd9da01d"} Jul 21 15:04:19.444848 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:19.444509 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-756c494bbd-kwhhc" event={"ID":"74f05f06-466c-4a88-bb75-58634a88d2d5","Type":"ContainerStarted","Data":"d39bab5e2da68c3e034b9f922a5326a7407d9ccf8030bfb045c95fdce183fc23"} Jul 21 15:04:19.444848 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:19.444654 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-756c494bbd-kwhhc" Jul 21 15:04:19.473641 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:19.473591 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-756c494bbd-kwhhc" podStartSLOduration=1.295671976 podStartE2EDuration="4.473559156s" totalImagesPullingTime="3.17788718s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 15:04:15 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 15:04:19.471364293 +0000 UTC m=+291.467149726" watchObservedRunningTime="2026-07-21 15:04:19.473559156 +0000 UTC m=+291.469344570" Jul 21 15:04:21.233865 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:21.233826 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-d5d6c8b9-z5c2j" Jul 21 15:04:21.234237 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:21.233982 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-d5d6c8b9-z5c2j" Jul 21 15:04:21.238539 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:21.238518 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-d5d6c8b9-z5c2j" Jul 21 15:04:21.455993 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:21.455964 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-d5d6c8b9-z5c2j" Jul 21 15:04:25.454936 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:25.454910 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-756c494bbd-kwhhc" Jul 21 15:04:25.474510 ip-10-0-135-176 kubenswrapper[2563]: E0721 15:04:25.474481 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-dh8jm" podUID="26de6bb3-eb05-461a-958f-406464dea020" Jul 21 15:04:27.439558 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:27.439528 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-dpzts" Jul 21 15:04:28.328900 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:28.328871 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kn9fn_02f3d0c6-4116-4134-ba51-1c02c73bef05/ovn-acl-logging/0.log" Jul 21 15:04:28.329662 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:28.329638 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kn9fn_02f3d0c6-4116-4134-ba51-1c02c73bef05/ovn-acl-logging/0.log" Jul 21 15:04:28.340767 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:28.340747 2563 kubelet.go:1648] "Image garbage collection succeeded" Jul 21 15:04:31.337827 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:31.337798 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-d5d6c8b9-z5c2j"] Jul 21 15:04:36.474992 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:36.474960 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dh8jm" Jul 21 15:04:54.565894 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:54.565806 2563 generic.go:356] "Generic (PLEG): container finished" podID="53f7dd9f-230e-4c5c-bc17-bd14332b1923" containerID="663588196fab0b13cc95f8ed768135df3a69cda11d66636a77c3a89ac4a30d32" exitCode=0 Jul 21 15:04:54.565894 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:54.565867 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-bfv6n" event={"ID":"53f7dd9f-230e-4c5c-bc17-bd14332b1923","Type":"ContainerDied","Data":"663588196fab0b13cc95f8ed768135df3a69cda11d66636a77c3a89ac4a30d32"} Jul 21 15:04:54.566356 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:54.566170 2563 scope.go:117] "RemoveContainer" containerID="663588196fab0b13cc95f8ed768135df3a69cda11d66636a77c3a89ac4a30d32" Jul 21 15:04:55.572489 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:55.572455 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-bfv6n" event={"ID":"53f7dd9f-230e-4c5c-bc17-bd14332b1923","Type":"ContainerStarted","Data":"0dfe643e8124c6f6fe7846bf778ea049a6dcb3fc9b1987856687a7cd3d128a51"} Jul 21 15:04:56.358058 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:56.358010 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-d5d6c8b9-z5c2j" podUID="a8dde5b2-9cc3-4592-ae2d-3f558c970e8a" containerName="console" containerID="cri-o://815d9586f3778cfd1f4ebf9c89dd0458d4393af1eaa9ec22e06143be9fa0d920" gracePeriod=15 Jul 21 15:04:56.577552 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:56.577522 2563 generic.go:356] "Generic (PLEG): container finished" podID="a8dde5b2-9cc3-4592-ae2d-3f558c970e8a" containerID="815d9586f3778cfd1f4ebf9c89dd0458d4393af1eaa9ec22e06143be9fa0d920" exitCode=0 Jul 21 15:04:56.577907 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:56.577596 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d5d6c8b9-z5c2j" event={"ID":"a8dde5b2-9cc3-4592-ae2d-3f558c970e8a","Type":"ContainerDied","Data":"815d9586f3778cfd1f4ebf9c89dd0458d4393af1eaa9ec22e06143be9fa0d920"} Jul 21 15:04:56.606179 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:56.606156 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d5d6c8b9-z5c2j" Jul 21 15:04:56.700755 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:56.700667 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a8dde5b2-9cc3-4592-ae2d-3f558c970e8a-console-serving-cert\") pod \"a8dde5b2-9cc3-4592-ae2d-3f558c970e8a\" (UID: \"a8dde5b2-9cc3-4592-ae2d-3f558c970e8a\") " Jul 21 15:04:56.700755 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:56.700707 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a8dde5b2-9cc3-4592-ae2d-3f558c970e8a-trusted-ca-bundle\") pod \"a8dde5b2-9cc3-4592-ae2d-3f558c970e8a\" (UID: \"a8dde5b2-9cc3-4592-ae2d-3f558c970e8a\") " Jul 21 15:04:56.700755 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:56.700728 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a8dde5b2-9cc3-4592-ae2d-3f558c970e8a-service-ca\") pod \"a8dde5b2-9cc3-4592-ae2d-3f558c970e8a\" (UID: \"a8dde5b2-9cc3-4592-ae2d-3f558c970e8a\") " Jul 21 15:04:56.700980 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:56.700766 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-msqlm\" (UniqueName: \"kubernetes.io/projected/a8dde5b2-9cc3-4592-ae2d-3f558c970e8a-kube-api-access-msqlm\") pod \"a8dde5b2-9cc3-4592-ae2d-3f558c970e8a\" (UID: \"a8dde5b2-9cc3-4592-ae2d-3f558c970e8a\") " Jul 21 15:04:56.700980 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:56.700794 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a8dde5b2-9cc3-4592-ae2d-3f558c970e8a-console-config\") pod \"a8dde5b2-9cc3-4592-ae2d-3f558c970e8a\" (UID: \"a8dde5b2-9cc3-4592-ae2d-3f558c970e8a\") " Jul 21 15:04:56.700980 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:56.700883 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a8dde5b2-9cc3-4592-ae2d-3f558c970e8a-oauth-serving-cert\") pod \"a8dde5b2-9cc3-4592-ae2d-3f558c970e8a\" (UID: \"a8dde5b2-9cc3-4592-ae2d-3f558c970e8a\") " Jul 21 15:04:56.700980 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:56.700907 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a8dde5b2-9cc3-4592-ae2d-3f558c970e8a-console-oauth-config\") pod \"a8dde5b2-9cc3-4592-ae2d-3f558c970e8a\" (UID: \"a8dde5b2-9cc3-4592-ae2d-3f558c970e8a\") " Jul 21 15:04:56.701416 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:56.701203 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8dde5b2-9cc3-4592-ae2d-3f558c970e8a-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "a8dde5b2-9cc3-4592-ae2d-3f558c970e8a" (UID: "a8dde5b2-9cc3-4592-ae2d-3f558c970e8a"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 15:04:56.701416 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:56.701240 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8dde5b2-9cc3-4592-ae2d-3f558c970e8a-service-ca" (OuterVolumeSpecName: "service-ca") pod "a8dde5b2-9cc3-4592-ae2d-3f558c970e8a" (UID: "a8dde5b2-9cc3-4592-ae2d-3f558c970e8a"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 15:04:56.701784 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:56.701314 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8dde5b2-9cc3-4592-ae2d-3f558c970e8a-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "a8dde5b2-9cc3-4592-ae2d-3f558c970e8a" (UID: "a8dde5b2-9cc3-4592-ae2d-3f558c970e8a"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 15:04:56.701784 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:56.701514 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8dde5b2-9cc3-4592-ae2d-3f558c970e8a-console-config" (OuterVolumeSpecName: "console-config") pod "a8dde5b2-9cc3-4592-ae2d-3f558c970e8a" (UID: "a8dde5b2-9cc3-4592-ae2d-3f558c970e8a"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 15:04:56.703235 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:56.703206 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8dde5b2-9cc3-4592-ae2d-3f558c970e8a-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "a8dde5b2-9cc3-4592-ae2d-3f558c970e8a" (UID: "a8dde5b2-9cc3-4592-ae2d-3f558c970e8a"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 15:04:56.703355 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:56.703244 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8dde5b2-9cc3-4592-ae2d-3f558c970e8a-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "a8dde5b2-9cc3-4592-ae2d-3f558c970e8a" (UID: "a8dde5b2-9cc3-4592-ae2d-3f558c970e8a"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 15:04:56.703355 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:56.703254 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8dde5b2-9cc3-4592-ae2d-3f558c970e8a-kube-api-access-msqlm" (OuterVolumeSpecName: "kube-api-access-msqlm") pod "a8dde5b2-9cc3-4592-ae2d-3f558c970e8a" (UID: "a8dde5b2-9cc3-4592-ae2d-3f558c970e8a"). InnerVolumeSpecName "kube-api-access-msqlm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 15:04:56.801601 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:56.801566 2563 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a8dde5b2-9cc3-4592-ae2d-3f558c970e8a-oauth-serving-cert\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 15:04:56.801601 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:56.801592 2563 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a8dde5b2-9cc3-4592-ae2d-3f558c970e8a-console-oauth-config\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 15:04:56.801601 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:56.801604 2563 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a8dde5b2-9cc3-4592-ae2d-3f558c970e8a-console-serving-cert\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 15:04:56.801827 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:56.801615 2563 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a8dde5b2-9cc3-4592-ae2d-3f558c970e8a-trusted-ca-bundle\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 15:04:56.801827 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:56.801624 2563 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a8dde5b2-9cc3-4592-ae2d-3f558c970e8a-service-ca\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 15:04:56.801827 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:56.801633 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-msqlm\" (UniqueName: \"kubernetes.io/projected/a8dde5b2-9cc3-4592-ae2d-3f558c970e8a-kube-api-access-msqlm\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 15:04:56.801827 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:56.801641 2563 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a8dde5b2-9cc3-4592-ae2d-3f558c970e8a-console-config\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 15:04:57.582201 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:57.582172 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d5d6c8b9-z5c2j" Jul 21 15:04:57.582201 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:57.582188 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d5d6c8b9-z5c2j" event={"ID":"a8dde5b2-9cc3-4592-ae2d-3f558c970e8a","Type":"ContainerDied","Data":"f60b1cf8ec1c9438e174282dfba03c805b07d5e068f5f591960eec46668c8d32"} Jul 21 15:04:57.582764 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:57.582251 2563 scope.go:117] "RemoveContainer" containerID="815d9586f3778cfd1f4ebf9c89dd0458d4393af1eaa9ec22e06143be9fa0d920" Jul 21 15:04:57.610185 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:57.610159 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-d5d6c8b9-z5c2j"] Jul 21 15:04:57.617194 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:57.617159 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-d5d6c8b9-z5c2j"] Jul 21 15:04:58.478781 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:04:58.478748 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a8dde5b2-9cc3-4592-ae2d-3f558c970e8a" path="/var/lib/kubelet/pods/a8dde5b2-9cc3-4592-ae2d-3f558c970e8a/volumes" Jul 21 15:05:41.252654 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:05:41.252605 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26de6bb3-eb05-461a-958f-406464dea020-metrics-certs\") pod \"network-metrics-daemon-dh8jm\" (UID: \"26de6bb3-eb05-461a-958f-406464dea020\") " pod="openshift-multus/network-metrics-daemon-dh8jm" Jul 21 15:05:41.255096 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:05:41.255066 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26de6bb3-eb05-461a-958f-406464dea020-metrics-certs\") pod \"network-metrics-daemon-dh8jm\" (UID: \"26de6bb3-eb05-461a-958f-406464dea020\") " pod="openshift-multus/network-metrics-daemon-dh8jm" Jul 21 15:05:41.279244 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:05:41.279213 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-x2gv2\"" Jul 21 15:05:41.286910 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:05:41.286883 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dh8jm" Jul 21 15:05:41.421618 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:05:41.421584 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-dh8jm"] Jul 21 15:05:41.425258 ip-10-0-135-176 kubenswrapper[2563]: W0721 15:05:41.425216 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26de6bb3_eb05_461a_958f_406464dea020.slice/crio-9aa5f6bbc6171acd5256e8af07021440fe843a38addd84a836b03f9beb805a5d WatchSource:0}: Error finding container 9aa5f6bbc6171acd5256e8af07021440fe843a38addd84a836b03f9beb805a5d: Status 404 returned error can't find the container with id 9aa5f6bbc6171acd5256e8af07021440fe843a38addd84a836b03f9beb805a5d Jul 21 15:05:41.427235 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:05:41.427216 2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 21 15:05:41.738789 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:05:41.738754 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-dh8jm" event={"ID":"26de6bb3-eb05-461a-958f-406464dea020","Type":"ContainerStarted","Data":"9aa5f6bbc6171acd5256e8af07021440fe843a38addd84a836b03f9beb805a5d"} Jul 21 15:05:42.568071 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:05:42.568035 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5486bc559b-kcq8x"] Jul 21 15:05:42.568524 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:05:42.568379 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8dde5b2-9cc3-4592-ae2d-3f558c970e8a" containerName="console" Jul 21 15:05:42.568524 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:05:42.568395 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8dde5b2-9cc3-4592-ae2d-3f558c970e8a" containerName="console" Jul 21 15:05:42.568524 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:05:42.568442 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8dde5b2-9cc3-4592-ae2d-3f558c970e8a" containerName="console" Jul 21 15:05:42.571587 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:05:42.571569 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5486bc559b-kcq8x" Jul 21 15:05:42.574799 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:05:42.574742 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 21 15:05:42.575256 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:05:42.575231 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 21 15:05:42.575388 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:05:42.575282 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 21 15:05:42.576435 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:05:42.576359 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-q9zfc\"" Jul 21 15:05:42.576617 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:05:42.576509 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 21 15:05:42.578104 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:05:42.578083 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 21 15:05:42.578214 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:05:42.578105 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 21 15:05:42.578492 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:05:42.578407 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 21 15:05:42.582248 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:05:42.582222 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 21 15:05:42.585208 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:05:42.585178 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5486bc559b-kcq8x"] Jul 21 15:05:42.663702 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:05:42.663656 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/48c27850-31a8-4fe8-8a0b-76f734fb9416-console-config\") pod \"console-5486bc559b-kcq8x\" (UID: \"48c27850-31a8-4fe8-8a0b-76f734fb9416\") " pod="openshift-console/console-5486bc559b-kcq8x" Jul 21 15:05:42.663912 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:05:42.663721 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/48c27850-31a8-4fe8-8a0b-76f734fb9416-oauth-serving-cert\") pod \"console-5486bc559b-kcq8x\" (UID: \"48c27850-31a8-4fe8-8a0b-76f734fb9416\") " pod="openshift-console/console-5486bc559b-kcq8x" Jul 21 15:05:42.663912 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:05:42.663751 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/48c27850-31a8-4fe8-8a0b-76f734fb9416-console-serving-cert\") pod \"console-5486bc559b-kcq8x\" (UID: \"48c27850-31a8-4fe8-8a0b-76f734fb9416\") " pod="openshift-console/console-5486bc559b-kcq8x" Jul 21 15:05:42.663912 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:05:42.663832 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/48c27850-31a8-4fe8-8a0b-76f734fb9416-service-ca\") pod \"console-5486bc559b-kcq8x\" (UID: \"48c27850-31a8-4fe8-8a0b-76f734fb9416\") " pod="openshift-console/console-5486bc559b-kcq8x" Jul 21 15:05:42.663912 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:05:42.663861 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/48c27850-31a8-4fe8-8a0b-76f734fb9416-trusted-ca-bundle\") pod \"console-5486bc559b-kcq8x\" (UID: \"48c27850-31a8-4fe8-8a0b-76f734fb9416\") " pod="openshift-console/console-5486bc559b-kcq8x" Jul 21 15:05:42.663912 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:05:42.663891 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/48c27850-31a8-4fe8-8a0b-76f734fb9416-console-oauth-config\") pod \"console-5486bc559b-kcq8x\" (UID: \"48c27850-31a8-4fe8-8a0b-76f734fb9416\") " pod="openshift-console/console-5486bc559b-kcq8x" Jul 21 15:05:42.663912 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:05:42.663910 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqgmm\" (UniqueName: \"kubernetes.io/projected/48c27850-31a8-4fe8-8a0b-76f734fb9416-kube-api-access-gqgmm\") pod \"console-5486bc559b-kcq8x\" (UID: \"48c27850-31a8-4fe8-8a0b-76f734fb9416\") " pod="openshift-console/console-5486bc559b-kcq8x" Jul 21 15:05:42.769904 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:05:42.769719 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/48c27850-31a8-4fe8-8a0b-76f734fb9416-console-config\") pod \"console-5486bc559b-kcq8x\" (UID: \"48c27850-31a8-4fe8-8a0b-76f734fb9416\") " pod="openshift-console/console-5486bc559b-kcq8x" Jul 21 15:05:42.770309 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:05:42.770280 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/48c27850-31a8-4fe8-8a0b-76f734fb9416-oauth-serving-cert\") pod \"console-5486bc559b-kcq8x\" (UID: \"48c27850-31a8-4fe8-8a0b-76f734fb9416\") " pod="openshift-console/console-5486bc559b-kcq8x" Jul 21 15:05:42.770428 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:05:42.770359 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/48c27850-31a8-4fe8-8a0b-76f734fb9416-console-serving-cert\") pod \"console-5486bc559b-kcq8x\" (UID: \"48c27850-31a8-4fe8-8a0b-76f734fb9416\") " pod="openshift-console/console-5486bc559b-kcq8x" Jul 21 15:05:42.770428 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:05:42.770418 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/48c27850-31a8-4fe8-8a0b-76f734fb9416-service-ca\") pod \"console-5486bc559b-kcq8x\" (UID: \"48c27850-31a8-4fe8-8a0b-76f734fb9416\") " pod="openshift-console/console-5486bc559b-kcq8x" Jul 21 15:05:42.770532 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:05:42.770445 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/48c27850-31a8-4fe8-8a0b-76f734fb9416-trusted-ca-bundle\") pod \"console-5486bc559b-kcq8x\" (UID: \"48c27850-31a8-4fe8-8a0b-76f734fb9416\") " pod="openshift-console/console-5486bc559b-kcq8x" Jul 21 15:05:42.770532 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:05:42.770478 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/48c27850-31a8-4fe8-8a0b-76f734fb9416-console-oauth-config\") pod \"console-5486bc559b-kcq8x\" (UID: \"48c27850-31a8-4fe8-8a0b-76f734fb9416\") " pod="openshift-console/console-5486bc559b-kcq8x" Jul 21 15:05:42.770532 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:05:42.770501 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqgmm\" (UniqueName: \"kubernetes.io/projected/48c27850-31a8-4fe8-8a0b-76f734fb9416-kube-api-access-gqgmm\") pod \"console-5486bc559b-kcq8x\" (UID: \"48c27850-31a8-4fe8-8a0b-76f734fb9416\") " pod="openshift-console/console-5486bc559b-kcq8x" Jul 21 15:05:42.770852 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:05:42.770828 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/48c27850-31a8-4fe8-8a0b-76f734fb9416-console-config\") pod \"console-5486bc559b-kcq8x\" (UID: \"48c27850-31a8-4fe8-8a0b-76f734fb9416\") " pod="openshift-console/console-5486bc559b-kcq8x" Jul 21 15:05:42.771145 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:05:42.771116 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/48c27850-31a8-4fe8-8a0b-76f734fb9416-service-ca\") pod \"console-5486bc559b-kcq8x\" (UID: \"48c27850-31a8-4fe8-8a0b-76f734fb9416\") " pod="openshift-console/console-5486bc559b-kcq8x" Jul 21 15:05:42.771242 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:05:42.771183 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/48c27850-31a8-4fe8-8a0b-76f734fb9416-oauth-serving-cert\") pod \"console-5486bc559b-kcq8x\" (UID: \"48c27850-31a8-4fe8-8a0b-76f734fb9416\") " pod="openshift-console/console-5486bc559b-kcq8x" Jul 21 15:05:42.771845 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:05:42.771821 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/48c27850-31a8-4fe8-8a0b-76f734fb9416-trusted-ca-bundle\") pod \"console-5486bc559b-kcq8x\" (UID: \"48c27850-31a8-4fe8-8a0b-76f734fb9416\") " pod="openshift-console/console-5486bc559b-kcq8x" Jul 21 15:05:42.773301 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:05:42.773272 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/48c27850-31a8-4fe8-8a0b-76f734fb9416-console-serving-cert\") pod \"console-5486bc559b-kcq8x\" (UID: \"48c27850-31a8-4fe8-8a0b-76f734fb9416\") " pod="openshift-console/console-5486bc559b-kcq8x" Jul 21 15:05:42.773452 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:05:42.773427 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/48c27850-31a8-4fe8-8a0b-76f734fb9416-console-oauth-config\") pod \"console-5486bc559b-kcq8x\" (UID: \"48c27850-31a8-4fe8-8a0b-76f734fb9416\") " pod="openshift-console/console-5486bc559b-kcq8x" Jul 21 15:05:42.780512 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:05:42.780490 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqgmm\" (UniqueName: \"kubernetes.io/projected/48c27850-31a8-4fe8-8a0b-76f734fb9416-kube-api-access-gqgmm\") pod \"console-5486bc559b-kcq8x\" (UID: \"48c27850-31a8-4fe8-8a0b-76f734fb9416\") " pod="openshift-console/console-5486bc559b-kcq8x" Jul 21 15:05:42.884109 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:05:42.884078 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5486bc559b-kcq8x" Jul 21 15:05:43.025697 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:05:43.025665 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5486bc559b-kcq8x"] Jul 21 15:05:43.028990 ip-10-0-135-176 kubenswrapper[2563]: W0721 15:05:43.028961 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48c27850_31a8_4fe8_8a0b_76f734fb9416.slice/crio-8756f283ee9f114ed3eee2ac1aace9ee48e9a68c026ae7d12865a39b720d6ade WatchSource:0}: Error finding container 8756f283ee9f114ed3eee2ac1aace9ee48e9a68c026ae7d12865a39b720d6ade: Status 404 returned error can't find the container with id 8756f283ee9f114ed3eee2ac1aace9ee48e9a68c026ae7d12865a39b720d6ade Jul 21 15:05:43.747960 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:05:43.747918 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-dh8jm" event={"ID":"26de6bb3-eb05-461a-958f-406464dea020","Type":"ContainerStarted","Data":"29859e71a5eb374f6036d083782f47e1e34afab12222cf1686fbd153d8aaa6f4"} Jul 21 15:05:43.748418 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:05:43.747985 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-dh8jm" event={"ID":"26de6bb3-eb05-461a-958f-406464dea020","Type":"ContainerStarted","Data":"db31a2e9938e50be4636e9fad4bbbf1a482c8865acb389397e8811260165e2c4"} Jul 21 15:05:43.749513 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:05:43.749487 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5486bc559b-kcq8x" event={"ID":"48c27850-31a8-4fe8-8a0b-76f734fb9416","Type":"ContainerStarted","Data":"4f458890b32c2b0c388d06d523a1f056c38cca4fc200bc7b4d877c1c3e0a4d58"} Jul 21 15:05:43.749644 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:05:43.749536 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5486bc559b-kcq8x" event={"ID":"48c27850-31a8-4fe8-8a0b-76f734fb9416","Type":"ContainerStarted","Data":"8756f283ee9f114ed3eee2ac1aace9ee48e9a68c026ae7d12865a39b720d6ade"} Jul 21 15:05:43.765396 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:05:43.765348 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-dh8jm" podStartSLOduration=374.463734578 podStartE2EDuration="6m15.765307663s" totalImagesPullingTime="1.301573085s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 14:59:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 15:05:43.764992941 +0000 UTC m=+375.760778379" watchObservedRunningTime="2026-07-21 15:05:43.765307663 +0000 UTC m=+375.761093080" Jul 21 15:05:43.790948 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:05:43.790895 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5486bc559b-kcq8x" podStartSLOduration=1.79087892 podStartE2EDuration="1.79087892s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 15:05:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 15:05:43.790777017 +0000 UTC m=+375.786562469" watchObservedRunningTime="2026-07-21 15:05:43.79087892 +0000 UTC m=+375.786664336" Jul 21 15:05:52.884524 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:05:52.884491 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5486bc559b-kcq8x" Jul 21 15:05:52.884884 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:05:52.884592 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5486bc559b-kcq8x" Jul 21 15:05:52.889294 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:05:52.889273 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5486bc559b-kcq8x" Jul 21 15:05:53.790779 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:05:53.790755 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5486bc559b-kcq8x" Jul 21 15:06:53.032086 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:06:53.032013 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6b687b7676-d9v9s"] Jul 21 15:06:53.035194 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:06:53.035174 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b687b7676-d9v9s" Jul 21 15:06:53.046373 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:06:53.046348 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6b687b7676-d9v9s"] Jul 21 15:06:53.118636 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:06:53.118597 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-968db\" (UniqueName: \"kubernetes.io/projected/7dd4bb5f-1fee-4053-a7e9-a9325ceec7bf-kube-api-access-968db\") pod \"console-6b687b7676-d9v9s\" (UID: \"7dd4bb5f-1fee-4053-a7e9-a9325ceec7bf\") " pod="openshift-console/console-6b687b7676-d9v9s" Jul 21 15:06:53.118636 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:06:53.118642 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7dd4bb5f-1fee-4053-a7e9-a9325ceec7bf-console-serving-cert\") pod \"console-6b687b7676-d9v9s\" (UID: \"7dd4bb5f-1fee-4053-a7e9-a9325ceec7bf\") " pod="openshift-console/console-6b687b7676-d9v9s" Jul 21 15:06:53.118860 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:06:53.118663 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7dd4bb5f-1fee-4053-a7e9-a9325ceec7bf-service-ca\") pod \"console-6b687b7676-d9v9s\" (UID: \"7dd4bb5f-1fee-4053-a7e9-a9325ceec7bf\") " pod="openshift-console/console-6b687b7676-d9v9s" Jul 21 15:06:53.118860 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:06:53.118708 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7dd4bb5f-1fee-4053-a7e9-a9325ceec7bf-trusted-ca-bundle\") pod \"console-6b687b7676-d9v9s\" (UID: \"7dd4bb5f-1fee-4053-a7e9-a9325ceec7bf\") " pod="openshift-console/console-6b687b7676-d9v9s" Jul 21 15:06:53.118860 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:06:53.118769 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7dd4bb5f-1fee-4053-a7e9-a9325ceec7bf-console-oauth-config\") pod \"console-6b687b7676-d9v9s\" (UID: \"7dd4bb5f-1fee-4053-a7e9-a9325ceec7bf\") " pod="openshift-console/console-6b687b7676-d9v9s" Jul 21 15:06:53.118860 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:06:53.118815 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7dd4bb5f-1fee-4053-a7e9-a9325ceec7bf-console-config\") pod \"console-6b687b7676-d9v9s\" (UID: \"7dd4bb5f-1fee-4053-a7e9-a9325ceec7bf\") " pod="openshift-console/console-6b687b7676-d9v9s" Jul 21 15:06:53.118860 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:06:53.118839 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7dd4bb5f-1fee-4053-a7e9-a9325ceec7bf-oauth-serving-cert\") pod \"console-6b687b7676-d9v9s\" (UID: \"7dd4bb5f-1fee-4053-a7e9-a9325ceec7bf\") " pod="openshift-console/console-6b687b7676-d9v9s" Jul 21 15:06:53.219922 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:06:53.219866 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7dd4bb5f-1fee-4053-a7e9-a9325ceec7bf-console-config\") pod \"console-6b687b7676-d9v9s\" (UID: \"7dd4bb5f-1fee-4053-a7e9-a9325ceec7bf\") " pod="openshift-console/console-6b687b7676-d9v9s" Jul 21 15:06:53.219922 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:06:53.219923 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7dd4bb5f-1fee-4053-a7e9-a9325ceec7bf-oauth-serving-cert\") pod \"console-6b687b7676-d9v9s\" (UID: \"7dd4bb5f-1fee-4053-a7e9-a9325ceec7bf\") " pod="openshift-console/console-6b687b7676-d9v9s" Jul 21 15:06:53.220162 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:06:53.219967 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-968db\" (UniqueName: \"kubernetes.io/projected/7dd4bb5f-1fee-4053-a7e9-a9325ceec7bf-kube-api-access-968db\") pod \"console-6b687b7676-d9v9s\" (UID: \"7dd4bb5f-1fee-4053-a7e9-a9325ceec7bf\") " pod="openshift-console/console-6b687b7676-d9v9s" Jul 21 15:06:53.220162 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:06:53.220000 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7dd4bb5f-1fee-4053-a7e9-a9325ceec7bf-console-serving-cert\") pod \"console-6b687b7676-d9v9s\" (UID: \"7dd4bb5f-1fee-4053-a7e9-a9325ceec7bf\") " pod="openshift-console/console-6b687b7676-d9v9s" Jul 21 15:06:53.220162 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:06:53.220030 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7dd4bb5f-1fee-4053-a7e9-a9325ceec7bf-service-ca\") pod \"console-6b687b7676-d9v9s\" (UID: \"7dd4bb5f-1fee-4053-a7e9-a9325ceec7bf\") " pod="openshift-console/console-6b687b7676-d9v9s" Jul 21 15:06:53.220162 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:06:53.220049 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7dd4bb5f-1fee-4053-a7e9-a9325ceec7bf-trusted-ca-bundle\") pod \"console-6b687b7676-d9v9s\" (UID: \"7dd4bb5f-1fee-4053-a7e9-a9325ceec7bf\") " pod="openshift-console/console-6b687b7676-d9v9s" Jul 21 15:06:53.220162 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:06:53.220084 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7dd4bb5f-1fee-4053-a7e9-a9325ceec7bf-console-oauth-config\") pod \"console-6b687b7676-d9v9s\" (UID: \"7dd4bb5f-1fee-4053-a7e9-a9325ceec7bf\") " pod="openshift-console/console-6b687b7676-d9v9s" Jul 21 15:06:53.220785 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:06:53.220753 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7dd4bb5f-1fee-4053-a7e9-a9325ceec7bf-console-config\") pod \"console-6b687b7676-d9v9s\" (UID: \"7dd4bb5f-1fee-4053-a7e9-a9325ceec7bf\") " pod="openshift-console/console-6b687b7676-d9v9s" Jul 21 15:06:53.220785 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:06:53.220775 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7dd4bb5f-1fee-4053-a7e9-a9325ceec7bf-service-ca\") pod \"console-6b687b7676-d9v9s\" (UID: \"7dd4bb5f-1fee-4053-a7e9-a9325ceec7bf\") " pod="openshift-console/console-6b687b7676-d9v9s" Jul 21 15:06:53.220923 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:06:53.220753 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7dd4bb5f-1fee-4053-a7e9-a9325ceec7bf-oauth-serving-cert\") pod \"console-6b687b7676-d9v9s\" (UID: \"7dd4bb5f-1fee-4053-a7e9-a9325ceec7bf\") " pod="openshift-console/console-6b687b7676-d9v9s" Jul 21 15:06:53.221007 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:06:53.220985 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7dd4bb5f-1fee-4053-a7e9-a9325ceec7bf-trusted-ca-bundle\") pod \"console-6b687b7676-d9v9s\" (UID: \"7dd4bb5f-1fee-4053-a7e9-a9325ceec7bf\") " pod="openshift-console/console-6b687b7676-d9v9s" Jul 21 15:06:53.223142 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:06:53.223120 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7dd4bb5f-1fee-4053-a7e9-a9325ceec7bf-console-serving-cert\") pod \"console-6b687b7676-d9v9s\" (UID: \"7dd4bb5f-1fee-4053-a7e9-a9325ceec7bf\") " pod="openshift-console/console-6b687b7676-d9v9s" Jul 21 15:06:53.223239 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:06:53.223151 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7dd4bb5f-1fee-4053-a7e9-a9325ceec7bf-console-oauth-config\") pod \"console-6b687b7676-d9v9s\" (UID: \"7dd4bb5f-1fee-4053-a7e9-a9325ceec7bf\") " pod="openshift-console/console-6b687b7676-d9v9s" Jul 21 15:06:53.230308 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:06:53.230283 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-968db\" (UniqueName: \"kubernetes.io/projected/7dd4bb5f-1fee-4053-a7e9-a9325ceec7bf-kube-api-access-968db\") pod \"console-6b687b7676-d9v9s\" (UID: \"7dd4bb5f-1fee-4053-a7e9-a9325ceec7bf\") " pod="openshift-console/console-6b687b7676-d9v9s" Jul 21 15:06:53.344722 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:06:53.344683 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b687b7676-d9v9s" Jul 21 15:06:53.477361 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:06:53.477284 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6b687b7676-d9v9s"] Jul 21 15:06:53.480017 ip-10-0-135-176 kubenswrapper[2563]: W0721 15:06:53.479985 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dd4bb5f_1fee_4053_a7e9_a9325ceec7bf.slice/crio-821ef0fb127b1e1eae0a2a839f1a40de8699cf90958ecb9ec3cbf0467a244b05 WatchSource:0}: Error finding container 821ef0fb127b1e1eae0a2a839f1a40de8699cf90958ecb9ec3cbf0467a244b05: Status 404 returned error can't find the container with id 821ef0fb127b1e1eae0a2a839f1a40de8699cf90958ecb9ec3cbf0467a244b05 Jul 21 15:06:53.996900 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:06:53.996868 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b687b7676-d9v9s" event={"ID":"7dd4bb5f-1fee-4053-a7e9-a9325ceec7bf","Type":"ContainerStarted","Data":"c1c92474de6f93517ebfe855058d0e7d7d5c8815f95f1255ba6f98c3e4f61bb3"} Jul 21 15:06:53.997088 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:06:53.996913 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b687b7676-d9v9s" event={"ID":"7dd4bb5f-1fee-4053-a7e9-a9325ceec7bf","Type":"ContainerStarted","Data":"821ef0fb127b1e1eae0a2a839f1a40de8699cf90958ecb9ec3cbf0467a244b05"} Jul 21 15:06:54.018034 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:06:54.017981 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6b687b7676-d9v9s" podStartSLOduration=1.017964081 podStartE2EDuration="1.017964081s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 15:06:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 15:06:54.016231978 +0000 UTC m=+446.012017395" watchObservedRunningTime="2026-07-21 15:06:54.017964081 +0000 UTC m=+446.013749497" Jul 21 15:07:03.344859 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:07:03.344822 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6b687b7676-d9v9s" Jul 21 15:07:03.345261 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:07:03.344887 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6b687b7676-d9v9s" Jul 21 15:07:03.349641 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:07:03.349615 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6b687b7676-d9v9s" Jul 21 15:07:04.034512 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:07:04.034467 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6b687b7676-d9v9s" Jul 21 15:07:04.081156 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:07:04.081118 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5486bc559b-kcq8x"] Jul 21 15:07:29.102399 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:07:29.102357 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5486bc559b-kcq8x" podUID="48c27850-31a8-4fe8-8a0b-76f734fb9416" containerName="console" containerID="cri-o://4f458890b32c2b0c388d06d523a1f056c38cca4fc200bc7b4d877c1c3e0a4d58" gracePeriod=15 Jul 21 15:07:29.337216 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:07:29.337185 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5486bc559b-kcq8x" Jul 21 15:07:29.413179 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:07:29.413086 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/48c27850-31a8-4fe8-8a0b-76f734fb9416-trusted-ca-bundle\") pod \"48c27850-31a8-4fe8-8a0b-76f734fb9416\" (UID: \"48c27850-31a8-4fe8-8a0b-76f734fb9416\") " Jul 21 15:07:29.413179 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:07:29.413123 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/48c27850-31a8-4fe8-8a0b-76f734fb9416-service-ca\") pod \"48c27850-31a8-4fe8-8a0b-76f734fb9416\" (UID: \"48c27850-31a8-4fe8-8a0b-76f734fb9416\") " Jul 21 15:07:29.413179 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:07:29.413160 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/48c27850-31a8-4fe8-8a0b-76f734fb9416-console-serving-cert\") pod \"48c27850-31a8-4fe8-8a0b-76f734fb9416\" (UID: \"48c27850-31a8-4fe8-8a0b-76f734fb9416\") " Jul 21 15:07:29.413484 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:07:29.413188 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/48c27850-31a8-4fe8-8a0b-76f734fb9416-oauth-serving-cert\") pod \"48c27850-31a8-4fe8-8a0b-76f734fb9416\" (UID: \"48c27850-31a8-4fe8-8a0b-76f734fb9416\") " Jul 21 15:07:29.413484 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:07:29.413216 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gqgmm\" (UniqueName: \"kubernetes.io/projected/48c27850-31a8-4fe8-8a0b-76f734fb9416-kube-api-access-gqgmm\") pod \"48c27850-31a8-4fe8-8a0b-76f734fb9416\" (UID: \"48c27850-31a8-4fe8-8a0b-76f734fb9416\") " Jul 21 15:07:29.413484 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:07:29.413256 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/48c27850-31a8-4fe8-8a0b-76f734fb9416-console-config\") pod \"48c27850-31a8-4fe8-8a0b-76f734fb9416\" (UID: \"48c27850-31a8-4fe8-8a0b-76f734fb9416\") " Jul 21 15:07:29.413484 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:07:29.413282 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/48c27850-31a8-4fe8-8a0b-76f734fb9416-console-oauth-config\") pod \"48c27850-31a8-4fe8-8a0b-76f734fb9416\" (UID: \"48c27850-31a8-4fe8-8a0b-76f734fb9416\") " Jul 21 15:07:29.413678 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:07:29.413602 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48c27850-31a8-4fe8-8a0b-76f734fb9416-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "48c27850-31a8-4fe8-8a0b-76f734fb9416" (UID: "48c27850-31a8-4fe8-8a0b-76f734fb9416"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 15:07:29.413737 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:07:29.413678 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48c27850-31a8-4fe8-8a0b-76f734fb9416-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "48c27850-31a8-4fe8-8a0b-76f734fb9416" (UID: "48c27850-31a8-4fe8-8a0b-76f734fb9416"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 15:07:29.413791 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:07:29.413758 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48c27850-31a8-4fe8-8a0b-76f734fb9416-console-config" (OuterVolumeSpecName: "console-config") pod "48c27850-31a8-4fe8-8a0b-76f734fb9416" (UID: "48c27850-31a8-4fe8-8a0b-76f734fb9416"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 15:07:29.413845 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:07:29.413823 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48c27850-31a8-4fe8-8a0b-76f734fb9416-service-ca" (OuterVolumeSpecName: "service-ca") pod "48c27850-31a8-4fe8-8a0b-76f734fb9416" (UID: "48c27850-31a8-4fe8-8a0b-76f734fb9416"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 15:07:29.415689 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:07:29.415659 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48c27850-31a8-4fe8-8a0b-76f734fb9416-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "48c27850-31a8-4fe8-8a0b-76f734fb9416" (UID: "48c27850-31a8-4fe8-8a0b-76f734fb9416"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 15:07:29.415689 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:07:29.415673 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48c27850-31a8-4fe8-8a0b-76f734fb9416-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "48c27850-31a8-4fe8-8a0b-76f734fb9416" (UID: "48c27850-31a8-4fe8-8a0b-76f734fb9416"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 15:07:29.415838 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:07:29.415793 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48c27850-31a8-4fe8-8a0b-76f734fb9416-kube-api-access-gqgmm" (OuterVolumeSpecName: "kube-api-access-gqgmm") pod "48c27850-31a8-4fe8-8a0b-76f734fb9416" (UID: "48c27850-31a8-4fe8-8a0b-76f734fb9416"). InnerVolumeSpecName "kube-api-access-gqgmm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 15:07:29.514483 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:07:29.514444 2563 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/48c27850-31a8-4fe8-8a0b-76f734fb9416-trusted-ca-bundle\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 15:07:29.514483 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:07:29.514478 2563 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/48c27850-31a8-4fe8-8a0b-76f734fb9416-service-ca\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 15:07:29.514483 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:07:29.514487 2563 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/48c27850-31a8-4fe8-8a0b-76f734fb9416-console-serving-cert\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 15:07:29.514483 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:07:29.514496 2563 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/48c27850-31a8-4fe8-8a0b-76f734fb9416-oauth-serving-cert\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 15:07:29.514757 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:07:29.514504 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqgmm\" (UniqueName: \"kubernetes.io/projected/48c27850-31a8-4fe8-8a0b-76f734fb9416-kube-api-access-gqgmm\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 15:07:29.514757 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:07:29.514514 2563 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/48c27850-31a8-4fe8-8a0b-76f734fb9416-console-config\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 15:07:29.514757 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:07:29.514522 2563 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/48c27850-31a8-4fe8-8a0b-76f734fb9416-console-oauth-config\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 15:07:30.124539 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:07:30.124505 2563 generic.go:356] "Generic (PLEG): container finished" podID="48c27850-31a8-4fe8-8a0b-76f734fb9416" containerID="4f458890b32c2b0c388d06d523a1f056c38cca4fc200bc7b4d877c1c3e0a4d58" exitCode=0 Jul 21 15:07:30.124955 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:07:30.124552 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5486bc559b-kcq8x" Jul 21 15:07:30.124955 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:07:30.124565 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5486bc559b-kcq8x" event={"ID":"48c27850-31a8-4fe8-8a0b-76f734fb9416","Type":"ContainerDied","Data":"4f458890b32c2b0c388d06d523a1f056c38cca4fc200bc7b4d877c1c3e0a4d58"} Jul 21 15:07:30.124955 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:07:30.124609 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5486bc559b-kcq8x" event={"ID":"48c27850-31a8-4fe8-8a0b-76f734fb9416","Type":"ContainerDied","Data":"8756f283ee9f114ed3eee2ac1aace9ee48e9a68c026ae7d12865a39b720d6ade"} Jul 21 15:07:30.124955 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:07:30.124632 2563 scope.go:117] "RemoveContainer" containerID="4f458890b32c2b0c388d06d523a1f056c38cca4fc200bc7b4d877c1c3e0a4d58" Jul 21 15:07:30.133395 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:07:30.133380 2563 scope.go:117] "RemoveContainer" containerID="4f458890b32c2b0c388d06d523a1f056c38cca4fc200bc7b4d877c1c3e0a4d58" Jul 21 15:07:30.133677 ip-10-0-135-176 kubenswrapper[2563]: E0721 15:07:30.133657 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f458890b32c2b0c388d06d523a1f056c38cca4fc200bc7b4d877c1c3e0a4d58\": container with ID starting with 4f458890b32c2b0c388d06d523a1f056c38cca4fc200bc7b4d877c1c3e0a4d58 not found: ID does not exist" containerID="4f458890b32c2b0c388d06d523a1f056c38cca4fc200bc7b4d877c1c3e0a4d58" Jul 21 15:07:30.133726 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:07:30.133686 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f458890b32c2b0c388d06d523a1f056c38cca4fc200bc7b4d877c1c3e0a4d58"} err="failed to get container status \"4f458890b32c2b0c388d06d523a1f056c38cca4fc200bc7b4d877c1c3e0a4d58\": rpc error: code = NotFound desc = could not find container \"4f458890b32c2b0c388d06d523a1f056c38cca4fc200bc7b4d877c1c3e0a4d58\": container with ID starting with 4f458890b32c2b0c388d06d523a1f056c38cca4fc200bc7b4d877c1c3e0a4d58 not found: ID does not exist" Jul 21 15:07:30.147835 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:07:30.147805 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5486bc559b-kcq8x"] Jul 21 15:07:30.152288 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:07:30.152264 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5486bc559b-kcq8x"] Jul 21 15:07:30.479721 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:07:30.479634 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="48c27850-31a8-4fe8-8a0b-76f734fb9416" path="/var/lib/kubelet/pods/48c27850-31a8-4fe8-8a0b-76f734fb9416/volumes" Jul 21 15:07:46.724727 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:07:46.724689 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-66ffb444fb-wpwvm"] Jul 21 15:07:46.725166 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:07:46.724957 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48c27850-31a8-4fe8-8a0b-76f734fb9416" containerName="console" Jul 21 15:07:46.725166 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:07:46.724967 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="48c27850-31a8-4fe8-8a0b-76f734fb9416" containerName="console" Jul 21 15:07:46.725166 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:07:46.725019 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="48c27850-31a8-4fe8-8a0b-76f734fb9416" containerName="console" Jul 21 15:07:46.729255 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:07:46.729238 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-66ffb444fb-wpwvm" Jul 21 15:07:46.732153 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:07:46.732133 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 21 15:07:46.733479 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:07:46.733462 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Jul 21 15:07:46.733581 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:07:46.733534 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 21 15:07:46.733638 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:07:46.733577 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 21 15:07:46.738127 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:07:46.738106 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-66ffb444fb-wpwvm"] Jul 21 15:07:46.846921 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:07:46.846889 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/715e3e38-35a1-49a9-82c6-ad8c366d9ad3-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-66ffb444fb-wpwvm\" (UID: \"715e3e38-35a1-49a9-82c6-ad8c366d9ad3\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-66ffb444fb-wpwvm" Jul 21 15:07:46.847089 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:07:46.847002 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mn8xm\" (UniqueName: \"kubernetes.io/projected/715e3e38-35a1-49a9-82c6-ad8c366d9ad3-kube-api-access-mn8xm\") pod \"managed-serviceaccount-addon-agent-66ffb444fb-wpwvm\" (UID: \"715e3e38-35a1-49a9-82c6-ad8c366d9ad3\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-66ffb444fb-wpwvm" Jul 21 15:07:46.948437 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:07:46.948394 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mn8xm\" (UniqueName: \"kubernetes.io/projected/715e3e38-35a1-49a9-82c6-ad8c366d9ad3-kube-api-access-mn8xm\") pod \"managed-serviceaccount-addon-agent-66ffb444fb-wpwvm\" (UID: \"715e3e38-35a1-49a9-82c6-ad8c366d9ad3\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-66ffb444fb-wpwvm" Jul 21 15:07:46.948640 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:07:46.948442 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/715e3e38-35a1-49a9-82c6-ad8c366d9ad3-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-66ffb444fb-wpwvm\" (UID: \"715e3e38-35a1-49a9-82c6-ad8c366d9ad3\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-66ffb444fb-wpwvm" Jul 21 15:07:46.951140 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:07:46.951113 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/715e3e38-35a1-49a9-82c6-ad8c366d9ad3-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-66ffb444fb-wpwvm\" (UID: \"715e3e38-35a1-49a9-82c6-ad8c366d9ad3\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-66ffb444fb-wpwvm" Jul 21 15:07:46.957367 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:07:46.957345 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mn8xm\" (UniqueName: \"kubernetes.io/projected/715e3e38-35a1-49a9-82c6-ad8c366d9ad3-kube-api-access-mn8xm\") pod \"managed-serviceaccount-addon-agent-66ffb444fb-wpwvm\" (UID: \"715e3e38-35a1-49a9-82c6-ad8c366d9ad3\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-66ffb444fb-wpwvm" Jul 21 15:07:47.048851 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:07:47.048759 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-66ffb444fb-wpwvm" Jul 21 15:07:47.187099 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:07:47.187066 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-66ffb444fb-wpwvm"] Jul 21 15:07:47.189869 ip-10-0-135-176 kubenswrapper[2563]: W0721 15:07:47.189844 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod715e3e38_35a1_49a9_82c6_ad8c366d9ad3.slice/crio-1e0cbdc9bf3ed08e9cacc6103926bb7738bedae22ea5b645f581514bb6755bcf WatchSource:0}: Error finding container 1e0cbdc9bf3ed08e9cacc6103926bb7738bedae22ea5b645f581514bb6755bcf: Status 404 returned error can't find the container with id 1e0cbdc9bf3ed08e9cacc6103926bb7738bedae22ea5b645f581514bb6755bcf Jul 21 15:07:48.190985 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:07:48.190948 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-66ffb444fb-wpwvm" event={"ID":"715e3e38-35a1-49a9-82c6-ad8c366d9ad3","Type":"ContainerStarted","Data":"1e0cbdc9bf3ed08e9cacc6103926bb7738bedae22ea5b645f581514bb6755bcf"} Jul 21 15:07:51.205472 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:07:51.205435 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-66ffb444fb-wpwvm" event={"ID":"715e3e38-35a1-49a9-82c6-ad8c366d9ad3","Type":"ContainerStarted","Data":"d2694691fb21c757f37b5f9953b64be27e612a5a78961a7147ada4421f14dcd8"} Jul 21 15:07:51.221855 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:07:51.221801 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-66ffb444fb-wpwvm" podStartSLOduration=1.910042722 podStartE2EDuration="5.22178452s" totalImagesPullingTime="3.311741798s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 15:07:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 15:07:51.22042359 +0000 UTC m=+503.216209007" watchObservedRunningTime="2026-07-21 15:07:51.22178452 +0000 UTC m=+503.217569937" Jul 21 15:08:21.623010 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:08:21.622930 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zdpmz"] Jul 21 15:08:21.629947 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:08:21.629927 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zdpmz" Jul 21 15:08:21.634715 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:08:21.634693 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 21 15:08:21.635944 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:08:21.635923 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 21 15:08:21.636626 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:08:21.636242 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-lkwhz\"" Jul 21 15:08:21.638758 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:08:21.638735 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zdpmz"] Jul 21 15:08:21.722119 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:08:21.722080 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/807e8526-473c-4511-8418-97eba53bebd8-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zdpmz\" (UID: \"807e8526-473c-4511-8418-97eba53bebd8\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zdpmz" Jul 21 15:08:21.722287 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:08:21.722148 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqpmm\" (UniqueName: \"kubernetes.io/projected/807e8526-473c-4511-8418-97eba53bebd8-kube-api-access-gqpmm\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zdpmz\" (UID: \"807e8526-473c-4511-8418-97eba53bebd8\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zdpmz" Jul 21 15:08:21.722287 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:08:21.722205 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/807e8526-473c-4511-8418-97eba53bebd8-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zdpmz\" (UID: \"807e8526-473c-4511-8418-97eba53bebd8\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zdpmz" Jul 21 15:08:21.822624 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:08:21.822582 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/807e8526-473c-4511-8418-97eba53bebd8-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zdpmz\" (UID: \"807e8526-473c-4511-8418-97eba53bebd8\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zdpmz" Jul 21 15:08:21.822801 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:08:21.822666 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqpmm\" (UniqueName: \"kubernetes.io/projected/807e8526-473c-4511-8418-97eba53bebd8-kube-api-access-gqpmm\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zdpmz\" (UID: \"807e8526-473c-4511-8418-97eba53bebd8\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zdpmz" Jul 21 15:08:21.822801 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:08:21.822690 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/807e8526-473c-4511-8418-97eba53bebd8-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zdpmz\" (UID: \"807e8526-473c-4511-8418-97eba53bebd8\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zdpmz" Jul 21 15:08:21.823045 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:08:21.823018 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/807e8526-473c-4511-8418-97eba53bebd8-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zdpmz\" (UID: \"807e8526-473c-4511-8418-97eba53bebd8\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zdpmz" Jul 21 15:08:21.823086 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:08:21.823074 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/807e8526-473c-4511-8418-97eba53bebd8-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zdpmz\" (UID: \"807e8526-473c-4511-8418-97eba53bebd8\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zdpmz" Jul 21 15:08:21.832053 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:08:21.832021 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqpmm\" (UniqueName: \"kubernetes.io/projected/807e8526-473c-4511-8418-97eba53bebd8-kube-api-access-gqpmm\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zdpmz\" (UID: \"807e8526-473c-4511-8418-97eba53bebd8\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zdpmz" Jul 21 15:08:21.939497 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:08:21.939396 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zdpmz" Jul 21 15:08:22.069249 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:08:22.069094 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zdpmz"] Jul 21 15:08:22.071918 ip-10-0-135-176 kubenswrapper[2563]: W0721 15:08:22.071885 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod807e8526_473c_4511_8418_97eba53bebd8.slice/crio-57f4494e8b2e78c88e136b3f6b5e275f8d410d19421f79952d48a6867b39be47 WatchSource:0}: Error finding container 57f4494e8b2e78c88e136b3f6b5e275f8d410d19421f79952d48a6867b39be47: Status 404 returned error can't find the container with id 57f4494e8b2e78c88e136b3f6b5e275f8d410d19421f79952d48a6867b39be47 Jul 21 15:08:22.315816 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:08:22.315726 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zdpmz" event={"ID":"807e8526-473c-4511-8418-97eba53bebd8","Type":"ContainerStarted","Data":"57f4494e8b2e78c88e136b3f6b5e275f8d410d19421f79952d48a6867b39be47"} Jul 21 15:08:29.346736 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:08:29.346704 2563 generic.go:356] "Generic (PLEG): container finished" podID="807e8526-473c-4511-8418-97eba53bebd8" containerID="d03423457d3b3ee481d208b5c22156f42f671c2b586c7f94caf1da60630e6f68" exitCode=0 Jul 21 15:08:29.347219 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:08:29.346752 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zdpmz" event={"ID":"807e8526-473c-4511-8418-97eba53bebd8","Type":"ContainerDied","Data":"d03423457d3b3ee481d208b5c22156f42f671c2b586c7f94caf1da60630e6f68"} Jul 21 15:08:34.368980 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:08:34.368938 2563 generic.go:356] "Generic (PLEG): container finished" podID="807e8526-473c-4511-8418-97eba53bebd8" containerID="a5c2475e8c46fecb5665c4d16662b9e8005d93f9cfe9bdee01d0e49d76a1119c" exitCode=0 Jul 21 15:08:34.369447 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:08:34.369022 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zdpmz" event={"ID":"807e8526-473c-4511-8418-97eba53bebd8","Type":"ContainerDied","Data":"a5c2475e8c46fecb5665c4d16662b9e8005d93f9cfe9bdee01d0e49d76a1119c"} Jul 21 15:08:42.408435 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:08:42.408401 2563 generic.go:356] "Generic (PLEG): container finished" podID="807e8526-473c-4511-8418-97eba53bebd8" containerID="5de2c7f91597c43543d6417e27f8912dc16a6f87fcc6c5f54354165b465c7a94" exitCode=0 Jul 21 15:08:42.408822 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:08:42.408457 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zdpmz" event={"ID":"807e8526-473c-4511-8418-97eba53bebd8","Type":"ContainerDied","Data":"5de2c7f91597c43543d6417e27f8912dc16a6f87fcc6c5f54354165b465c7a94"} Jul 21 15:08:43.561114 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:08:43.561080 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zdpmz" Jul 21 15:08:43.606042 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:08:43.606002 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/807e8526-473c-4511-8418-97eba53bebd8-bundle\") pod \"807e8526-473c-4511-8418-97eba53bebd8\" (UID: \"807e8526-473c-4511-8418-97eba53bebd8\") " Jul 21 15:08:43.606192 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:08:43.606059 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gqpmm\" (UniqueName: \"kubernetes.io/projected/807e8526-473c-4511-8418-97eba53bebd8-kube-api-access-gqpmm\") pod \"807e8526-473c-4511-8418-97eba53bebd8\" (UID: \"807e8526-473c-4511-8418-97eba53bebd8\") " Jul 21 15:08:43.606192 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:08:43.606083 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/807e8526-473c-4511-8418-97eba53bebd8-util\") pod \"807e8526-473c-4511-8418-97eba53bebd8\" (UID: \"807e8526-473c-4511-8418-97eba53bebd8\") " Jul 21 15:08:43.607771 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:08:43.607161 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/807e8526-473c-4511-8418-97eba53bebd8-bundle" (OuterVolumeSpecName: "bundle") pod "807e8526-473c-4511-8418-97eba53bebd8" (UID: "807e8526-473c-4511-8418-97eba53bebd8"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 15:08:43.612352 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:08:43.609411 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/807e8526-473c-4511-8418-97eba53bebd8-kube-api-access-gqpmm" (OuterVolumeSpecName: "kube-api-access-gqpmm") pod "807e8526-473c-4511-8418-97eba53bebd8" (UID: "807e8526-473c-4511-8418-97eba53bebd8"). InnerVolumeSpecName "kube-api-access-gqpmm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 15:08:43.612824 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:08:43.612801 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/807e8526-473c-4511-8418-97eba53bebd8-util" (OuterVolumeSpecName: "util") pod "807e8526-473c-4511-8418-97eba53bebd8" (UID: "807e8526-473c-4511-8418-97eba53bebd8"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 15:08:43.707575 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:08:43.707492 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqpmm\" (UniqueName: \"kubernetes.io/projected/807e8526-473c-4511-8418-97eba53bebd8-kube-api-access-gqpmm\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 15:08:43.707575 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:08:43.707520 2563 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/807e8526-473c-4511-8418-97eba53bebd8-util\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 15:08:43.707575 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:08:43.707531 2563 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/807e8526-473c-4511-8418-97eba53bebd8-bundle\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 15:08:44.417628 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:08:44.417600 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zdpmz" Jul 21 15:08:44.417628 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:08:44.417603 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zdpmz" event={"ID":"807e8526-473c-4511-8418-97eba53bebd8","Type":"ContainerDied","Data":"57f4494e8b2e78c88e136b3f6b5e275f8d410d19421f79952d48a6867b39be47"} Jul 21 15:08:44.417827 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:08:44.417672 2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="57f4494e8b2e78c88e136b3f6b5e275f8d410d19421f79952d48a6867b39be47" Jul 21 15:08:49.702540 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:08:49.702505 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-sp862"] Jul 21 15:08:49.702993 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:08:49.702799 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="807e8526-473c-4511-8418-97eba53bebd8" containerName="util" Jul 21 15:08:49.702993 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:08:49.702809 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="807e8526-473c-4511-8418-97eba53bebd8" containerName="util" Jul 21 15:08:49.702993 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:08:49.702816 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="807e8526-473c-4511-8418-97eba53bebd8" containerName="extract" Jul 21 15:08:49.702993 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:08:49.702821 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="807e8526-473c-4511-8418-97eba53bebd8" containerName="extract" Jul 21 15:08:49.702993 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:08:49.702839 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="807e8526-473c-4511-8418-97eba53bebd8" containerName="pull" Jul 21 15:08:49.702993 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:08:49.702844 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="807e8526-473c-4511-8418-97eba53bebd8" containerName="pull" Jul 21 15:08:49.702993 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:08:49.702888 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="807e8526-473c-4511-8418-97eba53bebd8" containerName="extract" Jul 21 15:08:49.707616 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:08:49.707599 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-sp862" Jul 21 15:08:49.710856 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:08:49.710837 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Jul 21 15:08:49.710984 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:08:49.710866 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-h9zfd\"" Jul 21 15:08:49.710984 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:08:49.710870 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Jul 21 15:08:49.717072 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:08:49.717049 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-sp862"] Jul 21 15:08:49.758094 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:08:49.758057 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jprbh\" (UniqueName: \"kubernetes.io/projected/aea127a7-6c27-425d-b904-a3d79a6d393a-kube-api-access-jprbh\") pod \"cert-manager-operator-controller-manager-55c444bf86-sp862\" (UID: \"aea127a7-6c27-425d-b904-a3d79a6d393a\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-sp862" Jul 21 15:08:49.758265 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:08:49.758106 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aea127a7-6c27-425d-b904-a3d79a6d393a-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-sp862\" (UID: \"aea127a7-6c27-425d-b904-a3d79a6d393a\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-sp862" Jul 21 15:08:49.858898 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:08:49.858853 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aea127a7-6c27-425d-b904-a3d79a6d393a-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-sp862\" (UID: \"aea127a7-6c27-425d-b904-a3d79a6d393a\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-sp862" Jul 21 15:08:49.859118 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:08:49.858937 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jprbh\" (UniqueName: \"kubernetes.io/projected/aea127a7-6c27-425d-b904-a3d79a6d393a-kube-api-access-jprbh\") pod \"cert-manager-operator-controller-manager-55c444bf86-sp862\" (UID: \"aea127a7-6c27-425d-b904-a3d79a6d393a\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-sp862" Jul 21 15:08:49.859250 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:08:49.859233 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aea127a7-6c27-425d-b904-a3d79a6d393a-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-sp862\" (UID: \"aea127a7-6c27-425d-b904-a3d79a6d393a\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-sp862" Jul 21 15:08:49.871341 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:08:49.871295 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jprbh\" (UniqueName: \"kubernetes.io/projected/aea127a7-6c27-425d-b904-a3d79a6d393a-kube-api-access-jprbh\") pod \"cert-manager-operator-controller-manager-55c444bf86-sp862\" (UID: \"aea127a7-6c27-425d-b904-a3d79a6d393a\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-sp862" Jul 21 15:08:50.016456 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:08:50.016368 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-sp862" Jul 21 15:08:50.158084 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:08:50.157981 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-sp862"] Jul 21 15:08:50.161103 ip-10-0-135-176 kubenswrapper[2563]: W0721 15:08:50.161065 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaea127a7_6c27_425d_b904_a3d79a6d393a.slice/crio-fa0a50e3226f94fcf2b1773f4a6c1b5886f89288da07e6fe642f25a647415be3 WatchSource:0}: Error finding container fa0a50e3226f94fcf2b1773f4a6c1b5886f89288da07e6fe642f25a647415be3: Status 404 returned error can't find the container with id fa0a50e3226f94fcf2b1773f4a6c1b5886f89288da07e6fe642f25a647415be3 Jul 21 15:08:50.441074 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:08:50.441038 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-sp862" event={"ID":"aea127a7-6c27-425d-b904-a3d79a6d393a","Type":"ContainerStarted","Data":"fa0a50e3226f94fcf2b1773f4a6c1b5886f89288da07e6fe642f25a647415be3"} Jul 21 15:09:09.516319 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:09:09.516270 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-sp862" event={"ID":"aea127a7-6c27-425d-b904-a3d79a6d393a","Type":"ContainerStarted","Data":"b9a146f05faeaad7580ca27aaeaebe253510532056dedfd161ac42ac1a3dfb14"} Jul 21 15:09:09.551966 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:09:09.551907 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-sp862" podStartSLOduration=1.778008297 podStartE2EDuration="20.5518924s" totalImagesPullingTime="18.773884103s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 15:08:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 15:09:09.549836154 +0000 UTC m=+581.545621590" watchObservedRunningTime="2026-07-21 15:09:09.5518924 +0000 UTC m=+581.547677818" Jul 21 15:09:10.196295 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:09:10.196266 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzcvxq"] Jul 21 15:09:10.199807 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:09:10.199790 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzcvxq" Jul 21 15:09:10.202870 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:09:10.202848 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 21 15:09:10.203015 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:09:10.202994 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 21 15:09:10.204669 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:09:10.204652 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-lkwhz\"" Jul 21 15:09:10.213873 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:09:10.213853 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzcvxq"] Jul 21 15:09:10.322582 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:09:10.322538 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6wmm\" (UniqueName: \"kubernetes.io/projected/15d47518-d027-48fb-a395-3c5a3039f9bf-kube-api-access-h6wmm\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzcvxq\" (UID: \"15d47518-d027-48fb-a395-3c5a3039f9bf\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzcvxq" Jul 21 15:09:10.322788 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:09:10.322600 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/15d47518-d027-48fb-a395-3c5a3039f9bf-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzcvxq\" (UID: \"15d47518-d027-48fb-a395-3c5a3039f9bf\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzcvxq" Jul 21 15:09:10.322788 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:09:10.322726 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/15d47518-d027-48fb-a395-3c5a3039f9bf-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzcvxq\" (UID: \"15d47518-d027-48fb-a395-3c5a3039f9bf\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzcvxq" Jul 21 15:09:10.423612 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:09:10.423573 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/15d47518-d027-48fb-a395-3c5a3039f9bf-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzcvxq\" (UID: \"15d47518-d027-48fb-a395-3c5a3039f9bf\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzcvxq" Jul 21 15:09:10.423822 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:09:10.423628 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/15d47518-d027-48fb-a395-3c5a3039f9bf-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzcvxq\" (UID: \"15d47518-d027-48fb-a395-3c5a3039f9bf\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzcvxq" Jul 21 15:09:10.423822 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:09:10.423666 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6wmm\" (UniqueName: \"kubernetes.io/projected/15d47518-d027-48fb-a395-3c5a3039f9bf-kube-api-access-h6wmm\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzcvxq\" (UID: \"15d47518-d027-48fb-a395-3c5a3039f9bf\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzcvxq" Jul 21 15:09:10.424063 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:09:10.424039 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/15d47518-d027-48fb-a395-3c5a3039f9bf-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzcvxq\" (UID: \"15d47518-d027-48fb-a395-3c5a3039f9bf\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzcvxq" Jul 21 15:09:10.424137 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:09:10.424046 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/15d47518-d027-48fb-a395-3c5a3039f9bf-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzcvxq\" (UID: \"15d47518-d027-48fb-a395-3c5a3039f9bf\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzcvxq" Jul 21 15:09:10.432733 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:09:10.432702 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6wmm\" (UniqueName: \"kubernetes.io/projected/15d47518-d027-48fb-a395-3c5a3039f9bf-kube-api-access-h6wmm\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzcvxq\" (UID: \"15d47518-d027-48fb-a395-3c5a3039f9bf\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzcvxq" Jul 21 15:09:10.508759 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:09:10.508668 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzcvxq" Jul 21 15:09:10.638705 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:09:10.638682 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzcvxq"] Jul 21 15:09:10.641390 ip-10-0-135-176 kubenswrapper[2563]: W0721 15:09:10.641365 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15d47518_d027_48fb_a395_3c5a3039f9bf.slice/crio-9dd380f921905053b86332fcbcb1f8388b9af76eab9d8167c7be265cc12eeee3 WatchSource:0}: Error finding container 9dd380f921905053b86332fcbcb1f8388b9af76eab9d8167c7be265cc12eeee3: Status 404 returned error can't find the container with id 9dd380f921905053b86332fcbcb1f8388b9af76eab9d8167c7be265cc12eeee3 Jul 21 15:09:11.525675 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:09:11.525640 2563 generic.go:356] "Generic (PLEG): container finished" podID="15d47518-d027-48fb-a395-3c5a3039f9bf" containerID="a8c1098244937dbba11113f73d2070253f46c281d534617a2234f87dda1a87e5" exitCode=0 Jul 21 15:09:11.525821 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:09:11.525699 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzcvxq" event={"ID":"15d47518-d027-48fb-a395-3c5a3039f9bf","Type":"ContainerDied","Data":"a8c1098244937dbba11113f73d2070253f46c281d534617a2234f87dda1a87e5"} Jul 21 15:09:11.525821 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:09:11.525735 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzcvxq" event={"ID":"15d47518-d027-48fb-a395-3c5a3039f9bf","Type":"ContainerStarted","Data":"9dd380f921905053b86332fcbcb1f8388b9af76eab9d8167c7be265cc12eeee3"} Jul 21 15:09:15.543755 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:09:15.543716 2563 generic.go:356] "Generic (PLEG): container finished" podID="15d47518-d027-48fb-a395-3c5a3039f9bf" containerID="1f05282d69f8fd7fc372fa86be36a607b5297ca128f5dd17140296c9c80c562f" exitCode=0 Jul 21 15:09:15.544221 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:09:15.543810 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzcvxq" event={"ID":"15d47518-d027-48fb-a395-3c5a3039f9bf","Type":"ContainerDied","Data":"1f05282d69f8fd7fc372fa86be36a607b5297ca128f5dd17140296c9c80c562f"} Jul 21 15:09:16.550386 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:09:16.550349 2563 generic.go:356] "Generic (PLEG): container finished" podID="15d47518-d027-48fb-a395-3c5a3039f9bf" containerID="2c9915fa46bbcac7c58900aec5ae22c7c6ce864a5b34ec75ee7f8ac69e6a3bb2" exitCode=0 Jul 21 15:09:16.550763 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:09:16.550402 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzcvxq" event={"ID":"15d47518-d027-48fb-a395-3c5a3039f9bf","Type":"ContainerDied","Data":"2c9915fa46bbcac7c58900aec5ae22c7c6ce864a5b34ec75ee7f8ac69e6a3bb2"} Jul 21 15:09:17.675559 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:09:17.675536 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzcvxq" Jul 21 15:09:17.784647 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:09:17.784618 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/15d47518-d027-48fb-a395-3c5a3039f9bf-bundle\") pod \"15d47518-d027-48fb-a395-3c5a3039f9bf\" (UID: \"15d47518-d027-48fb-a395-3c5a3039f9bf\") " Jul 21 15:09:17.784834 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:09:17.784672 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h6wmm\" (UniqueName: \"kubernetes.io/projected/15d47518-d027-48fb-a395-3c5a3039f9bf-kube-api-access-h6wmm\") pod \"15d47518-d027-48fb-a395-3c5a3039f9bf\" (UID: \"15d47518-d027-48fb-a395-3c5a3039f9bf\") " Jul 21 15:09:17.784834 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:09:17.784689 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/15d47518-d027-48fb-a395-3c5a3039f9bf-util\") pod \"15d47518-d027-48fb-a395-3c5a3039f9bf\" (UID: \"15d47518-d027-48fb-a395-3c5a3039f9bf\") " Jul 21 15:09:17.785060 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:09:17.785030 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15d47518-d027-48fb-a395-3c5a3039f9bf-bundle" (OuterVolumeSpecName: "bundle") pod "15d47518-d027-48fb-a395-3c5a3039f9bf" (UID: "15d47518-d027-48fb-a395-3c5a3039f9bf"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 15:09:17.786913 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:09:17.786888 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15d47518-d027-48fb-a395-3c5a3039f9bf-kube-api-access-h6wmm" (OuterVolumeSpecName: "kube-api-access-h6wmm") pod "15d47518-d027-48fb-a395-3c5a3039f9bf" (UID: "15d47518-d027-48fb-a395-3c5a3039f9bf"). InnerVolumeSpecName "kube-api-access-h6wmm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 15:09:17.789018 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:09:17.788994 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15d47518-d027-48fb-a395-3c5a3039f9bf-util" (OuterVolumeSpecName: "util") pod "15d47518-d027-48fb-a395-3c5a3039f9bf" (UID: "15d47518-d027-48fb-a395-3c5a3039f9bf"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 15:09:17.885485 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:09:17.885435 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6wmm\" (UniqueName: \"kubernetes.io/projected/15d47518-d027-48fb-a395-3c5a3039f9bf-kube-api-access-h6wmm\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 15:09:17.885485 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:09:17.885480 2563 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/15d47518-d027-48fb-a395-3c5a3039f9bf-util\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 15:09:17.885485 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:09:17.885489 2563 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/15d47518-d027-48fb-a395-3c5a3039f9bf-bundle\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 15:09:18.202819 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:09:18.202740 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-d9rj5"] Jul 21 15:09:18.203090 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:09:18.203074 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15d47518-d027-48fb-a395-3c5a3039f9bf" containerName="pull" Jul 21 15:09:18.203167 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:09:18.203095 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="15d47518-d027-48fb-a395-3c5a3039f9bf" containerName="pull" Jul 21 15:09:18.203167 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:09:18.203116 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15d47518-d027-48fb-a395-3c5a3039f9bf" containerName="extract" Jul 21 15:09:18.203167 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:09:18.203125 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="15d47518-d027-48fb-a395-3c5a3039f9bf" containerName="extract" Jul 21 15:09:18.203167 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:09:18.203134 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15d47518-d027-48fb-a395-3c5a3039f9bf" containerName="util" Jul 21 15:09:18.203167 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:09:18.203143 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="15d47518-d027-48fb-a395-3c5a3039f9bf" containerName="util" Jul 21 15:09:18.203447 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:09:18.203213 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="15d47518-d027-48fb-a395-3c5a3039f9bf" containerName="extract" Jul 21 15:09:18.207530 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:09:18.207509 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-d9rj5" Jul 21 15:09:18.210450 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:09:18.210426 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 21 15:09:18.211756 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:09:18.211736 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-jlc4g\"" Jul 21 15:09:18.211756 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:09:18.211753 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 21 15:09:18.214085 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:09:18.214066 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-d9rj5"] Jul 21 15:09:18.388927 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:09:18.388887 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xd9lj\" (UniqueName: \"kubernetes.io/projected/f3b35b7f-5d94-4d46-b203-18962f6de819-kube-api-access-xd9lj\") pod \"cert-manager-webhook-6486ffd997-d9rj5\" (UID: \"f3b35b7f-5d94-4d46-b203-18962f6de819\") " pod="cert-manager/cert-manager-webhook-6486ffd997-d9rj5" Jul 21 15:09:18.388927 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:09:18.388926 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f3b35b7f-5d94-4d46-b203-18962f6de819-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-d9rj5\" (UID: \"f3b35b7f-5d94-4d46-b203-18962f6de819\") " pod="cert-manager/cert-manager-webhook-6486ffd997-d9rj5" Jul 21 15:09:18.489951 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:09:18.489921 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xd9lj\" (UniqueName: \"kubernetes.io/projected/f3b35b7f-5d94-4d46-b203-18962f6de819-kube-api-access-xd9lj\") pod \"cert-manager-webhook-6486ffd997-d9rj5\" (UID: \"f3b35b7f-5d94-4d46-b203-18962f6de819\") " pod="cert-manager/cert-manager-webhook-6486ffd997-d9rj5" Jul 21 15:09:18.490090 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:09:18.489962 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f3b35b7f-5d94-4d46-b203-18962f6de819-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-d9rj5\" (UID: \"f3b35b7f-5d94-4d46-b203-18962f6de819\") " pod="cert-manager/cert-manager-webhook-6486ffd997-d9rj5" Jul 21 15:09:18.499047 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:09:18.499026 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f3b35b7f-5d94-4d46-b203-18962f6de819-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-d9rj5\" (UID: \"f3b35b7f-5d94-4d46-b203-18962f6de819\") " pod="cert-manager/cert-manager-webhook-6486ffd997-d9rj5" Jul 21 15:09:18.499217 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:09:18.499197 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xd9lj\" (UniqueName: \"kubernetes.io/projected/f3b35b7f-5d94-4d46-b203-18962f6de819-kube-api-access-xd9lj\") pod \"cert-manager-webhook-6486ffd997-d9rj5\" (UID: \"f3b35b7f-5d94-4d46-b203-18962f6de819\") " pod="cert-manager/cert-manager-webhook-6486ffd997-d9rj5" Jul 21 15:09:18.517229 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:09:18.517191 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-d9rj5" Jul 21 15:09:18.561706 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:09:18.561664 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzcvxq" event={"ID":"15d47518-d027-48fb-a395-3c5a3039f9bf","Type":"ContainerDied","Data":"9dd380f921905053b86332fcbcb1f8388b9af76eab9d8167c7be265cc12eeee3"} Jul 21 15:09:18.561706 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:09:18.561696 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzcvxq" Jul 21 15:09:18.561961 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:09:18.561716 2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9dd380f921905053b86332fcbcb1f8388b9af76eab9d8167c7be265cc12eeee3" Jul 21 15:09:18.650686 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:09:18.650501 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-d9rj5"] Jul 21 15:09:18.652888 ip-10-0-135-176 kubenswrapper[2563]: W0721 15:09:18.652864 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf3b35b7f_5d94_4d46_b203_18962f6de819.slice/crio-d013000ad18cb752d0fda1c9aa97cc1134da17b04bd3dc556413ffc4062cc23e WatchSource:0}: Error finding container d013000ad18cb752d0fda1c9aa97cc1134da17b04bd3dc556413ffc4062cc23e: Status 404 returned error can't find the container with id d013000ad18cb752d0fda1c9aa97cc1134da17b04bd3dc556413ffc4062cc23e Jul 21 15:09:19.567362 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:09:19.567308 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-d9rj5" event={"ID":"f3b35b7f-5d94-4d46-b203-18962f6de819","Type":"ContainerStarted","Data":"d013000ad18cb752d0fda1c9aa97cc1134da17b04bd3dc556413ffc4062cc23e"} Jul 21 15:09:20.395052 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:09:20.395014 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-7kkcg"] Jul 21 15:09:20.398932 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:09:20.398907 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-7kkcg" Jul 21 15:09:20.402026 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:09:20.402001 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-6l5s7\"" Jul 21 15:09:20.410203 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:09:20.410180 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-7kkcg"] Jul 21 15:09:20.509763 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:09:20.509731 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1b7929ad-360a-477a-aaa9-1eb7b407001b-bound-sa-token\") pod \"cert-manager-55b4d954d5-7kkcg\" (UID: \"1b7929ad-360a-477a-aaa9-1eb7b407001b\") " pod="cert-manager/cert-manager-55b4d954d5-7kkcg" Jul 21 15:09:20.509763 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:09:20.509770 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5tlg\" (UniqueName: \"kubernetes.io/projected/1b7929ad-360a-477a-aaa9-1eb7b407001b-kube-api-access-s5tlg\") pod \"cert-manager-55b4d954d5-7kkcg\" (UID: \"1b7929ad-360a-477a-aaa9-1eb7b407001b\") " pod="cert-manager/cert-manager-55b4d954d5-7kkcg" Jul 21 15:09:20.610812 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:09:20.610749 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1b7929ad-360a-477a-aaa9-1eb7b407001b-bound-sa-token\") pod \"cert-manager-55b4d954d5-7kkcg\" (UID: \"1b7929ad-360a-477a-aaa9-1eb7b407001b\") " pod="cert-manager/cert-manager-55b4d954d5-7kkcg" Jul 21 15:09:20.610812 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:09:20.610794 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5tlg\" (UniqueName: \"kubernetes.io/projected/1b7929ad-360a-477a-aaa9-1eb7b407001b-kube-api-access-s5tlg\") pod \"cert-manager-55b4d954d5-7kkcg\" (UID: \"1b7929ad-360a-477a-aaa9-1eb7b407001b\") " pod="cert-manager/cert-manager-55b4d954d5-7kkcg" Jul 21 15:09:20.623033 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:09:20.623007 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1b7929ad-360a-477a-aaa9-1eb7b407001b-bound-sa-token\") pod \"cert-manager-55b4d954d5-7kkcg\" (UID: \"1b7929ad-360a-477a-aaa9-1eb7b407001b\") " pod="cert-manager/cert-manager-55b4d954d5-7kkcg" Jul 21 15:09:20.623167 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:09:20.623139 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5tlg\" (UniqueName: \"kubernetes.io/projected/1b7929ad-360a-477a-aaa9-1eb7b407001b-kube-api-access-s5tlg\") pod \"cert-manager-55b4d954d5-7kkcg\" (UID: \"1b7929ad-360a-477a-aaa9-1eb7b407001b\") " pod="cert-manager/cert-manager-55b4d954d5-7kkcg" Jul 21 15:09:20.711670 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:09:20.711585 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-7kkcg" Jul 21 15:09:20.878028 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:09:20.878000 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-7kkcg"] Jul 21 15:09:21.745371 ip-10-0-135-176 kubenswrapper[2563]: W0721 15:09:21.745319 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1b7929ad_360a_477a_aaa9_1eb7b407001b.slice/crio-7f9940f225af330134b2caa6c15544d70430bbf069575c6b524db624d9ee8358 WatchSource:0}: Error finding container 7f9940f225af330134b2caa6c15544d70430bbf069575c6b524db624d9ee8358: Status 404 returned error can't find the container with id 7f9940f225af330134b2caa6c15544d70430bbf069575c6b524db624d9ee8358 Jul 21 15:09:22.582160 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:09:22.582123 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-7kkcg" event={"ID":"1b7929ad-360a-477a-aaa9-1eb7b407001b","Type":"ContainerStarted","Data":"47a248f8d8a4c30cf6f21f684ca97a69ebfd3e00cf190223912c38a0cf657c22"} Jul 21 15:09:22.582386 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:09:22.582174 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-7kkcg" event={"ID":"1b7929ad-360a-477a-aaa9-1eb7b407001b","Type":"ContainerStarted","Data":"7f9940f225af330134b2caa6c15544d70430bbf069575c6b524db624d9ee8358"} Jul 21 15:09:22.583714 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:09:22.583687 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-d9rj5" event={"ID":"f3b35b7f-5d94-4d46-b203-18962f6de819","Type":"ContainerStarted","Data":"b89fda7ada351e4b0bda62e0568c8a990182dd5e8be8bd125773b6a60c607b65"} Jul 21 15:09:22.583851 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:09:22.583792 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-d9rj5" Jul 21 15:09:22.599073 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:09:22.598984 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-7kkcg" podStartSLOduration=2.009273883 podStartE2EDuration="2.598967435s" totalImagesPullingTime="589.693552ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 15:09:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 15:09:22.597526821 +0000 UTC m=+594.593312242" watchObservedRunningTime="2026-07-21 15:09:22.598967435 +0000 UTC m=+594.594752846" Jul 21 15:09:22.613602 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:09:22.613548 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-d9rj5" podStartSLOduration=1.463606083 podStartE2EDuration="4.613528489s" totalImagesPullingTime="3.149922406s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 15:09:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 15:09:22.612825448 +0000 UTC m=+594.608610866" watchObservedRunningTime="2026-07-21 15:09:22.613528489 +0000 UTC m=+594.609313905" Jul 21 15:09:27.951850 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:09:27.951818 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkhhld"] Jul 21 15:09:27.955318 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:09:27.955303 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkhhld" Jul 21 15:09:27.958112 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:09:27.958091 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 21 15:09:27.958234 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:09:27.958120 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 21 15:09:27.958234 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:09:27.958124 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-lkwhz\"" Jul 21 15:09:27.963232 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:09:27.963206 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkhhld"] Jul 21 15:09:27.972604 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:09:27.972581 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4c8fc9fe-1404-4a88-8b3b-137f2fc3aae0-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkhhld\" (UID: \"4c8fc9fe-1404-4a88-8b3b-137f2fc3aae0\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkhhld" Jul 21 15:09:27.972735 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:09:27.972616 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4c8fc9fe-1404-4a88-8b3b-137f2fc3aae0-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkhhld\" (UID: \"4c8fc9fe-1404-4a88-8b3b-137f2fc3aae0\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkhhld" Jul 21 15:09:27.972798 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:09:27.972783 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6dth\" (UniqueName: \"kubernetes.io/projected/4c8fc9fe-1404-4a88-8b3b-137f2fc3aae0-kube-api-access-m6dth\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkhhld\" (UID: \"4c8fc9fe-1404-4a88-8b3b-137f2fc3aae0\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkhhld" Jul 21 15:09:28.073431 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:09:28.073397 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6dth\" (UniqueName: \"kubernetes.io/projected/4c8fc9fe-1404-4a88-8b3b-137f2fc3aae0-kube-api-access-m6dth\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkhhld\" (UID: \"4c8fc9fe-1404-4a88-8b3b-137f2fc3aae0\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkhhld" Jul 21 15:09:28.073623 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:09:28.073449 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4c8fc9fe-1404-4a88-8b3b-137f2fc3aae0-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkhhld\" (UID: \"4c8fc9fe-1404-4a88-8b3b-137f2fc3aae0\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkhhld" Jul 21 15:09:28.073623 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:09:28.073469 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4c8fc9fe-1404-4a88-8b3b-137f2fc3aae0-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkhhld\" (UID: \"4c8fc9fe-1404-4a88-8b3b-137f2fc3aae0\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkhhld" Jul 21 15:09:28.073849 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:09:28.073827 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4c8fc9fe-1404-4a88-8b3b-137f2fc3aae0-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkhhld\" (UID: \"4c8fc9fe-1404-4a88-8b3b-137f2fc3aae0\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkhhld" Jul 21 15:09:28.073849 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:09:28.073841 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4c8fc9fe-1404-4a88-8b3b-137f2fc3aae0-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkhhld\" (UID: \"4c8fc9fe-1404-4a88-8b3b-137f2fc3aae0\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkhhld" Jul 21 15:09:28.083188 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:09:28.083159 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6dth\" (UniqueName: \"kubernetes.io/projected/4c8fc9fe-1404-4a88-8b3b-137f2fc3aae0-kube-api-access-m6dth\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkhhld\" (UID: \"4c8fc9fe-1404-4a88-8b3b-137f2fc3aae0\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkhhld" Jul 21 15:09:28.265919 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:09:28.265826 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkhhld" Jul 21 15:09:28.365110 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:09:28.365083 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kn9fn_02f3d0c6-4116-4134-ba51-1c02c73bef05/ovn-acl-logging/0.log" Jul 21 15:09:28.366136 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:09:28.366117 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kn9fn_02f3d0c6-4116-4134-ba51-1c02c73bef05/ovn-acl-logging/0.log" Jul 21 15:09:28.404609 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:09:28.404552 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkhhld"] Jul 21 15:09:28.407182 ip-10-0-135-176 kubenswrapper[2563]: W0721 15:09:28.407160 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c8fc9fe_1404_4a88_8b3b_137f2fc3aae0.slice/crio-0ce02e955239c8fef2a4ebba7a6130cbeb01b26bf4690c8b1197fc4fee466a5d WatchSource:0}: Error finding container 0ce02e955239c8fef2a4ebba7a6130cbeb01b26bf4690c8b1197fc4fee466a5d: Status 404 returned error can't find the container with id 0ce02e955239c8fef2a4ebba7a6130cbeb01b26bf4690c8b1197fc4fee466a5d Jul 21 15:09:28.589801 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:09:28.589779 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-d9rj5" Jul 21 15:09:28.610620 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:09:28.610585 2563 generic.go:356] "Generic (PLEG): container finished" podID="4c8fc9fe-1404-4a88-8b3b-137f2fc3aae0" containerID="d7b45d4c0d6111e7157b78d930b9d8415a7f1ec8aa4423301b4d7013f1be610e" exitCode=0 Jul 21 15:09:28.610755 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:09:28.610676 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkhhld" event={"ID":"4c8fc9fe-1404-4a88-8b3b-137f2fc3aae0","Type":"ContainerDied","Data":"d7b45d4c0d6111e7157b78d930b9d8415a7f1ec8aa4423301b4d7013f1be610e"} Jul 21 15:09:28.610755 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:09:28.610722 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkhhld" event={"ID":"4c8fc9fe-1404-4a88-8b3b-137f2fc3aae0","Type":"ContainerStarted","Data":"0ce02e955239c8fef2a4ebba7a6130cbeb01b26bf4690c8b1197fc4fee466a5d"} Jul 21 15:09:29.617289 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:09:29.617189 2563 generic.go:356] "Generic (PLEG): container finished" podID="4c8fc9fe-1404-4a88-8b3b-137f2fc3aae0" containerID="7b729fa1ce2520f08df4a6b809e1962c73797eaddfafdf18b3e8dfe5eea9bf21" exitCode=0 Jul 21 15:09:29.617289 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:09:29.617276 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkhhld" event={"ID":"4c8fc9fe-1404-4a88-8b3b-137f2fc3aae0","Type":"ContainerDied","Data":"7b729fa1ce2520f08df4a6b809e1962c73797eaddfafdf18b3e8dfe5eea9bf21"} Jul 21 15:09:30.624310 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:09:30.624264 2563 generic.go:356] "Generic (PLEG): container finished" podID="4c8fc9fe-1404-4a88-8b3b-137f2fc3aae0" containerID="c674107a1f9526016be246e0b4f8d397bd6b073fdefa63cdde165bc77835e1d2" exitCode=0 Jul 21 15:09:30.624721 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:09:30.624358 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkhhld" event={"ID":"4c8fc9fe-1404-4a88-8b3b-137f2fc3aae0","Type":"ContainerDied","Data":"c674107a1f9526016be246e0b4f8d397bd6b073fdefa63cdde165bc77835e1d2"} Jul 21 15:09:31.754990 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:09:31.754964 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkhhld" Jul 21 15:09:31.804604 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:09:31.804580 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4c8fc9fe-1404-4a88-8b3b-137f2fc3aae0-util\") pod \"4c8fc9fe-1404-4a88-8b3b-137f2fc3aae0\" (UID: \"4c8fc9fe-1404-4a88-8b3b-137f2fc3aae0\") " Jul 21 15:09:31.804752 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:09:31.804614 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m6dth\" (UniqueName: \"kubernetes.io/projected/4c8fc9fe-1404-4a88-8b3b-137f2fc3aae0-kube-api-access-m6dth\") pod \"4c8fc9fe-1404-4a88-8b3b-137f2fc3aae0\" (UID: \"4c8fc9fe-1404-4a88-8b3b-137f2fc3aae0\") " Jul 21 15:09:31.804752 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:09:31.804640 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4c8fc9fe-1404-4a88-8b3b-137f2fc3aae0-bundle\") pod \"4c8fc9fe-1404-4a88-8b3b-137f2fc3aae0\" (UID: \"4c8fc9fe-1404-4a88-8b3b-137f2fc3aae0\") " Jul 21 15:09:31.805387 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:09:31.805347 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c8fc9fe-1404-4a88-8b3b-137f2fc3aae0-bundle" (OuterVolumeSpecName: "bundle") pod "4c8fc9fe-1404-4a88-8b3b-137f2fc3aae0" (UID: "4c8fc9fe-1404-4a88-8b3b-137f2fc3aae0"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 15:09:31.806734 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:09:31.806710 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c8fc9fe-1404-4a88-8b3b-137f2fc3aae0-kube-api-access-m6dth" (OuterVolumeSpecName: "kube-api-access-m6dth") pod "4c8fc9fe-1404-4a88-8b3b-137f2fc3aae0" (UID: "4c8fc9fe-1404-4a88-8b3b-137f2fc3aae0"). InnerVolumeSpecName "kube-api-access-m6dth". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 15:09:31.809911 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:09:31.809876 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c8fc9fe-1404-4a88-8b3b-137f2fc3aae0-util" (OuterVolumeSpecName: "util") pod "4c8fc9fe-1404-4a88-8b3b-137f2fc3aae0" (UID: "4c8fc9fe-1404-4a88-8b3b-137f2fc3aae0"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 15:09:31.905382 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:09:31.905265 2563 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4c8fc9fe-1404-4a88-8b3b-137f2fc3aae0-util\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 15:09:31.905382 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:09:31.905296 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6dth\" (UniqueName: \"kubernetes.io/projected/4c8fc9fe-1404-4a88-8b3b-137f2fc3aae0-kube-api-access-m6dth\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 15:09:31.905382 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:09:31.905306 2563 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4c8fc9fe-1404-4a88-8b3b-137f2fc3aae0-bundle\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 15:09:32.634573 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:09:32.634543 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkhhld" event={"ID":"4c8fc9fe-1404-4a88-8b3b-137f2fc3aae0","Type":"ContainerDied","Data":"0ce02e955239c8fef2a4ebba7a6130cbeb01b26bf4690c8b1197fc4fee466a5d"} Jul 21 15:09:32.634573 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:09:32.634563 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkhhld" Jul 21 15:09:32.634781 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:09:32.634587 2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0ce02e955239c8fef2a4ebba7a6130cbeb01b26bf4690c8b1197fc4fee466a5d" Jul 21 15:09:41.392235 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:09:41.392201 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-85b74964cc-5fl76"] Jul 21 15:09:41.392627 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:09:41.392522 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c8fc9fe-1404-4a88-8b3b-137f2fc3aae0" containerName="pull" Jul 21 15:09:41.392627 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:09:41.392533 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c8fc9fe-1404-4a88-8b3b-137f2fc3aae0" containerName="pull" Jul 21 15:09:41.392627 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:09:41.392541 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c8fc9fe-1404-4a88-8b3b-137f2fc3aae0" containerName="extract" Jul 21 15:09:41.392627 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:09:41.392546 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c8fc9fe-1404-4a88-8b3b-137f2fc3aae0" containerName="extract" Jul 21 15:09:41.392627 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:09:41.392555 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c8fc9fe-1404-4a88-8b3b-137f2fc3aae0" containerName="util" Jul 21 15:09:41.392627 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:09:41.392560 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c8fc9fe-1404-4a88-8b3b-137f2fc3aae0" containerName="util" Jul 21 15:09:41.392627 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:09:41.392602 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c8fc9fe-1404-4a88-8b3b-137f2fc3aae0" containerName="extract" Jul 21 15:09:41.398302 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:09:41.398286 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-85b74964cc-5fl76" Jul 21 15:09:41.401319 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:09:41.401298 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 21 15:09:41.401319 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:09:41.401302 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 21 15:09:41.402140 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:09:41.402114 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 21 15:09:41.402770 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:09:41.402755 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 21 15:09:41.402871 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:09:41.402794 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-2sbsb\"" Jul 21 15:09:41.414721 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:09:41.414697 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-85b74964cc-5fl76"] Jul 21 15:09:41.479891 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:09:41.479857 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bf078948-6939-460b-99f5-12aba0a5e33d-webhook-cert\") pod \"opendatahub-operator-controller-manager-85b74964cc-5fl76\" (UID: \"bf078948-6939-460b-99f5-12aba0a5e33d\") " pod="opendatahub/opendatahub-operator-controller-manager-85b74964cc-5fl76" Jul 21 15:09:41.479891 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:09:41.479891 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/bf078948-6939-460b-99f5-12aba0a5e33d-manifests\") pod \"opendatahub-operator-controller-manager-85b74964cc-5fl76\" (UID: \"bf078948-6939-460b-99f5-12aba0a5e33d\") " pod="opendatahub/opendatahub-operator-controller-manager-85b74964cc-5fl76" Jul 21 15:09:41.480135 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:09:41.479928 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bf078948-6939-460b-99f5-12aba0a5e33d-tmp\") pod \"opendatahub-operator-controller-manager-85b74964cc-5fl76\" (UID: \"bf078948-6939-460b-99f5-12aba0a5e33d\") " pod="opendatahub/opendatahub-operator-controller-manager-85b74964cc-5fl76" Jul 21 15:09:41.480135 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:09:41.479981 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/bf078948-6939-460b-99f5-12aba0a5e33d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-85b74964cc-5fl76\" (UID: \"bf078948-6939-460b-99f5-12aba0a5e33d\") " pod="opendatahub/opendatahub-operator-controller-manager-85b74964cc-5fl76" Jul 21 15:09:41.480135 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:09:41.480043 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxn4p\" (UniqueName: \"kubernetes.io/projected/bf078948-6939-460b-99f5-12aba0a5e33d-kube-api-access-jxn4p\") pod \"opendatahub-operator-controller-manager-85b74964cc-5fl76\" (UID: \"bf078948-6939-460b-99f5-12aba0a5e33d\") " pod="opendatahub/opendatahub-operator-controller-manager-85b74964cc-5fl76" Jul 21 15:09:41.581050 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:09:41.581014 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bf078948-6939-460b-99f5-12aba0a5e33d-tmp\") pod \"opendatahub-operator-controller-manager-85b74964cc-5fl76\" (UID: \"bf078948-6939-460b-99f5-12aba0a5e33d\") " pod="opendatahub/opendatahub-operator-controller-manager-85b74964cc-5fl76" Jul 21 15:09:41.581050 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:09:41.581051 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/bf078948-6939-460b-99f5-12aba0a5e33d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-85b74964cc-5fl76\" (UID: \"bf078948-6939-460b-99f5-12aba0a5e33d\") " pod="opendatahub/opendatahub-operator-controller-manager-85b74964cc-5fl76" Jul 21 15:09:41.581297 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:09:41.581144 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxn4p\" (UniqueName: \"kubernetes.io/projected/bf078948-6939-460b-99f5-12aba0a5e33d-kube-api-access-jxn4p\") pod \"opendatahub-operator-controller-manager-85b74964cc-5fl76\" (UID: \"bf078948-6939-460b-99f5-12aba0a5e33d\") " pod="opendatahub/opendatahub-operator-controller-manager-85b74964cc-5fl76" Jul 21 15:09:41.581297 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:09:41.581228 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bf078948-6939-460b-99f5-12aba0a5e33d-webhook-cert\") pod \"opendatahub-operator-controller-manager-85b74964cc-5fl76\" (UID: \"bf078948-6939-460b-99f5-12aba0a5e33d\") " pod="opendatahub/opendatahub-operator-controller-manager-85b74964cc-5fl76" Jul 21 15:09:41.581297 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:09:41.581258 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/bf078948-6939-460b-99f5-12aba0a5e33d-manifests\") pod \"opendatahub-operator-controller-manager-85b74964cc-5fl76\" (UID: \"bf078948-6939-460b-99f5-12aba0a5e33d\") " pod="opendatahub/opendatahub-operator-controller-manager-85b74964cc-5fl76" Jul 21 15:09:41.581675 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:09:41.581640 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/bf078948-6939-460b-99f5-12aba0a5e33d-manifests\") pod \"opendatahub-operator-controller-manager-85b74964cc-5fl76\" (UID: \"bf078948-6939-460b-99f5-12aba0a5e33d\") " pod="opendatahub/opendatahub-operator-controller-manager-85b74964cc-5fl76" Jul 21 15:09:41.583547 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:09:41.583529 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bf078948-6939-460b-99f5-12aba0a5e33d-tmp\") pod \"opendatahub-operator-controller-manager-85b74964cc-5fl76\" (UID: \"bf078948-6939-460b-99f5-12aba0a5e33d\") " pod="opendatahub/opendatahub-operator-controller-manager-85b74964cc-5fl76" Jul 21 15:09:41.583829 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:09:41.583812 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/bf078948-6939-460b-99f5-12aba0a5e33d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-85b74964cc-5fl76\" (UID: \"bf078948-6939-460b-99f5-12aba0a5e33d\") " pod="opendatahub/opendatahub-operator-controller-manager-85b74964cc-5fl76" Jul 21 15:09:41.583877 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:09:41.583859 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bf078948-6939-460b-99f5-12aba0a5e33d-webhook-cert\") pod \"opendatahub-operator-controller-manager-85b74964cc-5fl76\" (UID: \"bf078948-6939-460b-99f5-12aba0a5e33d\") " pod="opendatahub/opendatahub-operator-controller-manager-85b74964cc-5fl76" Jul 21 15:09:41.590036 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:09:41.590014 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxn4p\" (UniqueName: \"kubernetes.io/projected/bf078948-6939-460b-99f5-12aba0a5e33d-kube-api-access-jxn4p\") pod \"opendatahub-operator-controller-manager-85b74964cc-5fl76\" (UID: \"bf078948-6939-460b-99f5-12aba0a5e33d\") " pod="opendatahub/opendatahub-operator-controller-manager-85b74964cc-5fl76" Jul 21 15:09:41.708747 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:09:41.708659 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-85b74964cc-5fl76" Jul 21 15:09:41.847840 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:09:41.847816 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-85b74964cc-5fl76"] Jul 21 15:09:41.850021 ip-10-0-135-176 kubenswrapper[2563]: W0721 15:09:41.849979 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf078948_6939_460b_99f5_12aba0a5e33d.slice/crio-d42b027df283d5e66014b79bc4525f3e2e72b1767633d630732636f7807408aa WatchSource:0}: Error finding container d42b027df283d5e66014b79bc4525f3e2e72b1767633d630732636f7807408aa: Status 404 returned error can't find the container with id d42b027df283d5e66014b79bc4525f3e2e72b1767633d630732636f7807408aa Jul 21 15:09:42.679571 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:09:42.679534 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-85b74964cc-5fl76" event={"ID":"bf078948-6939-460b-99f5-12aba0a5e33d","Type":"ContainerStarted","Data":"d42b027df283d5e66014b79bc4525f3e2e72b1767633d630732636f7807408aa"} Jul 21 15:09:46.708679 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:09:46.708636 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-85b74964cc-5fl76" event={"ID":"bf078948-6939-460b-99f5-12aba0a5e33d","Type":"ContainerStarted","Data":"2a9f060e591106a9f0414e1ec85cb47d41026eabe3b84aaa5be2f3f05c26ee93"} Jul 21 15:09:47.714295 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:09:47.714260 2563 generic.go:356] "Generic (PLEG): container finished" podID="bf078948-6939-460b-99f5-12aba0a5e33d" containerID="2a9f060e591106a9f0414e1ec85cb47d41026eabe3b84aaa5be2f3f05c26ee93" exitCode=0 Jul 21 15:09:47.714726 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:09:47.714355 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-85b74964cc-5fl76" event={"ID":"bf078948-6939-460b-99f5-12aba0a5e33d","Type":"ContainerDied","Data":"2a9f060e591106a9f0414e1ec85cb47d41026eabe3b84aaa5be2f3f05c26ee93"} Jul 21 15:09:48.721238 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:09:48.721204 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-85b74964cc-5fl76" event={"ID":"bf078948-6939-460b-99f5-12aba0a5e33d","Type":"ContainerStarted","Data":"b38228dc78dc1e6855c7076c0e5630ae84ba108436b8ccee9ebbd44e77744f04"} Jul 21 15:09:48.721635 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:09:48.721453 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-85b74964cc-5fl76" Jul 21 15:09:48.744865 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:09:48.744819 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-85b74964cc-5fl76" podStartSLOduration=1.6783124630000001 podStartE2EDuration="7.744804639s" totalImagesPullingTime="5.015209545s" totalInitContainerRuntime="1.051282631s" isStatefulPod=false podCreationTimestamp="2026-07-21 15:09:41 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 15:09:48.742686275 +0000 UTC m=+620.738471691" watchObservedRunningTime="2026-07-21 15:09:48.744804639 +0000 UTC m=+620.740590056" Jul 21 15:09:59.728169 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:09:59.728134 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-85b74964cc-5fl76" Jul 21 15:10:01.949406 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:01.949376 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7db96775b-tckp8"] Jul 21 15:10:01.959124 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:01.959103 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7db96775b-tckp8" Jul 21 15:10:01.962982 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:01.962965 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 21 15:10:01.963089 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:01.963074 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 21 15:10:01.964018 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:01.964006 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-h9hwh\"" Jul 21 15:10:01.968500 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:01.968480 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7db96775b-tckp8"] Jul 21 15:10:02.055133 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:02.055102 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fmpr\" (UniqueName: \"kubernetes.io/projected/6a378105-bb9e-4ca6-9887-8a0f3d6c6c94-kube-api-access-7fmpr\") pod \"kube-auth-proxy-7db96775b-tckp8\" (UID: \"6a378105-bb9e-4ca6-9887-8a0f3d6c6c94\") " pod="openshift-ingress/kube-auth-proxy-7db96775b-tckp8" Jul 21 15:10:02.055320 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:02.055181 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a378105-bb9e-4ca6-9887-8a0f3d6c6c94-tls-certs\") pod \"kube-auth-proxy-7db96775b-tckp8\" (UID: \"6a378105-bb9e-4ca6-9887-8a0f3d6c6c94\") " pod="openshift-ingress/kube-auth-proxy-7db96775b-tckp8" Jul 21 15:10:02.055320 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:02.055208 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6a378105-bb9e-4ca6-9887-8a0f3d6c6c94-tmp\") pod \"kube-auth-proxy-7db96775b-tckp8\" (UID: \"6a378105-bb9e-4ca6-9887-8a0f3d6c6c94\") " pod="openshift-ingress/kube-auth-proxy-7db96775b-tckp8" Jul 21 15:10:02.146722 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:02.146691 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-7pcnb"] Jul 21 15:10:02.149912 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:02.149896 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-7pcnb" Jul 21 15:10:02.155418 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:02.155384 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 21 15:10:02.155636 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:02.155615 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a378105-bb9e-4ca6-9887-8a0f3d6c6c94-tls-certs\") pod \"kube-auth-proxy-7db96775b-tckp8\" (UID: \"6a378105-bb9e-4ca6-9887-8a0f3d6c6c94\") " pod="openshift-ingress/kube-auth-proxy-7db96775b-tckp8" Jul 21 15:10:02.155715 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:02.155645 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6a378105-bb9e-4ca6-9887-8a0f3d6c6c94-tmp\") pod \"kube-auth-proxy-7db96775b-tckp8\" (UID: \"6a378105-bb9e-4ca6-9887-8a0f3d6c6c94\") " pod="openshift-ingress/kube-auth-proxy-7db96775b-tckp8" Jul 21 15:10:02.155715 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:02.155679 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fmpr\" (UniqueName: \"kubernetes.io/projected/6a378105-bb9e-4ca6-9887-8a0f3d6c6c94-kube-api-access-7fmpr\") pod \"kube-auth-proxy-7db96775b-tckp8\" (UID: \"6a378105-bb9e-4ca6-9887-8a0f3d6c6c94\") " pod="openshift-ingress/kube-auth-proxy-7db96775b-tckp8" Jul 21 15:10:02.158117 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:02.158095 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6a378105-bb9e-4ca6-9887-8a0f3d6c6c94-tmp\") pod \"kube-auth-proxy-7db96775b-tckp8\" (UID: \"6a378105-bb9e-4ca6-9887-8a0f3d6c6c94\") " pod="openshift-ingress/kube-auth-proxy-7db96775b-tckp8" Jul 21 15:10:02.158234 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:02.158217 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a378105-bb9e-4ca6-9887-8a0f3d6c6c94-tls-certs\") pod \"kube-auth-proxy-7db96775b-tckp8\" (UID: \"6a378105-bb9e-4ca6-9887-8a0f3d6c6c94\") " pod="openshift-ingress/kube-auth-proxy-7db96775b-tckp8" Jul 21 15:10:02.158763 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:02.158744 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-tkql4\"" Jul 21 15:10:02.168806 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:02.168787 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-7pcnb"] Jul 21 15:10:02.173908 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:02.173884 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fmpr\" (UniqueName: \"kubernetes.io/projected/6a378105-bb9e-4ca6-9887-8a0f3d6c6c94-kube-api-access-7fmpr\") pod \"kube-auth-proxy-7db96775b-tckp8\" (UID: \"6a378105-bb9e-4ca6-9887-8a0f3d6c6c94\") " pod="openshift-ingress/kube-auth-proxy-7db96775b-tckp8" Jul 21 15:10:02.256117 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:02.256031 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcvpm\" (UniqueName: \"kubernetes.io/projected/3a2888a0-305c-4d9a-9f53-46bed879f2c3-kube-api-access-kcvpm\") pod \"odh-model-controller-7998c67dc4-7pcnb\" (UID: \"3a2888a0-305c-4d9a-9f53-46bed879f2c3\") " pod="opendatahub/odh-model-controller-7998c67dc4-7pcnb" Jul 21 15:10:02.256117 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:02.256066 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3a2888a0-305c-4d9a-9f53-46bed879f2c3-cert\") pod \"odh-model-controller-7998c67dc4-7pcnb\" (UID: \"3a2888a0-305c-4d9a-9f53-46bed879f2c3\") " pod="opendatahub/odh-model-controller-7998c67dc4-7pcnb" Jul 21 15:10:02.270002 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:02.269979 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7db96775b-tckp8" Jul 21 15:10:02.356871 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:02.356826 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcvpm\" (UniqueName: \"kubernetes.io/projected/3a2888a0-305c-4d9a-9f53-46bed879f2c3-kube-api-access-kcvpm\") pod \"odh-model-controller-7998c67dc4-7pcnb\" (UID: \"3a2888a0-305c-4d9a-9f53-46bed879f2c3\") " pod="opendatahub/odh-model-controller-7998c67dc4-7pcnb" Jul 21 15:10:02.356871 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:02.356867 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3a2888a0-305c-4d9a-9f53-46bed879f2c3-cert\") pod \"odh-model-controller-7998c67dc4-7pcnb\" (UID: \"3a2888a0-305c-4d9a-9f53-46bed879f2c3\") " pod="opendatahub/odh-model-controller-7998c67dc4-7pcnb" Jul 21 15:10:02.357066 ip-10-0-135-176 kubenswrapper[2563]: E0721 15:10:02.356988 2563 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Jul 21 15:10:02.357066 ip-10-0-135-176 kubenswrapper[2563]: E0721 15:10:02.357045 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a2888a0-305c-4d9a-9f53-46bed879f2c3-cert podName:3a2888a0-305c-4d9a-9f53-46bed879f2c3 nodeName:}" failed. No retries permitted until 2026-07-21 15:10:02.857028217 +0000 UTC m=+634.852813616 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/3a2888a0-305c-4d9a-9f53-46bed879f2c3-cert") pod "odh-model-controller-7998c67dc4-7pcnb" (UID: "3a2888a0-305c-4d9a-9f53-46bed879f2c3") : secret "odh-model-controller-webhook-cert" not found Jul 21 15:10:02.368604 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:02.368584 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcvpm\" (UniqueName: \"kubernetes.io/projected/3a2888a0-305c-4d9a-9f53-46bed879f2c3-kube-api-access-kcvpm\") pod \"odh-model-controller-7998c67dc4-7pcnb\" (UID: \"3a2888a0-305c-4d9a-9f53-46bed879f2c3\") " pod="opendatahub/odh-model-controller-7998c67dc4-7pcnb" Jul 21 15:10:02.409188 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:02.409163 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7db96775b-tckp8"] Jul 21 15:10:02.411350 ip-10-0-135-176 kubenswrapper[2563]: W0721 15:10:02.411295 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a378105_bb9e_4ca6_9887_8a0f3d6c6c94.slice/crio-2ba49d57cc92e8952afa612f2a83aab094b3da314afdfce1b2f65d81abfd893e WatchSource:0}: Error finding container 2ba49d57cc92e8952afa612f2a83aab094b3da314afdfce1b2f65d81abfd893e: Status 404 returned error can't find the container with id 2ba49d57cc92e8952afa612f2a83aab094b3da314afdfce1b2f65d81abfd893e Jul 21 15:10:02.785342 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:02.785297 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7db96775b-tckp8" event={"ID":"6a378105-bb9e-4ca6-9887-8a0f3d6c6c94","Type":"ContainerStarted","Data":"2ba49d57cc92e8952afa612f2a83aab094b3da314afdfce1b2f65d81abfd893e"} Jul 21 15:10:02.862019 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:02.861985 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3a2888a0-305c-4d9a-9f53-46bed879f2c3-cert\") pod \"odh-model-controller-7998c67dc4-7pcnb\" (UID: \"3a2888a0-305c-4d9a-9f53-46bed879f2c3\") " pod="opendatahub/odh-model-controller-7998c67dc4-7pcnb" Jul 21 15:10:02.862211 ip-10-0-135-176 kubenswrapper[2563]: E0721 15:10:02.862161 2563 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Jul 21 15:10:02.862282 ip-10-0-135-176 kubenswrapper[2563]: E0721 15:10:02.862236 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a2888a0-305c-4d9a-9f53-46bed879f2c3-cert podName:3a2888a0-305c-4d9a-9f53-46bed879f2c3 nodeName:}" failed. No retries permitted until 2026-07-21 15:10:03.862214305 +0000 UTC m=+635.857999701 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/3a2888a0-305c-4d9a-9f53-46bed879f2c3-cert") pod "odh-model-controller-7998c67dc4-7pcnb" (UID: "3a2888a0-305c-4d9a-9f53-46bed879f2c3") : secret "odh-model-controller-webhook-cert" not found Jul 21 15:10:03.293852 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:03.293819 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-rlxc7"] Jul 21 15:10:03.298618 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:03.298602 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-rlxc7" Jul 21 15:10:03.302586 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:03.302563 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 21 15:10:03.302785 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:03.302764 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 21 15:10:03.302897 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:03.302804 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 21 15:10:03.302897 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:03.302819 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-c2lcg\"" Jul 21 15:10:03.303022 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:03.303010 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 21 15:10:03.339270 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:03.339246 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-rlxc7"] Jul 21 15:10:03.467643 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:03.467618 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhqb8\" (UniqueName: \"kubernetes.io/projected/8814b301-5245-4a11-801d-592088a1b625-kube-api-access-mhqb8\") pod \"istiod-openshift-gateway-7574459656-rlxc7\" (UID: \"8814b301-5245-4a11-801d-592088a1b625\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-rlxc7" Jul 21 15:10:03.467818 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:03.467662 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8814b301-5245-4a11-801d-592088a1b625-istio-token\") pod \"istiod-openshift-gateway-7574459656-rlxc7\" (UID: \"8814b301-5245-4a11-801d-592088a1b625\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-rlxc7" Jul 21 15:10:03.467818 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:03.467691 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8814b301-5245-4a11-801d-592088a1b625-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-rlxc7\" (UID: \"8814b301-5245-4a11-801d-592088a1b625\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-rlxc7" Jul 21 15:10:03.467818 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:03.467714 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8814b301-5245-4a11-801d-592088a1b625-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-rlxc7\" (UID: \"8814b301-5245-4a11-801d-592088a1b625\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-rlxc7" Jul 21 15:10:03.467818 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:03.467792 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/8814b301-5245-4a11-801d-592088a1b625-cacerts\") pod \"istiod-openshift-gateway-7574459656-rlxc7\" (UID: \"8814b301-5245-4a11-801d-592088a1b625\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-rlxc7" Jul 21 15:10:03.467956 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:03.467840 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8814b301-5245-4a11-801d-592088a1b625-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-rlxc7\" (UID: \"8814b301-5245-4a11-801d-592088a1b625\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-rlxc7" Jul 21 15:10:03.467956 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:03.467894 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8814b301-5245-4a11-801d-592088a1b625-local-certs\") pod \"istiod-openshift-gateway-7574459656-rlxc7\" (UID: \"8814b301-5245-4a11-801d-592088a1b625\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-rlxc7" Jul 21 15:10:03.569181 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:03.569090 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8814b301-5245-4a11-801d-592088a1b625-local-certs\") pod \"istiod-openshift-gateway-7574459656-rlxc7\" (UID: \"8814b301-5245-4a11-801d-592088a1b625\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-rlxc7" Jul 21 15:10:03.569377 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:03.569254 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhqb8\" (UniqueName: \"kubernetes.io/projected/8814b301-5245-4a11-801d-592088a1b625-kube-api-access-mhqb8\") pod \"istiod-openshift-gateway-7574459656-rlxc7\" (UID: \"8814b301-5245-4a11-801d-592088a1b625\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-rlxc7" Jul 21 15:10:03.569466 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:03.569388 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8814b301-5245-4a11-801d-592088a1b625-istio-token\") pod \"istiod-openshift-gateway-7574459656-rlxc7\" (UID: \"8814b301-5245-4a11-801d-592088a1b625\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-rlxc7" Jul 21 15:10:03.569466 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:03.569426 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8814b301-5245-4a11-801d-592088a1b625-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-rlxc7\" (UID: \"8814b301-5245-4a11-801d-592088a1b625\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-rlxc7" Jul 21 15:10:03.569466 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:03.569456 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8814b301-5245-4a11-801d-592088a1b625-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-rlxc7\" (UID: \"8814b301-5245-4a11-801d-592088a1b625\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-rlxc7" Jul 21 15:10:03.569632 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:03.569618 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/8814b301-5245-4a11-801d-592088a1b625-cacerts\") pod \"istiod-openshift-gateway-7574459656-rlxc7\" (UID: \"8814b301-5245-4a11-801d-592088a1b625\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-rlxc7" Jul 21 15:10:03.569733 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:03.569703 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8814b301-5245-4a11-801d-592088a1b625-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-rlxc7\" (UID: \"8814b301-5245-4a11-801d-592088a1b625\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-rlxc7" Jul 21 15:10:03.570250 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:03.570186 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8814b301-5245-4a11-801d-592088a1b625-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-rlxc7\" (UID: \"8814b301-5245-4a11-801d-592088a1b625\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-rlxc7" Jul 21 15:10:03.571990 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:03.571967 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/8814b301-5245-4a11-801d-592088a1b625-cacerts\") pod \"istiod-openshift-gateway-7574459656-rlxc7\" (UID: \"8814b301-5245-4a11-801d-592088a1b625\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-rlxc7" Jul 21 15:10:03.571990 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:03.571977 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8814b301-5245-4a11-801d-592088a1b625-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-rlxc7\" (UID: \"8814b301-5245-4a11-801d-592088a1b625\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-rlxc7" Jul 21 15:10:03.572157 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:03.572075 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8814b301-5245-4a11-801d-592088a1b625-local-certs\") pod \"istiod-openshift-gateway-7574459656-rlxc7\" (UID: \"8814b301-5245-4a11-801d-592088a1b625\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-rlxc7" Jul 21 15:10:03.572279 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:03.572260 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8814b301-5245-4a11-801d-592088a1b625-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-rlxc7\" (UID: \"8814b301-5245-4a11-801d-592088a1b625\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-rlxc7" Jul 21 15:10:03.578014 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:03.577994 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhqb8\" (UniqueName: \"kubernetes.io/projected/8814b301-5245-4a11-801d-592088a1b625-kube-api-access-mhqb8\") pod \"istiod-openshift-gateway-7574459656-rlxc7\" (UID: \"8814b301-5245-4a11-801d-592088a1b625\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-rlxc7" Jul 21 15:10:03.578131 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:03.578115 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8814b301-5245-4a11-801d-592088a1b625-istio-token\") pod \"istiod-openshift-gateway-7574459656-rlxc7\" (UID: \"8814b301-5245-4a11-801d-592088a1b625\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-rlxc7" Jul 21 15:10:03.607874 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:03.607851 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-rlxc7" Jul 21 15:10:03.743412 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:03.743387 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-rlxc7"] Jul 21 15:10:03.745174 ip-10-0-135-176 kubenswrapper[2563]: W0721 15:10:03.745147 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8814b301_5245_4a11_801d_592088a1b625.slice/crio-b17fd24381eea63f4c79684d8c95d2d59b2cf09b9dc40a5a428d0d8a8c3e8261 WatchSource:0}: Error finding container b17fd24381eea63f4c79684d8c95d2d59b2cf09b9dc40a5a428d0d8a8c3e8261: Status 404 returned error can't find the container with id b17fd24381eea63f4c79684d8c95d2d59b2cf09b9dc40a5a428d0d8a8c3e8261 Jul 21 15:10:03.790486 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:03.790455 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-rlxc7" event={"ID":"8814b301-5245-4a11-801d-592088a1b625","Type":"ContainerStarted","Data":"b17fd24381eea63f4c79684d8c95d2d59b2cf09b9dc40a5a428d0d8a8c3e8261"} Jul 21 15:10:03.872516 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:03.872486 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3a2888a0-305c-4d9a-9f53-46bed879f2c3-cert\") pod \"odh-model-controller-7998c67dc4-7pcnb\" (UID: \"3a2888a0-305c-4d9a-9f53-46bed879f2c3\") " pod="opendatahub/odh-model-controller-7998c67dc4-7pcnb" Jul 21 15:10:03.874895 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:03.874876 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3a2888a0-305c-4d9a-9f53-46bed879f2c3-cert\") pod \"odh-model-controller-7998c67dc4-7pcnb\" (UID: \"3a2888a0-305c-4d9a-9f53-46bed879f2c3\") " pod="opendatahub/odh-model-controller-7998c67dc4-7pcnb" Jul 21 15:10:03.960536 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:03.960499 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-7pcnb" Jul 21 15:10:04.121788 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:04.121664 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-7pcnb"] Jul 21 15:10:04.124302 ip-10-0-135-176 kubenswrapper[2563]: W0721 15:10:04.124234 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a2888a0_305c_4d9a_9f53_46bed879f2c3.slice/crio-2199e408ea0dc8a6034a0654a87d988f874815d7e3b8e224c7b857d7aafc0185 WatchSource:0}: Error finding container 2199e408ea0dc8a6034a0654a87d988f874815d7e3b8e224c7b857d7aafc0185: Status 404 returned error can't find the container with id 2199e408ea0dc8a6034a0654a87d988f874815d7e3b8e224c7b857d7aafc0185 Jul 21 15:10:04.805914 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:04.805874 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-7pcnb" event={"ID":"3a2888a0-305c-4d9a-9f53-46bed879f2c3","Type":"ContainerStarted","Data":"2199e408ea0dc8a6034a0654a87d988f874815d7e3b8e224c7b857d7aafc0185"} Jul 21 15:10:07.000754 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:07.000715 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 21 15:10:07.001141 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:07.000797 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 21 15:10:07.823815 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:07.823774 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-rlxc7" event={"ID":"8814b301-5245-4a11-801d-592088a1b625","Type":"ContainerStarted","Data":"1a2e2398e58a361f4c85f4e0c505ed7801128d19d27ec5f3ed5d9aa8766f4e17"} Jul 21 15:10:07.824004 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:07.823861 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-rlxc7" Jul 21 15:10:07.828367 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:07.828311 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7db96775b-tckp8" event={"ID":"6a378105-bb9e-4ca6-9887-8a0f3d6c6c94","Type":"ContainerStarted","Data":"5136c754690e1b0fc7668dddb0e81cd1a177fddb2baa810037ec862d883f8c08"} Jul 21 15:10:07.852483 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:07.852439 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7574459656-rlxc7" podStartSLOduration=1.599053303 podStartE2EDuration="4.852422491s" totalImagesPullingTime="3.253369188s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 15:10:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 15:10:07.851690644 +0000 UTC m=+639.847476086" watchObservedRunningTime="2026-07-21 15:10:07.852422491 +0000 UTC m=+639.848207912" Jul 21 15:10:07.873956 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:07.873901 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7db96775b-tckp8" podStartSLOduration=2.288134418 podStartE2EDuration="6.873881927s" totalImagesPullingTime="4.585747509s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 15:10:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 15:10:07.872550817 +0000 UTC m=+639.868336235" watchObservedRunningTime="2026-07-21 15:10:07.873881927 +0000 UTC m=+639.869667361" Jul 21 15:10:08.836385 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:08.836350 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-rlxc7" Jul 21 15:10:08.917967 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:08.916913 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678qkp8b"] Jul 21 15:10:08.920825 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:08.920800 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678qkp8b" Jul 21 15:10:08.924983 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:08.924959 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-97bgb\"" Jul 21 15:10:08.936087 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:08.936045 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678qkp8b"] Jul 21 15:10:09.018337 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:09.018296 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72f38b39-1295-435c-b9ef-fc2c03990034-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678qkp8b\" (UID: \"72f38b39-1295-435c-b9ef-fc2c03990034\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678qkp8b" Jul 21 15:10:09.018439 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:09.018357 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72f38b39-1295-435c-b9ef-fc2c03990034-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678qkp8b\" (UID: \"72f38b39-1295-435c-b9ef-fc2c03990034\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678qkp8b" Jul 21 15:10:09.018439 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:09.018396 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72f38b39-1295-435c-b9ef-fc2c03990034-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678qkp8b\" (UID: \"72f38b39-1295-435c-b9ef-fc2c03990034\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678qkp8b" Jul 21 15:10:09.018533 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:09.018464 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2rcs\" (UniqueName: \"kubernetes.io/projected/72f38b39-1295-435c-b9ef-fc2c03990034-kube-api-access-v2rcs\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678qkp8b\" (UID: \"72f38b39-1295-435c-b9ef-fc2c03990034\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678qkp8b" Jul 21 15:10:09.018580 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:09.018538 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72f38b39-1295-435c-b9ef-fc2c03990034-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678qkp8b\" (UID: \"72f38b39-1295-435c-b9ef-fc2c03990034\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678qkp8b" Jul 21 15:10:09.018580 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:09.018567 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72f38b39-1295-435c-b9ef-fc2c03990034-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678qkp8b\" (UID: \"72f38b39-1295-435c-b9ef-fc2c03990034\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678qkp8b" Jul 21 15:10:09.018667 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:09.018629 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72f38b39-1295-435c-b9ef-fc2c03990034-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678qkp8b\" (UID: \"72f38b39-1295-435c-b9ef-fc2c03990034\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678qkp8b" Jul 21 15:10:09.018667 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:09.018650 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72f38b39-1295-435c-b9ef-fc2c03990034-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678qkp8b\" (UID: \"72f38b39-1295-435c-b9ef-fc2c03990034\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678qkp8b" Jul 21 15:10:09.018776 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:09.018675 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72f38b39-1295-435c-b9ef-fc2c03990034-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678qkp8b\" (UID: \"72f38b39-1295-435c-b9ef-fc2c03990034\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678qkp8b" Jul 21 15:10:09.119948 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:09.119869 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72f38b39-1295-435c-b9ef-fc2c03990034-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678qkp8b\" (UID: \"72f38b39-1295-435c-b9ef-fc2c03990034\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678qkp8b" Jul 21 15:10:09.119948 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:09.119926 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72f38b39-1295-435c-b9ef-fc2c03990034-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678qkp8b\" (UID: \"72f38b39-1295-435c-b9ef-fc2c03990034\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678qkp8b" Jul 21 15:10:09.120209 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:09.119982 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72f38b39-1295-435c-b9ef-fc2c03990034-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678qkp8b\" (UID: \"72f38b39-1295-435c-b9ef-fc2c03990034\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678qkp8b" Jul 21 15:10:09.120209 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:09.120028 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72f38b39-1295-435c-b9ef-fc2c03990034-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678qkp8b\" (UID: \"72f38b39-1295-435c-b9ef-fc2c03990034\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678qkp8b" Jul 21 15:10:09.120209 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:09.120064 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72f38b39-1295-435c-b9ef-fc2c03990034-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678qkp8b\" (UID: \"72f38b39-1295-435c-b9ef-fc2c03990034\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678qkp8b" Jul 21 15:10:09.120209 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:09.120086 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72f38b39-1295-435c-b9ef-fc2c03990034-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678qkp8b\" (UID: \"72f38b39-1295-435c-b9ef-fc2c03990034\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678qkp8b" Jul 21 15:10:09.120209 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:09.120115 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2rcs\" (UniqueName: \"kubernetes.io/projected/72f38b39-1295-435c-b9ef-fc2c03990034-kube-api-access-v2rcs\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678qkp8b\" (UID: \"72f38b39-1295-435c-b9ef-fc2c03990034\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678qkp8b" Jul 21 15:10:09.120209 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:09.120186 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72f38b39-1295-435c-b9ef-fc2c03990034-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678qkp8b\" (UID: \"72f38b39-1295-435c-b9ef-fc2c03990034\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678qkp8b" Jul 21 15:10:09.120586 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:09.120253 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72f38b39-1295-435c-b9ef-fc2c03990034-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678qkp8b\" (UID: \"72f38b39-1295-435c-b9ef-fc2c03990034\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678qkp8b" Jul 21 15:10:09.120586 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:09.120304 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72f38b39-1295-435c-b9ef-fc2c03990034-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678qkp8b\" (UID: \"72f38b39-1295-435c-b9ef-fc2c03990034\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678qkp8b" Jul 21 15:10:09.120586 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:09.120446 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72f38b39-1295-435c-b9ef-fc2c03990034-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678qkp8b\" (UID: \"72f38b39-1295-435c-b9ef-fc2c03990034\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678qkp8b" Jul 21 15:10:09.120803 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:09.120756 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72f38b39-1295-435c-b9ef-fc2c03990034-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678qkp8b\" (UID: \"72f38b39-1295-435c-b9ef-fc2c03990034\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678qkp8b" Jul 21 15:10:09.120928 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:09.120808 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72f38b39-1295-435c-b9ef-fc2c03990034-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678qkp8b\" (UID: \"72f38b39-1295-435c-b9ef-fc2c03990034\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678qkp8b" Jul 21 15:10:09.121154 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:09.121126 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72f38b39-1295-435c-b9ef-fc2c03990034-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678qkp8b\" (UID: \"72f38b39-1295-435c-b9ef-fc2c03990034\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678qkp8b" Jul 21 15:10:09.122377 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:09.122356 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72f38b39-1295-435c-b9ef-fc2c03990034-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678qkp8b\" (UID: \"72f38b39-1295-435c-b9ef-fc2c03990034\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678qkp8b" Jul 21 15:10:09.122687 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:09.122662 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72f38b39-1295-435c-b9ef-fc2c03990034-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678qkp8b\" (UID: \"72f38b39-1295-435c-b9ef-fc2c03990034\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678qkp8b" Jul 21 15:10:09.134181 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:09.134161 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72f38b39-1295-435c-b9ef-fc2c03990034-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678qkp8b\" (UID: \"72f38b39-1295-435c-b9ef-fc2c03990034\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678qkp8b" Jul 21 15:10:09.134932 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:09.134903 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2rcs\" (UniqueName: \"kubernetes.io/projected/72f38b39-1295-435c-b9ef-fc2c03990034-kube-api-access-v2rcs\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678qkp8b\" (UID: \"72f38b39-1295-435c-b9ef-fc2c03990034\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678qkp8b" Jul 21 15:10:09.234684 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:09.234652 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678qkp8b" Jul 21 15:10:09.412054 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:09.412029 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678qkp8b"] Jul 21 15:10:09.414414 ip-10-0-135-176 kubenswrapper[2563]: W0721 15:10:09.414390 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72f38b39_1295_435c_b9ef_fc2c03990034.slice/crio-a81d4ba3bb5220a1491524b7142baaae71f5527b3d82daff1cde83bd233d3d72 WatchSource:0}: Error finding container a81d4ba3bb5220a1491524b7142baaae71f5527b3d82daff1cde83bd233d3d72: Status 404 returned error can't find the container with id a81d4ba3bb5220a1491524b7142baaae71f5527b3d82daff1cde83bd233d3d72 Jul 21 15:10:09.845827 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:09.845787 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678qkp8b" event={"ID":"72f38b39-1295-435c-b9ef-fc2c03990034","Type":"ContainerStarted","Data":"a81d4ba3bb5220a1491524b7142baaae71f5527b3d82daff1cde83bd233d3d72"} Jul 21 15:10:09.847407 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:09.847360 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-7pcnb" event={"ID":"3a2888a0-305c-4d9a-9f53-46bed879f2c3","Type":"ContainerStarted","Data":"107fa1e1d15964b43bf8563f5590a4e0b92dd130523d2bb631a3e69cd5d19bc2"} Jul 21 15:10:09.865775 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:09.865728 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-7pcnb" podStartSLOduration=2.993394308 podStartE2EDuration="7.865713144s" totalImagesPullingTime="4.872318836s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 15:10:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 15:10:09.865489092 +0000 UTC m=+641.861274512" watchObservedRunningTime="2026-07-21 15:10:09.865713144 +0000 UTC m=+641.861498561" Jul 21 15:10:10.854105 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:10.854075 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-7pcnb" Jul 21 15:10:12.002810 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:12.002755 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 21 15:10:12.003164 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:12.002850 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 21 15:10:12.003164 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:12.002892 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 21 15:10:12.857020 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:12.856984 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-jrpr5"] Jul 21 15:10:12.860213 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:12.860196 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-jrpr5" Jul 21 15:10:12.864009 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:12.863991 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 21 15:10:12.864118 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:12.864029 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-gjm79\"" Jul 21 15:10:12.871485 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:12.871461 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-mfgpf"] Jul 21 15:10:12.875085 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:12.875060 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678qkp8b" event={"ID":"72f38b39-1295-435c-b9ef-fc2c03990034","Type":"ContainerStarted","Data":"47d5df859b600edb9e496ecb7da664e7a665f1136fe51deb9549bff0ddc8b6ec"} Jul 21 15:10:12.875192 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:12.875175 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-mfgpf" Jul 21 15:10:12.875634 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:12.875614 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-jrpr5"] Jul 21 15:10:12.879887 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:12.879868 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-j8lmt\"" Jul 21 15:10:12.879987 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:12.879909 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 21 15:10:12.895507 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:12.895485 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-mfgpf"] Jul 21 15:10:12.931066 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:12.930992 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678qkp8b" podStartSLOduration=2.344987493 podStartE2EDuration="4.930975012s" totalImagesPullingTime="2.585987519s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 15:10:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 15:10:12.930072221 +0000 UTC m=+644.925857639" watchObservedRunningTime="2026-07-21 15:10:12.930975012 +0000 UTC m=+644.926760431" Jul 21 15:10:12.952825 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:12.952790 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/41551a7a-bb35-4aac-8b24-d5a407c9364d-cert\") pod \"kserve-controller-manager-d6f4b65b8-jrpr5\" (UID: \"41551a7a-bb35-4aac-8b24-d5a407c9364d\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-jrpr5" Jul 21 15:10:12.952966 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:12.952844 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2v5f\" (UniqueName: \"kubernetes.io/projected/41551a7a-bb35-4aac-8b24-d5a407c9364d-kube-api-access-d2v5f\") pod \"kserve-controller-manager-d6f4b65b8-jrpr5\" (UID: \"41551a7a-bb35-4aac-8b24-d5a407c9364d\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-jrpr5" Jul 21 15:10:13.054149 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:13.054118 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2v5f\" (UniqueName: \"kubernetes.io/projected/41551a7a-bb35-4aac-8b24-d5a407c9364d-kube-api-access-d2v5f\") pod \"kserve-controller-manager-d6f4b65b8-jrpr5\" (UID: \"41551a7a-bb35-4aac-8b24-d5a407c9364d\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-jrpr5" Jul 21 15:10:13.054644 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:13.054179 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2vzn\" (UniqueName: \"kubernetes.io/projected/027898ef-0c27-4aa0-84d8-71a02817fb7d-kube-api-access-r2vzn\") pod \"llmisvc-controller-manager-777c6bd7d5-mfgpf\" (UID: \"027898ef-0c27-4aa0-84d8-71a02817fb7d\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-mfgpf" Jul 21 15:10:13.054644 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:13.054238 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/41551a7a-bb35-4aac-8b24-d5a407c9364d-cert\") pod \"kserve-controller-manager-d6f4b65b8-jrpr5\" (UID: \"41551a7a-bb35-4aac-8b24-d5a407c9364d\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-jrpr5" Jul 21 15:10:13.054644 ip-10-0-135-176 kubenswrapper[2563]: E0721 15:10:13.054306 2563 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found Jul 21 15:10:13.054644 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:13.054310 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/027898ef-0c27-4aa0-84d8-71a02817fb7d-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-mfgpf\" (UID: \"027898ef-0c27-4aa0-84d8-71a02817fb7d\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-mfgpf" Jul 21 15:10:13.054644 ip-10-0-135-176 kubenswrapper[2563]: E0721 15:10:13.054395 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41551a7a-bb35-4aac-8b24-d5a407c9364d-cert podName:41551a7a-bb35-4aac-8b24-d5a407c9364d nodeName:}" failed. No retries permitted until 2026-07-21 15:10:13.554378178 +0000 UTC m=+645.550163574 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/41551a7a-bb35-4aac-8b24-d5a407c9364d-cert") pod "kserve-controller-manager-d6f4b65b8-jrpr5" (UID: "41551a7a-bb35-4aac-8b24-d5a407c9364d") : secret "kserve-webhook-server-cert" not found Jul 21 15:10:13.077340 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:13.077297 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2v5f\" (UniqueName: \"kubernetes.io/projected/41551a7a-bb35-4aac-8b24-d5a407c9364d-kube-api-access-d2v5f\") pod \"kserve-controller-manager-d6f4b65b8-jrpr5\" (UID: \"41551a7a-bb35-4aac-8b24-d5a407c9364d\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-jrpr5" Jul 21 15:10:13.155508 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:13.155422 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/027898ef-0c27-4aa0-84d8-71a02817fb7d-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-mfgpf\" (UID: \"027898ef-0c27-4aa0-84d8-71a02817fb7d\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-mfgpf" Jul 21 15:10:13.155653 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:13.155506 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2vzn\" (UniqueName: \"kubernetes.io/projected/027898ef-0c27-4aa0-84d8-71a02817fb7d-kube-api-access-r2vzn\") pod \"llmisvc-controller-manager-777c6bd7d5-mfgpf\" (UID: \"027898ef-0c27-4aa0-84d8-71a02817fb7d\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-mfgpf" Jul 21 15:10:13.155653 ip-10-0-135-176 kubenswrapper[2563]: E0721 15:10:13.155570 2563 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found Jul 21 15:10:13.155653 ip-10-0-135-176 kubenswrapper[2563]: E0721 15:10:13.155643 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/027898ef-0c27-4aa0-84d8-71a02817fb7d-cert podName:027898ef-0c27-4aa0-84d8-71a02817fb7d nodeName:}" failed. No retries permitted until 2026-07-21 15:10:13.655626785 +0000 UTC m=+645.651412180 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/027898ef-0c27-4aa0-84d8-71a02817fb7d-cert") pod "llmisvc-controller-manager-777c6bd7d5-mfgpf" (UID: "027898ef-0c27-4aa0-84d8-71a02817fb7d") : secret "llmisvc-webhook-server-cert" not found Jul 21 15:10:13.179647 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:13.179624 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2vzn\" (UniqueName: \"kubernetes.io/projected/027898ef-0c27-4aa0-84d8-71a02817fb7d-kube-api-access-r2vzn\") pod \"llmisvc-controller-manager-777c6bd7d5-mfgpf\" (UID: \"027898ef-0c27-4aa0-84d8-71a02817fb7d\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-mfgpf" Jul 21 15:10:13.235004 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:13.234978 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678qkp8b" Jul 21 15:10:13.239677 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:13.239653 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678qkp8b" Jul 21 15:10:13.558863 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:13.558767 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/41551a7a-bb35-4aac-8b24-d5a407c9364d-cert\") pod \"kserve-controller-manager-d6f4b65b8-jrpr5\" (UID: \"41551a7a-bb35-4aac-8b24-d5a407c9364d\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-jrpr5" Jul 21 15:10:13.561243 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:13.561218 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/41551a7a-bb35-4aac-8b24-d5a407c9364d-cert\") pod \"kserve-controller-manager-d6f4b65b8-jrpr5\" (UID: \"41551a7a-bb35-4aac-8b24-d5a407c9364d\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-jrpr5" Jul 21 15:10:13.659453 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:13.659409 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/027898ef-0c27-4aa0-84d8-71a02817fb7d-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-mfgpf\" (UID: \"027898ef-0c27-4aa0-84d8-71a02817fb7d\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-mfgpf" Jul 21 15:10:13.661894 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:13.661867 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/027898ef-0c27-4aa0-84d8-71a02817fb7d-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-mfgpf\" (UID: \"027898ef-0c27-4aa0-84d8-71a02817fb7d\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-mfgpf" Jul 21 15:10:13.771535 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:13.771501 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-jrpr5" Jul 21 15:10:13.784762 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:13.784732 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-mfgpf" Jul 21 15:10:13.879574 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:13.879543 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678qkp8b" Jul 21 15:10:13.880510 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:13.880484 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678qkp8b" Jul 21 15:10:13.923913 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:13.923867 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-jrpr5"] Jul 21 15:10:13.926036 ip-10-0-135-176 kubenswrapper[2563]: W0721 15:10:13.926005 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41551a7a_bb35_4aac_8b24_d5a407c9364d.slice/crio-a7c004e7b8b1a56a61810a54ded1b5fed3b1ef8ef267ea7002606301f322e757 WatchSource:0}: Error finding container a7c004e7b8b1a56a61810a54ded1b5fed3b1ef8ef267ea7002606301f322e757: Status 404 returned error can't find the container with id a7c004e7b8b1a56a61810a54ded1b5fed3b1ef8ef267ea7002606301f322e757 Jul 21 15:10:13.952858 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:13.952833 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-mfgpf"] Jul 21 15:10:13.955022 ip-10-0-135-176 kubenswrapper[2563]: W0721 15:10:13.954998 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod027898ef_0c27_4aa0_84d8_71a02817fb7d.slice/crio-168f47f3d420e5c47e90ebaf8fb0910c36ad3a6532a4f0d22c6ef8ef114aa64c WatchSource:0}: Error finding container 168f47f3d420e5c47e90ebaf8fb0910c36ad3a6532a4f0d22c6ef8ef114aa64c: Status 404 returned error can't find the container with id 168f47f3d420e5c47e90ebaf8fb0910c36ad3a6532a4f0d22c6ef8ef114aa64c Jul 21 15:10:14.885454 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:14.885419 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-jrpr5" event={"ID":"41551a7a-bb35-4aac-8b24-d5a407c9364d","Type":"ContainerStarted","Data":"a7c004e7b8b1a56a61810a54ded1b5fed3b1ef8ef267ea7002606301f322e757"} Jul 21 15:10:14.886585 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:14.886549 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-mfgpf" event={"ID":"027898ef-0c27-4aa0-84d8-71a02817fb7d","Type":"ContainerStarted","Data":"168f47f3d420e5c47e90ebaf8fb0910c36ad3a6532a4f0d22c6ef8ef114aa64c"} Jul 21 15:10:15.975066 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:15.975033 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-rlxc7"] Jul 21 15:10:15.975461 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:15.975213 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-rlxc7" podUID="8814b301-5245-4a11-801d-592088a1b625" containerName="discovery" containerID="cri-o://1a2e2398e58a361f4c85f4e0c505ed7801128d19d27ec5f3ed5d9aa8766f4e17" gracePeriod=30 Jul 21 15:10:16.686396 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:16.686373 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-rlxc7" Jul 21 15:10:16.788725 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:16.788703 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8814b301-5245-4a11-801d-592088a1b625-istio-csr-ca-configmap\") pod \"8814b301-5245-4a11-801d-592088a1b625\" (UID: \"8814b301-5245-4a11-801d-592088a1b625\") " Jul 21 15:10:16.788825 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:16.788758 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8814b301-5245-4a11-801d-592088a1b625-istio-token\") pod \"8814b301-5245-4a11-801d-592088a1b625\" (UID: \"8814b301-5245-4a11-801d-592088a1b625\") " Jul 21 15:10:16.788825 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:16.788787 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8814b301-5245-4a11-801d-592088a1b625-istio-csr-dns-cert\") pod \"8814b301-5245-4a11-801d-592088a1b625\" (UID: \"8814b301-5245-4a11-801d-592088a1b625\") " Jul 21 15:10:16.788919 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:16.788823 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8814b301-5245-4a11-801d-592088a1b625-local-certs\") pod \"8814b301-5245-4a11-801d-592088a1b625\" (UID: \"8814b301-5245-4a11-801d-592088a1b625\") " Jul 21 15:10:16.788919 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:16.788852 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mhqb8\" (UniqueName: \"kubernetes.io/projected/8814b301-5245-4a11-801d-592088a1b625-kube-api-access-mhqb8\") pod \"8814b301-5245-4a11-801d-592088a1b625\" (UID: \"8814b301-5245-4a11-801d-592088a1b625\") " Jul 21 15:10:16.788992 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:16.788951 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8814b301-5245-4a11-801d-592088a1b625-istio-kubeconfig\") pod \"8814b301-5245-4a11-801d-592088a1b625\" (UID: \"8814b301-5245-4a11-801d-592088a1b625\") " Jul 21 15:10:16.789040 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:16.788995 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/8814b301-5245-4a11-801d-592088a1b625-cacerts\") pod \"8814b301-5245-4a11-801d-592088a1b625\" (UID: \"8814b301-5245-4a11-801d-592088a1b625\") " Jul 21 15:10:16.789094 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:16.789082 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8814b301-5245-4a11-801d-592088a1b625-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "8814b301-5245-4a11-801d-592088a1b625" (UID: "8814b301-5245-4a11-801d-592088a1b625"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 15:10:16.789417 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:16.789231 2563 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8814b301-5245-4a11-801d-592088a1b625-istio-csr-ca-configmap\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 15:10:16.791530 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:16.791487 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8814b301-5245-4a11-801d-592088a1b625-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "8814b301-5245-4a11-801d-592088a1b625" (UID: "8814b301-5245-4a11-801d-592088a1b625"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 15:10:16.791621 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:16.791540 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8814b301-5245-4a11-801d-592088a1b625-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "8814b301-5245-4a11-801d-592088a1b625" (UID: "8814b301-5245-4a11-801d-592088a1b625"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 15:10:16.791816 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:16.791786 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8814b301-5245-4a11-801d-592088a1b625-istio-token" (OuterVolumeSpecName: "istio-token") pod "8814b301-5245-4a11-801d-592088a1b625" (UID: "8814b301-5245-4a11-801d-592088a1b625"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 15:10:16.792031 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:16.792010 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8814b301-5245-4a11-801d-592088a1b625-cacerts" (OuterVolumeSpecName: "cacerts") pod "8814b301-5245-4a11-801d-592088a1b625" (UID: "8814b301-5245-4a11-801d-592088a1b625"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 15:10:16.792164 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:16.792145 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8814b301-5245-4a11-801d-592088a1b625-local-certs" (OuterVolumeSpecName: "local-certs") pod "8814b301-5245-4a11-801d-592088a1b625" (UID: "8814b301-5245-4a11-801d-592088a1b625"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 15:10:16.792241 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:16.792158 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8814b301-5245-4a11-801d-592088a1b625-kube-api-access-mhqb8" (OuterVolumeSpecName: "kube-api-access-mhqb8") pod "8814b301-5245-4a11-801d-592088a1b625" (UID: "8814b301-5245-4a11-801d-592088a1b625"). InnerVolumeSpecName "kube-api-access-mhqb8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 15:10:16.890484 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:16.890458 2563 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8814b301-5245-4a11-801d-592088a1b625-istio-kubeconfig\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 15:10:16.890484 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:16.890484 2563 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/8814b301-5245-4a11-801d-592088a1b625-cacerts\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 15:10:16.890658 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:16.890494 2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8814b301-5245-4a11-801d-592088a1b625-istio-token\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 15:10:16.890658 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:16.890503 2563 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8814b301-5245-4a11-801d-592088a1b625-istio-csr-dns-cert\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 15:10:16.890658 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:16.890517 2563 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8814b301-5245-4a11-801d-592088a1b625-local-certs\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 15:10:16.890658 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:16.890527 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhqb8\" (UniqueName: \"kubernetes.io/projected/8814b301-5245-4a11-801d-592088a1b625-kube-api-access-mhqb8\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 15:10:16.898591 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:16.898561 2563 generic.go:356] "Generic (PLEG): container finished" podID="8814b301-5245-4a11-801d-592088a1b625" containerID="1a2e2398e58a361f4c85f4e0c505ed7801128d19d27ec5f3ed5d9aa8766f4e17" exitCode=0 Jul 21 15:10:16.898725 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:16.898612 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-rlxc7" Jul 21 15:10:16.898725 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:16.898651 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-rlxc7" event={"ID":"8814b301-5245-4a11-801d-592088a1b625","Type":"ContainerDied","Data":"1a2e2398e58a361f4c85f4e0c505ed7801128d19d27ec5f3ed5d9aa8766f4e17"} Jul 21 15:10:16.898725 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:16.898698 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-rlxc7" event={"ID":"8814b301-5245-4a11-801d-592088a1b625","Type":"ContainerDied","Data":"b17fd24381eea63f4c79684d8c95d2d59b2cf09b9dc40a5a428d0d8a8c3e8261"} Jul 21 15:10:16.898934 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:16.898734 2563 scope.go:117] "RemoveContainer" containerID="1a2e2398e58a361f4c85f4e0c505ed7801128d19d27ec5f3ed5d9aa8766f4e17" Jul 21 15:10:16.900491 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:16.900467 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-jrpr5" event={"ID":"41551a7a-bb35-4aac-8b24-d5a407c9364d","Type":"ContainerStarted","Data":"46ea66c3adb0c2a2a4f2a0ca95a0a016d884a85bd92a31e923aad911476dab58"} Jul 21 15:10:16.900617 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:16.900603 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-jrpr5" Jul 21 15:10:16.907539 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:16.907516 2563 scope.go:117] "RemoveContainer" containerID="1a2e2398e58a361f4c85f4e0c505ed7801128d19d27ec5f3ed5d9aa8766f4e17" Jul 21 15:10:16.907781 ip-10-0-135-176 kubenswrapper[2563]: E0721 15:10:16.907762 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a2e2398e58a361f4c85f4e0c505ed7801128d19d27ec5f3ed5d9aa8766f4e17\": container with ID starting with 1a2e2398e58a361f4c85f4e0c505ed7801128d19d27ec5f3ed5d9aa8766f4e17 not found: ID does not exist" containerID="1a2e2398e58a361f4c85f4e0c505ed7801128d19d27ec5f3ed5d9aa8766f4e17" Jul 21 15:10:16.907837 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:16.907789 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a2e2398e58a361f4c85f4e0c505ed7801128d19d27ec5f3ed5d9aa8766f4e17"} err="failed to get container status \"1a2e2398e58a361f4c85f4e0c505ed7801128d19d27ec5f3ed5d9aa8766f4e17\": rpc error: code = NotFound desc = could not find container \"1a2e2398e58a361f4c85f4e0c505ed7801128d19d27ec5f3ed5d9aa8766f4e17\": container with ID starting with 1a2e2398e58a361f4c85f4e0c505ed7801128d19d27ec5f3ed5d9aa8766f4e17 not found: ID does not exist" Jul 21 15:10:16.937313 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:16.937256 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-rlxc7"] Jul 21 15:10:16.947982 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:16.947957 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-rlxc7"] Jul 21 15:10:18.484061 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:18.483403 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8814b301-5245-4a11-801d-592088a1b625" path="/var/lib/kubelet/pods/8814b301-5245-4a11-801d-592088a1b625/volumes" Jul 21 15:10:19.922263 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:19.922182 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-mfgpf" event={"ID":"027898ef-0c27-4aa0-84d8-71a02817fb7d","Type":"ContainerStarted","Data":"4b8429ef6728466438a6473bef02430be99304517143d25e0637182c431b77c1"} Jul 21 15:10:19.922685 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:19.922311 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-mfgpf" Jul 21 15:10:19.950736 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:19.950688 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-jrpr5" podStartSLOduration=5.158552095 podStartE2EDuration="7.950672477s" totalImagesPullingTime="2.792120382s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 15:10:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 15:10:17.035858799 +0000 UTC m=+649.031644216" watchObservedRunningTime="2026-07-21 15:10:19.950672477 +0000 UTC m=+651.946457891" Jul 21 15:10:19.950874 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:19.950769 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-mfgpf" podStartSLOduration=2.254021324 podStartE2EDuration="7.950760209s" totalImagesPullingTime="5.696738885s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 15:10:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 15:10:19.947419247 +0000 UTC m=+651.943204684" watchObservedRunningTime="2026-07-21 15:10:19.950760209 +0000 UTC m=+651.946545626" Jul 21 15:10:21.867798 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:21.867773 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-7pcnb" Jul 21 15:10:46.218192 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:46.218163 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-8558d588c-xbtgr"] Jul 21 15:10:46.218656 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:46.218552 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8814b301-5245-4a11-801d-592088a1b625" containerName="discovery" Jul 21 15:10:46.218656 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:46.218564 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8814b301-5245-4a11-801d-592088a1b625" containerName="discovery" Jul 21 15:10:46.218656 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:46.218624 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="8814b301-5245-4a11-801d-592088a1b625" containerName="discovery" Jul 21 15:10:46.232548 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:46.232524 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8558d588c-xbtgr" Jul 21 15:10:46.238581 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:46.238548 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8558d588c-xbtgr"] Jul 21 15:10:46.334704 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:46.334671 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5fc95f8b-070c-4199-9bbd-b184c6222f03-trusted-ca-bundle\") pod \"console-8558d588c-xbtgr\" (UID: \"5fc95f8b-070c-4199-9bbd-b184c6222f03\") " pod="openshift-console/console-8558d588c-xbtgr" Jul 21 15:10:46.334704 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:46.334711 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5fc95f8b-070c-4199-9bbd-b184c6222f03-console-oauth-config\") pod \"console-8558d588c-xbtgr\" (UID: \"5fc95f8b-070c-4199-9bbd-b184c6222f03\") " pod="openshift-console/console-8558d588c-xbtgr" Jul 21 15:10:46.334904 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:46.334727 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5fc95f8b-070c-4199-9bbd-b184c6222f03-service-ca\") pod \"console-8558d588c-xbtgr\" (UID: \"5fc95f8b-070c-4199-9bbd-b184c6222f03\") " pod="openshift-console/console-8558d588c-xbtgr" Jul 21 15:10:46.334904 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:46.334767 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5fc95f8b-070c-4199-9bbd-b184c6222f03-console-config\") pod \"console-8558d588c-xbtgr\" (UID: \"5fc95f8b-070c-4199-9bbd-b184c6222f03\") " pod="openshift-console/console-8558d588c-xbtgr" Jul 21 15:10:46.334904 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:46.334805 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4sfs\" (UniqueName: \"kubernetes.io/projected/5fc95f8b-070c-4199-9bbd-b184c6222f03-kube-api-access-q4sfs\") pod \"console-8558d588c-xbtgr\" (UID: \"5fc95f8b-070c-4199-9bbd-b184c6222f03\") " pod="openshift-console/console-8558d588c-xbtgr" Jul 21 15:10:46.335023 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:46.334909 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5fc95f8b-070c-4199-9bbd-b184c6222f03-oauth-serving-cert\") pod \"console-8558d588c-xbtgr\" (UID: \"5fc95f8b-070c-4199-9bbd-b184c6222f03\") " pod="openshift-console/console-8558d588c-xbtgr" Jul 21 15:10:46.335023 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:46.334937 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5fc95f8b-070c-4199-9bbd-b184c6222f03-console-serving-cert\") pod \"console-8558d588c-xbtgr\" (UID: \"5fc95f8b-070c-4199-9bbd-b184c6222f03\") " pod="openshift-console/console-8558d588c-xbtgr" Jul 21 15:10:46.435283 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:46.435252 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5fc95f8b-070c-4199-9bbd-b184c6222f03-oauth-serving-cert\") pod \"console-8558d588c-xbtgr\" (UID: \"5fc95f8b-070c-4199-9bbd-b184c6222f03\") " pod="openshift-console/console-8558d588c-xbtgr" Jul 21 15:10:46.435283 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:46.435285 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5fc95f8b-070c-4199-9bbd-b184c6222f03-console-serving-cert\") pod \"console-8558d588c-xbtgr\" (UID: \"5fc95f8b-070c-4199-9bbd-b184c6222f03\") " pod="openshift-console/console-8558d588c-xbtgr" Jul 21 15:10:46.435545 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:46.435306 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5fc95f8b-070c-4199-9bbd-b184c6222f03-trusted-ca-bundle\") pod \"console-8558d588c-xbtgr\" (UID: \"5fc95f8b-070c-4199-9bbd-b184c6222f03\") " pod="openshift-console/console-8558d588c-xbtgr" Jul 21 15:10:46.435545 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:46.435346 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5fc95f8b-070c-4199-9bbd-b184c6222f03-console-oauth-config\") pod \"console-8558d588c-xbtgr\" (UID: \"5fc95f8b-070c-4199-9bbd-b184c6222f03\") " pod="openshift-console/console-8558d588c-xbtgr" Jul 21 15:10:46.435545 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:46.435362 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5fc95f8b-070c-4199-9bbd-b184c6222f03-service-ca\") pod \"console-8558d588c-xbtgr\" (UID: \"5fc95f8b-070c-4199-9bbd-b184c6222f03\") " pod="openshift-console/console-8558d588c-xbtgr" Jul 21 15:10:46.435545 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:46.435388 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5fc95f8b-070c-4199-9bbd-b184c6222f03-console-config\") pod \"console-8558d588c-xbtgr\" (UID: \"5fc95f8b-070c-4199-9bbd-b184c6222f03\") " pod="openshift-console/console-8558d588c-xbtgr" Jul 21 15:10:46.435545 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:46.435516 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4sfs\" (UniqueName: \"kubernetes.io/projected/5fc95f8b-070c-4199-9bbd-b184c6222f03-kube-api-access-q4sfs\") pod \"console-8558d588c-xbtgr\" (UID: \"5fc95f8b-070c-4199-9bbd-b184c6222f03\") " pod="openshift-console/console-8558d588c-xbtgr" Jul 21 15:10:46.436203 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:46.436151 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5fc95f8b-070c-4199-9bbd-b184c6222f03-oauth-serving-cert\") pod \"console-8558d588c-xbtgr\" (UID: \"5fc95f8b-070c-4199-9bbd-b184c6222f03\") " pod="openshift-console/console-8558d588c-xbtgr" Jul 21 15:10:46.436350 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:46.436252 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5fc95f8b-070c-4199-9bbd-b184c6222f03-trusted-ca-bundle\") pod \"console-8558d588c-xbtgr\" (UID: \"5fc95f8b-070c-4199-9bbd-b184c6222f03\") " pod="openshift-console/console-8558d588c-xbtgr" Jul 21 15:10:46.436350 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:46.436316 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5fc95f8b-070c-4199-9bbd-b184c6222f03-service-ca\") pod \"console-8558d588c-xbtgr\" (UID: \"5fc95f8b-070c-4199-9bbd-b184c6222f03\") " pod="openshift-console/console-8558d588c-xbtgr" Jul 21 15:10:46.436662 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:46.436638 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5fc95f8b-070c-4199-9bbd-b184c6222f03-console-config\") pod \"console-8558d588c-xbtgr\" (UID: \"5fc95f8b-070c-4199-9bbd-b184c6222f03\") " pod="openshift-console/console-8558d588c-xbtgr" Jul 21 15:10:46.437926 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:46.437911 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5fc95f8b-070c-4199-9bbd-b184c6222f03-console-serving-cert\") pod \"console-8558d588c-xbtgr\" (UID: \"5fc95f8b-070c-4199-9bbd-b184c6222f03\") " pod="openshift-console/console-8558d588c-xbtgr" Jul 21 15:10:46.438057 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:46.438040 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5fc95f8b-070c-4199-9bbd-b184c6222f03-console-oauth-config\") pod \"console-8558d588c-xbtgr\" (UID: \"5fc95f8b-070c-4199-9bbd-b184c6222f03\") " pod="openshift-console/console-8558d588c-xbtgr" Jul 21 15:10:46.444750 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:46.444730 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4sfs\" (UniqueName: \"kubernetes.io/projected/5fc95f8b-070c-4199-9bbd-b184c6222f03-kube-api-access-q4sfs\") pod \"console-8558d588c-xbtgr\" (UID: \"5fc95f8b-070c-4199-9bbd-b184c6222f03\") " pod="openshift-console/console-8558d588c-xbtgr" Jul 21 15:10:46.543050 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:46.542970 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8558d588c-xbtgr" Jul 21 15:10:46.680143 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:46.680118 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8558d588c-xbtgr"] Jul 21 15:10:46.682048 ip-10-0-135-176 kubenswrapper[2563]: W0721 15:10:46.682020 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fc95f8b_070c_4199_9bbd_b184c6222f03.slice/crio-496846cf8eb3bc62f7a0fe8f3ff40158e95487256831e90d5261325aa3321575 WatchSource:0}: Error finding container 496846cf8eb3bc62f7a0fe8f3ff40158e95487256831e90d5261325aa3321575: Status 404 returned error can't find the container with id 496846cf8eb3bc62f7a0fe8f3ff40158e95487256831e90d5261325aa3321575 Jul 21 15:10:47.051778 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:47.051742 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8558d588c-xbtgr" event={"ID":"5fc95f8b-070c-4199-9bbd-b184c6222f03","Type":"ContainerStarted","Data":"c93879021efc9b68414724ea6b7ce2f427511afde6166c3c7e2146c4ccdfe900"} Jul 21 15:10:47.051964 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:47.051803 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8558d588c-xbtgr" event={"ID":"5fc95f8b-070c-4199-9bbd-b184c6222f03","Type":"ContainerStarted","Data":"496846cf8eb3bc62f7a0fe8f3ff40158e95487256831e90d5261325aa3321575"} Jul 21 15:10:47.071880 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:47.071833 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-8558d588c-xbtgr" podStartSLOduration=1.071816676 podStartE2EDuration="1.071816676s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 15:10:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 15:10:47.070127543 +0000 UTC m=+679.065912985" watchObservedRunningTime="2026-07-21 15:10:47.071816676 +0000 UTC m=+679.067602093" Jul 21 15:10:47.915488 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:47.915455 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-jrpr5" Jul 21 15:10:50.928909 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:50.928881 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-mfgpf" Jul 21 15:10:56.544019 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:56.543985 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-8558d588c-xbtgr" Jul 21 15:10:56.544548 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:56.544059 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-8558d588c-xbtgr" Jul 21 15:10:56.548789 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:56.548769 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-8558d588c-xbtgr" Jul 21 15:10:57.100529 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:57.100503 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-8558d588c-xbtgr" Jul 21 15:10:57.152398 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:10:57.152366 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6b687b7676-d9v9s"] Jul 21 15:11:04.581700 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:04.581664 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76npgkb"] Jul 21 15:11:04.585290 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:04.585272 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76npgkb" Jul 21 15:11:04.588250 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:04.588222 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 21 15:11:04.588384 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:04.588362 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-lkwhz\"" Jul 21 15:11:04.589637 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:04.589617 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 21 15:11:04.595416 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:04.595396 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76npgkb"] Jul 21 15:11:04.665650 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:04.665621 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efb4wrn"] Jul 21 15:11:04.669112 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:04.669095 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efb4wrn" Jul 21 15:11:04.679053 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:04.679033 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efb4wrn"] Jul 21 15:11:04.684850 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:04.684828 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3d3076b0-8a66-4d23-94a1-4571181bb4d1-util\") pod \"f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76npgkb\" (UID: \"3d3076b0-8a66-4d23-94a1-4571181bb4d1\") " pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76npgkb" Jul 21 15:11:04.684954 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:04.684884 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3d3076b0-8a66-4d23-94a1-4571181bb4d1-bundle\") pod \"f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76npgkb\" (UID: \"3d3076b0-8a66-4d23-94a1-4571181bb4d1\") " pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76npgkb" Jul 21 15:11:04.684954 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:04.684908 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tt4j7\" (UniqueName: \"kubernetes.io/projected/3d3076b0-8a66-4d23-94a1-4571181bb4d1-kube-api-access-tt4j7\") pod \"f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76npgkb\" (UID: \"3d3076b0-8a66-4d23-94a1-4571181bb4d1\") " pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76npgkb" Jul 21 15:11:04.770290 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:04.770259 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36svlbb"] Jul 21 15:11:04.773726 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:04.773710 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36svlbb" Jul 21 15:11:04.785273 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:04.785256 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3d3076b0-8a66-4d23-94a1-4571181bb4d1-util\") pod \"f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76npgkb\" (UID: \"3d3076b0-8a66-4d23-94a1-4571181bb4d1\") " pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76npgkb" Jul 21 15:11:04.785389 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:04.785284 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9e8aec79-3c5f-4b73-93b1-7c3c84a0f714-bundle\") pod \"299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efb4wrn\" (UID: \"9e8aec79-3c5f-4b73-93b1-7c3c84a0f714\") " pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efb4wrn" Jul 21 15:11:04.785432 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:04.785417 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3d3076b0-8a66-4d23-94a1-4571181bb4d1-bundle\") pod \"f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76npgkb\" (UID: \"3d3076b0-8a66-4d23-94a1-4571181bb4d1\") " pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76npgkb" Jul 21 15:11:04.785465 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:04.785440 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tt4j7\" (UniqueName: \"kubernetes.io/projected/3d3076b0-8a66-4d23-94a1-4571181bb4d1-kube-api-access-tt4j7\") pod \"f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76npgkb\" (UID: \"3d3076b0-8a66-4d23-94a1-4571181bb4d1\") " pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76npgkb" Jul 21 15:11:04.785501 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:04.785473 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9e8aec79-3c5f-4b73-93b1-7c3c84a0f714-util\") pod \"299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efb4wrn\" (UID: \"9e8aec79-3c5f-4b73-93b1-7c3c84a0f714\") " pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efb4wrn" Jul 21 15:11:04.785535 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:04.785512 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9wxf\" (UniqueName: \"kubernetes.io/projected/9e8aec79-3c5f-4b73-93b1-7c3c84a0f714-kube-api-access-n9wxf\") pod \"299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efb4wrn\" (UID: \"9e8aec79-3c5f-4b73-93b1-7c3c84a0f714\") " pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efb4wrn" Jul 21 15:11:04.785633 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:04.785603 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3d3076b0-8a66-4d23-94a1-4571181bb4d1-util\") pod \"f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76npgkb\" (UID: \"3d3076b0-8a66-4d23-94a1-4571181bb4d1\") " pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76npgkb" Jul 21 15:11:04.785729 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:04.785711 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3d3076b0-8a66-4d23-94a1-4571181bb4d1-bundle\") pod \"f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76npgkb\" (UID: \"3d3076b0-8a66-4d23-94a1-4571181bb4d1\") " pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76npgkb" Jul 21 15:11:04.791991 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:04.791973 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36svlbb"] Jul 21 15:11:04.795377 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:04.795354 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tt4j7\" (UniqueName: \"kubernetes.io/projected/3d3076b0-8a66-4d23-94a1-4571181bb4d1-kube-api-access-tt4j7\") pod \"f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76npgkb\" (UID: \"3d3076b0-8a66-4d23-94a1-4571181bb4d1\") " pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76npgkb" Jul 21 15:11:04.886684 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:04.886605 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a6a22243-b718-41a6-8e49-c7b897c0e8db-util\") pod \"0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36svlbb\" (UID: \"a6a22243-b718-41a6-8e49-c7b897c0e8db\") " pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36svlbb" Jul 21 15:11:04.886684 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:04.886634 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6z2w\" (UniqueName: \"kubernetes.io/projected/a6a22243-b718-41a6-8e49-c7b897c0e8db-kube-api-access-h6z2w\") pod \"0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36svlbb\" (UID: \"a6a22243-b718-41a6-8e49-c7b897c0e8db\") " pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36svlbb" Jul 21 15:11:04.886684 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:04.886667 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9e8aec79-3c5f-4b73-93b1-7c3c84a0f714-util\") pod \"299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efb4wrn\" (UID: \"9e8aec79-3c5f-4b73-93b1-7c3c84a0f714\") " pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efb4wrn" Jul 21 15:11:04.886954 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:04.886743 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9wxf\" (UniqueName: \"kubernetes.io/projected/9e8aec79-3c5f-4b73-93b1-7c3c84a0f714-kube-api-access-n9wxf\") pod \"299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efb4wrn\" (UID: \"9e8aec79-3c5f-4b73-93b1-7c3c84a0f714\") " pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efb4wrn" Jul 21 15:11:04.886954 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:04.886797 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a6a22243-b718-41a6-8e49-c7b897c0e8db-bundle\") pod \"0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36svlbb\" (UID: \"a6a22243-b718-41a6-8e49-c7b897c0e8db\") " pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36svlbb" Jul 21 15:11:04.886954 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:04.886838 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9e8aec79-3c5f-4b73-93b1-7c3c84a0f714-bundle\") pod \"299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efb4wrn\" (UID: \"9e8aec79-3c5f-4b73-93b1-7c3c84a0f714\") " pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efb4wrn" Jul 21 15:11:04.886954 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:04.886942 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9e8aec79-3c5f-4b73-93b1-7c3c84a0f714-util\") pod \"299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efb4wrn\" (UID: \"9e8aec79-3c5f-4b73-93b1-7c3c84a0f714\") " pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efb4wrn" Jul 21 15:11:04.887158 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:04.887129 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9e8aec79-3c5f-4b73-93b1-7c3c84a0f714-bundle\") pod \"299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efb4wrn\" (UID: \"9e8aec79-3c5f-4b73-93b1-7c3c84a0f714\") " pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efb4wrn" Jul 21 15:11:04.895582 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:04.895547 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76npgkb" Jul 21 15:11:04.899315 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:04.899299 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabxj885"] Jul 21 15:11:04.901447 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:04.901425 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9wxf\" (UniqueName: \"kubernetes.io/projected/9e8aec79-3c5f-4b73-93b1-7c3c84a0f714-kube-api-access-n9wxf\") pod \"299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efb4wrn\" (UID: \"9e8aec79-3c5f-4b73-93b1-7c3c84a0f714\") " pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efb4wrn" Jul 21 15:11:04.902971 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:04.902955 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabxj885" Jul 21 15:11:04.914146 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:04.914121 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabxj885"] Jul 21 15:11:04.978110 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:04.977950 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efb4wrn" Jul 21 15:11:04.987909 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:04.987878 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a6a22243-b718-41a6-8e49-c7b897c0e8db-bundle\") pod \"0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36svlbb\" (UID: \"a6a22243-b718-41a6-8e49-c7b897c0e8db\") " pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36svlbb" Jul 21 15:11:04.988030 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:04.987922 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9880cef9-389f-42fa-9b80-19c68132c706-bundle\") pod \"af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabxj885\" (UID: \"9880cef9-389f-42fa-9b80-19c68132c706\") " pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabxj885" Jul 21 15:11:04.988030 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:04.987958 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9880cef9-389f-42fa-9b80-19c68132c706-util\") pod \"af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabxj885\" (UID: \"9880cef9-389f-42fa-9b80-19c68132c706\") " pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabxj885" Jul 21 15:11:04.988145 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:04.988060 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a6a22243-b718-41a6-8e49-c7b897c0e8db-util\") pod \"0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36svlbb\" (UID: \"a6a22243-b718-41a6-8e49-c7b897c0e8db\") " pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36svlbb" Jul 21 15:11:04.988145 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:04.988091 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6z2w\" (UniqueName: \"kubernetes.io/projected/a6a22243-b718-41a6-8e49-c7b897c0e8db-kube-api-access-h6z2w\") pod \"0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36svlbb\" (UID: \"a6a22243-b718-41a6-8e49-c7b897c0e8db\") " pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36svlbb" Jul 21 15:11:04.988145 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:04.988118 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcr48\" (UniqueName: \"kubernetes.io/projected/9880cef9-389f-42fa-9b80-19c68132c706-kube-api-access-zcr48\") pod \"af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabxj885\" (UID: \"9880cef9-389f-42fa-9b80-19c68132c706\") " pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabxj885" Jul 21 15:11:04.988617 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:04.988562 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a6a22243-b718-41a6-8e49-c7b897c0e8db-bundle\") pod \"0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36svlbb\" (UID: \"a6a22243-b718-41a6-8e49-c7b897c0e8db\") " pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36svlbb" Jul 21 15:11:04.988746 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:04.988720 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a6a22243-b718-41a6-8e49-c7b897c0e8db-util\") pod \"0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36svlbb\" (UID: \"a6a22243-b718-41a6-8e49-c7b897c0e8db\") " pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36svlbb" Jul 21 15:11:05.001245 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:05.001222 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6z2w\" (UniqueName: \"kubernetes.io/projected/a6a22243-b718-41a6-8e49-c7b897c0e8db-kube-api-access-h6z2w\") pod \"0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36svlbb\" (UID: \"a6a22243-b718-41a6-8e49-c7b897c0e8db\") " pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36svlbb" Jul 21 15:11:05.053158 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:05.053028 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76npgkb"] Jul 21 15:11:05.056410 ip-10-0-135-176 kubenswrapper[2563]: W0721 15:11:05.056383 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d3076b0_8a66_4d23_94a1_4571181bb4d1.slice/crio-c8f3ffd9d834ea963db027985f6416a03d15ded64e03ec1474c9f7ff8ffc5c4b WatchSource:0}: Error finding container c8f3ffd9d834ea963db027985f6416a03d15ded64e03ec1474c9f7ff8ffc5c4b: Status 404 returned error can't find the container with id c8f3ffd9d834ea963db027985f6416a03d15ded64e03ec1474c9f7ff8ffc5c4b Jul 21 15:11:05.082847 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:05.082824 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36svlbb" Jul 21 15:11:05.089129 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:05.089100 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9880cef9-389f-42fa-9b80-19c68132c706-bundle\") pod \"af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabxj885\" (UID: \"9880cef9-389f-42fa-9b80-19c68132c706\") " pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabxj885" Jul 21 15:11:05.089270 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:05.089145 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9880cef9-389f-42fa-9b80-19c68132c706-util\") pod \"af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabxj885\" (UID: \"9880cef9-389f-42fa-9b80-19c68132c706\") " pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabxj885" Jul 21 15:11:05.089270 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:05.089246 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcr48\" (UniqueName: \"kubernetes.io/projected/9880cef9-389f-42fa-9b80-19c68132c706-kube-api-access-zcr48\") pod \"af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabxj885\" (UID: \"9880cef9-389f-42fa-9b80-19c68132c706\") " pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabxj885" Jul 21 15:11:05.089571 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:05.089547 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9880cef9-389f-42fa-9b80-19c68132c706-bundle\") pod \"af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabxj885\" (UID: \"9880cef9-389f-42fa-9b80-19c68132c706\") " pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabxj885" Jul 21 15:11:05.089710 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:05.089691 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9880cef9-389f-42fa-9b80-19c68132c706-util\") pod \"af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabxj885\" (UID: \"9880cef9-389f-42fa-9b80-19c68132c706\") " pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabxj885" Jul 21 15:11:05.103746 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:05.103721 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcr48\" (UniqueName: \"kubernetes.io/projected/9880cef9-389f-42fa-9b80-19c68132c706-kube-api-access-zcr48\") pod \"af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabxj885\" (UID: \"9880cef9-389f-42fa-9b80-19c68132c706\") " pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabxj885" Jul 21 15:11:05.140948 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:05.137293 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efb4wrn"] Jul 21 15:11:05.143212 ip-10-0-135-176 kubenswrapper[2563]: W0721 15:11:05.142558 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e8aec79_3c5f_4b73_93b1_7c3c84a0f714.slice/crio-256d5405bc489fb2693e3c1cb77844ecdc92396f1209dcd79cb43c0d09ffe6f0 WatchSource:0}: Error finding container 256d5405bc489fb2693e3c1cb77844ecdc92396f1209dcd79cb43c0d09ffe6f0: Status 404 returned error can't find the container with id 256d5405bc489fb2693e3c1cb77844ecdc92396f1209dcd79cb43c0d09ffe6f0 Jul 21 15:11:05.143212 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:05.142729 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76npgkb" event={"ID":"3d3076b0-8a66-4d23-94a1-4571181bb4d1","Type":"ContainerStarted","Data":"640597dfc1b427aaa5b8cee727ecb9db4c120cb825761a98fa05acafdbad2e47"} Jul 21 15:11:05.143212 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:05.142789 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76npgkb" event={"ID":"3d3076b0-8a66-4d23-94a1-4571181bb4d1","Type":"ContainerStarted","Data":"c8f3ffd9d834ea963db027985f6416a03d15ded64e03ec1474c9f7ff8ffc5c4b"} Jul 21 15:11:05.226283 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:05.226258 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabxj885" Jul 21 15:11:05.255578 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:05.255552 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36svlbb"] Jul 21 15:11:05.257174 ip-10-0-135-176 kubenswrapper[2563]: W0721 15:11:05.257146 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6a22243_b718_41a6_8e49_c7b897c0e8db.slice/crio-9d244df865a52ceb103a0a88fdafb616ff7d0ba5b47cc643f751f6efa0802b48 WatchSource:0}: Error finding container 9d244df865a52ceb103a0a88fdafb616ff7d0ba5b47cc643f751f6efa0802b48: Status 404 returned error can't find the container with id 9d244df865a52ceb103a0a88fdafb616ff7d0ba5b47cc643f751f6efa0802b48 Jul 21 15:11:05.389006 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:05.388953 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabxj885"] Jul 21 15:11:05.433056 ip-10-0-135-176 kubenswrapper[2563]: W0721 15:11:05.432973 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9880cef9_389f_42fa_9b80_19c68132c706.slice/crio-d973826c4661d83f473bea6723c6cf364c173b148ca007f97ec0ca6be07fd309 WatchSource:0}: Error finding container d973826c4661d83f473bea6723c6cf364c173b148ca007f97ec0ca6be07fd309: Status 404 returned error can't find the container with id d973826c4661d83f473bea6723c6cf364c173b148ca007f97ec0ca6be07fd309 Jul 21 15:11:06.149788 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:06.149756 2563 generic.go:356] "Generic (PLEG): container finished" podID="9880cef9-389f-42fa-9b80-19c68132c706" containerID="40553c331ae99d7d953ca240752dc73e5fbb045da8621be3929803473fae39e4" exitCode=0 Jul 21 15:11:06.150229 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:06.149844 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabxj885" event={"ID":"9880cef9-389f-42fa-9b80-19c68132c706","Type":"ContainerDied","Data":"40553c331ae99d7d953ca240752dc73e5fbb045da8621be3929803473fae39e4"} Jul 21 15:11:06.150229 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:06.149908 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabxj885" event={"ID":"9880cef9-389f-42fa-9b80-19c68132c706","Type":"ContainerStarted","Data":"d973826c4661d83f473bea6723c6cf364c173b148ca007f97ec0ca6be07fd309"} Jul 21 15:11:06.150966 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:06.150950 2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 21 15:11:06.151500 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:06.151442 2563 generic.go:356] "Generic (PLEG): container finished" podID="3d3076b0-8a66-4d23-94a1-4571181bb4d1" containerID="640597dfc1b427aaa5b8cee727ecb9db4c120cb825761a98fa05acafdbad2e47" exitCode=0 Jul 21 15:11:06.151557 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:06.151531 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76npgkb" event={"ID":"3d3076b0-8a66-4d23-94a1-4571181bb4d1","Type":"ContainerDied","Data":"640597dfc1b427aaa5b8cee727ecb9db4c120cb825761a98fa05acafdbad2e47"} Jul 21 15:11:06.153229 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:06.153209 2563 generic.go:356] "Generic (PLEG): container finished" podID="9e8aec79-3c5f-4b73-93b1-7c3c84a0f714" containerID="60c99e8cb8b764414b28f49964fa790dfcd33194a1603f58ca8888f3702576dd" exitCode=0 Jul 21 15:11:06.153306 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:06.153273 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efb4wrn" event={"ID":"9e8aec79-3c5f-4b73-93b1-7c3c84a0f714","Type":"ContainerDied","Data":"60c99e8cb8b764414b28f49964fa790dfcd33194a1603f58ca8888f3702576dd"} Jul 21 15:11:06.153391 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:06.153313 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efb4wrn" event={"ID":"9e8aec79-3c5f-4b73-93b1-7c3c84a0f714","Type":"ContainerStarted","Data":"256d5405bc489fb2693e3c1cb77844ecdc92396f1209dcd79cb43c0d09ffe6f0"} Jul 21 15:11:06.154977 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:06.154958 2563 generic.go:356] "Generic (PLEG): container finished" podID="a6a22243-b718-41a6-8e49-c7b897c0e8db" containerID="34722b35d5222d1d9d41abec305791b08e2d88a5793699118699cee1545c4ffa" exitCode=0 Jul 21 15:11:06.155063 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:06.155023 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36svlbb" event={"ID":"a6a22243-b718-41a6-8e49-c7b897c0e8db","Type":"ContainerDied","Data":"34722b35d5222d1d9d41abec305791b08e2d88a5793699118699cee1545c4ffa"} Jul 21 15:11:06.155063 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:06.155047 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36svlbb" event={"ID":"a6a22243-b718-41a6-8e49-c7b897c0e8db","Type":"ContainerStarted","Data":"9d244df865a52ceb103a0a88fdafb616ff7d0ba5b47cc643f751f6efa0802b48"} Jul 21 15:11:08.167209 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:08.167179 2563 generic.go:356] "Generic (PLEG): container finished" podID="9880cef9-389f-42fa-9b80-19c68132c706" containerID="ded9db3f465be04a6ad33085dc1f63cf172cb116cb4c69f5020258f51b59b722" exitCode=0 Jul 21 15:11:08.167659 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:08.167272 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabxj885" event={"ID":"9880cef9-389f-42fa-9b80-19c68132c706","Type":"ContainerDied","Data":"ded9db3f465be04a6ad33085dc1f63cf172cb116cb4c69f5020258f51b59b722"} Jul 21 15:11:08.169049 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:08.169027 2563 generic.go:356] "Generic (PLEG): container finished" podID="3d3076b0-8a66-4d23-94a1-4571181bb4d1" containerID="e0ef9f01312414b847e40439f9776e39a1c3fced76c06608cb04575deaf52a1c" exitCode=0 Jul 21 15:11:08.169183 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:08.169161 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76npgkb" event={"ID":"3d3076b0-8a66-4d23-94a1-4571181bb4d1","Type":"ContainerDied","Data":"e0ef9f01312414b847e40439f9776e39a1c3fced76c06608cb04575deaf52a1c"} Jul 21 15:11:08.171158 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:08.171137 2563 generic.go:356] "Generic (PLEG): container finished" podID="9e8aec79-3c5f-4b73-93b1-7c3c84a0f714" containerID="fbea4293021767e58bae729683bcf3007cf50fb569b82e43a7f930e28f72563d" exitCode=0 Jul 21 15:11:08.171300 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:08.171215 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efb4wrn" event={"ID":"9e8aec79-3c5f-4b73-93b1-7c3c84a0f714","Type":"ContainerDied","Data":"fbea4293021767e58bae729683bcf3007cf50fb569b82e43a7f930e28f72563d"} Jul 21 15:11:08.173164 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:08.173147 2563 generic.go:356] "Generic (PLEG): container finished" podID="a6a22243-b718-41a6-8e49-c7b897c0e8db" containerID="59ec1f6d442aa4877c6cfbe874e278c13b7376ef71d1dcac6c23279f4d9f3cbc" exitCode=0 Jul 21 15:11:08.173248 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:08.173175 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36svlbb" event={"ID":"a6a22243-b718-41a6-8e49-c7b897c0e8db","Type":"ContainerDied","Data":"59ec1f6d442aa4877c6cfbe874e278c13b7376ef71d1dcac6c23279f4d9f3cbc"} Jul 21 15:11:09.180261 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:09.180227 2563 generic.go:356] "Generic (PLEG): container finished" podID="9880cef9-389f-42fa-9b80-19c68132c706" containerID="83f5bd70dae119ba84cefef046fd62ad1ede3b38b7b338d1d0038092f9b4d623" exitCode=0 Jul 21 15:11:09.180690 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:09.180313 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabxj885" event={"ID":"9880cef9-389f-42fa-9b80-19c68132c706","Type":"ContainerDied","Data":"83f5bd70dae119ba84cefef046fd62ad1ede3b38b7b338d1d0038092f9b4d623"} Jul 21 15:11:09.182316 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:09.182297 2563 generic.go:356] "Generic (PLEG): container finished" podID="3d3076b0-8a66-4d23-94a1-4571181bb4d1" containerID="0aa4f4cca618baf0b967240055502e5087aa333590a3d370c78c80f3cd349036" exitCode=0 Jul 21 15:11:09.182423 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:09.182360 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76npgkb" event={"ID":"3d3076b0-8a66-4d23-94a1-4571181bb4d1","Type":"ContainerDied","Data":"0aa4f4cca618baf0b967240055502e5087aa333590a3d370c78c80f3cd349036"} Jul 21 15:11:09.184450 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:09.184431 2563 generic.go:356] "Generic (PLEG): container finished" podID="9e8aec79-3c5f-4b73-93b1-7c3c84a0f714" containerID="a931a34c355e17de0c9f247725620e29ff5decdb1ccdac43abea7850d974a350" exitCode=0 Jul 21 15:11:09.184549 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:09.184461 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efb4wrn" event={"ID":"9e8aec79-3c5f-4b73-93b1-7c3c84a0f714","Type":"ContainerDied","Data":"a931a34c355e17de0c9f247725620e29ff5decdb1ccdac43abea7850d974a350"} Jul 21 15:11:09.186465 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:09.186447 2563 generic.go:356] "Generic (PLEG): container finished" podID="a6a22243-b718-41a6-8e49-c7b897c0e8db" containerID="1f8448a19732432eac977dd28aa5ef738e9cea108ee9864d5dd80d03e85f0ed6" exitCode=0 Jul 21 15:11:09.186528 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:09.186501 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36svlbb" event={"ID":"a6a22243-b718-41a6-8e49-c7b897c0e8db","Type":"ContainerDied","Data":"1f8448a19732432eac977dd28aa5ef738e9cea108ee9864d5dd80d03e85f0ed6"} Jul 21 15:11:10.336633 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:10.336603 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabxj885" Jul 21 15:11:10.399911 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:10.399892 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36svlbb" Jul 21 15:11:10.402819 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:10.402793 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76npgkb" Jul 21 15:11:10.405918 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:10.405897 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efb4wrn" Jul 21 15:11:10.433779 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:10.433754 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9880cef9-389f-42fa-9b80-19c68132c706-util\") pod \"9880cef9-389f-42fa-9b80-19c68132c706\" (UID: \"9880cef9-389f-42fa-9b80-19c68132c706\") " Jul 21 15:11:10.433899 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:10.433807 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9880cef9-389f-42fa-9b80-19c68132c706-bundle\") pod \"9880cef9-389f-42fa-9b80-19c68132c706\" (UID: \"9880cef9-389f-42fa-9b80-19c68132c706\") " Jul 21 15:11:10.433899 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:10.433872 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zcr48\" (UniqueName: \"kubernetes.io/projected/9880cef9-389f-42fa-9b80-19c68132c706-kube-api-access-zcr48\") pod \"9880cef9-389f-42fa-9b80-19c68132c706\" (UID: \"9880cef9-389f-42fa-9b80-19c68132c706\") " Jul 21 15:11:10.434471 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:10.434446 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9880cef9-389f-42fa-9b80-19c68132c706-bundle" (OuterVolumeSpecName: "bundle") pod "9880cef9-389f-42fa-9b80-19c68132c706" (UID: "9880cef9-389f-42fa-9b80-19c68132c706"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 15:11:10.436013 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:10.435993 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9880cef9-389f-42fa-9b80-19c68132c706-kube-api-access-zcr48" (OuterVolumeSpecName: "kube-api-access-zcr48") pod "9880cef9-389f-42fa-9b80-19c68132c706" (UID: "9880cef9-389f-42fa-9b80-19c68132c706"). InnerVolumeSpecName "kube-api-access-zcr48". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 15:11:10.439900 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:10.439844 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9880cef9-389f-42fa-9b80-19c68132c706-util" (OuterVolumeSpecName: "util") pod "9880cef9-389f-42fa-9b80-19c68132c706" (UID: "9880cef9-389f-42fa-9b80-19c68132c706"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 15:11:10.534901 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:10.534877 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3d3076b0-8a66-4d23-94a1-4571181bb4d1-util\") pod \"3d3076b0-8a66-4d23-94a1-4571181bb4d1\" (UID: \"3d3076b0-8a66-4d23-94a1-4571181bb4d1\") " Jul 21 15:11:10.535052 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:10.534919 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tt4j7\" (UniqueName: \"kubernetes.io/projected/3d3076b0-8a66-4d23-94a1-4571181bb4d1-kube-api-access-tt4j7\") pod \"3d3076b0-8a66-4d23-94a1-4571181bb4d1\" (UID: \"3d3076b0-8a66-4d23-94a1-4571181bb4d1\") " Jul 21 15:11:10.535052 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:10.534935 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9e8aec79-3c5f-4b73-93b1-7c3c84a0f714-util\") pod \"9e8aec79-3c5f-4b73-93b1-7c3c84a0f714\" (UID: \"9e8aec79-3c5f-4b73-93b1-7c3c84a0f714\") " Jul 21 15:11:10.535052 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:10.534951 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h6z2w\" (UniqueName: \"kubernetes.io/projected/a6a22243-b718-41a6-8e49-c7b897c0e8db-kube-api-access-h6z2w\") pod \"a6a22243-b718-41a6-8e49-c7b897c0e8db\" (UID: \"a6a22243-b718-41a6-8e49-c7b897c0e8db\") " Jul 21 15:11:10.535052 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:10.534967 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a6a22243-b718-41a6-8e49-c7b897c0e8db-bundle\") pod \"a6a22243-b718-41a6-8e49-c7b897c0e8db\" (UID: \"a6a22243-b718-41a6-8e49-c7b897c0e8db\") " Jul 21 15:11:10.535052 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:10.534984 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9e8aec79-3c5f-4b73-93b1-7c3c84a0f714-bundle\") pod \"9e8aec79-3c5f-4b73-93b1-7c3c84a0f714\" (UID: \"9e8aec79-3c5f-4b73-93b1-7c3c84a0f714\") " Jul 21 15:11:10.535052 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:10.535004 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n9wxf\" (UniqueName: \"kubernetes.io/projected/9e8aec79-3c5f-4b73-93b1-7c3c84a0f714-kube-api-access-n9wxf\") pod \"9e8aec79-3c5f-4b73-93b1-7c3c84a0f714\" (UID: \"9e8aec79-3c5f-4b73-93b1-7c3c84a0f714\") " Jul 21 15:11:10.535052 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:10.535029 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a6a22243-b718-41a6-8e49-c7b897c0e8db-util\") pod \"a6a22243-b718-41a6-8e49-c7b897c0e8db\" (UID: \"a6a22243-b718-41a6-8e49-c7b897c0e8db\") " Jul 21 15:11:10.535451 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:10.535128 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3d3076b0-8a66-4d23-94a1-4571181bb4d1-bundle\") pod \"3d3076b0-8a66-4d23-94a1-4571181bb4d1\" (UID: \"3d3076b0-8a66-4d23-94a1-4571181bb4d1\") " Jul 21 15:11:10.535451 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:10.535388 2563 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9880cef9-389f-42fa-9b80-19c68132c706-util\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 15:11:10.535451 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:10.535405 2563 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9880cef9-389f-42fa-9b80-19c68132c706-bundle\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 15:11:10.535451 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:10.535418 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zcr48\" (UniqueName: \"kubernetes.io/projected/9880cef9-389f-42fa-9b80-19c68132c706-kube-api-access-zcr48\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 15:11:10.536133 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:10.535651 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6a22243-b718-41a6-8e49-c7b897c0e8db-bundle" (OuterVolumeSpecName: "bundle") pod "a6a22243-b718-41a6-8e49-c7b897c0e8db" (UID: "a6a22243-b718-41a6-8e49-c7b897c0e8db"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 15:11:10.536133 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:10.535894 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e8aec79-3c5f-4b73-93b1-7c3c84a0f714-bundle" (OuterVolumeSpecName: "bundle") pod "9e8aec79-3c5f-4b73-93b1-7c3c84a0f714" (UID: "9e8aec79-3c5f-4b73-93b1-7c3c84a0f714"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 15:11:10.536320 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:10.536207 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d3076b0-8a66-4d23-94a1-4571181bb4d1-bundle" (OuterVolumeSpecName: "bundle") pod "3d3076b0-8a66-4d23-94a1-4571181bb4d1" (UID: "3d3076b0-8a66-4d23-94a1-4571181bb4d1"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 15:11:10.537315 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:10.537287 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d3076b0-8a66-4d23-94a1-4571181bb4d1-kube-api-access-tt4j7" (OuterVolumeSpecName: "kube-api-access-tt4j7") pod "3d3076b0-8a66-4d23-94a1-4571181bb4d1" (UID: "3d3076b0-8a66-4d23-94a1-4571181bb4d1"). InnerVolumeSpecName "kube-api-access-tt4j7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 15:11:10.537510 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:10.537490 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6a22243-b718-41a6-8e49-c7b897c0e8db-kube-api-access-h6z2w" (OuterVolumeSpecName: "kube-api-access-h6z2w") pod "a6a22243-b718-41a6-8e49-c7b897c0e8db" (UID: "a6a22243-b718-41a6-8e49-c7b897c0e8db"). InnerVolumeSpecName "kube-api-access-h6z2w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 15:11:10.538074 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:10.538050 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e8aec79-3c5f-4b73-93b1-7c3c84a0f714-kube-api-access-n9wxf" (OuterVolumeSpecName: "kube-api-access-n9wxf") pod "9e8aec79-3c5f-4b73-93b1-7c3c84a0f714" (UID: "9e8aec79-3c5f-4b73-93b1-7c3c84a0f714"). InnerVolumeSpecName "kube-api-access-n9wxf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 15:11:10.541490 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:10.541468 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6a22243-b718-41a6-8e49-c7b897c0e8db-util" (OuterVolumeSpecName: "util") pod "a6a22243-b718-41a6-8e49-c7b897c0e8db" (UID: "a6a22243-b718-41a6-8e49-c7b897c0e8db"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 15:11:10.541938 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:10.541920 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d3076b0-8a66-4d23-94a1-4571181bb4d1-util" (OuterVolumeSpecName: "util") pod "3d3076b0-8a66-4d23-94a1-4571181bb4d1" (UID: "3d3076b0-8a66-4d23-94a1-4571181bb4d1"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 15:11:10.542060 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:10.542042 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e8aec79-3c5f-4b73-93b1-7c3c84a0f714-util" (OuterVolumeSpecName: "util") pod "9e8aec79-3c5f-4b73-93b1-7c3c84a0f714" (UID: "9e8aec79-3c5f-4b73-93b1-7c3c84a0f714"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 15:11:10.636317 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:10.636215 2563 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3d3076b0-8a66-4d23-94a1-4571181bb4d1-util\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 15:11:10.636317 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:10.636246 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tt4j7\" (UniqueName: \"kubernetes.io/projected/3d3076b0-8a66-4d23-94a1-4571181bb4d1-kube-api-access-tt4j7\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 15:11:10.636317 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:10.636261 2563 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9e8aec79-3c5f-4b73-93b1-7c3c84a0f714-util\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 15:11:10.636317 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:10.636274 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6z2w\" (UniqueName: \"kubernetes.io/projected/a6a22243-b718-41a6-8e49-c7b897c0e8db-kube-api-access-h6z2w\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 15:11:10.636720 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:10.636697 2563 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a6a22243-b718-41a6-8e49-c7b897c0e8db-bundle\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 15:11:10.636842 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:10.636734 2563 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9e8aec79-3c5f-4b73-93b1-7c3c84a0f714-bundle\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 15:11:10.636842 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:10.636751 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9wxf\" (UniqueName: \"kubernetes.io/projected/9e8aec79-3c5f-4b73-93b1-7c3c84a0f714-kube-api-access-n9wxf\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 15:11:10.636842 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:10.636778 2563 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a6a22243-b718-41a6-8e49-c7b897c0e8db-util\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 15:11:10.636842 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:10.636791 2563 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3d3076b0-8a66-4d23-94a1-4571181bb4d1-bundle\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 15:11:11.199440 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:11.199414 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efb4wrn" Jul 21 15:11:11.199597 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:11.199414 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efb4wrn" event={"ID":"9e8aec79-3c5f-4b73-93b1-7c3c84a0f714","Type":"ContainerDied","Data":"256d5405bc489fb2693e3c1cb77844ecdc92396f1209dcd79cb43c0d09ffe6f0"} Jul 21 15:11:11.199597 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:11.199555 2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="256d5405bc489fb2693e3c1cb77844ecdc92396f1209dcd79cb43c0d09ffe6f0" Jul 21 15:11:11.201201 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:11.201176 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36svlbb" Jul 21 15:11:11.201348 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:11.201208 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36svlbb" event={"ID":"a6a22243-b718-41a6-8e49-c7b897c0e8db","Type":"ContainerDied","Data":"9d244df865a52ceb103a0a88fdafb616ff7d0ba5b47cc643f751f6efa0802b48"} Jul 21 15:11:11.201348 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:11.201245 2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9d244df865a52ceb103a0a88fdafb616ff7d0ba5b47cc643f751f6efa0802b48" Jul 21 15:11:11.203125 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:11.203098 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabxj885" event={"ID":"9880cef9-389f-42fa-9b80-19c68132c706","Type":"ContainerDied","Data":"d973826c4661d83f473bea6723c6cf364c173b148ca007f97ec0ca6be07fd309"} Jul 21 15:11:11.203233 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:11.203148 2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d973826c4661d83f473bea6723c6cf364c173b148ca007f97ec0ca6be07fd309" Jul 21 15:11:11.203233 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:11.203106 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabxj885" Jul 21 15:11:11.205180 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:11.205159 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76npgkb" event={"ID":"3d3076b0-8a66-4d23-94a1-4571181bb4d1","Type":"ContainerDied","Data":"c8f3ffd9d834ea963db027985f6416a03d15ded64e03ec1474c9f7ff8ffc5c4b"} Jul 21 15:11:11.205180 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:11.205192 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76npgkb" Jul 21 15:11:11.205379 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:11.205198 2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c8f3ffd9d834ea963db027985f6416a03d15ded64e03ec1474c9f7ff8ffc5c4b" Jul 21 15:11:19.471124 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:19.471052 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-7d6bb9cf99-phgp7"] Jul 21 15:11:19.471497 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:19.471452 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9880cef9-389f-42fa-9b80-19c68132c706" containerName="util" Jul 21 15:11:19.471497 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:19.471466 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="9880cef9-389f-42fa-9b80-19c68132c706" containerName="util" Jul 21 15:11:19.471497 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:19.471475 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6a22243-b718-41a6-8e49-c7b897c0e8db" containerName="pull" Jul 21 15:11:19.471497 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:19.471481 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6a22243-b718-41a6-8e49-c7b897c0e8db" containerName="pull" Jul 21 15:11:19.471497 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:19.471491 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9880cef9-389f-42fa-9b80-19c68132c706" containerName="pull" Jul 21 15:11:19.471497 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:19.471496 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="9880cef9-389f-42fa-9b80-19c68132c706" containerName="pull" Jul 21 15:11:19.471683 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:19.471509 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d3076b0-8a66-4d23-94a1-4571181bb4d1" containerName="extract" Jul 21 15:11:19.471683 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:19.471514 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d3076b0-8a66-4d23-94a1-4571181bb4d1" containerName="extract" Jul 21 15:11:19.471683 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:19.471521 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6a22243-b718-41a6-8e49-c7b897c0e8db" containerName="extract" Jul 21 15:11:19.471683 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:19.471525 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6a22243-b718-41a6-8e49-c7b897c0e8db" containerName="extract" Jul 21 15:11:19.471683 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:19.471532 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e8aec79-3c5f-4b73-93b1-7c3c84a0f714" containerName="pull" Jul 21 15:11:19.471683 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:19.471538 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e8aec79-3c5f-4b73-93b1-7c3c84a0f714" containerName="pull" Jul 21 15:11:19.471683 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:19.471543 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e8aec79-3c5f-4b73-93b1-7c3c84a0f714" containerName="extract" Jul 21 15:11:19.471683 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:19.471548 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e8aec79-3c5f-4b73-93b1-7c3c84a0f714" containerName="extract" Jul 21 15:11:19.471683 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:19.471557 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d3076b0-8a66-4d23-94a1-4571181bb4d1" containerName="util" Jul 21 15:11:19.471683 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:19.471562 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d3076b0-8a66-4d23-94a1-4571181bb4d1" containerName="util" Jul 21 15:11:19.471683 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:19.471569 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9880cef9-389f-42fa-9b80-19c68132c706" containerName="extract" Jul 21 15:11:19.471683 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:19.471573 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="9880cef9-389f-42fa-9b80-19c68132c706" containerName="extract" Jul 21 15:11:19.471683 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:19.471580 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e8aec79-3c5f-4b73-93b1-7c3c84a0f714" containerName="util" Jul 21 15:11:19.471683 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:19.471585 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e8aec79-3c5f-4b73-93b1-7c3c84a0f714" containerName="util" Jul 21 15:11:19.471683 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:19.471591 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d3076b0-8a66-4d23-94a1-4571181bb4d1" containerName="pull" Jul 21 15:11:19.471683 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:19.471595 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d3076b0-8a66-4d23-94a1-4571181bb4d1" containerName="pull" Jul 21 15:11:19.471683 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:19.471602 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6a22243-b718-41a6-8e49-c7b897c0e8db" containerName="util" Jul 21 15:11:19.471683 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:19.471606 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6a22243-b718-41a6-8e49-c7b897c0e8db" containerName="util" Jul 21 15:11:19.471683 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:19.471659 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d3076b0-8a66-4d23-94a1-4571181bb4d1" containerName="extract" Jul 21 15:11:19.471683 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:19.471668 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a6a22243-b718-41a6-8e49-c7b897c0e8db" containerName="extract" Jul 21 15:11:19.471683 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:19.471673 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e8aec79-3c5f-4b73-93b1-7c3c84a0f714" containerName="extract" Jul 21 15:11:19.471683 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:19.471680 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="9880cef9-389f-42fa-9b80-19c68132c706" containerName="extract" Jul 21 15:11:19.476036 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:19.476020 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-7d6bb9cf99-phgp7" Jul 21 15:11:19.479371 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:19.479316 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 21 15:11:19.479493 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:19.479431 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-hdlvf\"" Jul 21 15:11:19.480837 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:19.480820 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 21 15:11:19.494094 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:19.494064 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-7d6bb9cf99-phgp7"] Jul 21 15:11:19.609913 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:19.609885 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgjxk\" (UniqueName: \"kubernetes.io/projected/0bffdd67-4688-4054-9e38-07f805fe570d-kube-api-access-kgjxk\") pod \"limitador-operator-controller-manager-7d6bb9cf99-phgp7\" (UID: \"0bffdd67-4688-4054-9e38-07f805fe570d\") " pod="kuadrant-system/limitador-operator-controller-manager-7d6bb9cf99-phgp7" Jul 21 15:11:19.710625 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:19.710593 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgjxk\" (UniqueName: \"kubernetes.io/projected/0bffdd67-4688-4054-9e38-07f805fe570d-kube-api-access-kgjxk\") pod \"limitador-operator-controller-manager-7d6bb9cf99-phgp7\" (UID: \"0bffdd67-4688-4054-9e38-07f805fe570d\") " pod="kuadrant-system/limitador-operator-controller-manager-7d6bb9cf99-phgp7" Jul 21 15:11:19.719915 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:19.719884 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgjxk\" (UniqueName: \"kubernetes.io/projected/0bffdd67-4688-4054-9e38-07f805fe570d-kube-api-access-kgjxk\") pod \"limitador-operator-controller-manager-7d6bb9cf99-phgp7\" (UID: \"0bffdd67-4688-4054-9e38-07f805fe570d\") " pod="kuadrant-system/limitador-operator-controller-manager-7d6bb9cf99-phgp7" Jul 21 15:11:19.785876 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:19.785804 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-7d6bb9cf99-phgp7" Jul 21 15:11:19.928753 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:19.928724 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-7d6bb9cf99-phgp7"] Jul 21 15:11:19.930181 ip-10-0-135-176 kubenswrapper[2563]: W0721 15:11:19.930154 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bffdd67_4688_4054_9e38_07f805fe570d.slice/crio-5a3a9695c0d46ee16c749dc67b08bbe064cf85e355501b8638b443cc09ba696e WatchSource:0}: Error finding container 5a3a9695c0d46ee16c749dc67b08bbe064cf85e355501b8638b443cc09ba696e: Status 404 returned error can't find the container with id 5a3a9695c0d46ee16c749dc67b08bbe064cf85e355501b8638b443cc09ba696e Jul 21 15:11:20.255282 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:20.255247 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-7d6bb9cf99-phgp7" event={"ID":"0bffdd67-4688-4054-9e38-07f805fe570d","Type":"ContainerStarted","Data":"5a3a9695c0d46ee16c749dc67b08bbe064cf85e355501b8638b443cc09ba696e"} Jul 21 15:11:22.180754 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:22.180650 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6b687b7676-d9v9s" podUID="7dd4bb5f-1fee-4053-a7e9-a9325ceec7bf" containerName="console" containerID="cri-o://c1c92474de6f93517ebfe855058d0e7d7d5c8815f95f1255ba6f98c3e4f61bb3" gracePeriod=15 Jul 21 15:11:22.215340 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:22.215295 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-558c6fb9c4-p288c"] Jul 21 15:11:22.218812 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:22.218795 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-558c6fb9c4-p288c" Jul 21 15:11:22.222944 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:22.222927 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-cqckr\"" Jul 21 15:11:22.231318 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:22.231295 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-558c6fb9c4-p288c"] Jul 21 15:11:22.275495 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:22.275464 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-7d6bb9cf99-phgp7" event={"ID":"0bffdd67-4688-4054-9e38-07f805fe570d","Type":"ContainerStarted","Data":"b032218750c2d6fc78c0e3e92566e0e400a95908fb5fa75b81ab77bd810f3bab"} Jul 21 15:11:22.275608 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:22.275570 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-7d6bb9cf99-phgp7" Jul 21 15:11:22.300826 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:22.300786 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-7d6bb9cf99-phgp7" podStartSLOduration=1.394442162 podStartE2EDuration="3.300771954s" totalImagesPullingTime="1.906329792s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 15:11:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 15:11:22.298057036 +0000 UTC m=+714.293842453" watchObservedRunningTime="2026-07-21 15:11:22.300771954 +0000 UTC m=+714.296557374" Jul 21 15:11:22.334045 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:22.334003 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rmp7\" (UniqueName: \"kubernetes.io/projected/e4b9fea5-b9a4-4844-888a-9a4835b73382-kube-api-access-5rmp7\") pod \"authorino-operator-558c6fb9c4-p288c\" (UID: \"e4b9fea5-b9a4-4844-888a-9a4835b73382\") " pod="kuadrant-system/authorino-operator-558c6fb9c4-p288c" Jul 21 15:11:22.424786 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:22.424763 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b687b7676-d9v9s" Jul 21 15:11:22.435198 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:22.435144 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rmp7\" (UniqueName: \"kubernetes.io/projected/e4b9fea5-b9a4-4844-888a-9a4835b73382-kube-api-access-5rmp7\") pod \"authorino-operator-558c6fb9c4-p288c\" (UID: \"e4b9fea5-b9a4-4844-888a-9a4835b73382\") " pod="kuadrant-system/authorino-operator-558c6fb9c4-p288c" Jul 21 15:11:22.444737 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:22.444716 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rmp7\" (UniqueName: \"kubernetes.io/projected/e4b9fea5-b9a4-4844-888a-9a4835b73382-kube-api-access-5rmp7\") pod \"authorino-operator-558c6fb9c4-p288c\" (UID: \"e4b9fea5-b9a4-4844-888a-9a4835b73382\") " pod="kuadrant-system/authorino-operator-558c6fb9c4-p288c" Jul 21 15:11:22.536488 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:22.536459 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7dd4bb5f-1fee-4053-a7e9-a9325ceec7bf-console-oauth-config\") pod \"7dd4bb5f-1fee-4053-a7e9-a9325ceec7bf\" (UID: \"7dd4bb5f-1fee-4053-a7e9-a9325ceec7bf\") " Jul 21 15:11:22.536652 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:22.536509 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7dd4bb5f-1fee-4053-a7e9-a9325ceec7bf-service-ca\") pod \"7dd4bb5f-1fee-4053-a7e9-a9325ceec7bf\" (UID: \"7dd4bb5f-1fee-4053-a7e9-a9325ceec7bf\") " Jul 21 15:11:22.536652 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:22.536535 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7dd4bb5f-1fee-4053-a7e9-a9325ceec7bf-oauth-serving-cert\") pod \"7dd4bb5f-1fee-4053-a7e9-a9325ceec7bf\" (UID: \"7dd4bb5f-1fee-4053-a7e9-a9325ceec7bf\") " Jul 21 15:11:22.536782 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:22.536657 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7dd4bb5f-1fee-4053-a7e9-a9325ceec7bf-trusted-ca-bundle\") pod \"7dd4bb5f-1fee-4053-a7e9-a9325ceec7bf\" (UID: \"7dd4bb5f-1fee-4053-a7e9-a9325ceec7bf\") " Jul 21 15:11:22.536782 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:22.536722 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7dd4bb5f-1fee-4053-a7e9-a9325ceec7bf-console-config\") pod \"7dd4bb5f-1fee-4053-a7e9-a9325ceec7bf\" (UID: \"7dd4bb5f-1fee-4053-a7e9-a9325ceec7bf\") " Jul 21 15:11:22.536782 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:22.536752 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-968db\" (UniqueName: \"kubernetes.io/projected/7dd4bb5f-1fee-4053-a7e9-a9325ceec7bf-kube-api-access-968db\") pod \"7dd4bb5f-1fee-4053-a7e9-a9325ceec7bf\" (UID: \"7dd4bb5f-1fee-4053-a7e9-a9325ceec7bf\") " Jul 21 15:11:22.536782 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:22.536774 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7dd4bb5f-1fee-4053-a7e9-a9325ceec7bf-console-serving-cert\") pod \"7dd4bb5f-1fee-4053-a7e9-a9325ceec7bf\" (UID: \"7dd4bb5f-1fee-4053-a7e9-a9325ceec7bf\") " Jul 21 15:11:22.536970 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:22.536900 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7dd4bb5f-1fee-4053-a7e9-a9325ceec7bf-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "7dd4bb5f-1fee-4053-a7e9-a9325ceec7bf" (UID: "7dd4bb5f-1fee-4053-a7e9-a9325ceec7bf"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 15:11:22.537088 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:22.537062 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7dd4bb5f-1fee-4053-a7e9-a9325ceec7bf-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "7dd4bb5f-1fee-4053-a7e9-a9325ceec7bf" (UID: "7dd4bb5f-1fee-4053-a7e9-a9325ceec7bf"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 15:11:22.537144 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:22.537067 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7dd4bb5f-1fee-4053-a7e9-a9325ceec7bf-service-ca" (OuterVolumeSpecName: "service-ca") pod "7dd4bb5f-1fee-4053-a7e9-a9325ceec7bf" (UID: "7dd4bb5f-1fee-4053-a7e9-a9325ceec7bf"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 15:11:22.537144 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:22.537108 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7dd4bb5f-1fee-4053-a7e9-a9325ceec7bf-console-config" (OuterVolumeSpecName: "console-config") pod "7dd4bb5f-1fee-4053-a7e9-a9325ceec7bf" (UID: "7dd4bb5f-1fee-4053-a7e9-a9325ceec7bf"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 15:11:22.537242 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:22.537168 2563 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7dd4bb5f-1fee-4053-a7e9-a9325ceec7bf-trusted-ca-bundle\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 15:11:22.537242 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:22.537183 2563 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7dd4bb5f-1fee-4053-a7e9-a9325ceec7bf-console-config\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 15:11:22.537242 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:22.537196 2563 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7dd4bb5f-1fee-4053-a7e9-a9325ceec7bf-service-ca\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 15:11:22.537242 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:22.537207 2563 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7dd4bb5f-1fee-4053-a7e9-a9325ceec7bf-oauth-serving-cert\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 15:11:22.538886 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:22.538866 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7dd4bb5f-1fee-4053-a7e9-a9325ceec7bf-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "7dd4bb5f-1fee-4053-a7e9-a9325ceec7bf" (UID: "7dd4bb5f-1fee-4053-a7e9-a9325ceec7bf"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 15:11:22.539013 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:22.538983 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7dd4bb5f-1fee-4053-a7e9-a9325ceec7bf-kube-api-access-968db" (OuterVolumeSpecName: "kube-api-access-968db") pod "7dd4bb5f-1fee-4053-a7e9-a9325ceec7bf" (UID: "7dd4bb5f-1fee-4053-a7e9-a9325ceec7bf"). InnerVolumeSpecName "kube-api-access-968db". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 15:11:22.539057 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:22.539011 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7dd4bb5f-1fee-4053-a7e9-a9325ceec7bf-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "7dd4bb5f-1fee-4053-a7e9-a9325ceec7bf" (UID: "7dd4bb5f-1fee-4053-a7e9-a9325ceec7bf"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 15:11:22.563252 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:22.563228 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-558c6fb9c4-p288c" Jul 21 15:11:22.638492 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:22.638451 2563 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7dd4bb5f-1fee-4053-a7e9-a9325ceec7bf-console-oauth-config\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 15:11:22.638492 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:22.638481 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-968db\" (UniqueName: \"kubernetes.io/projected/7dd4bb5f-1fee-4053-a7e9-a9325ceec7bf-kube-api-access-968db\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 15:11:22.638492 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:22.638494 2563 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7dd4bb5f-1fee-4053-a7e9-a9325ceec7bf-console-serving-cert\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 15:11:22.703526 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:22.703504 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-558c6fb9c4-p288c"] Jul 21 15:11:22.705013 ip-10-0-135-176 kubenswrapper[2563]: W0721 15:11:22.704991 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4b9fea5_b9a4_4844_888a_9a4835b73382.slice/crio-fd6e5cd2816ddbcd4c372a311f11cf30806e5fb14a01b5db74dc25672fe73766 WatchSource:0}: Error finding container fd6e5cd2816ddbcd4c372a311f11cf30806e5fb14a01b5db74dc25672fe73766: Status 404 returned error can't find the container with id fd6e5cd2816ddbcd4c372a311f11cf30806e5fb14a01b5db74dc25672fe73766 Jul 21 15:11:23.281805 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:23.281774 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-558c6fb9c4-p288c" event={"ID":"e4b9fea5-b9a4-4844-888a-9a4835b73382","Type":"ContainerStarted","Data":"fd6e5cd2816ddbcd4c372a311f11cf30806e5fb14a01b5db74dc25672fe73766"} Jul 21 15:11:23.283010 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:23.282988 2563 generic.go:356] "Generic (PLEG): container finished" podID="7dd4bb5f-1fee-4053-a7e9-a9325ceec7bf" containerID="c1c92474de6f93517ebfe855058d0e7d7d5c8815f95f1255ba6f98c3e4f61bb3" exitCode=0 Jul 21 15:11:23.283128 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:23.283045 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b687b7676-d9v9s" Jul 21 15:11:23.283128 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:23.283074 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b687b7676-d9v9s" event={"ID":"7dd4bb5f-1fee-4053-a7e9-a9325ceec7bf","Type":"ContainerDied","Data":"c1c92474de6f93517ebfe855058d0e7d7d5c8815f95f1255ba6f98c3e4f61bb3"} Jul 21 15:11:23.283128 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:23.283115 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b687b7676-d9v9s" event={"ID":"7dd4bb5f-1fee-4053-a7e9-a9325ceec7bf","Type":"ContainerDied","Data":"821ef0fb127b1e1eae0a2a839f1a40de8699cf90958ecb9ec3cbf0467a244b05"} Jul 21 15:11:23.283272 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:23.283136 2563 scope.go:117] "RemoveContainer" containerID="c1c92474de6f93517ebfe855058d0e7d7d5c8815f95f1255ba6f98c3e4f61bb3" Jul 21 15:11:23.291982 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:23.291968 2563 scope.go:117] "RemoveContainer" containerID="c1c92474de6f93517ebfe855058d0e7d7d5c8815f95f1255ba6f98c3e4f61bb3" Jul 21 15:11:23.292223 ip-10-0-135-176 kubenswrapper[2563]: E0721 15:11:23.292207 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1c92474de6f93517ebfe855058d0e7d7d5c8815f95f1255ba6f98c3e4f61bb3\": container with ID starting with c1c92474de6f93517ebfe855058d0e7d7d5c8815f95f1255ba6f98c3e4f61bb3 not found: ID does not exist" containerID="c1c92474de6f93517ebfe855058d0e7d7d5c8815f95f1255ba6f98c3e4f61bb3" Jul 21 15:11:23.292275 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:23.292230 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1c92474de6f93517ebfe855058d0e7d7d5c8815f95f1255ba6f98c3e4f61bb3"} err="failed to get container status \"c1c92474de6f93517ebfe855058d0e7d7d5c8815f95f1255ba6f98c3e4f61bb3\": rpc error: code = NotFound desc = could not find container \"c1c92474de6f93517ebfe855058d0e7d7d5c8815f95f1255ba6f98c3e4f61bb3\": container with ID starting with c1c92474de6f93517ebfe855058d0e7d7d5c8815f95f1255ba6f98c3e4f61bb3 not found: ID does not exist" Jul 21 15:11:23.313688 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:23.313660 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6b687b7676-d9v9s"] Jul 21 15:11:23.327418 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:23.327393 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6b687b7676-d9v9s"] Jul 21 15:11:24.479715 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:24.479681 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7dd4bb5f-1fee-4053-a7e9-a9325ceec7bf" path="/var/lib/kubelet/pods/7dd4bb5f-1fee-4053-a7e9-a9325ceec7bf/volumes" Jul 21 15:11:26.642689 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:26.642646 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6c647fc79-7rvpr"] Jul 21 15:11:26.643134 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:26.643114 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7dd4bb5f-1fee-4053-a7e9-a9325ceec7bf" containerName="console" Jul 21 15:11:26.643239 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:26.643138 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="7dd4bb5f-1fee-4053-a7e9-a9325ceec7bf" containerName="console" Jul 21 15:11:26.643239 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:26.643224 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="7dd4bb5f-1fee-4053-a7e9-a9325ceec7bf" containerName="console" Jul 21 15:11:26.646416 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:26.646394 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6c647fc79-7rvpr" Jul 21 15:11:26.649194 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:26.649159 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-fkrwq\"" Jul 21 15:11:26.659398 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:26.659375 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6c647fc79-7rvpr"] Jul 21 15:11:26.670019 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:26.669920 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/43144fbf-2c62-4246-adf2-e308216061b5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6c647fc79-7rvpr\" (UID: \"43144fbf-2c62-4246-adf2-e308216061b5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6c647fc79-7rvpr" Jul 21 15:11:26.670019 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:26.669962 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bxkl\" (UniqueName: \"kubernetes.io/projected/43144fbf-2c62-4246-adf2-e308216061b5-kube-api-access-7bxkl\") pod \"kuadrant-operator-controller-manager-6c647fc79-7rvpr\" (UID: \"43144fbf-2c62-4246-adf2-e308216061b5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6c647fc79-7rvpr" Jul 21 15:11:26.770604 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:26.770538 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/43144fbf-2c62-4246-adf2-e308216061b5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6c647fc79-7rvpr\" (UID: \"43144fbf-2c62-4246-adf2-e308216061b5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6c647fc79-7rvpr" Jul 21 15:11:26.770604 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:26.770583 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bxkl\" (UniqueName: \"kubernetes.io/projected/43144fbf-2c62-4246-adf2-e308216061b5-kube-api-access-7bxkl\") pod \"kuadrant-operator-controller-manager-6c647fc79-7rvpr\" (UID: \"43144fbf-2c62-4246-adf2-e308216061b5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6c647fc79-7rvpr" Jul 21 15:11:26.770975 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:26.770947 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/43144fbf-2c62-4246-adf2-e308216061b5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6c647fc79-7rvpr\" (UID: \"43144fbf-2c62-4246-adf2-e308216061b5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6c647fc79-7rvpr" Jul 21 15:11:26.780951 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:26.780921 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bxkl\" (UniqueName: \"kubernetes.io/projected/43144fbf-2c62-4246-adf2-e308216061b5-kube-api-access-7bxkl\") pod \"kuadrant-operator-controller-manager-6c647fc79-7rvpr\" (UID: \"43144fbf-2c62-4246-adf2-e308216061b5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6c647fc79-7rvpr" Jul 21 15:11:26.961033 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:26.960950 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6c647fc79-7rvpr" Jul 21 15:11:27.122583 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:27.122555 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6c647fc79-7rvpr"] Jul 21 15:11:27.124427 ip-10-0-135-176 kubenswrapper[2563]: W0721 15:11:27.124398 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43144fbf_2c62_4246_adf2_e308216061b5.slice/crio-dc2c40f7392fb87332217a8358023a7d753d10840c07839d613c66842db4f08b WatchSource:0}: Error finding container dc2c40f7392fb87332217a8358023a7d753d10840c07839d613c66842db4f08b: Status 404 returned error can't find the container with id dc2c40f7392fb87332217a8358023a7d753d10840c07839d613c66842db4f08b Jul 21 15:11:27.306253 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:27.306179 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6c647fc79-7rvpr" event={"ID":"43144fbf-2c62-4246-adf2-e308216061b5","Type":"ContainerStarted","Data":"dc2c40f7392fb87332217a8358023a7d753d10840c07839d613c66842db4f08b"} Jul 21 15:11:28.313242 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:28.313210 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-558c6fb9c4-p288c" event={"ID":"e4b9fea5-b9a4-4844-888a-9a4835b73382","Type":"ContainerStarted","Data":"e48110633b12b897728a08567fa6e1d71bab17c30953144f2424c81fb6330cf8"} Jul 21 15:11:28.313702 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:28.313281 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-558c6fb9c4-p288c" Jul 21 15:11:28.333146 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:28.333103 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-558c6fb9c4-p288c" podStartSLOduration=0.843113432 podStartE2EDuration="6.333088732s" totalImagesPullingTime="5.4899753s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 15:11:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 15:11:28.33182237 +0000 UTC m=+720.327607788" watchObservedRunningTime="2026-07-21 15:11:28.333088732 +0000 UTC m=+720.328874163" Jul 21 15:11:33.286627 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:33.286597 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-7d6bb9cf99-phgp7" Jul 21 15:11:35.353948 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:35.353914 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6c647fc79-7rvpr" event={"ID":"43144fbf-2c62-4246-adf2-e308216061b5","Type":"ContainerStarted","Data":"47c570d65f74db85c6911aed158e5d498f844013a5b2992751478948005e0728"} Jul 21 15:11:35.354362 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:35.354055 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6c647fc79-7rvpr" Jul 21 15:11:35.382561 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:35.382515 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6c647fc79-7rvpr" podStartSLOduration=1.508404166 podStartE2EDuration="9.382498497s" totalImagesPullingTime="7.874094331s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 15:11:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 15:11:35.379830541 +0000 UTC m=+727.375615957" watchObservedRunningTime="2026-07-21 15:11:35.382498497 +0000 UTC m=+727.378283913" Jul 21 15:11:37.599864 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:37.599830 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-c9z78"] Jul 21 15:11:37.604854 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:37.604832 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-c9z78" Jul 21 15:11:37.607756 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:37.607734 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-8gjhb\"" Jul 21 15:11:37.607903 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:37.607792 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 21 15:11:37.608058 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:37.608042 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 21 15:11:37.613727 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:37.613703 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-c9z78"] Jul 21 15:11:37.669474 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:37.669446 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qx22\" (UniqueName: \"kubernetes.io/projected/1da5d3b2-039e-41b4-9df7-23a127b47a57-kube-api-access-5qx22\") pod \"kuadrant-console-plugin-85fd4b8b9d-c9z78\" (UID: \"1da5d3b2-039e-41b4-9df7-23a127b47a57\") " pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-c9z78" Jul 21 15:11:37.669474 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:37.669478 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1da5d3b2-039e-41b4-9df7-23a127b47a57-nginx-conf\") pod \"kuadrant-console-plugin-85fd4b8b9d-c9z78\" (UID: \"1da5d3b2-039e-41b4-9df7-23a127b47a57\") " pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-c9z78" Jul 21 15:11:37.669775 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:37.669575 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/1da5d3b2-039e-41b4-9df7-23a127b47a57-plugin-serving-cert\") pod \"kuadrant-console-plugin-85fd4b8b9d-c9z78\" (UID: \"1da5d3b2-039e-41b4-9df7-23a127b47a57\") " pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-c9z78" Jul 21 15:11:37.770633 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:37.770595 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/1da5d3b2-039e-41b4-9df7-23a127b47a57-plugin-serving-cert\") pod \"kuadrant-console-plugin-85fd4b8b9d-c9z78\" (UID: \"1da5d3b2-039e-41b4-9df7-23a127b47a57\") " pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-c9z78" Jul 21 15:11:37.770788 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:37.770659 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qx22\" (UniqueName: \"kubernetes.io/projected/1da5d3b2-039e-41b4-9df7-23a127b47a57-kube-api-access-5qx22\") pod \"kuadrant-console-plugin-85fd4b8b9d-c9z78\" (UID: \"1da5d3b2-039e-41b4-9df7-23a127b47a57\") " pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-c9z78" Jul 21 15:11:37.770788 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:37.770676 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1da5d3b2-039e-41b4-9df7-23a127b47a57-nginx-conf\") pod \"kuadrant-console-plugin-85fd4b8b9d-c9z78\" (UID: \"1da5d3b2-039e-41b4-9df7-23a127b47a57\") " pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-c9z78" Jul 21 15:11:37.770788 ip-10-0-135-176 kubenswrapper[2563]: E0721 15:11:37.770753 2563 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found Jul 21 15:11:37.770892 ip-10-0-135-176 kubenswrapper[2563]: E0721 15:11:37.770835 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1da5d3b2-039e-41b4-9df7-23a127b47a57-plugin-serving-cert podName:1da5d3b2-039e-41b4-9df7-23a127b47a57 nodeName:}" failed. No retries permitted until 2026-07-21 15:11:38.270814721 +0000 UTC m=+730.266600117 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/1da5d3b2-039e-41b4-9df7-23a127b47a57-plugin-serving-cert") pod "kuadrant-console-plugin-85fd4b8b9d-c9z78" (UID: "1da5d3b2-039e-41b4-9df7-23a127b47a57") : secret "plugin-serving-cert" not found Jul 21 15:11:37.771221 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:37.771206 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1da5d3b2-039e-41b4-9df7-23a127b47a57-nginx-conf\") pod \"kuadrant-console-plugin-85fd4b8b9d-c9z78\" (UID: \"1da5d3b2-039e-41b4-9df7-23a127b47a57\") " pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-c9z78" Jul 21 15:11:37.782046 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:37.782024 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qx22\" (UniqueName: \"kubernetes.io/projected/1da5d3b2-039e-41b4-9df7-23a127b47a57-kube-api-access-5qx22\") pod \"kuadrant-console-plugin-85fd4b8b9d-c9z78\" (UID: \"1da5d3b2-039e-41b4-9df7-23a127b47a57\") " pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-c9z78" Jul 21 15:11:38.275847 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:38.275813 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/1da5d3b2-039e-41b4-9df7-23a127b47a57-plugin-serving-cert\") pod \"kuadrant-console-plugin-85fd4b8b9d-c9z78\" (UID: \"1da5d3b2-039e-41b4-9df7-23a127b47a57\") " pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-c9z78" Jul 21 15:11:38.278277 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:38.278259 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/1da5d3b2-039e-41b4-9df7-23a127b47a57-plugin-serving-cert\") pod \"kuadrant-console-plugin-85fd4b8b9d-c9z78\" (UID: \"1da5d3b2-039e-41b4-9df7-23a127b47a57\") " pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-c9z78" Jul 21 15:11:38.515196 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:38.515167 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-c9z78" Jul 21 15:11:38.663970 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:38.663942 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-c9z78"] Jul 21 15:11:38.664482 ip-10-0-135-176 kubenswrapper[2563]: W0721 15:11:38.664459 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1da5d3b2_039e_41b4_9df7_23a127b47a57.slice/crio-c11ae8f31e1cdcf576007233a551fdd905f461f411f4b6c1e17f0b3dc14dd575 WatchSource:0}: Error finding container c11ae8f31e1cdcf576007233a551fdd905f461f411f4b6c1e17f0b3dc14dd575: Status 404 returned error can't find the container with id c11ae8f31e1cdcf576007233a551fdd905f461f411f4b6c1e17f0b3dc14dd575 Jul 21 15:11:39.321027 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:39.320994 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-558c6fb9c4-p288c" Jul 21 15:11:39.377075 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:39.377044 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-c9z78" event={"ID":"1da5d3b2-039e-41b4-9df7-23a127b47a57","Type":"ContainerStarted","Data":"c11ae8f31e1cdcf576007233a551fdd905f461f411f4b6c1e17f0b3dc14dd575"} Jul 21 15:11:46.362023 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:46.361994 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6c647fc79-7rvpr" Jul 21 15:11:46.419498 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:46.419466 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-c9z78" event={"ID":"1da5d3b2-039e-41b4-9df7-23a127b47a57","Type":"ContainerStarted","Data":"7938d9fc8e0f76bf1f814e3582be7b274fe3d3d53c5ec747601822d64ec716a5"} Jul 21 15:11:46.438345 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:46.438286 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-c9z78" podStartSLOduration=2.539520989 podStartE2EDuration="9.438271879s" totalImagesPullingTime="6.89875089s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 15:11:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 15:11:46.435902741 +0000 UTC m=+738.431688160" watchObservedRunningTime="2026-07-21 15:11:46.438271879 +0000 UTC m=+738.434057346" Jul 21 15:11:48.216619 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:48.216588 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6c647fc79-7rvpr"] Jul 21 15:11:48.217046 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:48.216776 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6c647fc79-7rvpr" podUID="43144fbf-2c62-4246-adf2-e308216061b5" containerName="manager" containerID="cri-o://47c570d65f74db85c6911aed158e5d498f844013a5b2992751478948005e0728" gracePeriod=2 Jul 21 15:11:48.231442 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:48.231413 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6c647fc79-7rvpr"] Jul 21 15:11:48.240570 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:48.240543 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6c647fc79-lvnxg"] Jul 21 15:11:48.240894 ip-10-0-135-176 kubenswrapper[2563]: E0721 15:11:48.240867 2563 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"43144fbf-2c62-4246-adf2-e308216061b5\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-6c647fc79-7rvpr\": pods \"kuadrant-operator-controller-manager-6c647fc79-7rvpr\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-6c647fc79-7rvpr" Jul 21 15:11:48.240985 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:48.240971 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43144fbf-2c62-4246-adf2-e308216061b5" containerName="manager" Jul 21 15:11:48.240985 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:48.240984 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="43144fbf-2c62-4246-adf2-e308216061b5" containerName="manager" Jul 21 15:11:48.241093 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:48.241053 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="43144fbf-2c62-4246-adf2-e308216061b5" containerName="manager" Jul 21 15:11:48.244366 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:48.244313 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6c647fc79-lvnxg" Jul 21 15:11:48.247126 ip-10-0-135-176 kubenswrapper[2563]: E0721 15:11:48.247098 2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6c647fc79-7rvpr\" is forbidden: User \"system:node:ip-10-0-135-176.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-176.ec2.internal' and this object" podUID="43144fbf-2c62-4246-adf2-e308216061b5" pod="kuadrant-system/kuadrant-operator-controller-manager-6c647fc79-7rvpr" Jul 21 15:11:48.262392 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:48.262365 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6c647fc79-lvnxg"] Jul 21 15:11:48.268061 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:48.268036 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-7d6bb9cf99-phgp7"] Jul 21 15:11:48.268301 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:48.268277 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-7d6bb9cf99-phgp7" podUID="0bffdd67-4688-4054-9e38-07f805fe570d" containerName="manager" containerID="cri-o://b032218750c2d6fc78c0e3e92566e0e400a95908fb5fa75b81ab77bd810f3bab" gracePeriod=2 Jul 21 15:11:48.286810 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:48.286785 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-7d6bb9cf99-phgp7"] Jul 21 15:11:48.294753 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:48.294726 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-7d6bb9cf99-xw7fm"] Jul 21 15:11:48.295208 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:48.295189 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0bffdd67-4688-4054-9e38-07f805fe570d" containerName="manager" Jul 21 15:11:48.295208 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:48.295207 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0bffdd67-4688-4054-9e38-07f805fe570d" containerName="manager" Jul 21 15:11:48.295361 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:48.295274 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="0bffdd67-4688-4054-9e38-07f805fe570d" containerName="manager" Jul 21 15:11:48.297455 ip-10-0-135-176 kubenswrapper[2563]: E0721 15:11:48.297427 2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6c647fc79-7rvpr\" is forbidden: User \"system:node:ip-10-0-135-176.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-176.ec2.internal' and this object" podUID="43144fbf-2c62-4246-adf2-e308216061b5" pod="kuadrant-system/kuadrant-operator-controller-manager-6c647fc79-7rvpr" Jul 21 15:11:48.298566 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:48.298552 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-7d6bb9cf99-xw7fm" Jul 21 15:11:48.300517 ip-10-0-135-176 kubenswrapper[2563]: E0721 15:11:48.300494 2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-7d6bb9cf99-phgp7\" is forbidden: User \"system:node:ip-10-0-135-176.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-176.ec2.internal' and this object" podUID="0bffdd67-4688-4054-9e38-07f805fe570d" pod="kuadrant-system/limitador-operator-controller-manager-7d6bb9cf99-phgp7" Jul 21 15:11:48.302784 ip-10-0-135-176 kubenswrapper[2563]: E0721 15:11:48.302761 2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-7d6bb9cf99-phgp7\" is forbidden: User \"system:node:ip-10-0-135-176.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-176.ec2.internal' and this object" podUID="0bffdd67-4688-4054-9e38-07f805fe570d" pod="kuadrant-system/limitador-operator-controller-manager-7d6bb9cf99-phgp7" Jul 21 15:11:48.321787 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:48.321763 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-7d6bb9cf99-xw7fm"] Jul 21 15:11:48.337002 ip-10-0-135-176 kubenswrapper[2563]: E0721 15:11:48.336974 2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6c647fc79-7rvpr\" is forbidden: User \"system:node:ip-10-0-135-176.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-176.ec2.internal' and this object" podUID="43144fbf-2c62-4246-adf2-e308216061b5" pod="kuadrant-system/kuadrant-operator-controller-manager-6c647fc79-7rvpr" Jul 21 15:11:48.365981 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:48.365946 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmhvf\" (UniqueName: \"kubernetes.io/projected/b7a3e4c2-4cf1-48c9-a69c-5f3ba844253d-kube-api-access-nmhvf\") pod \"limitador-operator-controller-manager-7d6bb9cf99-xw7fm\" (UID: \"b7a3e4c2-4cf1-48c9-a69c-5f3ba844253d\") " pod="kuadrant-system/limitador-operator-controller-manager-7d6bb9cf99-xw7fm" Jul 21 15:11:48.366105 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:48.365995 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8v6d\" (UniqueName: \"kubernetes.io/projected/fb438ee9-195d-4be2-82b5-017f494087b6-kube-api-access-w8v6d\") pod \"kuadrant-operator-controller-manager-6c647fc79-lvnxg\" (UID: \"fb438ee9-195d-4be2-82b5-017f494087b6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6c647fc79-lvnxg" Jul 21 15:11:48.366105 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:48.366016 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fb438ee9-195d-4be2-82b5-017f494087b6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6c647fc79-lvnxg\" (UID: \"fb438ee9-195d-4be2-82b5-017f494087b6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6c647fc79-lvnxg" Jul 21 15:11:48.438915 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:48.438544 2563 generic.go:356] "Generic (PLEG): container finished" podID="43144fbf-2c62-4246-adf2-e308216061b5" containerID="47c570d65f74db85c6911aed158e5d498f844013a5b2992751478948005e0728" exitCode=0 Jul 21 15:11:48.443676 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:48.443637 2563 generic.go:356] "Generic (PLEG): container finished" podID="0bffdd67-4688-4054-9e38-07f805fe570d" containerID="b032218750c2d6fc78c0e3e92566e0e400a95908fb5fa75b81ab77bd810f3bab" exitCode=0 Jul 21 15:11:48.466615 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:48.466582 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmhvf\" (UniqueName: \"kubernetes.io/projected/b7a3e4c2-4cf1-48c9-a69c-5f3ba844253d-kube-api-access-nmhvf\") pod \"limitador-operator-controller-manager-7d6bb9cf99-xw7fm\" (UID: \"b7a3e4c2-4cf1-48c9-a69c-5f3ba844253d\") " pod="kuadrant-system/limitador-operator-controller-manager-7d6bb9cf99-xw7fm" Jul 21 15:11:48.466795 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:48.466630 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8v6d\" (UniqueName: \"kubernetes.io/projected/fb438ee9-195d-4be2-82b5-017f494087b6-kube-api-access-w8v6d\") pod \"kuadrant-operator-controller-manager-6c647fc79-lvnxg\" (UID: \"fb438ee9-195d-4be2-82b5-017f494087b6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6c647fc79-lvnxg" Jul 21 15:11:48.466795 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:48.466758 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fb438ee9-195d-4be2-82b5-017f494087b6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6c647fc79-lvnxg\" (UID: \"fb438ee9-195d-4be2-82b5-017f494087b6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6c647fc79-lvnxg" Jul 21 15:11:48.467263 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:48.467244 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fb438ee9-195d-4be2-82b5-017f494087b6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6c647fc79-lvnxg\" (UID: \"fb438ee9-195d-4be2-82b5-017f494087b6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6c647fc79-lvnxg" Jul 21 15:11:48.477136 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:48.477103 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8v6d\" (UniqueName: \"kubernetes.io/projected/fb438ee9-195d-4be2-82b5-017f494087b6-kube-api-access-w8v6d\") pod \"kuadrant-operator-controller-manager-6c647fc79-lvnxg\" (UID: \"fb438ee9-195d-4be2-82b5-017f494087b6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6c647fc79-lvnxg" Jul 21 15:11:48.480195 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:48.480167 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmhvf\" (UniqueName: \"kubernetes.io/projected/b7a3e4c2-4cf1-48c9-a69c-5f3ba844253d-kube-api-access-nmhvf\") pod \"limitador-operator-controller-manager-7d6bb9cf99-xw7fm\" (UID: \"b7a3e4c2-4cf1-48c9-a69c-5f3ba844253d\") " pod="kuadrant-system/limitador-operator-controller-manager-7d6bb9cf99-xw7fm" Jul 21 15:11:48.481090 ip-10-0-135-176 kubenswrapper[2563]: E0721 15:11:48.481056 2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6c647fc79-7rvpr\" is forbidden: User \"system:node:ip-10-0-135-176.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-176.ec2.internal' and this object" podUID="43144fbf-2c62-4246-adf2-e308216061b5" pod="kuadrant-system/kuadrant-operator-controller-manager-6c647fc79-7rvpr" Jul 21 15:11:48.483440 ip-10-0-135-176 kubenswrapper[2563]: E0721 15:11:48.483409 2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-7d6bb9cf99-phgp7\" is forbidden: User \"system:node:ip-10-0-135-176.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-176.ec2.internal' and this object" podUID="0bffdd67-4688-4054-9e38-07f805fe570d" pod="kuadrant-system/limitador-operator-controller-manager-7d6bb9cf99-phgp7" Jul 21 15:11:48.508061 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:48.508033 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6c647fc79-7rvpr" Jul 21 15:11:48.510277 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:48.510262 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-7d6bb9cf99-phgp7" Jul 21 15:11:48.567136 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:48.567110 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kgjxk\" (UniqueName: \"kubernetes.io/projected/0bffdd67-4688-4054-9e38-07f805fe570d-kube-api-access-kgjxk\") pod \"0bffdd67-4688-4054-9e38-07f805fe570d\" (UID: \"0bffdd67-4688-4054-9e38-07f805fe570d\") " Jul 21 15:11:48.567304 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:48.567203 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7bxkl\" (UniqueName: \"kubernetes.io/projected/43144fbf-2c62-4246-adf2-e308216061b5-kube-api-access-7bxkl\") pod \"43144fbf-2c62-4246-adf2-e308216061b5\" (UID: \"43144fbf-2c62-4246-adf2-e308216061b5\") " Jul 21 15:11:48.567304 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:48.567264 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/43144fbf-2c62-4246-adf2-e308216061b5-extensions-socket-volume\") pod \"43144fbf-2c62-4246-adf2-e308216061b5\" (UID: \"43144fbf-2c62-4246-adf2-e308216061b5\") " Jul 21 15:11:48.567739 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:48.567707 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43144fbf-2c62-4246-adf2-e308216061b5-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "43144fbf-2c62-4246-adf2-e308216061b5" (UID: "43144fbf-2c62-4246-adf2-e308216061b5"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 15:11:48.569485 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:48.569466 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43144fbf-2c62-4246-adf2-e308216061b5-kube-api-access-7bxkl" (OuterVolumeSpecName: "kube-api-access-7bxkl") pod "43144fbf-2c62-4246-adf2-e308216061b5" (UID: "43144fbf-2c62-4246-adf2-e308216061b5"). InnerVolumeSpecName "kube-api-access-7bxkl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 15:11:48.569707 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:48.569685 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bffdd67-4688-4054-9e38-07f805fe570d-kube-api-access-kgjxk" (OuterVolumeSpecName: "kube-api-access-kgjxk") pod "0bffdd67-4688-4054-9e38-07f805fe570d" (UID: "0bffdd67-4688-4054-9e38-07f805fe570d"). InnerVolumeSpecName "kube-api-access-kgjxk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 15:11:48.646467 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:48.646439 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6c647fc79-lvnxg" Jul 21 15:11:48.653589 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:48.653567 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-7d6bb9cf99-xw7fm" Jul 21 15:11:48.668871 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:48.668841 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7bxkl\" (UniqueName: \"kubernetes.io/projected/43144fbf-2c62-4246-adf2-e308216061b5-kube-api-access-7bxkl\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 15:11:48.668973 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:48.668876 2563 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/43144fbf-2c62-4246-adf2-e308216061b5-extensions-socket-volume\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 15:11:48.668973 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:48.668894 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kgjxk\" (UniqueName: \"kubernetes.io/projected/0bffdd67-4688-4054-9e38-07f805fe570d-kube-api-access-kgjxk\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 15:11:48.794880 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:48.794857 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6c647fc79-lvnxg"] Jul 21 15:11:48.796516 ip-10-0-135-176 kubenswrapper[2563]: W0721 15:11:48.796486 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb438ee9_195d_4be2_82b5_017f494087b6.slice/crio-817a8ce2fd3e0b6292b5da00a1a8f75e25e90b48d9423a25b605d7563f8f9b99 WatchSource:0}: Error finding container 817a8ce2fd3e0b6292b5da00a1a8f75e25e90b48d9423a25b605d7563f8f9b99: Status 404 returned error can't find the container with id 817a8ce2fd3e0b6292b5da00a1a8f75e25e90b48d9423a25b605d7563f8f9b99 Jul 21 15:11:48.816849 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:48.816774 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-7d6bb9cf99-xw7fm"] Jul 21 15:11:48.817848 ip-10-0-135-176 kubenswrapper[2563]: W0721 15:11:48.817821 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7a3e4c2_4cf1_48c9_a69c_5f3ba844253d.slice/crio-9cc551a78579cea1b9dab6922b4821995cba193aae438b5202eb62cd405d84d4 WatchSource:0}: Error finding container 9cc551a78579cea1b9dab6922b4821995cba193aae438b5202eb62cd405d84d4: Status 404 returned error can't find the container with id 9cc551a78579cea1b9dab6922b4821995cba193aae438b5202eb62cd405d84d4 Jul 21 15:11:49.451488 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:49.451460 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6c647fc79-7rvpr" Jul 21 15:11:49.451488 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:49.451478 2563 scope.go:117] "RemoveContainer" containerID="47c570d65f74db85c6911aed158e5d498f844013a5b2992751478948005e0728" Jul 21 15:11:49.453075 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:49.453056 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-7d6bb9cf99-phgp7" Jul 21 15:11:49.455194 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:49.455162 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-7d6bb9cf99-xw7fm" event={"ID":"b7a3e4c2-4cf1-48c9-a69c-5f3ba844253d","Type":"ContainerStarted","Data":"bb1431967bfd71db2b93d73daea978ca2abe5a5a54b8d91d1963a701003313ff"} Jul 21 15:11:49.455346 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:49.455224 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-7d6bb9cf99-xw7fm" Jul 21 15:11:49.455346 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:49.455250 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-7d6bb9cf99-xw7fm" event={"ID":"b7a3e4c2-4cf1-48c9-a69c-5f3ba844253d","Type":"ContainerStarted","Data":"9cc551a78579cea1b9dab6922b4821995cba193aae438b5202eb62cd405d84d4"} Jul 21 15:11:49.457216 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:49.457187 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6c647fc79-lvnxg" event={"ID":"fb438ee9-195d-4be2-82b5-017f494087b6","Type":"ContainerStarted","Data":"4653fe1bca4cfebfe6b0012efad097f11cc0d6196da380f81f387d9bae862568"} Jul 21 15:11:49.457376 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:49.457238 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6c647fc79-lvnxg" event={"ID":"fb438ee9-195d-4be2-82b5-017f494087b6","Type":"ContainerStarted","Data":"817a8ce2fd3e0b6292b5da00a1a8f75e25e90b48d9423a25b605d7563f8f9b99"} Jul 21 15:11:49.457376 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:49.457284 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6c647fc79-lvnxg" Jul 21 15:11:49.462467 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:49.462450 2563 scope.go:117] "RemoveContainer" containerID="b032218750c2d6fc78c0e3e92566e0e400a95908fb5fa75b81ab77bd810f3bab" Jul 21 15:11:49.502751 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:49.502680 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-7d6bb9cf99-xw7fm" podStartSLOduration=1.50266561 podStartE2EDuration="1.50266561s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 15:11:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 15:11:49.496882991 +0000 UTC m=+741.492668408" watchObservedRunningTime="2026-07-21 15:11:49.50266561 +0000 UTC m=+741.498451027" Jul 21 15:11:49.540379 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:49.540311 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6c647fc79-lvnxg" podStartSLOduration=1.540292332 podStartE2EDuration="1.540292332s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 15:11:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 15:11:49.53708425 +0000 UTC m=+741.532869665" watchObservedRunningTime="2026-07-21 15:11:49.540292332 +0000 UTC m=+741.536077752" Jul 21 15:11:50.480341 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:50.480289 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0bffdd67-4688-4054-9e38-07f805fe570d" path="/var/lib/kubelet/pods/0bffdd67-4688-4054-9e38-07f805fe570d/volumes" Jul 21 15:11:50.480713 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:11:50.480636 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="43144fbf-2c62-4246-adf2-e308216061b5" path="/var/lib/kubelet/pods/43144fbf-2c62-4246-adf2-e308216061b5/volumes" Jul 21 15:12:00.467531 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:12:00.467498 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-7d6bb9cf99-xw7fm" Jul 21 15:12:00.467993 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:12:00.467587 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6c647fc79-lvnxg" Jul 21 15:12:05.056038 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:12:05.056003 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6c647fc79-lvnxg"] Jul 21 15:12:05.056550 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:12:05.056187 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6c647fc79-lvnxg" podUID="fb438ee9-195d-4be2-82b5-017f494087b6" containerName="manager" containerID="cri-o://4653fe1bca4cfebfe6b0012efad097f11cc0d6196da380f81f387d9bae862568" gracePeriod=10 Jul 21 15:12:05.317874 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:12:05.317824 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6c647fc79-lvnxg" Jul 21 15:12:05.324954 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:12:05.324934 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7569bc7c69-g6dgt"] Jul 21 15:12:05.325267 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:12:05.325255 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb438ee9-195d-4be2-82b5-017f494087b6" containerName="manager" Jul 21 15:12:05.325313 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:12:05.325269 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb438ee9-195d-4be2-82b5-017f494087b6" containerName="manager" Jul 21 15:12:05.325370 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:12:05.325359 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb438ee9-195d-4be2-82b5-017f494087b6" containerName="manager" Jul 21 15:12:05.328666 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:12:05.328652 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7569bc7c69-g6dgt" Jul 21 15:12:05.354240 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:12:05.354210 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7569bc7c69-g6dgt"] Jul 21 15:12:05.408063 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:12:05.408038 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fb438ee9-195d-4be2-82b5-017f494087b6-extensions-socket-volume\") pod \"fb438ee9-195d-4be2-82b5-017f494087b6\" (UID: \"fb438ee9-195d-4be2-82b5-017f494087b6\") " Jul 21 15:12:05.408231 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:12:05.408105 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w8v6d\" (UniqueName: \"kubernetes.io/projected/fb438ee9-195d-4be2-82b5-017f494087b6-kube-api-access-w8v6d\") pod \"fb438ee9-195d-4be2-82b5-017f494087b6\" (UID: \"fb438ee9-195d-4be2-82b5-017f494087b6\") " Jul 21 15:12:05.408231 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:12:05.408218 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/84f73758-06c8-4238-8b98-d87c0937e1f0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7569bc7c69-g6dgt\" (UID: \"84f73758-06c8-4238-8b98-d87c0937e1f0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7569bc7c69-g6dgt" Jul 21 15:12:05.408377 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:12:05.408255 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncfk8\" (UniqueName: \"kubernetes.io/projected/84f73758-06c8-4238-8b98-d87c0937e1f0-kube-api-access-ncfk8\") pod \"kuadrant-operator-controller-manager-7569bc7c69-g6dgt\" (UID: \"84f73758-06c8-4238-8b98-d87c0937e1f0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7569bc7c69-g6dgt" Jul 21 15:12:05.408567 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:12:05.408543 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb438ee9-195d-4be2-82b5-017f494087b6-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "fb438ee9-195d-4be2-82b5-017f494087b6" (UID: "fb438ee9-195d-4be2-82b5-017f494087b6"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 15:12:05.410283 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:12:05.410260 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb438ee9-195d-4be2-82b5-017f494087b6-kube-api-access-w8v6d" (OuterVolumeSpecName: "kube-api-access-w8v6d") pod "fb438ee9-195d-4be2-82b5-017f494087b6" (UID: "fb438ee9-195d-4be2-82b5-017f494087b6"). InnerVolumeSpecName "kube-api-access-w8v6d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 15:12:05.509086 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:12:05.509061 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/84f73758-06c8-4238-8b98-d87c0937e1f0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7569bc7c69-g6dgt\" (UID: \"84f73758-06c8-4238-8b98-d87c0937e1f0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7569bc7c69-g6dgt" Jul 21 15:12:05.509179 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:12:05.509099 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncfk8\" (UniqueName: \"kubernetes.io/projected/84f73758-06c8-4238-8b98-d87c0937e1f0-kube-api-access-ncfk8\") pod \"kuadrant-operator-controller-manager-7569bc7c69-g6dgt\" (UID: \"84f73758-06c8-4238-8b98-d87c0937e1f0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7569bc7c69-g6dgt" Jul 21 15:12:05.509179 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:12:05.509141 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8v6d\" (UniqueName: \"kubernetes.io/projected/fb438ee9-195d-4be2-82b5-017f494087b6-kube-api-access-w8v6d\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 15:12:05.509179 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:12:05.509152 2563 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fb438ee9-195d-4be2-82b5-017f494087b6-extensions-socket-volume\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 15:12:05.509423 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:12:05.509406 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/84f73758-06c8-4238-8b98-d87c0937e1f0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7569bc7c69-g6dgt\" (UID: \"84f73758-06c8-4238-8b98-d87c0937e1f0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7569bc7c69-g6dgt" Jul 21 15:12:05.518955 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:12:05.518937 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncfk8\" (UniqueName: \"kubernetes.io/projected/84f73758-06c8-4238-8b98-d87c0937e1f0-kube-api-access-ncfk8\") pod \"kuadrant-operator-controller-manager-7569bc7c69-g6dgt\" (UID: \"84f73758-06c8-4238-8b98-d87c0937e1f0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7569bc7c69-g6dgt" Jul 21 15:12:05.544586 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:12:05.544559 2563 generic.go:356] "Generic (PLEG): container finished" podID="fb438ee9-195d-4be2-82b5-017f494087b6" containerID="4653fe1bca4cfebfe6b0012efad097f11cc0d6196da380f81f387d9bae862568" exitCode=0 Jul 21 15:12:05.544683 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:12:05.544598 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6c647fc79-lvnxg" Jul 21 15:12:05.544683 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:12:05.544642 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6c647fc79-lvnxg" event={"ID":"fb438ee9-195d-4be2-82b5-017f494087b6","Type":"ContainerDied","Data":"4653fe1bca4cfebfe6b0012efad097f11cc0d6196da380f81f387d9bae862568"} Jul 21 15:12:05.544773 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:12:05.544693 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6c647fc79-lvnxg" event={"ID":"fb438ee9-195d-4be2-82b5-017f494087b6","Type":"ContainerDied","Data":"817a8ce2fd3e0b6292b5da00a1a8f75e25e90b48d9423a25b605d7563f8f9b99"} Jul 21 15:12:05.544773 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:12:05.544718 2563 scope.go:117] "RemoveContainer" containerID="4653fe1bca4cfebfe6b0012efad097f11cc0d6196da380f81f387d9bae862568" Jul 21 15:12:05.553737 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:12:05.553720 2563 scope.go:117] "RemoveContainer" containerID="4653fe1bca4cfebfe6b0012efad097f11cc0d6196da380f81f387d9bae862568" Jul 21 15:12:05.553977 ip-10-0-135-176 kubenswrapper[2563]: E0721 15:12:05.553961 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4653fe1bca4cfebfe6b0012efad097f11cc0d6196da380f81f387d9bae862568\": container with ID starting with 4653fe1bca4cfebfe6b0012efad097f11cc0d6196da380f81f387d9bae862568 not found: ID does not exist" containerID="4653fe1bca4cfebfe6b0012efad097f11cc0d6196da380f81f387d9bae862568" Jul 21 15:12:05.554025 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:12:05.553985 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4653fe1bca4cfebfe6b0012efad097f11cc0d6196da380f81f387d9bae862568"} err="failed to get container status \"4653fe1bca4cfebfe6b0012efad097f11cc0d6196da380f81f387d9bae862568\": rpc error: code = NotFound desc = could not find container \"4653fe1bca4cfebfe6b0012efad097f11cc0d6196da380f81f387d9bae862568\": container with ID starting with 4653fe1bca4cfebfe6b0012efad097f11cc0d6196da380f81f387d9bae862568 not found: ID does not exist" Jul 21 15:12:05.574759 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:12:05.574681 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6c647fc79-lvnxg"] Jul 21 15:12:05.576694 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:12:05.576678 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6c647fc79-lvnxg"] Jul 21 15:12:05.637753 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:12:05.637733 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7569bc7c69-g6dgt" Jul 21 15:12:05.806994 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:12:05.806971 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7569bc7c69-g6dgt"] Jul 21 15:12:05.808230 ip-10-0-135-176 kubenswrapper[2563]: W0721 15:12:05.808204 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84f73758_06c8_4238_8b98_d87c0937e1f0.slice/crio-72b2448bbe5c0b098b52283f0817c339e3b5d4d0d891dfa4589a6b359d5dede3 WatchSource:0}: Error finding container 72b2448bbe5c0b098b52283f0817c339e3b5d4d0d891dfa4589a6b359d5dede3: Status 404 returned error can't find the container with id 72b2448bbe5c0b098b52283f0817c339e3b5d4d0d891dfa4589a6b359d5dede3 Jul 21 15:12:06.486474 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:12:06.486057 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb438ee9-195d-4be2-82b5-017f494087b6" path="/var/lib/kubelet/pods/fb438ee9-195d-4be2-82b5-017f494087b6/volumes" Jul 21 15:12:06.552774 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:12:06.552737 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-7569bc7c69-g6dgt" event={"ID":"84f73758-06c8-4238-8b98-d87c0937e1f0","Type":"ContainerStarted","Data":"bc5c15dbc89c230cd15d0eb3309a3c5383eecef7fbc9040d8ff66743119f6161"} Jul 21 15:12:06.552931 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:12:06.552885 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-7569bc7c69-g6dgt" Jul 21 15:12:06.553006 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:12:06.552928 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-7569bc7c69-g6dgt" event={"ID":"84f73758-06c8-4238-8b98-d87c0937e1f0","Type":"ContainerStarted","Data":"72b2448bbe5c0b098b52283f0817c339e3b5d4d0d891dfa4589a6b359d5dede3"} Jul 21 15:12:06.586363 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:12:06.586306 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-7569bc7c69-g6dgt" podStartSLOduration=1.586292531 podStartE2EDuration="1.586292531s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 15:12:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 15:12:06.584553528 +0000 UTC m=+758.580338944" watchObservedRunningTime="2026-07-21 15:12:06.586292531 +0000 UTC m=+758.582077949" Jul 21 15:12:17.561481 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:12:17.561449 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-7569bc7c69-g6dgt" Jul 21 15:12:21.591282 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:12:21.591254 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-klhxd"] Jul 21 15:12:21.613643 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:12:21.613612 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-klhxd" Jul 21 15:12:21.615660 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:12:21.615632 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-klhxd"] Jul 21 15:12:21.620068 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:12:21.620051 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-wd9fw\"" Jul 21 15:12:21.732706 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:12:21.732680 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2e9e18c-5caf-48ef-82d0-16411ada9674-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5896d4dd49-klhxd\" (UID: \"e2e9e18c-5caf-48ef-82d0-16411ada9674\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-klhxd" Jul 21 15:12:21.732863 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:12:21.732725 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2e9e18c-5caf-48ef-82d0-16411ada9674-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5896d4dd49-klhxd\" (UID: \"e2e9e18c-5caf-48ef-82d0-16411ada9674\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-klhxd" Jul 21 15:12:21.732863 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:12:21.732763 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqg2p\" (UniqueName: \"kubernetes.io/projected/e2e9e18c-5caf-48ef-82d0-16411ada9674-kube-api-access-dqg2p\") pod \"maas-default-gateway-openshift-default-5896d4dd49-klhxd\" (UID: \"e2e9e18c-5caf-48ef-82d0-16411ada9674\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-klhxd" Jul 21 15:12:21.732863 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:12:21.732803 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2e9e18c-5caf-48ef-82d0-16411ada9674-istio-token\") pod \"maas-default-gateway-openshift-default-5896d4dd49-klhxd\" (UID: \"e2e9e18c-5caf-48ef-82d0-16411ada9674\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-klhxd" Jul 21 15:12:21.732863 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:12:21.732828 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2e9e18c-5caf-48ef-82d0-16411ada9674-istio-data\") pod \"maas-default-gateway-openshift-default-5896d4dd49-klhxd\" (UID: \"e2e9e18c-5caf-48ef-82d0-16411ada9674\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-klhxd" Jul 21 15:12:21.733051 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:12:21.732869 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2e9e18c-5caf-48ef-82d0-16411ada9674-workload-certs\") pod \"maas-default-gateway-openshift-default-5896d4dd49-klhxd\" (UID: \"e2e9e18c-5caf-48ef-82d0-16411ada9674\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-klhxd" Jul 21 15:12:21.733051 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:12:21.732902 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2e9e18c-5caf-48ef-82d0-16411ada9674-workload-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-klhxd\" (UID: \"e2e9e18c-5caf-48ef-82d0-16411ada9674\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-klhxd" Jul 21 15:12:21.733051 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:12:21.732927 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2e9e18c-5caf-48ef-82d0-16411ada9674-credential-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-klhxd\" (UID: \"e2e9e18c-5caf-48ef-82d0-16411ada9674\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-klhxd" Jul 21 15:12:21.733051 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:12:21.732984 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2e9e18c-5caf-48ef-82d0-16411ada9674-istio-envoy\") pod \"maas-default-gateway-openshift-default-5896d4dd49-klhxd\" (UID: \"e2e9e18c-5caf-48ef-82d0-16411ada9674\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-klhxd" Jul 21 15:12:21.833866 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:12:21.833842 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2e9e18c-5caf-48ef-82d0-16411ada9674-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5896d4dd49-klhxd\" (UID: \"e2e9e18c-5caf-48ef-82d0-16411ada9674\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-klhxd" Jul 21 15:12:21.834001 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:12:21.833876 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2e9e18c-5caf-48ef-82d0-16411ada9674-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5896d4dd49-klhxd\" (UID: \"e2e9e18c-5caf-48ef-82d0-16411ada9674\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-klhxd" Jul 21 15:12:21.834001 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:12:21.833898 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqg2p\" (UniqueName: \"kubernetes.io/projected/e2e9e18c-5caf-48ef-82d0-16411ada9674-kube-api-access-dqg2p\") pod \"maas-default-gateway-openshift-default-5896d4dd49-klhxd\" (UID: \"e2e9e18c-5caf-48ef-82d0-16411ada9674\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-klhxd" Jul 21 15:12:21.834001 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:12:21.833923 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2e9e18c-5caf-48ef-82d0-16411ada9674-istio-token\") pod \"maas-default-gateway-openshift-default-5896d4dd49-klhxd\" (UID: \"e2e9e18c-5caf-48ef-82d0-16411ada9674\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-klhxd" Jul 21 15:12:21.834001 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:12:21.833944 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2e9e18c-5caf-48ef-82d0-16411ada9674-istio-data\") pod \"maas-default-gateway-openshift-default-5896d4dd49-klhxd\" (UID: \"e2e9e18c-5caf-48ef-82d0-16411ada9674\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-klhxd" Jul 21 15:12:21.834001 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:12:21.833983 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2e9e18c-5caf-48ef-82d0-16411ada9674-workload-certs\") pod \"maas-default-gateway-openshift-default-5896d4dd49-klhxd\" (UID: \"e2e9e18c-5caf-48ef-82d0-16411ada9674\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-klhxd" Jul 21 15:12:21.834239 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:12:21.834013 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2e9e18c-5caf-48ef-82d0-16411ada9674-workload-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-klhxd\" (UID: \"e2e9e18c-5caf-48ef-82d0-16411ada9674\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-klhxd" Jul 21 15:12:21.834239 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:12:21.834036 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2e9e18c-5caf-48ef-82d0-16411ada9674-credential-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-klhxd\" (UID: \"e2e9e18c-5caf-48ef-82d0-16411ada9674\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-klhxd" Jul 21 15:12:21.834239 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:12:21.834076 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2e9e18c-5caf-48ef-82d0-16411ada9674-istio-envoy\") pod \"maas-default-gateway-openshift-default-5896d4dd49-klhxd\" (UID: \"e2e9e18c-5caf-48ef-82d0-16411ada9674\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-klhxd" Jul 21 15:12:21.834441 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:12:21.834417 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2e9e18c-5caf-48ef-82d0-16411ada9674-workload-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-klhxd\" (UID: \"e2e9e18c-5caf-48ef-82d0-16411ada9674\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-klhxd" Jul 21 15:12:21.834557 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:12:21.834490 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2e9e18c-5caf-48ef-82d0-16411ada9674-workload-certs\") pod \"maas-default-gateway-openshift-default-5896d4dd49-klhxd\" (UID: \"e2e9e18c-5caf-48ef-82d0-16411ada9674\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-klhxd" Jul 21 15:12:21.834608 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:12:21.834556 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2e9e18c-5caf-48ef-82d0-16411ada9674-istio-data\") pod \"maas-default-gateway-openshift-default-5896d4dd49-klhxd\" (UID: \"e2e9e18c-5caf-48ef-82d0-16411ada9674\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-klhxd" Jul 21 15:12:21.834608 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:12:21.834589 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2e9e18c-5caf-48ef-82d0-16411ada9674-credential-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-klhxd\" (UID: \"e2e9e18c-5caf-48ef-82d0-16411ada9674\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-klhxd" Jul 21 15:12:21.834854 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:12:21.834838 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2e9e18c-5caf-48ef-82d0-16411ada9674-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5896d4dd49-klhxd\" (UID: \"e2e9e18c-5caf-48ef-82d0-16411ada9674\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-klhxd" Jul 21 15:12:21.836913 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:12:21.836893 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2e9e18c-5caf-48ef-82d0-16411ada9674-istio-envoy\") pod \"maas-default-gateway-openshift-default-5896d4dd49-klhxd\" (UID: \"e2e9e18c-5caf-48ef-82d0-16411ada9674\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-klhxd" Jul 21 15:12:21.837200 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:12:21.837180 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2e9e18c-5caf-48ef-82d0-16411ada9674-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5896d4dd49-klhxd\" (UID: \"e2e9e18c-5caf-48ef-82d0-16411ada9674\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-klhxd" Jul 21 15:12:21.842832 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:12:21.842781 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqg2p\" (UniqueName: \"kubernetes.io/projected/e2e9e18c-5caf-48ef-82d0-16411ada9674-kube-api-access-dqg2p\") pod \"maas-default-gateway-openshift-default-5896d4dd49-klhxd\" (UID: \"e2e9e18c-5caf-48ef-82d0-16411ada9674\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-klhxd" Jul 21 15:12:21.842832 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:12:21.842784 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2e9e18c-5caf-48ef-82d0-16411ada9674-istio-token\") pod \"maas-default-gateway-openshift-default-5896d4dd49-klhxd\" (UID: \"e2e9e18c-5caf-48ef-82d0-16411ada9674\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-klhxd" Jul 21 15:12:21.925338 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:12:21.925304 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-klhxd" Jul 21 15:12:22.059968 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:12:22.059944 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-klhxd"] Jul 21 15:12:22.061928 ip-10-0-135-176 kubenswrapper[2563]: W0721 15:12:22.061895 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2e9e18c_5caf_48ef_82d0_16411ada9674.slice/crio-9a397899ec4e527636e997c667c08c56c21b212bf4582ede091923977f8f4f17 WatchSource:0}: Error finding container 9a397899ec4e527636e997c667c08c56c21b212bf4582ede091923977f8f4f17: Status 404 returned error can't find the container with id 9a397899ec4e527636e997c667c08c56c21b212bf4582ede091923977f8f4f17 Jul 21 15:12:22.063970 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:12:22.063938 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 21 15:12:22.064079 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:12:22.064024 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 21 15:12:22.064079 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:12:22.064068 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 21 15:12:22.640180 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:12:22.640151 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-klhxd" event={"ID":"e2e9e18c-5caf-48ef-82d0-16411ada9674","Type":"ContainerStarted","Data":"846d2a33abad9f847c1ccd0484507bf14c8752b00fec2bafacc9fce95959cb91"} Jul 21 15:12:22.640560 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:12:22.640211 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-klhxd" event={"ID":"e2e9e18c-5caf-48ef-82d0-16411ada9674","Type":"ContainerStarted","Data":"9a397899ec4e527636e997c667c08c56c21b212bf4582ede091923977f8f4f17"} Jul 21 15:12:22.663513 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:12:22.663469 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-klhxd" podStartSLOduration=1.663453589 podStartE2EDuration="1.663453589s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 15:12:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 15:12:22.661354242 +0000 UTC m=+774.657139654" watchObservedRunningTime="2026-07-21 15:12:22.663453589 +0000 UTC m=+774.659239006" Jul 21 15:12:22.925456 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:12:22.925379 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-klhxd" Jul 21 15:12:23.930521 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:12:23.930490 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-klhxd" Jul 21 15:12:24.651293 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:12:24.651261 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-klhxd" Jul 21 15:12:24.652108 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:12:24.652090 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-klhxd" Jul 21 15:12:37.388821 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:12:37.388783 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-j4c8p"] Jul 21 15:12:37.393913 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:12:37.393890 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-69574b596d-j4c8p" Jul 21 15:12:37.396905 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:12:37.396885 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 21 15:12:37.402692 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:12:37.402671 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-j4c8p"] Jul 21 15:12:37.445126 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:12:37.445096 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-j4c8p"] Jul 21 15:12:37.460450 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:12:37.460421 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-545dm\" (UniqueName: \"kubernetes.io/projected/40d18d90-5c0e-48f2-bf99-eed3392a66ee-kube-api-access-545dm\") pod \"limitador-limitador-69574b596d-j4c8p\" (UID: \"40d18d90-5c0e-48f2-bf99-eed3392a66ee\") " pod="kuadrant-system/limitador-limitador-69574b596d-j4c8p" Jul 21 15:12:37.460571 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:12:37.460459 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/40d18d90-5c0e-48f2-bf99-eed3392a66ee-config-file\") pod \"limitador-limitador-69574b596d-j4c8p\" (UID: \"40d18d90-5c0e-48f2-bf99-eed3392a66ee\") " pod="kuadrant-system/limitador-limitador-69574b596d-j4c8p" Jul 21 15:12:37.561813 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:12:37.561784 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-545dm\" (UniqueName: \"kubernetes.io/projected/40d18d90-5c0e-48f2-bf99-eed3392a66ee-kube-api-access-545dm\") pod \"limitador-limitador-69574b596d-j4c8p\" (UID: \"40d18d90-5c0e-48f2-bf99-eed3392a66ee\") " pod="kuadrant-system/limitador-limitador-69574b596d-j4c8p" Jul 21 15:12:37.561960 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:12:37.561844 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/40d18d90-5c0e-48f2-bf99-eed3392a66ee-config-file\") pod \"limitador-limitador-69574b596d-j4c8p\" (UID: \"40d18d90-5c0e-48f2-bf99-eed3392a66ee\") " pod="kuadrant-system/limitador-limitador-69574b596d-j4c8p" Jul 21 15:12:37.562493 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:12:37.562472 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/40d18d90-5c0e-48f2-bf99-eed3392a66ee-config-file\") pod \"limitador-limitador-69574b596d-j4c8p\" (UID: \"40d18d90-5c0e-48f2-bf99-eed3392a66ee\") " pod="kuadrant-system/limitador-limitador-69574b596d-j4c8p" Jul 21 15:12:37.570565 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:12:37.570540 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-545dm\" (UniqueName: \"kubernetes.io/projected/40d18d90-5c0e-48f2-bf99-eed3392a66ee-kube-api-access-545dm\") pod \"limitador-limitador-69574b596d-j4c8p\" (UID: \"40d18d90-5c0e-48f2-bf99-eed3392a66ee\") " pod="kuadrant-system/limitador-limitador-69574b596d-j4c8p" Jul 21 15:12:37.704700 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:12:37.704631 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-69574b596d-j4c8p" Jul 21 15:12:37.845113 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:12:37.845089 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-j4c8p"] Jul 21 15:12:37.846595 ip-10-0-135-176 kubenswrapper[2563]: W0721 15:12:37.846564 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40d18d90_5c0e_48f2_bf99_eed3392a66ee.slice/crio-4482d3c3121d8c4568a4fa3bd9916caea6bb266c021f88e6a46222ae75801c9d WatchSource:0}: Error finding container 4482d3c3121d8c4568a4fa3bd9916caea6bb266c021f88e6a46222ae75801c9d: Status 404 returned error can't find the container with id 4482d3c3121d8c4568a4fa3bd9916caea6bb266c021f88e6a46222ae75801c9d Jul 21 15:12:38.725102 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:12:38.725068 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-69574b596d-j4c8p" event={"ID":"40d18d90-5c0e-48f2-bf99-eed3392a66ee","Type":"ContainerStarted","Data":"4482d3c3121d8c4568a4fa3bd9916caea6bb266c021f88e6a46222ae75801c9d"} Jul 21 15:12:42.750245 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:12:42.750211 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-69574b596d-j4c8p" event={"ID":"40d18d90-5c0e-48f2-bf99-eed3392a66ee","Type":"ContainerStarted","Data":"dee6a636895e75c78fc6623167036df57c7050674860c9474bae0c1ecdb83d0c"} Jul 21 15:12:42.750633 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:12:42.750358 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-69574b596d-j4c8p" Jul 21 15:12:42.767828 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:12:42.767786 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-69574b596d-j4c8p" podStartSLOduration=1.5869059650000001 podStartE2EDuration="5.767774727s" totalImagesPullingTime="4.180868762s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 15:12:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 15:12:42.766968988 +0000 UTC m=+794.762754422" watchObservedRunningTime="2026-07-21 15:12:42.767774727 +0000 UTC m=+794.763560142" Jul 21 15:12:53.757498 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:12:53.757424 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-69574b596d-j4c8p" Jul 21 15:13:09.261867 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:13:09.261828 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7c8458985d-gkl6v"] Jul 21 15:13:09.280026 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:13:09.279971 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7c8458985d-gkl6v"] Jul 21 15:13:09.280026 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:13:09.280016 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7c8458985d-gkl6v" Jul 21 15:13:09.284666 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:13:09.284646 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 21 15:13:09.284812 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:13:09.284691 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-rmczl\"" Jul 21 15:13:09.447904 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:13:09.447869 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znlwp\" (UniqueName: \"kubernetes.io/projected/9be5a277-9187-4315-8c00-493432ae68dc-kube-api-access-znlwp\") pod \"authorino-7c8458985d-gkl6v\" (UID: \"9be5a277-9187-4315-8c00-493432ae68dc\") " pod="kuadrant-system/authorino-7c8458985d-gkl6v" Jul 21 15:13:09.448155 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:13:09.447985 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9be5a277-9187-4315-8c00-493432ae68dc-tls-cert\") pod \"authorino-7c8458985d-gkl6v\" (UID: \"9be5a277-9187-4315-8c00-493432ae68dc\") " pod="kuadrant-system/authorino-7c8458985d-gkl6v" Jul 21 15:13:09.548619 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:13:09.548520 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znlwp\" (UniqueName: \"kubernetes.io/projected/9be5a277-9187-4315-8c00-493432ae68dc-kube-api-access-znlwp\") pod \"authorino-7c8458985d-gkl6v\" (UID: \"9be5a277-9187-4315-8c00-493432ae68dc\") " pod="kuadrant-system/authorino-7c8458985d-gkl6v" Jul 21 15:13:09.548795 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:13:09.548629 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9be5a277-9187-4315-8c00-493432ae68dc-tls-cert\") pod \"authorino-7c8458985d-gkl6v\" (UID: \"9be5a277-9187-4315-8c00-493432ae68dc\") " pod="kuadrant-system/authorino-7c8458985d-gkl6v" Jul 21 15:13:09.551161 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:13:09.551137 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9be5a277-9187-4315-8c00-493432ae68dc-tls-cert\") pod \"authorino-7c8458985d-gkl6v\" (UID: \"9be5a277-9187-4315-8c00-493432ae68dc\") " pod="kuadrant-system/authorino-7c8458985d-gkl6v" Jul 21 15:13:09.556773 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:13:09.556749 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-znlwp\" (UniqueName: \"kubernetes.io/projected/9be5a277-9187-4315-8c00-493432ae68dc-kube-api-access-znlwp\") pod \"authorino-7c8458985d-gkl6v\" (UID: \"9be5a277-9187-4315-8c00-493432ae68dc\") " pod="kuadrant-system/authorino-7c8458985d-gkl6v" Jul 21 15:13:09.589776 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:13:09.589745 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7c8458985d-gkl6v" Jul 21 15:13:09.730202 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:13:09.730162 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7c8458985d-gkl6v"] Jul 21 15:13:09.732446 ip-10-0-135-176 kubenswrapper[2563]: W0721 15:13:09.732421 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9be5a277_9187_4315_8c00_493432ae68dc.slice/crio-f75bc68219b62dad55fec5c3bf777ec26d80835a1d6ad9bde3e67490ab3849ae WatchSource:0}: Error finding container f75bc68219b62dad55fec5c3bf777ec26d80835a1d6ad9bde3e67490ab3849ae: Status 404 returned error can't find the container with id f75bc68219b62dad55fec5c3bf777ec26d80835a1d6ad9bde3e67490ab3849ae Jul 21 15:13:09.895625 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:13:09.895577 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7c8458985d-gkl6v" event={"ID":"9be5a277-9187-4315-8c00-493432ae68dc","Type":"ContainerStarted","Data":"f75bc68219b62dad55fec5c3bf777ec26d80835a1d6ad9bde3e67490ab3849ae"} Jul 21 15:13:12.914828 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:13:12.914797 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7c8458985d-gkl6v" event={"ID":"9be5a277-9187-4315-8c00-493432ae68dc","Type":"ContainerStarted","Data":"c33776a0751235c2777dd32f0356ee5486fb1b31b2d1669fd5056c24e62a36d7"} Jul 21 15:13:12.933413 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:13:12.933362 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7c8458985d-gkl6v" podStartSLOduration=0.932014615 podStartE2EDuration="3.933318254s" totalImagesPullingTime="3.001303639s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 15:13:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 15:13:12.931481084 +0000 UTC m=+824.927266552" watchObservedRunningTime="2026-07-21 15:13:12.933318254 +0000 UTC m=+824.929103668" Jul 21 15:13:55.696336 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:13:55.696297 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-j4c8p"] Jul 21 15:14:05.198654 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:14:05.198620 2563 generic.go:356] "Generic (PLEG): container finished" podID="e2e9e18c-5caf-48ef-82d0-16411ada9674" containerID="846d2a33abad9f847c1ccd0484507bf14c8752b00fec2bafacc9fce95959cb91" exitCode=0 Jul 21 15:14:05.199078 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:14:05.198697 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-klhxd" event={"ID":"e2e9e18c-5caf-48ef-82d0-16411ada9674","Type":"ContainerDied","Data":"846d2a33abad9f847c1ccd0484507bf14c8752b00fec2bafacc9fce95959cb91"} Jul 21 15:14:05.199078 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:14:05.199071 2563 scope.go:117] "RemoveContainer" containerID="846d2a33abad9f847c1ccd0484507bf14c8752b00fec2bafacc9fce95959cb91" Jul 21 15:14:05.200052 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:14:05.200021 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 21 15:14:05.200105 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:14:05.200092 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 21 15:14:05.200144 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:14:05.200128 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 21 15:14:06.209409 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:14:06.209375 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-klhxd" event={"ID":"e2e9e18c-5caf-48ef-82d0-16411ada9674","Type":"ContainerStarted","Data":"36e20843f14b65c5ee0f766baada354ef3151c21a690c2c625cb8f9cb71799fb"} Jul 21 15:14:06.925556 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:14:06.925515 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-klhxd" Jul 21 15:14:06.926790 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:14:06.926767 2563 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-klhxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Jul 21 15:14:06.926845 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:14:06.926821 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-klhxd" podUID="e2e9e18c-5caf-48ef-82d0-16411ada9674" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Jul 21 15:14:07.926736 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:14:07.926698 2563 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-klhxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Jul 21 15:14:07.927150 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:14:07.926757 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-klhxd" podUID="e2e9e18c-5caf-48ef-82d0-16411ada9674" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Jul 21 15:14:08.926066 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:14:08.926024 2563 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-klhxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Jul 21 15:14:08.926249 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:14:08.926088 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-klhxd" podUID="e2e9e18c-5caf-48ef-82d0-16411ada9674" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Jul 21 15:14:09.651368 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:14:09.651309 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-klhxd" Jul 21 15:14:09.925806 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:14:09.925704 2563 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-klhxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Jul 21 15:14:09.925806 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:14:09.925766 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-klhxd" podUID="e2e9e18c-5caf-48ef-82d0-16411ada9674" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Jul 21 15:14:10.926740 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:14:10.926698 2563 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-klhxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Jul 21 15:14:10.927179 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:14:10.926770 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-klhxd" podUID="e2e9e18c-5caf-48ef-82d0-16411ada9674" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Jul 21 15:14:11.930111 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:14:11.930077 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-klhxd" Jul 21 15:14:11.931114 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:14:11.931099 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-klhxd" Jul 21 15:14:22.143432 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:14:22.143393 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-j4c8p"] Jul 21 15:14:26.338699 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:14:26.338660 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-j4c8p"] Jul 21 15:14:28.402404 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:14:28.402373 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kn9fn_02f3d0c6-4116-4134-ba51-1c02c73bef05/ovn-acl-logging/0.log" Jul 21 15:14:28.405010 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:14:28.404984 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kn9fn_02f3d0c6-4116-4134-ba51-1c02c73bef05/ovn-acl-logging/0.log" Jul 21 15:14:38.544912 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:14:38.544876 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-j4c8p"] Jul 21 15:14:53.244642 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:14:53.244601 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-j4c8p"] Jul 21 15:15:03.747433 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:15:03.747402 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-j4c8p"] Jul 21 15:15:16.046471 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:15:16.046434 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-dfzpn"] Jul 21 15:15:16.050280 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:15:16.050264 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dfzpn" Jul 21 15:15:16.055241 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:15:16.055213 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 21 15:15:16.055394 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:15:16.055254 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 21 15:15:16.055394 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:15:16.055220 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Jul 21 15:15:16.055394 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:15:16.055229 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-wcwjp\"" Jul 21 15:15:16.061268 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:15:16.061246 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-dfzpn"] Jul 21 15:15:16.176079 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:15:16.176049 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04b26501-4761-44b0-a36a-48caa3a51787-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-dfzpn\" (UID: \"04b26501-4761-44b0-a36a-48caa3a51787\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dfzpn" Jul 21 15:15:16.176261 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:15:16.176091 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04b26501-4761-44b0-a36a-48caa3a51787-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-dfzpn\" (UID: \"04b26501-4761-44b0-a36a-48caa3a51787\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dfzpn" Jul 21 15:15:16.176261 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:15:16.176111 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjqst\" (UniqueName: \"kubernetes.io/projected/04b26501-4761-44b0-a36a-48caa3a51787-kube-api-access-vjqst\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-dfzpn\" (UID: \"04b26501-4761-44b0-a36a-48caa3a51787\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dfzpn" Jul 21 15:15:16.176261 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:15:16.176209 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04b26501-4761-44b0-a36a-48caa3a51787-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-dfzpn\" (UID: \"04b26501-4761-44b0-a36a-48caa3a51787\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dfzpn" Jul 21 15:15:16.176261 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:15:16.176246 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04b26501-4761-44b0-a36a-48caa3a51787-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-dfzpn\" (UID: \"04b26501-4761-44b0-a36a-48caa3a51787\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dfzpn" Jul 21 15:15:16.176429 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:15:16.176308 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04b26501-4761-44b0-a36a-48caa3a51787-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-dfzpn\" (UID: \"04b26501-4761-44b0-a36a-48caa3a51787\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dfzpn" Jul 21 15:15:16.277419 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:15:16.277382 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04b26501-4761-44b0-a36a-48caa3a51787-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-dfzpn\" (UID: \"04b26501-4761-44b0-a36a-48caa3a51787\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dfzpn" Jul 21 15:15:16.277619 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:15:16.277435 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04b26501-4761-44b0-a36a-48caa3a51787-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-dfzpn\" (UID: \"04b26501-4761-44b0-a36a-48caa3a51787\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dfzpn" Jul 21 15:15:16.277619 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:15:16.277481 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04b26501-4761-44b0-a36a-48caa3a51787-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-dfzpn\" (UID: \"04b26501-4761-44b0-a36a-48caa3a51787\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dfzpn" Jul 21 15:15:16.277619 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:15:16.277534 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04b26501-4761-44b0-a36a-48caa3a51787-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-dfzpn\" (UID: \"04b26501-4761-44b0-a36a-48caa3a51787\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dfzpn" Jul 21 15:15:16.277619 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:15:16.277576 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04b26501-4761-44b0-a36a-48caa3a51787-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-dfzpn\" (UID: \"04b26501-4761-44b0-a36a-48caa3a51787\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dfzpn" Jul 21 15:15:16.277619 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:15:16.277599 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjqst\" (UniqueName: \"kubernetes.io/projected/04b26501-4761-44b0-a36a-48caa3a51787-kube-api-access-vjqst\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-dfzpn\" (UID: \"04b26501-4761-44b0-a36a-48caa3a51787\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dfzpn" Jul 21 15:15:16.277949 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:15:16.277912 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04b26501-4761-44b0-a36a-48caa3a51787-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-dfzpn\" (UID: \"04b26501-4761-44b0-a36a-48caa3a51787\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dfzpn" Jul 21 15:15:16.278010 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:15:16.277966 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04b26501-4761-44b0-a36a-48caa3a51787-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-dfzpn\" (UID: \"04b26501-4761-44b0-a36a-48caa3a51787\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dfzpn" Jul 21 15:15:16.278058 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:15:16.278009 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04b26501-4761-44b0-a36a-48caa3a51787-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-dfzpn\" (UID: \"04b26501-4761-44b0-a36a-48caa3a51787\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dfzpn" Jul 21 15:15:16.279845 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:15:16.279820 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04b26501-4761-44b0-a36a-48caa3a51787-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-dfzpn\" (UID: \"04b26501-4761-44b0-a36a-48caa3a51787\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dfzpn" Jul 21 15:15:16.280004 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:15:16.279987 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04b26501-4761-44b0-a36a-48caa3a51787-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-dfzpn\" (UID: \"04b26501-4761-44b0-a36a-48caa3a51787\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dfzpn" Jul 21 15:15:16.287720 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:15:16.287688 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjqst\" (UniqueName: \"kubernetes.io/projected/04b26501-4761-44b0-a36a-48caa3a51787-kube-api-access-vjqst\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-dfzpn\" (UID: \"04b26501-4761-44b0-a36a-48caa3a51787\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dfzpn" Jul 21 15:15:16.359507 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:15:16.359471 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dfzpn" Jul 21 15:15:16.500738 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:15:16.500706 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-dfzpn"] Jul 21 15:15:16.502204 ip-10-0-135-176 kubenswrapper[2563]: W0721 15:15:16.502168 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod04b26501_4761_44b0_a36a_48caa3a51787.slice/crio-b699f420a87f4def8d887ec4446b554bcc83ee04a1aa67dce4c85069a7d83c0b WatchSource:0}: Error finding container b699f420a87f4def8d887ec4446b554bcc83ee04a1aa67dce4c85069a7d83c0b: Status 404 returned error can't find the container with id b699f420a87f4def8d887ec4446b554bcc83ee04a1aa67dce4c85069a7d83c0b Jul 21 15:15:16.601450 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:15:16.601415 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dfzpn" event={"ID":"04b26501-4761-44b0-a36a-48caa3a51787","Type":"ContainerStarted","Data":"b699f420a87f4def8d887ec4446b554bcc83ee04a1aa67dce4c85069a7d83c0b"} Jul 21 15:15:18.149000 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:15:18.148965 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-j4c8p"] Jul 21 15:15:18.614703 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:15:18.614665 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dfzpn" event={"ID":"04b26501-4761-44b0-a36a-48caa3a51787","Type":"ContainerStarted","Data":"b0f3840bda94501ab69a9f094ad17880e1269fcfac1ad896cbcca12cf5424f00"} Jul 21 15:15:18.635592 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:15:18.635542 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dfzpn" podStartSLOduration=1.41364443 podStartE2EDuration="2.635526789s" totalImagesPullingTime="1.221882359s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 15:15:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 15:15:18.633795373 +0000 UTC m=+950.629580790" watchObservedRunningTime="2026-07-21 15:15:18.635526789 +0000 UTC m=+950.631312205" Jul 21 15:15:26.359567 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:15:26.359533 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dfzpn" Jul 21 15:15:26.360053 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:15:26.359583 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dfzpn" Jul 21 15:15:26.372436 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:15:26.372405 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dfzpn" Jul 21 15:15:26.670559 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:15:26.670480 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dfzpn" Jul 21 15:16:18.327790 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:16:18.327705 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7c8458985d-gkl6v"] Jul 21 15:16:18.328314 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:16:18.327905 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7c8458985d-gkl6v" podUID="9be5a277-9187-4315-8c00-493432ae68dc" containerName="authorino" containerID="cri-o://c33776a0751235c2777dd32f0356ee5486fb1b31b2d1669fd5056c24e62a36d7" gracePeriod=30 Jul 21 15:16:18.577786 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:16:18.577766 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7c8458985d-gkl6v" Jul 21 15:16:18.677979 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:16:18.677907 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9be5a277-9187-4315-8c00-493432ae68dc-tls-cert\") pod \"9be5a277-9187-4315-8c00-493432ae68dc\" (UID: \"9be5a277-9187-4315-8c00-493432ae68dc\") " Jul 21 15:16:18.677979 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:16:18.677951 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-znlwp\" (UniqueName: \"kubernetes.io/projected/9be5a277-9187-4315-8c00-493432ae68dc-kube-api-access-znlwp\") pod \"9be5a277-9187-4315-8c00-493432ae68dc\" (UID: \"9be5a277-9187-4315-8c00-493432ae68dc\") " Jul 21 15:16:18.680076 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:16:18.680051 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9be5a277-9187-4315-8c00-493432ae68dc-kube-api-access-znlwp" (OuterVolumeSpecName: "kube-api-access-znlwp") pod "9be5a277-9187-4315-8c00-493432ae68dc" (UID: "9be5a277-9187-4315-8c00-493432ae68dc"). InnerVolumeSpecName "kube-api-access-znlwp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 15:16:18.688909 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:16:18.688879 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9be5a277-9187-4315-8c00-493432ae68dc-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "9be5a277-9187-4315-8c00-493432ae68dc" (UID: "9be5a277-9187-4315-8c00-493432ae68dc"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 15:16:18.778714 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:16:18.778676 2563 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9be5a277-9187-4315-8c00-493432ae68dc-tls-cert\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 15:16:18.778714 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:16:18.778711 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-znlwp\" (UniqueName: \"kubernetes.io/projected/9be5a277-9187-4315-8c00-493432ae68dc-kube-api-access-znlwp\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 15:16:18.942517 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:16:18.942448 2563 generic.go:356] "Generic (PLEG): container finished" podID="9be5a277-9187-4315-8c00-493432ae68dc" containerID="c33776a0751235c2777dd32f0356ee5486fb1b31b2d1669fd5056c24e62a36d7" exitCode=0 Jul 21 15:16:18.942517 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:16:18.942490 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7c8458985d-gkl6v" Jul 21 15:16:18.942517 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:16:18.942500 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7c8458985d-gkl6v" event={"ID":"9be5a277-9187-4315-8c00-493432ae68dc","Type":"ContainerDied","Data":"c33776a0751235c2777dd32f0356ee5486fb1b31b2d1669fd5056c24e62a36d7"} Jul 21 15:16:18.942711 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:16:18.942530 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7c8458985d-gkl6v" event={"ID":"9be5a277-9187-4315-8c00-493432ae68dc","Type":"ContainerDied","Data":"f75bc68219b62dad55fec5c3bf777ec26d80835a1d6ad9bde3e67490ab3849ae"} Jul 21 15:16:18.942711 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:16:18.942547 2563 scope.go:117] "RemoveContainer" containerID="c33776a0751235c2777dd32f0356ee5486fb1b31b2d1669fd5056c24e62a36d7" Jul 21 15:16:18.952475 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:16:18.952454 2563 scope.go:117] "RemoveContainer" containerID="c33776a0751235c2777dd32f0356ee5486fb1b31b2d1669fd5056c24e62a36d7" Jul 21 15:16:18.952738 ip-10-0-135-176 kubenswrapper[2563]: E0721 15:16:18.952717 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c33776a0751235c2777dd32f0356ee5486fb1b31b2d1669fd5056c24e62a36d7\": container with ID starting with c33776a0751235c2777dd32f0356ee5486fb1b31b2d1669fd5056c24e62a36d7 not found: ID does not exist" containerID="c33776a0751235c2777dd32f0356ee5486fb1b31b2d1669fd5056c24e62a36d7" Jul 21 15:16:18.952833 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:16:18.952743 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c33776a0751235c2777dd32f0356ee5486fb1b31b2d1669fd5056c24e62a36d7"} err="failed to get container status \"c33776a0751235c2777dd32f0356ee5486fb1b31b2d1669fd5056c24e62a36d7\": rpc error: code = NotFound desc = could not find container \"c33776a0751235c2777dd32f0356ee5486fb1b31b2d1669fd5056c24e62a36d7\": container with ID starting with c33776a0751235c2777dd32f0356ee5486fb1b31b2d1669fd5056c24e62a36d7 not found: ID does not exist" Jul 21 15:16:18.987166 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:16:18.987141 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7c8458985d-gkl6v"] Jul 21 15:16:18.994940 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:16:18.994921 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7c8458985d-gkl6v"] Jul 21 15:16:20.482630 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:16:20.481797 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9be5a277-9187-4315-8c00-493432ae68dc" path="/var/lib/kubelet/pods/9be5a277-9187-4315-8c00-493432ae68dc/volumes" Jul 21 15:16:56.840649 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:16:56.840615 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-j4c8p"] Jul 21 15:17:06.441170 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:17:06.441135 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-j4c8p"] Jul 21 15:17:18.145682 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:17:18.145652 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-j4c8p"] Jul 21 15:18:55.495119 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:18:55.495033 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-j4c8p"] Jul 21 15:19:04.544819 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:19:04.544783 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-j4c8p"] Jul 21 15:19:28.436715 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:19:28.436689 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kn9fn_02f3d0c6-4116-4134-ba51-1c02c73bef05/ovn-acl-logging/0.log" Jul 21 15:19:28.441015 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:19:28.440994 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kn9fn_02f3d0c6-4116-4134-ba51-1c02c73bef05/ovn-acl-logging/0.log" Jul 21 15:19:40.938118 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:19:40.938082 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-j4c8p"] Jul 21 15:19:47.738860 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:19:47.738827 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-j4c8p"] Jul 21 15:19:53.638437 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:19:53.638405 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-j4c8p"] Jul 21 15:20:00.443046 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:20:00.443013 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-j4c8p"] Jul 21 15:20:09.946507 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:20:09.946463 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-j4c8p"] Jul 21 15:20:17.243225 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:20:17.243194 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-j4c8p"] Jul 21 15:20:30.646615 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:20:30.646536 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-j4c8p"] Jul 21 15:20:37.236201 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:20:37.236168 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-j4c8p"] Jul 21 15:20:43.040225 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:20:43.040192 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-j4c8p"] Jul 21 15:21:02.791270 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:21:02.791221 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-j4c8p"] Jul 21 15:21:10.144347 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:21:10.144297 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-j4c8p"] Jul 21 15:21:35.047706 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:21:35.047670 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-j4c8p"] Jul 21 15:21:42.443707 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:21:42.443675 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-j4c8p"] Jul 21 15:21:50.148277 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:21:50.148197 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-j4c8p"] Jul 21 15:21:57.542198 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:21:57.542163 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-j4c8p"] Jul 21 15:22:11.737362 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:22:11.737305 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-j4c8p"] Jul 21 15:22:17.538967 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:22:17.538935 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-j4c8p"] Jul 21 15:22:27.637254 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:22:27.637220 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-j4c8p"] Jul 21 15:22:32.341168 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:22:32.341140 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-j4c8p"] Jul 21 15:23:05.496657 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:23:05.496616 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7569bc7c69-g6dgt"] Jul 21 15:23:05.497169 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:23:05.496800 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-7569bc7c69-g6dgt" podUID="84f73758-06c8-4238-8b98-d87c0937e1f0" containerName="manager" containerID="cri-o://bc5c15dbc89c230cd15d0eb3309a3c5383eecef7fbc9040d8ff66743119f6161" gracePeriod=10 Jul 21 15:23:06.660213 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:23:06.660189 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7569bc7c69-g6dgt" Jul 21 15:23:06.709529 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:23:06.709503 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ncfk8\" (UniqueName: \"kubernetes.io/projected/84f73758-06c8-4238-8b98-d87c0937e1f0-kube-api-access-ncfk8\") pod \"84f73758-06c8-4238-8b98-d87c0937e1f0\" (UID: \"84f73758-06c8-4238-8b98-d87c0937e1f0\") " Jul 21 15:23:06.709683 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:23:06.709668 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/84f73758-06c8-4238-8b98-d87c0937e1f0-extensions-socket-volume\") pod \"84f73758-06c8-4238-8b98-d87c0937e1f0\" (UID: \"84f73758-06c8-4238-8b98-d87c0937e1f0\") " Jul 21 15:23:06.710014 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:23:06.709988 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84f73758-06c8-4238-8b98-d87c0937e1f0-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "84f73758-06c8-4238-8b98-d87c0937e1f0" (UID: "84f73758-06c8-4238-8b98-d87c0937e1f0"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 15:23:06.711696 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:23:06.711663 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84f73758-06c8-4238-8b98-d87c0937e1f0-kube-api-access-ncfk8" (OuterVolumeSpecName: "kube-api-access-ncfk8") pod "84f73758-06c8-4238-8b98-d87c0937e1f0" (UID: "84f73758-06c8-4238-8b98-d87c0937e1f0"). InnerVolumeSpecName "kube-api-access-ncfk8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 15:23:06.810400 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:23:06.810343 2563 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/84f73758-06c8-4238-8b98-d87c0937e1f0-extensions-socket-volume\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 15:23:06.810400 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:23:06.810365 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ncfk8\" (UniqueName: \"kubernetes.io/projected/84f73758-06c8-4238-8b98-d87c0937e1f0-kube-api-access-ncfk8\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 15:23:07.133703 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:23:07.133672 2563 generic.go:356] "Generic (PLEG): container finished" podID="84f73758-06c8-4238-8b98-d87c0937e1f0" containerID="bc5c15dbc89c230cd15d0eb3309a3c5383eecef7fbc9040d8ff66743119f6161" exitCode=0 Jul 21 15:23:07.133861 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:23:07.133717 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7569bc7c69-g6dgt" Jul 21 15:23:07.133861 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:23:07.133764 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-7569bc7c69-g6dgt" event={"ID":"84f73758-06c8-4238-8b98-d87c0937e1f0","Type":"ContainerDied","Data":"bc5c15dbc89c230cd15d0eb3309a3c5383eecef7fbc9040d8ff66743119f6161"} Jul 21 15:23:07.133861 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:23:07.133824 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-7569bc7c69-g6dgt" event={"ID":"84f73758-06c8-4238-8b98-d87c0937e1f0","Type":"ContainerDied","Data":"72b2448bbe5c0b098b52283f0817c339e3b5d4d0d891dfa4589a6b359d5dede3"} Jul 21 15:23:07.133861 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:23:07.133853 2563 scope.go:117] "RemoveContainer" containerID="bc5c15dbc89c230cd15d0eb3309a3c5383eecef7fbc9040d8ff66743119f6161" Jul 21 15:23:07.144251 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:23:07.144231 2563 scope.go:117] "RemoveContainer" containerID="bc5c15dbc89c230cd15d0eb3309a3c5383eecef7fbc9040d8ff66743119f6161" Jul 21 15:23:07.144530 ip-10-0-135-176 kubenswrapper[2563]: E0721 15:23:07.144514 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc5c15dbc89c230cd15d0eb3309a3c5383eecef7fbc9040d8ff66743119f6161\": container with ID starting with bc5c15dbc89c230cd15d0eb3309a3c5383eecef7fbc9040d8ff66743119f6161 not found: ID does not exist" containerID="bc5c15dbc89c230cd15d0eb3309a3c5383eecef7fbc9040d8ff66743119f6161" Jul 21 15:23:07.144578 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:23:07.144539 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc5c15dbc89c230cd15d0eb3309a3c5383eecef7fbc9040d8ff66743119f6161"} err="failed to get container status \"bc5c15dbc89c230cd15d0eb3309a3c5383eecef7fbc9040d8ff66743119f6161\": rpc error: code = NotFound desc = could not find container \"bc5c15dbc89c230cd15d0eb3309a3c5383eecef7fbc9040d8ff66743119f6161\": container with ID starting with bc5c15dbc89c230cd15d0eb3309a3c5383eecef7fbc9040d8ff66743119f6161 not found: ID does not exist" Jul 21 15:23:07.156440 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:23:07.156418 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7569bc7c69-g6dgt"] Jul 21 15:23:07.160965 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:23:07.160947 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7569bc7c69-g6dgt"] Jul 21 15:23:08.480287 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:23:08.480256 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="84f73758-06c8-4238-8b98-d87c0937e1f0" path="/var/lib/kubelet/pods/84f73758-06c8-4238-8b98-d87c0937e1f0/volumes" Jul 21 15:24:06.669886 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:24:06.669843 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7569bc7c69-z7r6n"] Jul 21 15:24:06.670371 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:24:06.670217 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84f73758-06c8-4238-8b98-d87c0937e1f0" containerName="manager" Jul 21 15:24:06.670371 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:24:06.670228 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="84f73758-06c8-4238-8b98-d87c0937e1f0" containerName="manager" Jul 21 15:24:06.670371 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:24:06.670241 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9be5a277-9187-4315-8c00-493432ae68dc" containerName="authorino" Jul 21 15:24:06.670371 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:24:06.670246 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="9be5a277-9187-4315-8c00-493432ae68dc" containerName="authorino" Jul 21 15:24:06.670371 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:24:06.670309 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="84f73758-06c8-4238-8b98-d87c0937e1f0" containerName="manager" Jul 21 15:24:06.670371 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:24:06.670318 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="9be5a277-9187-4315-8c00-493432ae68dc" containerName="authorino" Jul 21 15:24:06.673480 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:24:06.673461 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7569bc7c69-z7r6n" Jul 21 15:24:06.677377 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:24:06.677356 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-fkrwq\"" Jul 21 15:24:06.688097 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:24:06.688076 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7569bc7c69-z7r6n"] Jul 21 15:24:06.758371 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:24:06.758346 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2f00f1fa-667b-4345-911f-8dd3c0404da1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7569bc7c69-z7r6n\" (UID: \"2f00f1fa-667b-4345-911f-8dd3c0404da1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7569bc7c69-z7r6n" Jul 21 15:24:06.758480 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:24:06.758379 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9g8s9\" (UniqueName: \"kubernetes.io/projected/2f00f1fa-667b-4345-911f-8dd3c0404da1-kube-api-access-9g8s9\") pod \"kuadrant-operator-controller-manager-7569bc7c69-z7r6n\" (UID: \"2f00f1fa-667b-4345-911f-8dd3c0404da1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7569bc7c69-z7r6n" Jul 21 15:24:06.859501 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:24:06.859471 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2f00f1fa-667b-4345-911f-8dd3c0404da1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7569bc7c69-z7r6n\" (UID: \"2f00f1fa-667b-4345-911f-8dd3c0404da1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7569bc7c69-z7r6n" Jul 21 15:24:06.859501 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:24:06.859502 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9g8s9\" (UniqueName: \"kubernetes.io/projected/2f00f1fa-667b-4345-911f-8dd3c0404da1-kube-api-access-9g8s9\") pod \"kuadrant-operator-controller-manager-7569bc7c69-z7r6n\" (UID: \"2f00f1fa-667b-4345-911f-8dd3c0404da1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7569bc7c69-z7r6n" Jul 21 15:24:06.859820 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:24:06.859803 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2f00f1fa-667b-4345-911f-8dd3c0404da1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7569bc7c69-z7r6n\" (UID: \"2f00f1fa-667b-4345-911f-8dd3c0404da1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7569bc7c69-z7r6n" Jul 21 15:24:06.868732 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:24:06.868714 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9g8s9\" (UniqueName: \"kubernetes.io/projected/2f00f1fa-667b-4345-911f-8dd3c0404da1-kube-api-access-9g8s9\") pod \"kuadrant-operator-controller-manager-7569bc7c69-z7r6n\" (UID: \"2f00f1fa-667b-4345-911f-8dd3c0404da1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7569bc7c69-z7r6n" Jul 21 15:24:06.983411 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:24:06.983346 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7569bc7c69-z7r6n" Jul 21 15:24:07.136684 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:24:07.136656 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7569bc7c69-z7r6n"] Jul 21 15:24:07.137005 ip-10-0-135-176 kubenswrapper[2563]: W0721 15:24:07.136981 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f00f1fa_667b_4345_911f_8dd3c0404da1.slice/crio-49cc217a0cdb63042604ec718d78b39ec8967434725d7c858320664ff056d1d9 WatchSource:0}: Error finding container 49cc217a0cdb63042604ec718d78b39ec8967434725d7c858320664ff056d1d9: Status 404 returned error can't find the container with id 49cc217a0cdb63042604ec718d78b39ec8967434725d7c858320664ff056d1d9 Jul 21 15:24:07.435336 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:24:07.435302 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-7569bc7c69-z7r6n" event={"ID":"2f00f1fa-667b-4345-911f-8dd3c0404da1","Type":"ContainerStarted","Data":"62929ededa5cae57ea58ba4b33e687d4a39e6964a99746c8c5d41b3ee4e0fcc8"} Jul 21 15:24:07.435471 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:24:07.435378 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-7569bc7c69-z7r6n" event={"ID":"2f00f1fa-667b-4345-911f-8dd3c0404da1","Type":"ContainerStarted","Data":"49cc217a0cdb63042604ec718d78b39ec8967434725d7c858320664ff056d1d9"} Jul 21 15:24:07.457340 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:24:07.457267 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-7569bc7c69-z7r6n" podStartSLOduration=1.45724901 podStartE2EDuration="1.45724901s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 15:24:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 15:24:07.456612769 +0000 UTC m=+1479.452398195" watchObservedRunningTime="2026-07-21 15:24:07.45724901 +0000 UTC m=+1479.453034431" Jul 21 15:24:08.440300 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:24:08.440272 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-7569bc7c69-z7r6n" Jul 21 15:24:19.446814 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:24:19.446785 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-7569bc7c69-z7r6n" Jul 21 15:24:26.638363 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:24:26.638306 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-j4c8p"] Jul 21 15:24:28.470285 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:24:28.470179 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kn9fn_02f3d0c6-4116-4134-ba51-1c02c73bef05/ovn-acl-logging/0.log" Jul 21 15:24:28.482391 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:24:28.482368 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kn9fn_02f3d0c6-4116-4134-ba51-1c02c73bef05/ovn-acl-logging/0.log" Jul 21 15:24:34.139827 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:24:34.139795 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-j4c8p"] Jul 21 15:24:44.344627 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:24:44.344595 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-j4c8p"] Jul 21 15:24:50.044353 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:24:50.044261 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-j4c8p"] Jul 21 15:24:55.743177 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:24:55.743146 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-j4c8p"] Jul 21 15:25:05.041126 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:25:05.041097 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-j4c8p"] Jul 21 15:25:11.341281 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:25:11.341248 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-j4c8p"] Jul 21 15:25:20.447827 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:25:20.447793 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-j4c8p"] Jul 21 15:25:26.737282 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:25:26.737244 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-j4c8p"] Jul 21 15:25:35.843414 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:25:35.843380 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-j4c8p"] Jul 21 15:25:42.142746 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:25:42.142708 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-j4c8p"] Jul 21 15:25:51.338263 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:25:51.338231 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-j4c8p"] Jul 21 15:25:57.738982 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:25:57.738952 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-j4c8p"] Jul 21 15:26:05.938474 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:26:05.938442 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-j4c8p"] Jul 21 15:26:29.041800 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:26:29.041723 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-j4c8p"] Jul 21 15:26:36.743364 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:26:36.743311 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-j4c8p"] Jul 21 15:26:44.749039 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:26:44.749010 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-57f86f6c-kserve-68b6c968dc-f6tf9"] Jul 21 15:26:44.752523 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:26:44.752507 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-57f86f6c-kserve-68b6c968dc-f6tf9" Jul 21 15:26:44.755300 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:26:44.755282 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-57f86f6c-kserve-self-signed-certs\"" Jul 21 15:26:44.768418 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:26:44.768399 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-57f86f6c-kserve-68b6c968dc-f6tf9"] Jul 21 15:26:44.873742 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:26:44.873711 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b35644bc-a938-4d59-b916-3c5327e8e02d-tmp-dir\") pod \"e2e-same-id-a-57f86f6c-kserve-68b6c968dc-f6tf9\" (UID: \"b35644bc-a938-4d59-b916-3c5327e8e02d\") " pod="llm/e2e-same-id-a-57f86f6c-kserve-68b6c968dc-f6tf9" Jul 21 15:26:44.873874 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:26:44.873745 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b35644bc-a938-4d59-b916-3c5327e8e02d-tls-certs\") pod \"e2e-same-id-a-57f86f6c-kserve-68b6c968dc-f6tf9\" (UID: \"b35644bc-a938-4d59-b916-3c5327e8e02d\") " pod="llm/e2e-same-id-a-57f86f6c-kserve-68b6c968dc-f6tf9" Jul 21 15:26:44.873874 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:26:44.873772 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b35644bc-a938-4d59-b916-3c5327e8e02d-dshm\") pod \"e2e-same-id-a-57f86f6c-kserve-68b6c968dc-f6tf9\" (UID: \"b35644bc-a938-4d59-b916-3c5327e8e02d\") " pod="llm/e2e-same-id-a-57f86f6c-kserve-68b6c968dc-f6tf9" Jul 21 15:26:44.873999 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:26:44.873875 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b35644bc-a938-4d59-b916-3c5327e8e02d-model-cache\") pod \"e2e-same-id-a-57f86f6c-kserve-68b6c968dc-f6tf9\" (UID: \"b35644bc-a938-4d59-b916-3c5327e8e02d\") " pod="llm/e2e-same-id-a-57f86f6c-kserve-68b6c968dc-f6tf9" Jul 21 15:26:44.873999 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:26:44.873929 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b35644bc-a938-4d59-b916-3c5327e8e02d-home\") pod \"e2e-same-id-a-57f86f6c-kserve-68b6c968dc-f6tf9\" (UID: \"b35644bc-a938-4d59-b916-3c5327e8e02d\") " pod="llm/e2e-same-id-a-57f86f6c-kserve-68b6c968dc-f6tf9" Jul 21 15:26:44.873999 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:26:44.873955 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrqcj\" (UniqueName: \"kubernetes.io/projected/b35644bc-a938-4d59-b916-3c5327e8e02d-kube-api-access-xrqcj\") pod \"e2e-same-id-a-57f86f6c-kserve-68b6c968dc-f6tf9\" (UID: \"b35644bc-a938-4d59-b916-3c5327e8e02d\") " pod="llm/e2e-same-id-a-57f86f6c-kserve-68b6c968dc-f6tf9" Jul 21 15:26:44.975411 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:26:44.975378 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b35644bc-a938-4d59-b916-3c5327e8e02d-model-cache\") pod \"e2e-same-id-a-57f86f6c-kserve-68b6c968dc-f6tf9\" (UID: \"b35644bc-a938-4d59-b916-3c5327e8e02d\") " pod="llm/e2e-same-id-a-57f86f6c-kserve-68b6c968dc-f6tf9" Jul 21 15:26:44.975589 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:26:44.975418 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b35644bc-a938-4d59-b916-3c5327e8e02d-home\") pod \"e2e-same-id-a-57f86f6c-kserve-68b6c968dc-f6tf9\" (UID: \"b35644bc-a938-4d59-b916-3c5327e8e02d\") " pod="llm/e2e-same-id-a-57f86f6c-kserve-68b6c968dc-f6tf9" Jul 21 15:26:44.975589 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:26:44.975433 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrqcj\" (UniqueName: \"kubernetes.io/projected/b35644bc-a938-4d59-b916-3c5327e8e02d-kube-api-access-xrqcj\") pod \"e2e-same-id-a-57f86f6c-kserve-68b6c968dc-f6tf9\" (UID: \"b35644bc-a938-4d59-b916-3c5327e8e02d\") " pod="llm/e2e-same-id-a-57f86f6c-kserve-68b6c968dc-f6tf9" Jul 21 15:26:44.975589 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:26:44.975476 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b35644bc-a938-4d59-b916-3c5327e8e02d-tmp-dir\") pod \"e2e-same-id-a-57f86f6c-kserve-68b6c968dc-f6tf9\" (UID: \"b35644bc-a938-4d59-b916-3c5327e8e02d\") " pod="llm/e2e-same-id-a-57f86f6c-kserve-68b6c968dc-f6tf9" Jul 21 15:26:44.975589 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:26:44.975490 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b35644bc-a938-4d59-b916-3c5327e8e02d-tls-certs\") pod \"e2e-same-id-a-57f86f6c-kserve-68b6c968dc-f6tf9\" (UID: \"b35644bc-a938-4d59-b916-3c5327e8e02d\") " pod="llm/e2e-same-id-a-57f86f6c-kserve-68b6c968dc-f6tf9" Jul 21 15:26:44.975589 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:26:44.975506 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b35644bc-a938-4d59-b916-3c5327e8e02d-dshm\") pod \"e2e-same-id-a-57f86f6c-kserve-68b6c968dc-f6tf9\" (UID: \"b35644bc-a938-4d59-b916-3c5327e8e02d\") " pod="llm/e2e-same-id-a-57f86f6c-kserve-68b6c968dc-f6tf9" Jul 21 15:26:44.975853 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:26:44.975830 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b35644bc-a938-4d59-b916-3c5327e8e02d-home\") pod \"e2e-same-id-a-57f86f6c-kserve-68b6c968dc-f6tf9\" (UID: \"b35644bc-a938-4d59-b916-3c5327e8e02d\") " pod="llm/e2e-same-id-a-57f86f6c-kserve-68b6c968dc-f6tf9" Jul 21 15:26:44.975924 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:26:44.975854 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b35644bc-a938-4d59-b916-3c5327e8e02d-tmp-dir\") pod \"e2e-same-id-a-57f86f6c-kserve-68b6c968dc-f6tf9\" (UID: \"b35644bc-a938-4d59-b916-3c5327e8e02d\") " pod="llm/e2e-same-id-a-57f86f6c-kserve-68b6c968dc-f6tf9" Jul 21 15:26:44.975924 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:26:44.975894 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b35644bc-a938-4d59-b916-3c5327e8e02d-model-cache\") pod \"e2e-same-id-a-57f86f6c-kserve-68b6c968dc-f6tf9\" (UID: \"b35644bc-a938-4d59-b916-3c5327e8e02d\") " pod="llm/e2e-same-id-a-57f86f6c-kserve-68b6c968dc-f6tf9" Jul 21 15:26:44.977908 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:26:44.977881 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b35644bc-a938-4d59-b916-3c5327e8e02d-dshm\") pod \"e2e-same-id-a-57f86f6c-kserve-68b6c968dc-f6tf9\" (UID: \"b35644bc-a938-4d59-b916-3c5327e8e02d\") " pod="llm/e2e-same-id-a-57f86f6c-kserve-68b6c968dc-f6tf9" Jul 21 15:26:44.978095 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:26:44.978077 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b35644bc-a938-4d59-b916-3c5327e8e02d-tls-certs\") pod \"e2e-same-id-a-57f86f6c-kserve-68b6c968dc-f6tf9\" (UID: \"b35644bc-a938-4d59-b916-3c5327e8e02d\") " pod="llm/e2e-same-id-a-57f86f6c-kserve-68b6c968dc-f6tf9" Jul 21 15:26:44.985275 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:26:44.985247 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrqcj\" (UniqueName: \"kubernetes.io/projected/b35644bc-a938-4d59-b916-3c5327e8e02d-kube-api-access-xrqcj\") pod \"e2e-same-id-a-57f86f6c-kserve-68b6c968dc-f6tf9\" (UID: \"b35644bc-a938-4d59-b916-3c5327e8e02d\") " pod="llm/e2e-same-id-a-57f86f6c-kserve-68b6c968dc-f6tf9" Jul 21 15:26:45.061193 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:26:45.061127 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-57f86f6c-kserve-68b6c968dc-f6tf9" Jul 21 15:26:45.198915 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:26:45.198885 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-57f86f6c-kserve-68b6c968dc-f6tf9"] Jul 21 15:26:45.200125 ip-10-0-135-176 kubenswrapper[2563]: W0721 15:26:45.200098 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb35644bc_a938_4d59_b916_3c5327e8e02d.slice/crio-142a2928fdea5b4d82bbc7346179b54cf8a1c780ba20328614b9fc9b95d20d2b WatchSource:0}: Error finding container 142a2928fdea5b4d82bbc7346179b54cf8a1c780ba20328614b9fc9b95d20d2b: Status 404 returned error can't find the container with id 142a2928fdea5b4d82bbc7346179b54cf8a1c780ba20328614b9fc9b95d20d2b Jul 21 15:26:45.276455 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:26:45.276423 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-57f86f6c-kserve-68b6c968dc-f6tf9" event={"ID":"b35644bc-a938-4d59-b916-3c5327e8e02d","Type":"ContainerStarted","Data":"26121eeaa4793c638b7bd46537d71faf04831ad014782846ad8b59d2b3964e52"} Jul 21 15:26:45.276553 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:26:45.276471 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-57f86f6c-kserve-68b6c968dc-f6tf9" event={"ID":"b35644bc-a938-4d59-b916-3c5327e8e02d","Type":"ContainerStarted","Data":"142a2928fdea5b4d82bbc7346179b54cf8a1c780ba20328614b9fc9b95d20d2b"} Jul 21 15:26:45.298355 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:26:45.297351 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-57f86f6c-kserve-68b6c968dc-f6tf9" podStartSLOduration=1.297310236 podStartE2EDuration="1.297310236s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 15:26:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 15:26:45.293128576 +0000 UTC m=+1637.288913994" watchObservedRunningTime="2026-07-21 15:26:45.297310236 +0000 UTC m=+1637.293095651" Jul 21 15:26:46.862411 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:26:46.862380 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-j4c8p"] Jul 21 15:26:55.061477 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:26:55.061397 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-57f86f6c-kserve-68b6c968dc-f6tf9" Jul 21 15:26:55.061477 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:26:55.061443 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-57f86f6c-kserve-68b6c968dc-f6tf9" Jul 21 15:26:55.073474 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:26:55.073455 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-57f86f6c-kserve-68b6c968dc-f6tf9" Jul 21 15:26:55.340377 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:26:55.340354 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-57f86f6c-kserve-68b6c968dc-f6tf9" Jul 21 15:27:04.837392 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:27:04.837360 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-57f86f6c-kserve-75c4bff487-sbkld"] Jul 21 15:27:04.841263 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:27:04.841242 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-57f86f6c-kserve-75c4bff487-sbkld" Jul 21 15:27:04.844119 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:27:04.844100 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-57f86f6c-kserve-self-signed-certs\"" Jul 21 15:27:04.848815 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:27:04.848782 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-57f86f6c-kserve-75c4bff487-sbkld"] Jul 21 15:27:04.927408 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:27:04.927376 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/048a18a0-ffff-47cf-8147-1b28b8cb8ef6-tmp-dir\") pod \"e2e-same-id-b-57f86f6c-kserve-75c4bff487-sbkld\" (UID: \"048a18a0-ffff-47cf-8147-1b28b8cb8ef6\") " pod="llm/e2e-same-id-b-57f86f6c-kserve-75c4bff487-sbkld" Jul 21 15:27:04.927544 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:27:04.927415 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54vg9\" (UniqueName: \"kubernetes.io/projected/048a18a0-ffff-47cf-8147-1b28b8cb8ef6-kube-api-access-54vg9\") pod \"e2e-same-id-b-57f86f6c-kserve-75c4bff487-sbkld\" (UID: \"048a18a0-ffff-47cf-8147-1b28b8cb8ef6\") " pod="llm/e2e-same-id-b-57f86f6c-kserve-75c4bff487-sbkld" Jul 21 15:27:04.927544 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:27:04.927442 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/048a18a0-ffff-47cf-8147-1b28b8cb8ef6-home\") pod \"e2e-same-id-b-57f86f6c-kserve-75c4bff487-sbkld\" (UID: \"048a18a0-ffff-47cf-8147-1b28b8cb8ef6\") " pod="llm/e2e-same-id-b-57f86f6c-kserve-75c4bff487-sbkld" Jul 21 15:27:04.927544 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:27:04.927489 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/048a18a0-ffff-47cf-8147-1b28b8cb8ef6-dshm\") pod \"e2e-same-id-b-57f86f6c-kserve-75c4bff487-sbkld\" (UID: \"048a18a0-ffff-47cf-8147-1b28b8cb8ef6\") " pod="llm/e2e-same-id-b-57f86f6c-kserve-75c4bff487-sbkld" Jul 21 15:27:04.927674 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:27:04.927550 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/048a18a0-ffff-47cf-8147-1b28b8cb8ef6-tls-certs\") pod \"e2e-same-id-b-57f86f6c-kserve-75c4bff487-sbkld\" (UID: \"048a18a0-ffff-47cf-8147-1b28b8cb8ef6\") " pod="llm/e2e-same-id-b-57f86f6c-kserve-75c4bff487-sbkld" Jul 21 15:27:04.927674 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:27:04.927600 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/048a18a0-ffff-47cf-8147-1b28b8cb8ef6-model-cache\") pod \"e2e-same-id-b-57f86f6c-kserve-75c4bff487-sbkld\" (UID: \"048a18a0-ffff-47cf-8147-1b28b8cb8ef6\") " pod="llm/e2e-same-id-b-57f86f6c-kserve-75c4bff487-sbkld" Jul 21 15:27:05.028486 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:27:05.028449 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/048a18a0-ffff-47cf-8147-1b28b8cb8ef6-tmp-dir\") pod \"e2e-same-id-b-57f86f6c-kserve-75c4bff487-sbkld\" (UID: \"048a18a0-ffff-47cf-8147-1b28b8cb8ef6\") " pod="llm/e2e-same-id-b-57f86f6c-kserve-75c4bff487-sbkld" Jul 21 15:27:05.028675 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:27:05.028491 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54vg9\" (UniqueName: \"kubernetes.io/projected/048a18a0-ffff-47cf-8147-1b28b8cb8ef6-kube-api-access-54vg9\") pod \"e2e-same-id-b-57f86f6c-kserve-75c4bff487-sbkld\" (UID: \"048a18a0-ffff-47cf-8147-1b28b8cb8ef6\") " pod="llm/e2e-same-id-b-57f86f6c-kserve-75c4bff487-sbkld" Jul 21 15:27:05.028675 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:27:05.028520 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/048a18a0-ffff-47cf-8147-1b28b8cb8ef6-home\") pod \"e2e-same-id-b-57f86f6c-kserve-75c4bff487-sbkld\" (UID: \"048a18a0-ffff-47cf-8147-1b28b8cb8ef6\") " pod="llm/e2e-same-id-b-57f86f6c-kserve-75c4bff487-sbkld" Jul 21 15:27:05.028675 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:27:05.028625 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/048a18a0-ffff-47cf-8147-1b28b8cb8ef6-dshm\") pod \"e2e-same-id-b-57f86f6c-kserve-75c4bff487-sbkld\" (UID: \"048a18a0-ffff-47cf-8147-1b28b8cb8ef6\") " pod="llm/e2e-same-id-b-57f86f6c-kserve-75c4bff487-sbkld" Jul 21 15:27:05.028675 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:27:05.028668 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/048a18a0-ffff-47cf-8147-1b28b8cb8ef6-tls-certs\") pod \"e2e-same-id-b-57f86f6c-kserve-75c4bff487-sbkld\" (UID: \"048a18a0-ffff-47cf-8147-1b28b8cb8ef6\") " pod="llm/e2e-same-id-b-57f86f6c-kserve-75c4bff487-sbkld" Jul 21 15:27:05.028897 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:27:05.028720 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/048a18a0-ffff-47cf-8147-1b28b8cb8ef6-model-cache\") pod \"e2e-same-id-b-57f86f6c-kserve-75c4bff487-sbkld\" (UID: \"048a18a0-ffff-47cf-8147-1b28b8cb8ef6\") " pod="llm/e2e-same-id-b-57f86f6c-kserve-75c4bff487-sbkld" Jul 21 15:27:05.028986 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:27:05.028896 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/048a18a0-ffff-47cf-8147-1b28b8cb8ef6-home\") pod \"e2e-same-id-b-57f86f6c-kserve-75c4bff487-sbkld\" (UID: \"048a18a0-ffff-47cf-8147-1b28b8cb8ef6\") " pod="llm/e2e-same-id-b-57f86f6c-kserve-75c4bff487-sbkld" Jul 21 15:27:05.028986 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:27:05.028977 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/048a18a0-ffff-47cf-8147-1b28b8cb8ef6-tmp-dir\") pod \"e2e-same-id-b-57f86f6c-kserve-75c4bff487-sbkld\" (UID: \"048a18a0-ffff-47cf-8147-1b28b8cb8ef6\") " pod="llm/e2e-same-id-b-57f86f6c-kserve-75c4bff487-sbkld" Jul 21 15:27:05.029060 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:27:05.029041 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/048a18a0-ffff-47cf-8147-1b28b8cb8ef6-model-cache\") pod \"e2e-same-id-b-57f86f6c-kserve-75c4bff487-sbkld\" (UID: \"048a18a0-ffff-47cf-8147-1b28b8cb8ef6\") " pod="llm/e2e-same-id-b-57f86f6c-kserve-75c4bff487-sbkld" Jul 21 15:27:05.030864 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:27:05.030839 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/048a18a0-ffff-47cf-8147-1b28b8cb8ef6-dshm\") pod \"e2e-same-id-b-57f86f6c-kserve-75c4bff487-sbkld\" (UID: \"048a18a0-ffff-47cf-8147-1b28b8cb8ef6\") " pod="llm/e2e-same-id-b-57f86f6c-kserve-75c4bff487-sbkld" Jul 21 15:27:05.031197 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:27:05.031181 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/048a18a0-ffff-47cf-8147-1b28b8cb8ef6-tls-certs\") pod \"e2e-same-id-b-57f86f6c-kserve-75c4bff487-sbkld\" (UID: \"048a18a0-ffff-47cf-8147-1b28b8cb8ef6\") " pod="llm/e2e-same-id-b-57f86f6c-kserve-75c4bff487-sbkld" Jul 21 15:27:05.037097 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:27:05.037075 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-54vg9\" (UniqueName: \"kubernetes.io/projected/048a18a0-ffff-47cf-8147-1b28b8cb8ef6-kube-api-access-54vg9\") pod \"e2e-same-id-b-57f86f6c-kserve-75c4bff487-sbkld\" (UID: \"048a18a0-ffff-47cf-8147-1b28b8cb8ef6\") " pod="llm/e2e-same-id-b-57f86f6c-kserve-75c4bff487-sbkld" Jul 21 15:27:05.153606 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:27:05.153551 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-57f86f6c-kserve-75c4bff487-sbkld" Jul 21 15:27:05.292092 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:27:05.292067 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-57f86f6c-kserve-75c4bff487-sbkld"] Jul 21 15:27:05.293989 ip-10-0-135-176 kubenswrapper[2563]: W0721 15:27:05.293962 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod048a18a0_ffff_47cf_8147_1b28b8cb8ef6.slice/crio-d7727d0ceb23eb0d558e3bf455b264b5d59437378e7fafd01a9e48165996e56d WatchSource:0}: Error finding container d7727d0ceb23eb0d558e3bf455b264b5d59437378e7fafd01a9e48165996e56d: Status 404 returned error can't find the container with id d7727d0ceb23eb0d558e3bf455b264b5d59437378e7fafd01a9e48165996e56d Jul 21 15:27:05.384601 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:27:05.384549 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-57f86f6c-kserve-75c4bff487-sbkld" event={"ID":"048a18a0-ffff-47cf-8147-1b28b8cb8ef6","Type":"ContainerStarted","Data":"35db67d47dbf22eb2b81eacfc2a5a9ac236ab81f6ca7f41b557d69b92cc25ea6"} Jul 21 15:27:05.384601 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:27:05.384599 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-57f86f6c-kserve-75c4bff487-sbkld" event={"ID":"048a18a0-ffff-47cf-8147-1b28b8cb8ef6","Type":"ContainerStarted","Data":"d7727d0ceb23eb0d558e3bf455b264b5d59437378e7fafd01a9e48165996e56d"} Jul 21 15:27:05.405144 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:27:05.405060 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-57f86f6c-kserve-75c4bff487-sbkld" podStartSLOduration=1.405040196 podStartE2EDuration="1.405040196s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 15:27:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 15:27:05.404167588 +0000 UTC m=+1657.399953004" watchObservedRunningTime="2026-07-21 15:27:05.405040196 +0000 UTC m=+1657.400825610" Jul 21 15:27:07.349555 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:27:07.349527 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-j4c8p"] Jul 21 15:27:15.154276 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:27:15.154242 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-57f86f6c-kserve-75c4bff487-sbkld" Jul 21 15:27:15.154705 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:27:15.154297 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-57f86f6c-kserve-75c4bff487-sbkld" Jul 21 15:27:15.166419 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:27:15.166397 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-57f86f6c-kserve-75c4bff487-sbkld" Jul 21 15:27:15.447507 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:27:15.447433 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-57f86f6c-kserve-75c4bff487-sbkld" Jul 21 15:27:28.970950 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:27:28.970909 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-57f86f6c-kserve-68b6c968dc-f6tf9"] Jul 21 15:27:30.231889 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:27:30.231861 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-57f86f6c-kserve-75c4bff487-sbkld"] Jul 21 15:27:30.741098 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:27:30.741065 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-j4c8p"] Jul 21 15:27:39.943144 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:27:39.943114 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-j4c8p"] Jul 21 15:27:43.991441 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:27:43.991387 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-57f86f6c-kserve-68b6c968dc-f6tf9" podUID="b35644bc-a938-4d59-b916-3c5327e8e02d" containerName="main" containerID="cri-o://26121eeaa4793c638b7bd46537d71faf04831ad014782846ad8b59d2b3964e52" gracePeriod=45 Jul 21 15:27:44.261939 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:27:44.261913 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-57f86f6c-kserve-68b6c968dc-f6tf9" Jul 21 15:27:44.347521 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:27:44.347498 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b35644bc-a938-4d59-b916-3c5327e8e02d-model-cache\") pod \"b35644bc-a938-4d59-b916-3c5327e8e02d\" (UID: \"b35644bc-a938-4d59-b916-3c5327e8e02d\") " Jul 21 15:27:44.347674 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:27:44.347548 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b35644bc-a938-4d59-b916-3c5327e8e02d-tmp-dir\") pod \"b35644bc-a938-4d59-b916-3c5327e8e02d\" (UID: \"b35644bc-a938-4d59-b916-3c5327e8e02d\") " Jul 21 15:27:44.347674 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:27:44.347572 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b35644bc-a938-4d59-b916-3c5327e8e02d-dshm\") pod \"b35644bc-a938-4d59-b916-3c5327e8e02d\" (UID: \"b35644bc-a938-4d59-b916-3c5327e8e02d\") " Jul 21 15:27:44.347674 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:27:44.347594 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xrqcj\" (UniqueName: \"kubernetes.io/projected/b35644bc-a938-4d59-b916-3c5327e8e02d-kube-api-access-xrqcj\") pod \"b35644bc-a938-4d59-b916-3c5327e8e02d\" (UID: \"b35644bc-a938-4d59-b916-3c5327e8e02d\") " Jul 21 15:27:44.347674 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:27:44.347661 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b35644bc-a938-4d59-b916-3c5327e8e02d-tls-certs\") pod \"b35644bc-a938-4d59-b916-3c5327e8e02d\" (UID: \"b35644bc-a938-4d59-b916-3c5327e8e02d\") " Jul 21 15:27:44.347886 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:27:44.347688 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b35644bc-a938-4d59-b916-3c5327e8e02d-home\") pod \"b35644bc-a938-4d59-b916-3c5327e8e02d\" (UID: \"b35644bc-a938-4d59-b916-3c5327e8e02d\") " Jul 21 15:27:44.347886 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:27:44.347832 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b35644bc-a938-4d59-b916-3c5327e8e02d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b35644bc-a938-4d59-b916-3c5327e8e02d" (UID: "b35644bc-a938-4d59-b916-3c5327e8e02d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 15:27:44.347886 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:27:44.347840 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b35644bc-a938-4d59-b916-3c5327e8e02d-model-cache" (OuterVolumeSpecName: "model-cache") pod "b35644bc-a938-4d59-b916-3c5327e8e02d" (UID: "b35644bc-a938-4d59-b916-3c5327e8e02d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 15:27:44.348044 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:27:44.348028 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b35644bc-a938-4d59-b916-3c5327e8e02d-home" (OuterVolumeSpecName: "home") pod "b35644bc-a938-4d59-b916-3c5327e8e02d" (UID: "b35644bc-a938-4d59-b916-3c5327e8e02d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 15:27:44.348096 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:27:44.348052 2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b35644bc-a938-4d59-b916-3c5327e8e02d-model-cache\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 15:27:44.348096 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:27:44.348093 2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b35644bc-a938-4d59-b916-3c5327e8e02d-tmp-dir\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 15:27:44.349959 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:27:44.349934 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b35644bc-a938-4d59-b916-3c5327e8e02d-dshm" (OuterVolumeSpecName: "dshm") pod "b35644bc-a938-4d59-b916-3c5327e8e02d" (UID: "b35644bc-a938-4d59-b916-3c5327e8e02d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 15:27:44.350068 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:27:44.349957 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b35644bc-a938-4d59-b916-3c5327e8e02d-kube-api-access-xrqcj" (OuterVolumeSpecName: "kube-api-access-xrqcj") pod "b35644bc-a938-4d59-b916-3c5327e8e02d" (UID: "b35644bc-a938-4d59-b916-3c5327e8e02d"). InnerVolumeSpecName "kube-api-access-xrqcj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 15:27:44.350117 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:27:44.350094 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b35644bc-a938-4d59-b916-3c5327e8e02d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b35644bc-a938-4d59-b916-3c5327e8e02d" (UID: "b35644bc-a938-4d59-b916-3c5327e8e02d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 15:27:44.448466 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:27:44.448438 2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b35644bc-a938-4d59-b916-3c5327e8e02d-tls-certs\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 15:27:44.448466 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:27:44.448464 2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b35644bc-a938-4d59-b916-3c5327e8e02d-home\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 15:27:44.448606 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:27:44.448476 2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b35644bc-a938-4d59-b916-3c5327e8e02d-dshm\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 15:27:44.448606 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:27:44.448489 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrqcj\" (UniqueName: \"kubernetes.io/projected/b35644bc-a938-4d59-b916-3c5327e8e02d-kube-api-access-xrqcj\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 15:27:44.587284 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:27:44.587253 2563 generic.go:356] "Generic (PLEG): container finished" podID="b35644bc-a938-4d59-b916-3c5327e8e02d" containerID="26121eeaa4793c638b7bd46537d71faf04831ad014782846ad8b59d2b3964e52" exitCode=0 Jul 21 15:27:44.587409 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:27:44.587292 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-57f86f6c-kserve-68b6c968dc-f6tf9" event={"ID":"b35644bc-a938-4d59-b916-3c5327e8e02d","Type":"ContainerDied","Data":"26121eeaa4793c638b7bd46537d71faf04831ad014782846ad8b59d2b3964e52"} Jul 21 15:27:44.587409 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:27:44.587356 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-57f86f6c-kserve-68b6c968dc-f6tf9" event={"ID":"b35644bc-a938-4d59-b916-3c5327e8e02d","Type":"ContainerDied","Data":"142a2928fdea5b4d82bbc7346179b54cf8a1c780ba20328614b9fc9b95d20d2b"} Jul 21 15:27:44.587409 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:27:44.587296 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-57f86f6c-kserve-68b6c968dc-f6tf9" Jul 21 15:27:44.587409 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:27:44.587381 2563 scope.go:117] "RemoveContainer" containerID="26121eeaa4793c638b7bd46537d71faf04831ad014782846ad8b59d2b3964e52" Jul 21 15:27:44.601194 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:27:44.601177 2563 scope.go:117] "RemoveContainer" containerID="26121eeaa4793c638b7bd46537d71faf04831ad014782846ad8b59d2b3964e52" Jul 21 15:27:44.601472 ip-10-0-135-176 kubenswrapper[2563]: E0721 15:27:44.601454 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26121eeaa4793c638b7bd46537d71faf04831ad014782846ad8b59d2b3964e52\": container with ID starting with 26121eeaa4793c638b7bd46537d71faf04831ad014782846ad8b59d2b3964e52 not found: ID does not exist" containerID="26121eeaa4793c638b7bd46537d71faf04831ad014782846ad8b59d2b3964e52" Jul 21 15:27:44.601538 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:27:44.601485 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26121eeaa4793c638b7bd46537d71faf04831ad014782846ad8b59d2b3964e52"} err="failed to get container status \"26121eeaa4793c638b7bd46537d71faf04831ad014782846ad8b59d2b3964e52\": rpc error: code = NotFound desc = could not find container \"26121eeaa4793c638b7bd46537d71faf04831ad014782846ad8b59d2b3964e52\": container with ID starting with 26121eeaa4793c638b7bd46537d71faf04831ad014782846ad8b59d2b3964e52 not found: ID does not exist" Jul 21 15:27:44.605384 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:27:44.605353 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-57f86f6c-kserve-68b6c968dc-f6tf9"] Jul 21 15:27:44.611545 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:27:44.611524 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-57f86f6c-kserve-68b6c968dc-f6tf9"] Jul 21 15:27:45.251942 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:27:45.251889 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-57f86f6c-kserve-75c4bff487-sbkld" podUID="048a18a0-ffff-47cf-8147-1b28b8cb8ef6" containerName="main" containerID="cri-o://35db67d47dbf22eb2b81eacfc2a5a9ac236ab81f6ca7f41b557d69b92cc25ea6" gracePeriod=45 Jul 21 15:27:45.492148 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:27:45.492126 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-57f86f6c-kserve-75c4bff487-sbkld" Jul 21 15:27:45.537756 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:27:45.537692 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-j4c8p"] Jul 21 15:27:45.555297 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:27:45.555272 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/048a18a0-ffff-47cf-8147-1b28b8cb8ef6-tmp-dir\") pod \"048a18a0-ffff-47cf-8147-1b28b8cb8ef6\" (UID: \"048a18a0-ffff-47cf-8147-1b28b8cb8ef6\") " Jul 21 15:27:45.555297 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:27:45.555297 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/048a18a0-ffff-47cf-8147-1b28b8cb8ef6-dshm\") pod \"048a18a0-ffff-47cf-8147-1b28b8cb8ef6\" (UID: \"048a18a0-ffff-47cf-8147-1b28b8cb8ef6\") " Jul 21 15:27:45.555475 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:27:45.555351 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-54vg9\" (UniqueName: \"kubernetes.io/projected/048a18a0-ffff-47cf-8147-1b28b8cb8ef6-kube-api-access-54vg9\") pod \"048a18a0-ffff-47cf-8147-1b28b8cb8ef6\" (UID: \"048a18a0-ffff-47cf-8147-1b28b8cb8ef6\") " Jul 21 15:27:45.555475 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:27:45.555367 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/048a18a0-ffff-47cf-8147-1b28b8cb8ef6-tls-certs\") pod \"048a18a0-ffff-47cf-8147-1b28b8cb8ef6\" (UID: \"048a18a0-ffff-47cf-8147-1b28b8cb8ef6\") " Jul 21 15:27:45.555475 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:27:45.555382 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/048a18a0-ffff-47cf-8147-1b28b8cb8ef6-model-cache\") pod \"048a18a0-ffff-47cf-8147-1b28b8cb8ef6\" (UID: \"048a18a0-ffff-47cf-8147-1b28b8cb8ef6\") " Jul 21 15:27:45.555475 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:27:45.555402 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/048a18a0-ffff-47cf-8147-1b28b8cb8ef6-home\") pod \"048a18a0-ffff-47cf-8147-1b28b8cb8ef6\" (UID: \"048a18a0-ffff-47cf-8147-1b28b8cb8ef6\") " Jul 21 15:27:45.555681 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:27:45.555570 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/048a18a0-ffff-47cf-8147-1b28b8cb8ef6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "048a18a0-ffff-47cf-8147-1b28b8cb8ef6" (UID: "048a18a0-ffff-47cf-8147-1b28b8cb8ef6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 15:27:45.555681 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:27:45.555648 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/048a18a0-ffff-47cf-8147-1b28b8cb8ef6-model-cache" (OuterVolumeSpecName: "model-cache") pod "048a18a0-ffff-47cf-8147-1b28b8cb8ef6" (UID: "048a18a0-ffff-47cf-8147-1b28b8cb8ef6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 15:27:45.555789 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:27:45.555763 2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/048a18a0-ffff-47cf-8147-1b28b8cb8ef6-model-cache\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 15:27:45.555789 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:27:45.555768 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/048a18a0-ffff-47cf-8147-1b28b8cb8ef6-home" (OuterVolumeSpecName: "home") pod "048a18a0-ffff-47cf-8147-1b28b8cb8ef6" (UID: "048a18a0-ffff-47cf-8147-1b28b8cb8ef6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 15:27:45.555789 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:27:45.555781 2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/048a18a0-ffff-47cf-8147-1b28b8cb8ef6-tmp-dir\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 15:27:45.557599 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:27:45.557572 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/048a18a0-ffff-47cf-8147-1b28b8cb8ef6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "048a18a0-ffff-47cf-8147-1b28b8cb8ef6" (UID: "048a18a0-ffff-47cf-8147-1b28b8cb8ef6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 15:27:45.557700 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:27:45.557679 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/048a18a0-ffff-47cf-8147-1b28b8cb8ef6-dshm" (OuterVolumeSpecName: "dshm") pod "048a18a0-ffff-47cf-8147-1b28b8cb8ef6" (UID: "048a18a0-ffff-47cf-8147-1b28b8cb8ef6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 15:27:45.557700 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:27:45.557691 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/048a18a0-ffff-47cf-8147-1b28b8cb8ef6-kube-api-access-54vg9" (OuterVolumeSpecName: "kube-api-access-54vg9") pod "048a18a0-ffff-47cf-8147-1b28b8cb8ef6" (UID: "048a18a0-ffff-47cf-8147-1b28b8cb8ef6"). InnerVolumeSpecName "kube-api-access-54vg9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 15:27:45.594887 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:27:45.594862 2563 generic.go:356] "Generic (PLEG): container finished" podID="048a18a0-ffff-47cf-8147-1b28b8cb8ef6" containerID="35db67d47dbf22eb2b81eacfc2a5a9ac236ab81f6ca7f41b557d69b92cc25ea6" exitCode=0 Jul 21 15:27:45.595036 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:27:45.594913 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-57f86f6c-kserve-75c4bff487-sbkld" Jul 21 15:27:45.595036 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:27:45.594947 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-57f86f6c-kserve-75c4bff487-sbkld" event={"ID":"048a18a0-ffff-47cf-8147-1b28b8cb8ef6","Type":"ContainerDied","Data":"35db67d47dbf22eb2b81eacfc2a5a9ac236ab81f6ca7f41b557d69b92cc25ea6"} Jul 21 15:27:45.595036 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:27:45.594991 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-57f86f6c-kserve-75c4bff487-sbkld" event={"ID":"048a18a0-ffff-47cf-8147-1b28b8cb8ef6","Type":"ContainerDied","Data":"d7727d0ceb23eb0d558e3bf455b264b5d59437378e7fafd01a9e48165996e56d"} Jul 21 15:27:45.595036 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:27:45.595010 2563 scope.go:117] "RemoveContainer" containerID="35db67d47dbf22eb2b81eacfc2a5a9ac236ab81f6ca7f41b557d69b92cc25ea6" Jul 21 15:27:45.604389 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:27:45.604373 2563 scope.go:117] "RemoveContainer" containerID="35db67d47dbf22eb2b81eacfc2a5a9ac236ab81f6ca7f41b557d69b92cc25ea6" Jul 21 15:27:45.604675 ip-10-0-135-176 kubenswrapper[2563]: E0721 15:27:45.604655 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35db67d47dbf22eb2b81eacfc2a5a9ac236ab81f6ca7f41b557d69b92cc25ea6\": container with ID starting with 35db67d47dbf22eb2b81eacfc2a5a9ac236ab81f6ca7f41b557d69b92cc25ea6 not found: ID does not exist" containerID="35db67d47dbf22eb2b81eacfc2a5a9ac236ab81f6ca7f41b557d69b92cc25ea6" Jul 21 15:27:45.604757 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:27:45.604682 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35db67d47dbf22eb2b81eacfc2a5a9ac236ab81f6ca7f41b557d69b92cc25ea6"} err="failed to get container status \"35db67d47dbf22eb2b81eacfc2a5a9ac236ab81f6ca7f41b557d69b92cc25ea6\": rpc error: code = NotFound desc = could not find container \"35db67d47dbf22eb2b81eacfc2a5a9ac236ab81f6ca7f41b557d69b92cc25ea6\": container with ID starting with 35db67d47dbf22eb2b81eacfc2a5a9ac236ab81f6ca7f41b557d69b92cc25ea6 not found: ID does not exist" Jul 21 15:27:45.618696 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:27:45.618671 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-57f86f6c-kserve-75c4bff487-sbkld"] Jul 21 15:27:45.621947 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:27:45.621929 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-57f86f6c-kserve-75c4bff487-sbkld"] Jul 21 15:27:45.656306 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:27:45.656289 2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/048a18a0-ffff-47cf-8147-1b28b8cb8ef6-dshm\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 15:27:45.656432 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:27:45.656364 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54vg9\" (UniqueName: \"kubernetes.io/projected/048a18a0-ffff-47cf-8147-1b28b8cb8ef6-kube-api-access-54vg9\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 15:27:45.656495 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:27:45.656483 2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/048a18a0-ffff-47cf-8147-1b28b8cb8ef6-tls-certs\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 15:27:45.656551 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:27:45.656503 2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/048a18a0-ffff-47cf-8147-1b28b8cb8ef6-home\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 15:27:46.488626 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:27:46.485833 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="048a18a0-ffff-47cf-8147-1b28b8cb8ef6" path="/var/lib/kubelet/pods/048a18a0-ffff-47cf-8147-1b28b8cb8ef6/volumes" Jul 21 15:27:46.488626 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:27:46.486451 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b35644bc-a938-4d59-b916-3c5327e8e02d" path="/var/lib/kubelet/pods/b35644bc-a938-4d59-b916-3c5327e8e02d/volumes" Jul 21 15:27:50.437089 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:27:50.437010 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="llm/e2e-same-id-b-57f86f6c-kserve-75c4bff487-sbkld" podUID="048a18a0-ffff-47cf-8147-1b28b8cb8ef6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Jul 21 15:27:51.439300 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:27:51.439271 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-j4c8p"] Jul 21 15:27:59.642387 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:27:59.642351 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-j4c8p"] Jul 21 15:28:09.938250 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:28:09.938216 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-j4c8p"] Jul 21 15:28:16.742417 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:28:16.742386 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-j4c8p"] Jul 21 15:28:25.845152 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:28:25.845125 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-j4c8p"] Jul 21 15:28:33.443005 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:28:33.442972 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-j4c8p"] Jul 21 15:28:48.941898 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:28:48.941869 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-j4c8p"] Jul 21 15:28:55.447377 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:28:55.447334 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-j4c8p"] Jul 21 15:29:07.051725 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:29:07.051686 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-j4c8p"] Jul 21 15:29:14.744083 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:29:14.744049 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-j4c8p"] Jul 21 15:29:20.647022 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:29:20.646940 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-j4c8p"] Jul 21 15:29:26.244561 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:29:26.244530 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-j4c8p"] Jul 21 15:29:28.511230 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:29:28.511132 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kn9fn_02f3d0c6-4116-4134-ba51-1c02c73bef05/ovn-acl-logging/0.log" Jul 21 15:29:28.520632 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:29:28.520614 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kn9fn_02f3d0c6-4116-4134-ba51-1c02c73bef05/ovn-acl-logging/0.log" Jul 21 15:29:32.042980 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:29:32.042949 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-j4c8p"] Jul 21 15:29:40.043318 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:29:40.043281 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-j4c8p"] Jul 21 15:29:45.942849 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:29:45.942814 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-j4c8p"] Jul 21 15:29:54.050417 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:29:54.050383 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-j4c8p"] Jul 21 15:30:00.945383 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:30:00.945349 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-j4c8p"] Jul 21 15:30:06.042234 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:30:06.042204 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-j4c8p"] Jul 21 15:30:12.448919 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:30:12.448883 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-j4c8p"] Jul 21 15:30:19.047865 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:30:19.047824 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-j4c8p"] Jul 21 15:30:32.443810 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:30:32.443776 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-81110a4b-openshift-default-84c7556bc-qx8lq"] Jul 21 15:30:32.444258 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:30:32.444123 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="048a18a0-ffff-47cf-8147-1b28b8cb8ef6" containerName="main" Jul 21 15:30:32.444258 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:30:32.444135 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="048a18a0-ffff-47cf-8147-1b28b8cb8ef6" containerName="main" Jul 21 15:30:32.444258 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:30:32.444151 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b35644bc-a938-4d59-b916-3c5327e8e02d" containerName="main" Jul 21 15:30:32.444258 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:30:32.444157 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b35644bc-a938-4d59-b916-3c5327e8e02d" containerName="main" Jul 21 15:30:32.444258 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:30:32.444219 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="048a18a0-ffff-47cf-8147-1b28b8cb8ef6" containerName="main" Jul 21 15:30:32.444258 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:30:32.444226 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="b35644bc-a938-4d59-b916-3c5327e8e02d" containerName="main" Jul 21 15:30:32.447174 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:30:32.447159 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-81110a4b-openshift-default-84c7556bc-qx8lq" Jul 21 15:30:32.450048 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:30:32.450026 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-81110a4b-openshift-default-dockercfg-m87bc\"" Jul 21 15:30:32.460403 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:30:32.460379 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-81110a4b-openshift-default-84c7556bc-qx8lq"] Jul 21 15:30:32.534583 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:30:32.534545 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96fd54f9-c100-467b-b848-dcf6fbb4a403-credential-socket\") pod \"e2e-ait-81110a4b-openshift-default-84c7556bc-qx8lq\" (UID: \"96fd54f9-c100-467b-b848-dcf6fbb4a403\") " pod="openshift-ingress/e2e-ait-81110a4b-openshift-default-84c7556bc-qx8lq" Jul 21 15:30:32.534583 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:30:32.534587 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96fd54f9-c100-467b-b848-dcf6fbb4a403-workload-certs\") pod \"e2e-ait-81110a4b-openshift-default-84c7556bc-qx8lq\" (UID: \"96fd54f9-c100-467b-b848-dcf6fbb4a403\") " pod="openshift-ingress/e2e-ait-81110a4b-openshift-default-84c7556bc-qx8lq" Jul 21 15:30:32.534809 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:30:32.534610 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96fd54f9-c100-467b-b848-dcf6fbb4a403-istio-token\") pod \"e2e-ait-81110a4b-openshift-default-84c7556bc-qx8lq\" (UID: \"96fd54f9-c100-467b-b848-dcf6fbb4a403\") " pod="openshift-ingress/e2e-ait-81110a4b-openshift-default-84c7556bc-qx8lq" Jul 21 15:30:32.534809 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:30:32.534643 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4vwp\" (UniqueName: \"kubernetes.io/projected/96fd54f9-c100-467b-b848-dcf6fbb4a403-kube-api-access-d4vwp\") pod \"e2e-ait-81110a4b-openshift-default-84c7556bc-qx8lq\" (UID: \"96fd54f9-c100-467b-b848-dcf6fbb4a403\") " pod="openshift-ingress/e2e-ait-81110a4b-openshift-default-84c7556bc-qx8lq" Jul 21 15:30:32.534809 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:30:32.534696 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96fd54f9-c100-467b-b848-dcf6fbb4a403-istio-podinfo\") pod \"e2e-ait-81110a4b-openshift-default-84c7556bc-qx8lq\" (UID: \"96fd54f9-c100-467b-b848-dcf6fbb4a403\") " pod="openshift-ingress/e2e-ait-81110a4b-openshift-default-84c7556bc-qx8lq" Jul 21 15:30:32.534809 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:30:32.534789 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96fd54f9-c100-467b-b848-dcf6fbb4a403-istiod-ca-cert\") pod \"e2e-ait-81110a4b-openshift-default-84c7556bc-qx8lq\" (UID: \"96fd54f9-c100-467b-b848-dcf6fbb4a403\") " pod="openshift-ingress/e2e-ait-81110a4b-openshift-default-84c7556bc-qx8lq" Jul 21 15:30:32.534977 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:30:32.534840 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96fd54f9-c100-467b-b848-dcf6fbb4a403-istio-envoy\") pod \"e2e-ait-81110a4b-openshift-default-84c7556bc-qx8lq\" (UID: \"96fd54f9-c100-467b-b848-dcf6fbb4a403\") " pod="openshift-ingress/e2e-ait-81110a4b-openshift-default-84c7556bc-qx8lq" Jul 21 15:30:32.534977 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:30:32.534868 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96fd54f9-c100-467b-b848-dcf6fbb4a403-istio-data\") pod \"e2e-ait-81110a4b-openshift-default-84c7556bc-qx8lq\" (UID: \"96fd54f9-c100-467b-b848-dcf6fbb4a403\") " pod="openshift-ingress/e2e-ait-81110a4b-openshift-default-84c7556bc-qx8lq" Jul 21 15:30:32.534977 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:30:32.534902 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96fd54f9-c100-467b-b848-dcf6fbb4a403-workload-socket\") pod \"e2e-ait-81110a4b-openshift-default-84c7556bc-qx8lq\" (UID: \"96fd54f9-c100-467b-b848-dcf6fbb4a403\") " pod="openshift-ingress/e2e-ait-81110a4b-openshift-default-84c7556bc-qx8lq" Jul 21 15:30:32.636261 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:30:32.636229 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96fd54f9-c100-467b-b848-dcf6fbb4a403-workload-socket\") pod \"e2e-ait-81110a4b-openshift-default-84c7556bc-qx8lq\" (UID: \"96fd54f9-c100-467b-b848-dcf6fbb4a403\") " pod="openshift-ingress/e2e-ait-81110a4b-openshift-default-84c7556bc-qx8lq" Jul 21 15:30:32.636473 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:30:32.636295 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96fd54f9-c100-467b-b848-dcf6fbb4a403-credential-socket\") pod \"e2e-ait-81110a4b-openshift-default-84c7556bc-qx8lq\" (UID: \"96fd54f9-c100-467b-b848-dcf6fbb4a403\") " pod="openshift-ingress/e2e-ait-81110a4b-openshift-default-84c7556bc-qx8lq" Jul 21 15:30:32.636473 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:30:32.636316 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96fd54f9-c100-467b-b848-dcf6fbb4a403-workload-certs\") pod \"e2e-ait-81110a4b-openshift-default-84c7556bc-qx8lq\" (UID: \"96fd54f9-c100-467b-b848-dcf6fbb4a403\") " pod="openshift-ingress/e2e-ait-81110a4b-openshift-default-84c7556bc-qx8lq" Jul 21 15:30:32.636473 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:30:32.636346 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96fd54f9-c100-467b-b848-dcf6fbb4a403-istio-token\") pod \"e2e-ait-81110a4b-openshift-default-84c7556bc-qx8lq\" (UID: \"96fd54f9-c100-467b-b848-dcf6fbb4a403\") " pod="openshift-ingress/e2e-ait-81110a4b-openshift-default-84c7556bc-qx8lq" Jul 21 15:30:32.636473 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:30:32.636361 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4vwp\" (UniqueName: \"kubernetes.io/projected/96fd54f9-c100-467b-b848-dcf6fbb4a403-kube-api-access-d4vwp\") pod \"e2e-ait-81110a4b-openshift-default-84c7556bc-qx8lq\" (UID: \"96fd54f9-c100-467b-b848-dcf6fbb4a403\") " pod="openshift-ingress/e2e-ait-81110a4b-openshift-default-84c7556bc-qx8lq" Jul 21 15:30:32.636473 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:30:32.636384 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96fd54f9-c100-467b-b848-dcf6fbb4a403-istio-podinfo\") pod \"e2e-ait-81110a4b-openshift-default-84c7556bc-qx8lq\" (UID: \"96fd54f9-c100-467b-b848-dcf6fbb4a403\") " pod="openshift-ingress/e2e-ait-81110a4b-openshift-default-84c7556bc-qx8lq" Jul 21 15:30:32.636473 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:30:32.636427 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96fd54f9-c100-467b-b848-dcf6fbb4a403-istiod-ca-cert\") pod \"e2e-ait-81110a4b-openshift-default-84c7556bc-qx8lq\" (UID: \"96fd54f9-c100-467b-b848-dcf6fbb4a403\") " pod="openshift-ingress/e2e-ait-81110a4b-openshift-default-84c7556bc-qx8lq" Jul 21 15:30:32.636795 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:30:32.636477 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96fd54f9-c100-467b-b848-dcf6fbb4a403-istio-envoy\") pod \"e2e-ait-81110a4b-openshift-default-84c7556bc-qx8lq\" (UID: \"96fd54f9-c100-467b-b848-dcf6fbb4a403\") " pod="openshift-ingress/e2e-ait-81110a4b-openshift-default-84c7556bc-qx8lq" Jul 21 15:30:32.636795 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:30:32.636509 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96fd54f9-c100-467b-b848-dcf6fbb4a403-istio-data\") pod \"e2e-ait-81110a4b-openshift-default-84c7556bc-qx8lq\" (UID: \"96fd54f9-c100-467b-b848-dcf6fbb4a403\") " pod="openshift-ingress/e2e-ait-81110a4b-openshift-default-84c7556bc-qx8lq" Jul 21 15:30:32.636795 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:30:32.636693 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96fd54f9-c100-467b-b848-dcf6fbb4a403-workload-socket\") pod \"e2e-ait-81110a4b-openshift-default-84c7556bc-qx8lq\" (UID: \"96fd54f9-c100-467b-b848-dcf6fbb4a403\") " pod="openshift-ingress/e2e-ait-81110a4b-openshift-default-84c7556bc-qx8lq" Jul 21 15:30:32.636795 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:30:32.636714 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96fd54f9-c100-467b-b848-dcf6fbb4a403-credential-socket\") pod \"e2e-ait-81110a4b-openshift-default-84c7556bc-qx8lq\" (UID: \"96fd54f9-c100-467b-b848-dcf6fbb4a403\") " pod="openshift-ingress/e2e-ait-81110a4b-openshift-default-84c7556bc-qx8lq" Jul 21 15:30:32.636795 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:30:32.636735 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96fd54f9-c100-467b-b848-dcf6fbb4a403-workload-certs\") pod \"e2e-ait-81110a4b-openshift-default-84c7556bc-qx8lq\" (UID: \"96fd54f9-c100-467b-b848-dcf6fbb4a403\") " pod="openshift-ingress/e2e-ait-81110a4b-openshift-default-84c7556bc-qx8lq" Jul 21 15:30:32.636795 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:30:32.636790 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96fd54f9-c100-467b-b848-dcf6fbb4a403-istio-data\") pod \"e2e-ait-81110a4b-openshift-default-84c7556bc-qx8lq\" (UID: \"96fd54f9-c100-467b-b848-dcf6fbb4a403\") " pod="openshift-ingress/e2e-ait-81110a4b-openshift-default-84c7556bc-qx8lq" Jul 21 15:30:32.637173 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:30:32.637154 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96fd54f9-c100-467b-b848-dcf6fbb4a403-istiod-ca-cert\") pod \"e2e-ait-81110a4b-openshift-default-84c7556bc-qx8lq\" (UID: \"96fd54f9-c100-467b-b848-dcf6fbb4a403\") " pod="openshift-ingress/e2e-ait-81110a4b-openshift-default-84c7556bc-qx8lq" Jul 21 15:30:32.638700 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:30:32.638679 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96fd54f9-c100-467b-b848-dcf6fbb4a403-istio-envoy\") pod \"e2e-ait-81110a4b-openshift-default-84c7556bc-qx8lq\" (UID: \"96fd54f9-c100-467b-b848-dcf6fbb4a403\") " pod="openshift-ingress/e2e-ait-81110a4b-openshift-default-84c7556bc-qx8lq" Jul 21 15:30:32.639004 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:30:32.638985 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96fd54f9-c100-467b-b848-dcf6fbb4a403-istio-podinfo\") pod \"e2e-ait-81110a4b-openshift-default-84c7556bc-qx8lq\" (UID: \"96fd54f9-c100-467b-b848-dcf6fbb4a403\") " pod="openshift-ingress/e2e-ait-81110a4b-openshift-default-84c7556bc-qx8lq" Jul 21 15:30:32.644368 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:30:32.644318 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96fd54f9-c100-467b-b848-dcf6fbb4a403-istio-token\") pod \"e2e-ait-81110a4b-openshift-default-84c7556bc-qx8lq\" (UID: \"96fd54f9-c100-467b-b848-dcf6fbb4a403\") " pod="openshift-ingress/e2e-ait-81110a4b-openshift-default-84c7556bc-qx8lq" Jul 21 15:30:32.644528 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:30:32.644510 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4vwp\" (UniqueName: \"kubernetes.io/projected/96fd54f9-c100-467b-b848-dcf6fbb4a403-kube-api-access-d4vwp\") pod \"e2e-ait-81110a4b-openshift-default-84c7556bc-qx8lq\" (UID: \"96fd54f9-c100-467b-b848-dcf6fbb4a403\") " pod="openshift-ingress/e2e-ait-81110a4b-openshift-default-84c7556bc-qx8lq" Jul 21 15:30:32.757651 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:30:32.757595 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-81110a4b-openshift-default-84c7556bc-qx8lq" Jul 21 15:30:32.894085 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:30:32.894059 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-81110a4b-openshift-default-84c7556bc-qx8lq"] Jul 21 15:30:32.895201 ip-10-0-135-176 kubenswrapper[2563]: W0721 15:30:32.895177 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96fd54f9_c100_467b_b848_dcf6fbb4a403.slice/crio-4d8e93cc7b29f55f79747a26bb802284ad7aca51e4c2cd27a1a02ea416021498 WatchSource:0}: Error finding container 4d8e93cc7b29f55f79747a26bb802284ad7aca51e4c2cd27a1a02ea416021498: Status 404 returned error can't find the container with id 4d8e93cc7b29f55f79747a26bb802284ad7aca51e4c2cd27a1a02ea416021498 Jul 21 15:30:32.897536 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:30:32.897501 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 21 15:30:32.897625 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:30:32.897564 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 21 15:30:32.897625 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:30:32.897594 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 21 15:30:33.484642 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:30:33.484607 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-81110a4b-openshift-default-84c7556bc-qx8lq" event={"ID":"96fd54f9-c100-467b-b848-dcf6fbb4a403","Type":"ContainerStarted","Data":"cf34bf10e66bae9bd3e56778244dbba2b6462a548575cf5cedc2e55d5a83209f"} Jul 21 15:30:33.485000 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:30:33.484666 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-81110a4b-openshift-default-84c7556bc-qx8lq" event={"ID":"96fd54f9-c100-467b-b848-dcf6fbb4a403","Type":"ContainerStarted","Data":"4d8e93cc7b29f55f79747a26bb802284ad7aca51e4c2cd27a1a02ea416021498"} Jul 21 15:30:33.523947 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:30:33.523910 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-81110a4b-openshift-default-84c7556bc-qx8lq" podStartSLOduration=1.523895706 podStartE2EDuration="1.523895706s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 15:30:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 15:30:33.520563197 +0000 UTC m=+1865.516348614" watchObservedRunningTime="2026-07-21 15:30:33.523895706 +0000 UTC m=+1865.519681121" Jul 21 15:30:33.757849 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:30:33.757776 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-81110a4b-openshift-default-84c7556bc-qx8lq" Jul 21 15:30:33.762285 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:30:33.762263 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-81110a4b-openshift-default-84c7556bc-qx8lq" Jul 21 15:30:34.490166 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:30:34.490137 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-81110a4b-openshift-default-84c7556bc-qx8lq" Jul 21 15:30:34.490927 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:30:34.490908 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-81110a4b-openshift-default-84c7556bc-qx8lq" Jul 21 15:30:38.345106 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:30:38.345075 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-j4c8p"] Jul 21 15:30:47.563716 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:30:47.563679 2563 generic.go:356] "Generic (PLEG): container finished" podID="96fd54f9-c100-467b-b848-dcf6fbb4a403" containerID="cf34bf10e66bae9bd3e56778244dbba2b6462a548575cf5cedc2e55d5a83209f" exitCode=0 Jul 21 15:30:47.564105 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:30:47.563751 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-81110a4b-openshift-default-84c7556bc-qx8lq" event={"ID":"96fd54f9-c100-467b-b848-dcf6fbb4a403","Type":"ContainerDied","Data":"cf34bf10e66bae9bd3e56778244dbba2b6462a548575cf5cedc2e55d5a83209f"} Jul 21 15:30:47.564105 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:30:47.564080 2563 scope.go:117] "RemoveContainer" containerID="cf34bf10e66bae9bd3e56778244dbba2b6462a548575cf5cedc2e55d5a83209f" Jul 21 15:30:47.565012 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:30:47.564983 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 21 15:30:47.565118 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:30:47.565047 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 21 15:30:47.565118 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:30:47.565079 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 21 15:30:48.571073 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:30:48.571036 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-81110a4b-openshift-default-84c7556bc-qx8lq" event={"ID":"96fd54f9-c100-467b-b848-dcf6fbb4a403","Type":"ContainerStarted","Data":"9a22bad39562b74dbe630561756b4a378ea8269b40437b79919cd9df313f5efb"} Jul 21 15:30:48.758499 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:30:48.758470 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-81110a4b-openshift-default-84c7556bc-qx8lq" Jul 21 15:30:48.759730 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:30:48.759694 2563 patch_prober.go:28] interesting pod/e2e-ait-81110a4b-openshift-default-84c7556bc-qx8lq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Jul 21 15:30:48.759862 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:30:48.759783 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-81110a4b-openshift-default-84c7556bc-qx8lq" podUID="96fd54f9-c100-467b-b848-dcf6fbb4a403" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Jul 21 15:30:49.491243 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:30:49.491152 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-81110a4b-openshift-default-84c7556bc-qx8lq" Jul 21 15:30:49.758783 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:30:49.758705 2563 patch_prober.go:28] interesting pod/e2e-ait-81110a4b-openshift-default-84c7556bc-qx8lq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Jul 21 15:30:49.759113 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:30:49.758781 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-81110a4b-openshift-default-84c7556bc-qx8lq" podUID="96fd54f9-c100-467b-b848-dcf6fbb4a403" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Jul 21 15:30:50.758121 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:30:50.758083 2563 patch_prober.go:28] interesting pod/e2e-ait-81110a4b-openshift-default-84c7556bc-qx8lq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Jul 21 15:30:50.758318 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:30:50.758141 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-81110a4b-openshift-default-84c7556bc-qx8lq" podUID="96fd54f9-c100-467b-b848-dcf6fbb4a403" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Jul 21 15:30:51.758691 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:30:51.758658 2563 patch_prober.go:28] interesting pod/e2e-ait-81110a4b-openshift-default-84c7556bc-qx8lq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Jul 21 15:30:51.759076 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:30:51.758705 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-81110a4b-openshift-default-84c7556bc-qx8lq" podUID="96fd54f9-c100-467b-b848-dcf6fbb4a403" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Jul 21 15:30:52.758935 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:30:52.758901 2563 patch_prober.go:28] interesting pod/e2e-ait-81110a4b-openshift-default-84c7556bc-qx8lq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Jul 21 15:30:52.759320 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:30:52.758948 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-81110a4b-openshift-default-84c7556bc-qx8lq" podUID="96fd54f9-c100-467b-b848-dcf6fbb4a403" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Jul 21 15:30:53.761685 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:30:53.761659 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-81110a4b-openshift-default-84c7556bc-qx8lq" Jul 21 15:30:53.762646 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:30:53.762630 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-81110a4b-openshift-default-84c7556bc-qx8lq" Jul 21 15:31:13.340310 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:31:13.340275 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-j4c8p"] Jul 21 15:34:00.999175 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:34:00.999097 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-81110a4b-openshift-default-84c7556bc-qx8lq"] Jul 21 15:34:00.999707 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:34:00.999364 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-81110a4b-openshift-default-84c7556bc-qx8lq" podUID="96fd54f9-c100-467b-b848-dcf6fbb4a403" containerName="istio-proxy" containerID="cri-o://9a22bad39562b74dbe630561756b4a378ea8269b40437b79919cd9df313f5efb" gracePeriod=30 Jul 21 15:34:02.046722 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:34:02.046690 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-719155ca-gw-openshift-default-8578dbf467-lcfrg"] Jul 21 15:34:02.050132 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:34:02.050115 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-719155ca-gw-openshift-default-8578dbf467-lcfrg" Jul 21 15:34:02.053198 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:34:02.053180 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-719155ca-gw-openshift-default-dockercfg-cnhqj\"" Jul 21 15:34:02.060784 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:34:02.060764 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-719155ca-gw-openshift-default-8578dbf467-lcfrg"] Jul 21 15:34:02.172805 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:34:02.172772 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cc65bf1-45a7-447e-aeba-a6ab9b43992f-istio-token\") pod \"e2e-migrate-719155ca-gw-openshift-default-8578dbf467-lcfrg\" (UID: \"0cc65bf1-45a7-447e-aeba-a6ab9b43992f\") " pod="openshift-ingress/e2e-migrate-719155ca-gw-openshift-default-8578dbf467-lcfrg" Jul 21 15:34:02.173025 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:34:02.172818 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cc65bf1-45a7-447e-aeba-a6ab9b43992f-credential-socket\") pod \"e2e-migrate-719155ca-gw-openshift-default-8578dbf467-lcfrg\" (UID: \"0cc65bf1-45a7-447e-aeba-a6ab9b43992f\") " pod="openshift-ingress/e2e-migrate-719155ca-gw-openshift-default-8578dbf467-lcfrg" Jul 21 15:34:02.173025 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:34:02.172889 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cc65bf1-45a7-447e-aeba-a6ab9b43992f-workload-socket\") pod \"e2e-migrate-719155ca-gw-openshift-default-8578dbf467-lcfrg\" (UID: \"0cc65bf1-45a7-447e-aeba-a6ab9b43992f\") " pod="openshift-ingress/e2e-migrate-719155ca-gw-openshift-default-8578dbf467-lcfrg" Jul 21 15:34:02.173025 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:34:02.172932 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cc65bf1-45a7-447e-aeba-a6ab9b43992f-istiod-ca-cert\") pod \"e2e-migrate-719155ca-gw-openshift-default-8578dbf467-lcfrg\" (UID: \"0cc65bf1-45a7-447e-aeba-a6ab9b43992f\") " pod="openshift-ingress/e2e-migrate-719155ca-gw-openshift-default-8578dbf467-lcfrg" Jul 21 15:34:02.173025 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:34:02.172961 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cc65bf1-45a7-447e-aeba-a6ab9b43992f-istio-data\") pod \"e2e-migrate-719155ca-gw-openshift-default-8578dbf467-lcfrg\" (UID: \"0cc65bf1-45a7-447e-aeba-a6ab9b43992f\") " pod="openshift-ingress/e2e-migrate-719155ca-gw-openshift-default-8578dbf467-lcfrg" Jul 21 15:34:02.173025 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:34:02.172982 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cc65bf1-45a7-447e-aeba-a6ab9b43992f-workload-certs\") pod \"e2e-migrate-719155ca-gw-openshift-default-8578dbf467-lcfrg\" (UID: \"0cc65bf1-45a7-447e-aeba-a6ab9b43992f\") " pod="openshift-ingress/e2e-migrate-719155ca-gw-openshift-default-8578dbf467-lcfrg" Jul 21 15:34:02.173194 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:34:02.173091 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cc65bf1-45a7-447e-aeba-a6ab9b43992f-istio-podinfo\") pod \"e2e-migrate-719155ca-gw-openshift-default-8578dbf467-lcfrg\" (UID: \"0cc65bf1-45a7-447e-aeba-a6ab9b43992f\") " pod="openshift-ingress/e2e-migrate-719155ca-gw-openshift-default-8578dbf467-lcfrg" Jul 21 15:34:02.173194 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:34:02.173116 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmkq4\" (UniqueName: \"kubernetes.io/projected/0cc65bf1-45a7-447e-aeba-a6ab9b43992f-kube-api-access-gmkq4\") pod \"e2e-migrate-719155ca-gw-openshift-default-8578dbf467-lcfrg\" (UID: \"0cc65bf1-45a7-447e-aeba-a6ab9b43992f\") " pod="openshift-ingress/e2e-migrate-719155ca-gw-openshift-default-8578dbf467-lcfrg" Jul 21 15:34:02.173194 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:34:02.173176 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cc65bf1-45a7-447e-aeba-a6ab9b43992f-istio-envoy\") pod \"e2e-migrate-719155ca-gw-openshift-default-8578dbf467-lcfrg\" (UID: \"0cc65bf1-45a7-447e-aeba-a6ab9b43992f\") " pod="openshift-ingress/e2e-migrate-719155ca-gw-openshift-default-8578dbf467-lcfrg" Jul 21 15:34:02.274347 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:34:02.274293 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cc65bf1-45a7-447e-aeba-a6ab9b43992f-istio-data\") pod \"e2e-migrate-719155ca-gw-openshift-default-8578dbf467-lcfrg\" (UID: \"0cc65bf1-45a7-447e-aeba-a6ab9b43992f\") " pod="openshift-ingress/e2e-migrate-719155ca-gw-openshift-default-8578dbf467-lcfrg" Jul 21 15:34:02.274528 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:34:02.274357 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cc65bf1-45a7-447e-aeba-a6ab9b43992f-workload-certs\") pod \"e2e-migrate-719155ca-gw-openshift-default-8578dbf467-lcfrg\" (UID: \"0cc65bf1-45a7-447e-aeba-a6ab9b43992f\") " pod="openshift-ingress/e2e-migrate-719155ca-gw-openshift-default-8578dbf467-lcfrg" Jul 21 15:34:02.274528 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:34:02.274412 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cc65bf1-45a7-447e-aeba-a6ab9b43992f-istio-podinfo\") pod \"e2e-migrate-719155ca-gw-openshift-default-8578dbf467-lcfrg\" (UID: \"0cc65bf1-45a7-447e-aeba-a6ab9b43992f\") " pod="openshift-ingress/e2e-migrate-719155ca-gw-openshift-default-8578dbf467-lcfrg" Jul 21 15:34:02.274528 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:34:02.274433 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmkq4\" (UniqueName: \"kubernetes.io/projected/0cc65bf1-45a7-447e-aeba-a6ab9b43992f-kube-api-access-gmkq4\") pod \"e2e-migrate-719155ca-gw-openshift-default-8578dbf467-lcfrg\" (UID: \"0cc65bf1-45a7-447e-aeba-a6ab9b43992f\") " pod="openshift-ingress/e2e-migrate-719155ca-gw-openshift-default-8578dbf467-lcfrg" Jul 21 15:34:02.274528 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:34:02.274475 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cc65bf1-45a7-447e-aeba-a6ab9b43992f-istio-envoy\") pod \"e2e-migrate-719155ca-gw-openshift-default-8578dbf467-lcfrg\" (UID: \"0cc65bf1-45a7-447e-aeba-a6ab9b43992f\") " pod="openshift-ingress/e2e-migrate-719155ca-gw-openshift-default-8578dbf467-lcfrg" Jul 21 15:34:02.274528 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:34:02.274489 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cc65bf1-45a7-447e-aeba-a6ab9b43992f-istio-token\") pod \"e2e-migrate-719155ca-gw-openshift-default-8578dbf467-lcfrg\" (UID: \"0cc65bf1-45a7-447e-aeba-a6ab9b43992f\") " pod="openshift-ingress/e2e-migrate-719155ca-gw-openshift-default-8578dbf467-lcfrg" Jul 21 15:34:02.274528 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:34:02.274518 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cc65bf1-45a7-447e-aeba-a6ab9b43992f-credential-socket\") pod \"e2e-migrate-719155ca-gw-openshift-default-8578dbf467-lcfrg\" (UID: \"0cc65bf1-45a7-447e-aeba-a6ab9b43992f\") " pod="openshift-ingress/e2e-migrate-719155ca-gw-openshift-default-8578dbf467-lcfrg" Jul 21 15:34:02.274838 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:34:02.274540 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cc65bf1-45a7-447e-aeba-a6ab9b43992f-workload-socket\") pod \"e2e-migrate-719155ca-gw-openshift-default-8578dbf467-lcfrg\" (UID: \"0cc65bf1-45a7-447e-aeba-a6ab9b43992f\") " pod="openshift-ingress/e2e-migrate-719155ca-gw-openshift-default-8578dbf467-lcfrg" Jul 21 15:34:02.274838 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:34:02.274644 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cc65bf1-45a7-447e-aeba-a6ab9b43992f-istiod-ca-cert\") pod \"e2e-migrate-719155ca-gw-openshift-default-8578dbf467-lcfrg\" (UID: \"0cc65bf1-45a7-447e-aeba-a6ab9b43992f\") " pod="openshift-ingress/e2e-migrate-719155ca-gw-openshift-default-8578dbf467-lcfrg" Jul 21 15:34:02.274838 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:34:02.274759 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cc65bf1-45a7-447e-aeba-a6ab9b43992f-istio-data\") pod \"e2e-migrate-719155ca-gw-openshift-default-8578dbf467-lcfrg\" (UID: \"0cc65bf1-45a7-447e-aeba-a6ab9b43992f\") " pod="openshift-ingress/e2e-migrate-719155ca-gw-openshift-default-8578dbf467-lcfrg" Jul 21 15:34:02.274838 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:34:02.274777 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cc65bf1-45a7-447e-aeba-a6ab9b43992f-workload-certs\") pod \"e2e-migrate-719155ca-gw-openshift-default-8578dbf467-lcfrg\" (UID: \"0cc65bf1-45a7-447e-aeba-a6ab9b43992f\") " pod="openshift-ingress/e2e-migrate-719155ca-gw-openshift-default-8578dbf467-lcfrg" Jul 21 15:34:02.275095 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:34:02.275074 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cc65bf1-45a7-447e-aeba-a6ab9b43992f-workload-socket\") pod \"e2e-migrate-719155ca-gw-openshift-default-8578dbf467-lcfrg\" (UID: \"0cc65bf1-45a7-447e-aeba-a6ab9b43992f\") " pod="openshift-ingress/e2e-migrate-719155ca-gw-openshift-default-8578dbf467-lcfrg" Jul 21 15:34:02.275363 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:34:02.275320 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cc65bf1-45a7-447e-aeba-a6ab9b43992f-credential-socket\") pod \"e2e-migrate-719155ca-gw-openshift-default-8578dbf467-lcfrg\" (UID: \"0cc65bf1-45a7-447e-aeba-a6ab9b43992f\") " pod="openshift-ingress/e2e-migrate-719155ca-gw-openshift-default-8578dbf467-lcfrg" Jul 21 15:34:02.275445 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:34:02.275348 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cc65bf1-45a7-447e-aeba-a6ab9b43992f-istiod-ca-cert\") pod \"e2e-migrate-719155ca-gw-openshift-default-8578dbf467-lcfrg\" (UID: \"0cc65bf1-45a7-447e-aeba-a6ab9b43992f\") " pod="openshift-ingress/e2e-migrate-719155ca-gw-openshift-default-8578dbf467-lcfrg" Jul 21 15:34:02.276800 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:34:02.276783 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cc65bf1-45a7-447e-aeba-a6ab9b43992f-istio-envoy\") pod \"e2e-migrate-719155ca-gw-openshift-default-8578dbf467-lcfrg\" (UID: \"0cc65bf1-45a7-447e-aeba-a6ab9b43992f\") " pod="openshift-ingress/e2e-migrate-719155ca-gw-openshift-default-8578dbf467-lcfrg" Jul 21 15:34:02.277079 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:34:02.277062 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cc65bf1-45a7-447e-aeba-a6ab9b43992f-istio-podinfo\") pod \"e2e-migrate-719155ca-gw-openshift-default-8578dbf467-lcfrg\" (UID: \"0cc65bf1-45a7-447e-aeba-a6ab9b43992f\") " pod="openshift-ingress/e2e-migrate-719155ca-gw-openshift-default-8578dbf467-lcfrg" Jul 21 15:34:02.287385 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:34:02.287356 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cc65bf1-45a7-447e-aeba-a6ab9b43992f-istio-token\") pod \"e2e-migrate-719155ca-gw-openshift-default-8578dbf467-lcfrg\" (UID: \"0cc65bf1-45a7-447e-aeba-a6ab9b43992f\") " pod="openshift-ingress/e2e-migrate-719155ca-gw-openshift-default-8578dbf467-lcfrg" Jul 21 15:34:02.288158 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:34:02.288139 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmkq4\" (UniqueName: \"kubernetes.io/projected/0cc65bf1-45a7-447e-aeba-a6ab9b43992f-kube-api-access-gmkq4\") pod \"e2e-migrate-719155ca-gw-openshift-default-8578dbf467-lcfrg\" (UID: \"0cc65bf1-45a7-447e-aeba-a6ab9b43992f\") " pod="openshift-ingress/e2e-migrate-719155ca-gw-openshift-default-8578dbf467-lcfrg" Jul 21 15:34:02.360682 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:34:02.360656 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-719155ca-gw-openshift-default-8578dbf467-lcfrg" Jul 21 15:34:02.513302 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:34:02.513276 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-719155ca-gw-openshift-default-8578dbf467-lcfrg"] Jul 21 15:34:02.515152 ip-10-0-135-176 kubenswrapper[2563]: W0721 15:34:02.515129 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cc65bf1_45a7_447e_aeba_a6ab9b43992f.slice/crio-e733784a5e4da7c6a2c733d44197f53545d01ce8ba77b033d05ca0095251449c WatchSource:0}: Error finding container e733784a5e4da7c6a2c733d44197f53545d01ce8ba77b033d05ca0095251449c: Status 404 returned error can't find the container with id e733784a5e4da7c6a2c733d44197f53545d01ce8ba77b033d05ca0095251449c Jul 21 15:34:02.517112 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:34:02.517082 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 21 15:34:02.517195 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:34:02.517150 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 21 15:34:02.517195 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:34:02.517181 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 21 15:34:02.602135 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:34:02.602097 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-719155ca-gw-openshift-default-8578dbf467-lcfrg" event={"ID":"0cc65bf1-45a7-447e-aeba-a6ab9b43992f","Type":"ContainerStarted","Data":"0efe42190699b2a26d862ab36e46ef0a42059194f73a7b9bb212296b3f4efa8d"} Jul 21 15:34:02.602286 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:34:02.602166 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-719155ca-gw-openshift-default-8578dbf467-lcfrg" event={"ID":"0cc65bf1-45a7-447e-aeba-a6ab9b43992f","Type":"ContainerStarted","Data":"e733784a5e4da7c6a2c733d44197f53545d01ce8ba77b033d05ca0095251449c"} Jul 21 15:34:02.625849 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:34:02.625751 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-719155ca-gw-openshift-default-8578dbf467-lcfrg" podStartSLOduration=0.625729388 podStartE2EDuration="625.729388ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 15:34:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 15:34:02.622696495 +0000 UTC m=+2074.618481930" watchObservedRunningTime="2026-07-21 15:34:02.625729388 +0000 UTC m=+2074.621514806" Jul 21 15:34:03.363584 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:34:03.363532 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-719155ca-gw-openshift-default-8578dbf467-lcfrg" Jul 21 15:34:03.368966 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:34:03.368938 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-719155ca-gw-openshift-default-8578dbf467-lcfrg" Jul 21 15:34:03.607472 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:34:03.607446 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-719155ca-gw-openshift-default-8578dbf467-lcfrg" Jul 21 15:34:03.608469 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:34:03.608443 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-719155ca-gw-openshift-default-8578dbf467-lcfrg" Jul 21 15:34:06.253119 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:34:06.253101 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-81110a4b-openshift-default-84c7556bc-qx8lq" Jul 21 15:34:06.412380 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:34:06.412310 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96fd54f9-c100-467b-b848-dcf6fbb4a403-istio-envoy\") pod \"96fd54f9-c100-467b-b848-dcf6fbb4a403\" (UID: \"96fd54f9-c100-467b-b848-dcf6fbb4a403\") " Jul 21 15:34:06.412541 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:34:06.412389 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96fd54f9-c100-467b-b848-dcf6fbb4a403-istio-data\") pod \"96fd54f9-c100-467b-b848-dcf6fbb4a403\" (UID: \"96fd54f9-c100-467b-b848-dcf6fbb4a403\") " Jul 21 15:34:06.412541 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:34:06.412414 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96fd54f9-c100-467b-b848-dcf6fbb4a403-workload-socket\") pod \"96fd54f9-c100-467b-b848-dcf6fbb4a403\" (UID: \"96fd54f9-c100-467b-b848-dcf6fbb4a403\") " Jul 21 15:34:06.412541 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:34:06.412459 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96fd54f9-c100-467b-b848-dcf6fbb4a403-credential-socket\") pod \"96fd54f9-c100-467b-b848-dcf6fbb4a403\" (UID: \"96fd54f9-c100-467b-b848-dcf6fbb4a403\") " Jul 21 15:34:06.412541 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:34:06.412483 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d4vwp\" (UniqueName: \"kubernetes.io/projected/96fd54f9-c100-467b-b848-dcf6fbb4a403-kube-api-access-d4vwp\") pod \"96fd54f9-c100-467b-b848-dcf6fbb4a403\" (UID: \"96fd54f9-c100-467b-b848-dcf6fbb4a403\") " Jul 21 15:34:06.412541 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:34:06.412528 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96fd54f9-c100-467b-b848-dcf6fbb4a403-workload-certs\") pod \"96fd54f9-c100-467b-b848-dcf6fbb4a403\" (UID: \"96fd54f9-c100-467b-b848-dcf6fbb4a403\") " Jul 21 15:34:06.412809 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:34:06.412565 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96fd54f9-c100-467b-b848-dcf6fbb4a403-istiod-ca-cert\") pod \"96fd54f9-c100-467b-b848-dcf6fbb4a403\" (UID: \"96fd54f9-c100-467b-b848-dcf6fbb4a403\") " Jul 21 15:34:06.412809 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:34:06.412591 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96fd54f9-c100-467b-b848-dcf6fbb4a403-istio-podinfo\") pod \"96fd54f9-c100-467b-b848-dcf6fbb4a403\" (UID: \"96fd54f9-c100-467b-b848-dcf6fbb4a403\") " Jul 21 15:34:06.412809 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:34:06.412640 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96fd54f9-c100-467b-b848-dcf6fbb4a403-istio-token\") pod \"96fd54f9-c100-467b-b848-dcf6fbb4a403\" (UID: \"96fd54f9-c100-467b-b848-dcf6fbb4a403\") " Jul 21 15:34:06.412809 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:34:06.412754 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96fd54f9-c100-467b-b848-dcf6fbb4a403-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "96fd54f9-c100-467b-b848-dcf6fbb4a403" (UID: "96fd54f9-c100-467b-b848-dcf6fbb4a403"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 15:34:06.413017 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:34:06.412999 2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96fd54f9-c100-467b-b848-dcf6fbb4a403-workload-socket\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 15:34:06.413101 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:34:06.413070 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96fd54f9-c100-467b-b848-dcf6fbb4a403-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "96fd54f9-c100-467b-b848-dcf6fbb4a403" (UID: "96fd54f9-c100-467b-b848-dcf6fbb4a403"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 15:34:06.413435 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:34:06.413409 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96fd54f9-c100-467b-b848-dcf6fbb4a403-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "96fd54f9-c100-467b-b848-dcf6fbb4a403" (UID: "96fd54f9-c100-467b-b848-dcf6fbb4a403"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 15:34:06.413435 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:34:06.413420 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96fd54f9-c100-467b-b848-dcf6fbb4a403-istio-data" (OuterVolumeSpecName: "istio-data") pod "96fd54f9-c100-467b-b848-dcf6fbb4a403" (UID: "96fd54f9-c100-467b-b848-dcf6fbb4a403"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 15:34:06.413628 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:34:06.413447 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96fd54f9-c100-467b-b848-dcf6fbb4a403-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "96fd54f9-c100-467b-b848-dcf6fbb4a403" (UID: "96fd54f9-c100-467b-b848-dcf6fbb4a403"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 15:34:06.415157 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:34:06.415117 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96fd54f9-c100-467b-b848-dcf6fbb4a403-kube-api-access-d4vwp" (OuterVolumeSpecName: "kube-api-access-d4vwp") pod "96fd54f9-c100-467b-b848-dcf6fbb4a403" (UID: "96fd54f9-c100-467b-b848-dcf6fbb4a403"). InnerVolumeSpecName "kube-api-access-d4vwp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 15:34:06.415379 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:34:06.415357 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/96fd54f9-c100-467b-b848-dcf6fbb4a403-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "96fd54f9-c100-467b-b848-dcf6fbb4a403" (UID: "96fd54f9-c100-467b-b848-dcf6fbb4a403"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 21 15:34:06.415469 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:34:06.415452 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96fd54f9-c100-467b-b848-dcf6fbb4a403-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "96fd54f9-c100-467b-b848-dcf6fbb4a403" (UID: "96fd54f9-c100-467b-b848-dcf6fbb4a403"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 15:34:06.415534 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:34:06.415503 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96fd54f9-c100-467b-b848-dcf6fbb4a403-istio-token" (OuterVolumeSpecName: "istio-token") pod "96fd54f9-c100-467b-b848-dcf6fbb4a403" (UID: "96fd54f9-c100-467b-b848-dcf6fbb4a403"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 15:34:06.514105 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:34:06.514084 2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96fd54f9-c100-467b-b848-dcf6fbb4a403-istio-envoy\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 15:34:06.514192 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:34:06.514112 2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96fd54f9-c100-467b-b848-dcf6fbb4a403-istio-data\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 15:34:06.514192 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:34:06.514125 2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96fd54f9-c100-467b-b848-dcf6fbb4a403-credential-socket\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 15:34:06.514192 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:34:06.514135 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4vwp\" (UniqueName: \"kubernetes.io/projected/96fd54f9-c100-467b-b848-dcf6fbb4a403-kube-api-access-d4vwp\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 15:34:06.514192 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:34:06.514143 2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96fd54f9-c100-467b-b848-dcf6fbb4a403-workload-certs\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 15:34:06.514192 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:34:06.514151 2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96fd54f9-c100-467b-b848-dcf6fbb4a403-istiod-ca-cert\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 15:34:06.514192 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:34:06.514159 2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96fd54f9-c100-467b-b848-dcf6fbb4a403-istio-podinfo\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 15:34:06.514192 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:34:06.514166 2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96fd54f9-c100-467b-b848-dcf6fbb4a403-istio-token\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 15:34:06.624807 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:34:06.624782 2563 generic.go:356] "Generic (PLEG): container finished" podID="96fd54f9-c100-467b-b848-dcf6fbb4a403" containerID="9a22bad39562b74dbe630561756b4a378ea8269b40437b79919cd9df313f5efb" exitCode=0 Jul 21 15:34:06.624938 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:34:06.624825 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-81110a4b-openshift-default-84c7556bc-qx8lq" Jul 21 15:34:06.624938 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:34:06.624826 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-81110a4b-openshift-default-84c7556bc-qx8lq" event={"ID":"96fd54f9-c100-467b-b848-dcf6fbb4a403","Type":"ContainerDied","Data":"9a22bad39562b74dbe630561756b4a378ea8269b40437b79919cd9df313f5efb"} Jul 21 15:34:06.624938 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:34:06.624895 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-81110a4b-openshift-default-84c7556bc-qx8lq" event={"ID":"96fd54f9-c100-467b-b848-dcf6fbb4a403","Type":"ContainerDied","Data":"4d8e93cc7b29f55f79747a26bb802284ad7aca51e4c2cd27a1a02ea416021498"} Jul 21 15:34:06.624938 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:34:06.624927 2563 scope.go:117] "RemoveContainer" containerID="9a22bad39562b74dbe630561756b4a378ea8269b40437b79919cd9df313f5efb" Jul 21 15:34:06.634400 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:34:06.634383 2563 scope.go:117] "RemoveContainer" containerID="cf34bf10e66bae9bd3e56778244dbba2b6462a548575cf5cedc2e55d5a83209f" Jul 21 15:34:06.642912 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:34:06.642896 2563 scope.go:117] "RemoveContainer" containerID="9a22bad39562b74dbe630561756b4a378ea8269b40437b79919cd9df313f5efb" Jul 21 15:34:06.643115 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:34:06.643103 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-81110a4b-openshift-default-84c7556bc-qx8lq"] Jul 21 15:34:06.643219 ip-10-0-135-176 kubenswrapper[2563]: E0721 15:34:06.643183 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a22bad39562b74dbe630561756b4a378ea8269b40437b79919cd9df313f5efb\": container with ID starting with 9a22bad39562b74dbe630561756b4a378ea8269b40437b79919cd9df313f5efb not found: ID does not exist" containerID="9a22bad39562b74dbe630561756b4a378ea8269b40437b79919cd9df313f5efb" Jul 21 15:34:06.643285 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:34:06.643212 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a22bad39562b74dbe630561756b4a378ea8269b40437b79919cd9df313f5efb"} err="failed to get container status \"9a22bad39562b74dbe630561756b4a378ea8269b40437b79919cd9df313f5efb\": rpc error: code = NotFound desc = could not find container \"9a22bad39562b74dbe630561756b4a378ea8269b40437b79919cd9df313f5efb\": container with ID starting with 9a22bad39562b74dbe630561756b4a378ea8269b40437b79919cd9df313f5efb not found: ID does not exist" Jul 21 15:34:06.643285 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:34:06.643234 2563 scope.go:117] "RemoveContainer" containerID="cf34bf10e66bae9bd3e56778244dbba2b6462a548575cf5cedc2e55d5a83209f" Jul 21 15:34:06.643514 ip-10-0-135-176 kubenswrapper[2563]: E0721 15:34:06.643492 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf34bf10e66bae9bd3e56778244dbba2b6462a548575cf5cedc2e55d5a83209f\": container with ID starting with cf34bf10e66bae9bd3e56778244dbba2b6462a548575cf5cedc2e55d5a83209f not found: ID does not exist" containerID="cf34bf10e66bae9bd3e56778244dbba2b6462a548575cf5cedc2e55d5a83209f" Jul 21 15:34:06.643572 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:34:06.643521 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf34bf10e66bae9bd3e56778244dbba2b6462a548575cf5cedc2e55d5a83209f"} err="failed to get container status \"cf34bf10e66bae9bd3e56778244dbba2b6462a548575cf5cedc2e55d5a83209f\": rpc error: code = NotFound desc = could not find container \"cf34bf10e66bae9bd3e56778244dbba2b6462a548575cf5cedc2e55d5a83209f\": container with ID starting with cf34bf10e66bae9bd3e56778244dbba2b6462a548575cf5cedc2e55d5a83209f not found: ID does not exist" Jul 21 15:34:06.645421 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:34:06.645403 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-81110a4b-openshift-default-84c7556bc-qx8lq"] Jul 21 15:34:07.234236 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:34:07.234203 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-j4c8p"] Jul 21 15:34:08.480996 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:34:08.480947 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="96fd54f9-c100-467b-b848-dcf6fbb4a403" path="/var/lib/kubelet/pods/96fd54f9-c100-467b-b848-dcf6fbb4a403/volumes" Jul 21 15:34:16.683175 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:34:16.683079 2563 generic.go:356] "Generic (PLEG): container finished" podID="0cc65bf1-45a7-447e-aeba-a6ab9b43992f" containerID="0efe42190699b2a26d862ab36e46ef0a42059194f73a7b9bb212296b3f4efa8d" exitCode=0 Jul 21 15:34:16.683175 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:34:16.683157 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-719155ca-gw-openshift-default-8578dbf467-lcfrg" event={"ID":"0cc65bf1-45a7-447e-aeba-a6ab9b43992f","Type":"ContainerDied","Data":"0efe42190699b2a26d862ab36e46ef0a42059194f73a7b9bb212296b3f4efa8d"} Jul 21 15:34:16.683697 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:34:16.683579 2563 scope.go:117] "RemoveContainer" containerID="0efe42190699b2a26d862ab36e46ef0a42059194f73a7b9bb212296b3f4efa8d" Jul 21 15:34:16.684576 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:34:16.684548 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 21 15:34:16.684682 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:34:16.684631 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 21 15:34:16.684748 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:34:16.684703 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 21 15:34:17.695773 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:34:17.695730 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-719155ca-gw-openshift-default-8578dbf467-lcfrg" event={"ID":"0cc65bf1-45a7-447e-aeba-a6ab9b43992f","Type":"ContainerStarted","Data":"0bd6538d3fae3eeba42597eaddf3e5754e902dcc209ae28def7b33d30da84698"} Jul 21 15:34:18.360978 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:34:18.360938 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-719155ca-gw-openshift-default-8578dbf467-lcfrg" Jul 21 15:34:18.362233 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:34:18.362210 2563 patch_prober.go:28] interesting pod/e2e-migrate-719155ca-gw-openshift-default-8578dbf467-lcfrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Jul 21 15:34:18.362362 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:34:18.362264 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-719155ca-gw-openshift-default-8578dbf467-lcfrg" podUID="0cc65bf1-45a7-447e-aeba-a6ab9b43992f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Jul 21 15:34:18.607439 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:34:18.607393 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-719155ca-gw-openshift-default-8578dbf467-lcfrg" Jul 21 15:34:19.361587 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:34:19.361546 2563 patch_prober.go:28] interesting pod/e2e-migrate-719155ca-gw-openshift-default-8578dbf467-lcfrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Jul 21 15:34:19.362057 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:34:19.361608 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-719155ca-gw-openshift-default-8578dbf467-lcfrg" podUID="0cc65bf1-45a7-447e-aeba-a6ab9b43992f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Jul 21 15:34:20.361576 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:34:20.361541 2563 patch_prober.go:28] interesting pod/e2e-migrate-719155ca-gw-openshift-default-8578dbf467-lcfrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Jul 21 15:34:20.361765 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:34:20.361598 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-719155ca-gw-openshift-default-8578dbf467-lcfrg" podUID="0cc65bf1-45a7-447e-aeba-a6ab9b43992f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Jul 21 15:34:21.361768 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:34:21.361725 2563 patch_prober.go:28] interesting pod/e2e-migrate-719155ca-gw-openshift-default-8578dbf467-lcfrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Jul 21 15:34:21.362255 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:34:21.361794 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-719155ca-gw-openshift-default-8578dbf467-lcfrg" podUID="0cc65bf1-45a7-447e-aeba-a6ab9b43992f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Jul 21 15:34:22.361709 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:34:22.361668 2563 patch_prober.go:28] interesting pod/e2e-migrate-719155ca-gw-openshift-default-8578dbf467-lcfrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Jul 21 15:34:22.361901 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:34:22.361752 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-719155ca-gw-openshift-default-8578dbf467-lcfrg" podUID="0cc65bf1-45a7-447e-aeba-a6ab9b43992f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Jul 21 15:34:23.365382 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:34:23.365348 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-719155ca-gw-openshift-default-8578dbf467-lcfrg" Jul 21 15:34:23.366236 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:34:23.366219 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-719155ca-gw-openshift-default-8578dbf467-lcfrg" Jul 21 15:34:28.549149 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:34:28.549124 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kn9fn_02f3d0c6-4116-4134-ba51-1c02c73bef05/ovn-acl-logging/0.log" Jul 21 15:34:28.560436 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:34:28.560412 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kn9fn_02f3d0c6-4116-4134-ba51-1c02c73bef05/ovn-acl-logging/0.log" Jul 21 15:34:40.848262 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:34:40.848227 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-j4c8p"] Jul 21 15:37:29.282152 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:37:29.282111 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-719155ca-gw-openshift-default-8578dbf467-lcfrg"] Jul 21 15:37:29.282761 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:37:29.282392 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-719155ca-gw-openshift-default-8578dbf467-lcfrg" podUID="0cc65bf1-45a7-447e-aeba-a6ab9b43992f" containerName="istio-proxy" containerID="cri-o://0bd6538d3fae3eeba42597eaddf3e5754e902dcc209ae28def7b33d30da84698" gracePeriod=30 Jul 21 15:37:30.108165 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:37:30.108131 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-ed4302cf-openshift-default-7784d75b45-5zct2"] Jul 21 15:37:30.108564 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:37:30.108550 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96fd54f9-c100-467b-b848-dcf6fbb4a403" containerName="istio-proxy" Jul 21 15:37:30.108631 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:37:30.108568 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="96fd54f9-c100-467b-b848-dcf6fbb4a403" containerName="istio-proxy" Jul 21 15:37:30.108631 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:37:30.108587 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96fd54f9-c100-467b-b848-dcf6fbb4a403" containerName="istio-proxy" Jul 21 15:37:30.108631 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:37:30.108593 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="96fd54f9-c100-467b-b848-dcf6fbb4a403" containerName="istio-proxy" Jul 21 15:37:30.108730 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:37:30.108667 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="96fd54f9-c100-467b-b848-dcf6fbb4a403" containerName="istio-proxy" Jul 21 15:37:30.111601 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:37:30.111584 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ed4302cf-openshift-default-7784d75b45-5zct2" Jul 21 15:37:30.114373 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:37:30.114356 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-ed4302cf-openshift-default-dockercfg-f5pbd\"" Jul 21 15:37:30.123508 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:37:30.123487 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-ed4302cf-openshift-default-7784d75b45-5zct2"] Jul 21 15:37:30.248434 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:37:30.248401 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5c452e5-24ee-4782-8098-41d4922d0548-workload-socket\") pod \"e2e-ait-ed4302cf-openshift-default-7784d75b45-5zct2\" (UID: \"c5c452e5-24ee-4782-8098-41d4922d0548\") " pod="openshift-ingress/e2e-ait-ed4302cf-openshift-default-7784d75b45-5zct2" Jul 21 15:37:30.248591 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:37:30.248463 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5c452e5-24ee-4782-8098-41d4922d0548-istio-data\") pod \"e2e-ait-ed4302cf-openshift-default-7784d75b45-5zct2\" (UID: \"c5c452e5-24ee-4782-8098-41d4922d0548\") " pod="openshift-ingress/e2e-ait-ed4302cf-openshift-default-7784d75b45-5zct2" Jul 21 15:37:30.248591 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:37:30.248517 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5c452e5-24ee-4782-8098-41d4922d0548-istiod-ca-cert\") pod \"e2e-ait-ed4302cf-openshift-default-7784d75b45-5zct2\" (UID: \"c5c452e5-24ee-4782-8098-41d4922d0548\") " pod="openshift-ingress/e2e-ait-ed4302cf-openshift-default-7784d75b45-5zct2" Jul 21 15:37:30.248591 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:37:30.248555 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5c452e5-24ee-4782-8098-41d4922d0548-istio-envoy\") pod \"e2e-ait-ed4302cf-openshift-default-7784d75b45-5zct2\" (UID: \"c5c452e5-24ee-4782-8098-41d4922d0548\") " pod="openshift-ingress/e2e-ait-ed4302cf-openshift-default-7784d75b45-5zct2" Jul 21 15:37:30.248591 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:37:30.248584 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5c452e5-24ee-4782-8098-41d4922d0548-istio-token\") pod \"e2e-ait-ed4302cf-openshift-default-7784d75b45-5zct2\" (UID: \"c5c452e5-24ee-4782-8098-41d4922d0548\") " pod="openshift-ingress/e2e-ait-ed4302cf-openshift-default-7784d75b45-5zct2" Jul 21 15:37:30.248817 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:37:30.248605 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5c452e5-24ee-4782-8098-41d4922d0548-credential-socket\") pod \"e2e-ait-ed4302cf-openshift-default-7784d75b45-5zct2\" (UID: \"c5c452e5-24ee-4782-8098-41d4922d0548\") " pod="openshift-ingress/e2e-ait-ed4302cf-openshift-default-7784d75b45-5zct2" Jul 21 15:37:30.248817 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:37:30.248633 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5c452e5-24ee-4782-8098-41d4922d0548-workload-certs\") pod \"e2e-ait-ed4302cf-openshift-default-7784d75b45-5zct2\" (UID: \"c5c452e5-24ee-4782-8098-41d4922d0548\") " pod="openshift-ingress/e2e-ait-ed4302cf-openshift-default-7784d75b45-5zct2" Jul 21 15:37:30.248817 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:37:30.248652 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9zq9\" (UniqueName: \"kubernetes.io/projected/c5c452e5-24ee-4782-8098-41d4922d0548-kube-api-access-s9zq9\") pod \"e2e-ait-ed4302cf-openshift-default-7784d75b45-5zct2\" (UID: \"c5c452e5-24ee-4782-8098-41d4922d0548\") " pod="openshift-ingress/e2e-ait-ed4302cf-openshift-default-7784d75b45-5zct2" Jul 21 15:37:30.248817 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:37:30.248679 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5c452e5-24ee-4782-8098-41d4922d0548-istio-podinfo\") pod \"e2e-ait-ed4302cf-openshift-default-7784d75b45-5zct2\" (UID: \"c5c452e5-24ee-4782-8098-41d4922d0548\") " pod="openshift-ingress/e2e-ait-ed4302cf-openshift-default-7784d75b45-5zct2" Jul 21 15:37:30.349911 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:37:30.349880 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5c452e5-24ee-4782-8098-41d4922d0548-credential-socket\") pod \"e2e-ait-ed4302cf-openshift-default-7784d75b45-5zct2\" (UID: \"c5c452e5-24ee-4782-8098-41d4922d0548\") " pod="openshift-ingress/e2e-ait-ed4302cf-openshift-default-7784d75b45-5zct2" Jul 21 15:37:30.349911 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:37:30.349913 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5c452e5-24ee-4782-8098-41d4922d0548-workload-certs\") pod \"e2e-ait-ed4302cf-openshift-default-7784d75b45-5zct2\" (UID: \"c5c452e5-24ee-4782-8098-41d4922d0548\") " pod="openshift-ingress/e2e-ait-ed4302cf-openshift-default-7784d75b45-5zct2" Jul 21 15:37:30.350361 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:37:30.349930 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9zq9\" (UniqueName: \"kubernetes.io/projected/c5c452e5-24ee-4782-8098-41d4922d0548-kube-api-access-s9zq9\") pod \"e2e-ait-ed4302cf-openshift-default-7784d75b45-5zct2\" (UID: \"c5c452e5-24ee-4782-8098-41d4922d0548\") " pod="openshift-ingress/e2e-ait-ed4302cf-openshift-default-7784d75b45-5zct2" Jul 21 15:37:30.350361 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:37:30.350058 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5c452e5-24ee-4782-8098-41d4922d0548-istio-podinfo\") pod \"e2e-ait-ed4302cf-openshift-default-7784d75b45-5zct2\" (UID: \"c5c452e5-24ee-4782-8098-41d4922d0548\") " pod="openshift-ingress/e2e-ait-ed4302cf-openshift-default-7784d75b45-5zct2" Jul 21 15:37:30.350361 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:37:30.350142 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5c452e5-24ee-4782-8098-41d4922d0548-workload-socket\") pod \"e2e-ait-ed4302cf-openshift-default-7784d75b45-5zct2\" (UID: \"c5c452e5-24ee-4782-8098-41d4922d0548\") " pod="openshift-ingress/e2e-ait-ed4302cf-openshift-default-7784d75b45-5zct2" Jul 21 15:37:30.350361 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:37:30.350203 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5c452e5-24ee-4782-8098-41d4922d0548-istio-data\") pod \"e2e-ait-ed4302cf-openshift-default-7784d75b45-5zct2\" (UID: \"c5c452e5-24ee-4782-8098-41d4922d0548\") " pod="openshift-ingress/e2e-ait-ed4302cf-openshift-default-7784d75b45-5zct2" Jul 21 15:37:30.350361 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:37:30.350249 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5c452e5-24ee-4782-8098-41d4922d0548-istiod-ca-cert\") pod \"e2e-ait-ed4302cf-openshift-default-7784d75b45-5zct2\" (UID: \"c5c452e5-24ee-4782-8098-41d4922d0548\") " pod="openshift-ingress/e2e-ait-ed4302cf-openshift-default-7784d75b45-5zct2" Jul 21 15:37:30.350361 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:37:30.350292 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5c452e5-24ee-4782-8098-41d4922d0548-istio-envoy\") pod \"e2e-ait-ed4302cf-openshift-default-7784d75b45-5zct2\" (UID: \"c5c452e5-24ee-4782-8098-41d4922d0548\") " pod="openshift-ingress/e2e-ait-ed4302cf-openshift-default-7784d75b45-5zct2" Jul 21 15:37:30.350361 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:37:30.350308 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5c452e5-24ee-4782-8098-41d4922d0548-credential-socket\") pod \"e2e-ait-ed4302cf-openshift-default-7784d75b45-5zct2\" (UID: \"c5c452e5-24ee-4782-8098-41d4922d0548\") " pod="openshift-ingress/e2e-ait-ed4302cf-openshift-default-7784d75b45-5zct2" Jul 21 15:37:30.350715 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:37:30.350387 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5c452e5-24ee-4782-8098-41d4922d0548-istio-token\") pod \"e2e-ait-ed4302cf-openshift-default-7784d75b45-5zct2\" (UID: \"c5c452e5-24ee-4782-8098-41d4922d0548\") " pod="openshift-ingress/e2e-ait-ed4302cf-openshift-default-7784d75b45-5zct2" Jul 21 15:37:30.350715 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:37:30.350412 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5c452e5-24ee-4782-8098-41d4922d0548-workload-certs\") pod \"e2e-ait-ed4302cf-openshift-default-7784d75b45-5zct2\" (UID: \"c5c452e5-24ee-4782-8098-41d4922d0548\") " pod="openshift-ingress/e2e-ait-ed4302cf-openshift-default-7784d75b45-5zct2" Jul 21 15:37:30.350715 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:37:30.350534 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5c452e5-24ee-4782-8098-41d4922d0548-workload-socket\") pod \"e2e-ait-ed4302cf-openshift-default-7784d75b45-5zct2\" (UID: \"c5c452e5-24ee-4782-8098-41d4922d0548\") " pod="openshift-ingress/e2e-ait-ed4302cf-openshift-default-7784d75b45-5zct2" Jul 21 15:37:30.350715 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:37:30.350609 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5c452e5-24ee-4782-8098-41d4922d0548-istio-data\") pod \"e2e-ait-ed4302cf-openshift-default-7784d75b45-5zct2\" (UID: \"c5c452e5-24ee-4782-8098-41d4922d0548\") " pod="openshift-ingress/e2e-ait-ed4302cf-openshift-default-7784d75b45-5zct2" Jul 21 15:37:30.350968 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:37:30.350890 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5c452e5-24ee-4782-8098-41d4922d0548-istiod-ca-cert\") pod \"e2e-ait-ed4302cf-openshift-default-7784d75b45-5zct2\" (UID: \"c5c452e5-24ee-4782-8098-41d4922d0548\") " pod="openshift-ingress/e2e-ait-ed4302cf-openshift-default-7784d75b45-5zct2" Jul 21 15:37:30.352597 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:37:30.352577 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5c452e5-24ee-4782-8098-41d4922d0548-istio-envoy\") pod \"e2e-ait-ed4302cf-openshift-default-7784d75b45-5zct2\" (UID: \"c5c452e5-24ee-4782-8098-41d4922d0548\") " pod="openshift-ingress/e2e-ait-ed4302cf-openshift-default-7784d75b45-5zct2" Jul 21 15:37:30.352700 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:37:30.352683 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5c452e5-24ee-4782-8098-41d4922d0548-istio-podinfo\") pod \"e2e-ait-ed4302cf-openshift-default-7784d75b45-5zct2\" (UID: \"c5c452e5-24ee-4782-8098-41d4922d0548\") " pod="openshift-ingress/e2e-ait-ed4302cf-openshift-default-7784d75b45-5zct2" Jul 21 15:37:30.359432 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:37:30.359373 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5c452e5-24ee-4782-8098-41d4922d0548-istio-token\") pod \"e2e-ait-ed4302cf-openshift-default-7784d75b45-5zct2\" (UID: \"c5c452e5-24ee-4782-8098-41d4922d0548\") " pod="openshift-ingress/e2e-ait-ed4302cf-openshift-default-7784d75b45-5zct2" Jul 21 15:37:30.359661 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:37:30.359643 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9zq9\" (UniqueName: \"kubernetes.io/projected/c5c452e5-24ee-4782-8098-41d4922d0548-kube-api-access-s9zq9\") pod \"e2e-ait-ed4302cf-openshift-default-7784d75b45-5zct2\" (UID: \"c5c452e5-24ee-4782-8098-41d4922d0548\") " pod="openshift-ingress/e2e-ait-ed4302cf-openshift-default-7784d75b45-5zct2" Jul 21 15:37:30.421757 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:37:30.421737 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ed4302cf-openshift-default-7784d75b45-5zct2" Jul 21 15:37:30.567762 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:37:30.567739 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-ed4302cf-openshift-default-7784d75b45-5zct2"] Jul 21 15:37:30.569970 ip-10-0-135-176 kubenswrapper[2563]: W0721 15:37:30.569940 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5c452e5_24ee_4782_8098_41d4922d0548.slice/crio-a24b6fe27243f560de4419ff2513bc543ed5e5fb1b2c81c09d05c5e4cc8536a1 WatchSource:0}: Error finding container a24b6fe27243f560de4419ff2513bc543ed5e5fb1b2c81c09d05c5e4cc8536a1: Status 404 returned error can't find the container with id a24b6fe27243f560de4419ff2513bc543ed5e5fb1b2c81c09d05c5e4cc8536a1 Jul 21 15:37:30.572129 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:37:30.572091 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 21 15:37:30.572225 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:37:30.572159 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 21 15:37:30.572225 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:37:30.572193 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 21 15:37:30.776696 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:37:30.776655 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ed4302cf-openshift-default-7784d75b45-5zct2" event={"ID":"c5c452e5-24ee-4782-8098-41d4922d0548","Type":"ContainerStarted","Data":"baced6578f936d2562bed8185edf1654f44f04f13aafc965c9d94d222e4516af"} Jul 21 15:37:30.776886 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:37:30.776720 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ed4302cf-openshift-default-7784d75b45-5zct2" event={"ID":"c5c452e5-24ee-4782-8098-41d4922d0548","Type":"ContainerStarted","Data":"a24b6fe27243f560de4419ff2513bc543ed5e5fb1b2c81c09d05c5e4cc8536a1"} Jul 21 15:37:30.799374 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:37:30.799300 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-ed4302cf-openshift-default-7784d75b45-5zct2" podStartSLOduration=0.799281646 podStartE2EDuration="799.281646ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 15:37:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 15:37:30.797138829 +0000 UTC m=+2282.792924266" watchObservedRunningTime="2026-07-21 15:37:30.799281646 +0000 UTC m=+2282.795067064" Jul 21 15:37:31.422255 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:37:31.422221 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-ed4302cf-openshift-default-7784d75b45-5zct2" Jul 21 15:37:31.427374 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:37:31.427349 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-ed4302cf-openshift-default-7784d75b45-5zct2" Jul 21 15:37:31.782852 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:37:31.782760 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-ed4302cf-openshift-default-7784d75b45-5zct2" Jul 21 15:37:31.783947 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:37:31.783924 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-ed4302cf-openshift-default-7784d75b45-5zct2" Jul 21 15:37:34.542539 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:37:34.542476 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-719155ca-gw-openshift-default-8578dbf467-lcfrg" Jul 21 15:37:34.689731 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:37:34.689695 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cc65bf1-45a7-447e-aeba-a6ab9b43992f-istio-token\") pod \"0cc65bf1-45a7-447e-aeba-a6ab9b43992f\" (UID: \"0cc65bf1-45a7-447e-aeba-a6ab9b43992f\") " Jul 21 15:37:34.689731 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:37:34.689734 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cc65bf1-45a7-447e-aeba-a6ab9b43992f-istio-data\") pod \"0cc65bf1-45a7-447e-aeba-a6ab9b43992f\" (UID: \"0cc65bf1-45a7-447e-aeba-a6ab9b43992f\") " Jul 21 15:37:34.689984 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:37:34.689750 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cc65bf1-45a7-447e-aeba-a6ab9b43992f-istiod-ca-cert\") pod \"0cc65bf1-45a7-447e-aeba-a6ab9b43992f\" (UID: \"0cc65bf1-45a7-447e-aeba-a6ab9b43992f\") " Jul 21 15:37:34.689984 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:37:34.689772 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cc65bf1-45a7-447e-aeba-a6ab9b43992f-credential-socket\") pod \"0cc65bf1-45a7-447e-aeba-a6ab9b43992f\" (UID: \"0cc65bf1-45a7-447e-aeba-a6ab9b43992f\") " Jul 21 15:37:34.689984 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:37:34.689797 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cc65bf1-45a7-447e-aeba-a6ab9b43992f-workload-certs\") pod \"0cc65bf1-45a7-447e-aeba-a6ab9b43992f\" (UID: \"0cc65bf1-45a7-447e-aeba-a6ab9b43992f\") " Jul 21 15:37:34.689984 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:37:34.689820 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gmkq4\" (UniqueName: \"kubernetes.io/projected/0cc65bf1-45a7-447e-aeba-a6ab9b43992f-kube-api-access-gmkq4\") pod \"0cc65bf1-45a7-447e-aeba-a6ab9b43992f\" (UID: \"0cc65bf1-45a7-447e-aeba-a6ab9b43992f\") " Jul 21 15:37:34.689984 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:37:34.689850 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cc65bf1-45a7-447e-aeba-a6ab9b43992f-istio-podinfo\") pod \"0cc65bf1-45a7-447e-aeba-a6ab9b43992f\" (UID: \"0cc65bf1-45a7-447e-aeba-a6ab9b43992f\") " Jul 21 15:37:34.689984 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:37:34.689874 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cc65bf1-45a7-447e-aeba-a6ab9b43992f-istio-envoy\") pod \"0cc65bf1-45a7-447e-aeba-a6ab9b43992f\" (UID: \"0cc65bf1-45a7-447e-aeba-a6ab9b43992f\") " Jul 21 15:37:34.689984 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:37:34.689949 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cc65bf1-45a7-447e-aeba-a6ab9b43992f-workload-socket\") pod \"0cc65bf1-45a7-447e-aeba-a6ab9b43992f\" (UID: \"0cc65bf1-45a7-447e-aeba-a6ab9b43992f\") " Jul 21 15:37:34.690374 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:37:34.690031 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cc65bf1-45a7-447e-aeba-a6ab9b43992f-istio-data" (OuterVolumeSpecName: "istio-data") pod "0cc65bf1-45a7-447e-aeba-a6ab9b43992f" (UID: "0cc65bf1-45a7-447e-aeba-a6ab9b43992f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 15:37:34.690374 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:37:34.690087 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cc65bf1-45a7-447e-aeba-a6ab9b43992f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0cc65bf1-45a7-447e-aeba-a6ab9b43992f" (UID: "0cc65bf1-45a7-447e-aeba-a6ab9b43992f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 15:37:34.690374 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:37:34.690275 2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cc65bf1-45a7-447e-aeba-a6ab9b43992f-istio-data\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 15:37:34.690374 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:37:34.690280 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cc65bf1-45a7-447e-aeba-a6ab9b43992f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0cc65bf1-45a7-447e-aeba-a6ab9b43992f" (UID: "0cc65bf1-45a7-447e-aeba-a6ab9b43992f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 15:37:34.690374 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:37:34.690295 2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cc65bf1-45a7-447e-aeba-a6ab9b43992f-workload-certs\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 15:37:34.690374 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:37:34.690301 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cc65bf1-45a7-447e-aeba-a6ab9b43992f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0cc65bf1-45a7-447e-aeba-a6ab9b43992f" (UID: "0cc65bf1-45a7-447e-aeba-a6ab9b43992f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 15:37:34.690647 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:37:34.690461 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cc65bf1-45a7-447e-aeba-a6ab9b43992f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0cc65bf1-45a7-447e-aeba-a6ab9b43992f" (UID: "0cc65bf1-45a7-447e-aeba-a6ab9b43992f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 15:37:34.692724 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:37:34.692699 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0cc65bf1-45a7-447e-aeba-a6ab9b43992f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0cc65bf1-45a7-447e-aeba-a6ab9b43992f" (UID: "0cc65bf1-45a7-447e-aeba-a6ab9b43992f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 21 15:37:34.692724 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:37:34.692714 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cc65bf1-45a7-447e-aeba-a6ab9b43992f-kube-api-access-gmkq4" (OuterVolumeSpecName: "kube-api-access-gmkq4") pod "0cc65bf1-45a7-447e-aeba-a6ab9b43992f" (UID: "0cc65bf1-45a7-447e-aeba-a6ab9b43992f"). InnerVolumeSpecName "kube-api-access-gmkq4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 15:37:34.692854 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:37:34.692813 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cc65bf1-45a7-447e-aeba-a6ab9b43992f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0cc65bf1-45a7-447e-aeba-a6ab9b43992f" (UID: "0cc65bf1-45a7-447e-aeba-a6ab9b43992f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 15:37:34.692854 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:37:34.692819 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cc65bf1-45a7-447e-aeba-a6ab9b43992f-istio-token" (OuterVolumeSpecName: "istio-token") pod "0cc65bf1-45a7-447e-aeba-a6ab9b43992f" (UID: "0cc65bf1-45a7-447e-aeba-a6ab9b43992f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 15:37:34.791631 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:37:34.791599 2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cc65bf1-45a7-447e-aeba-a6ab9b43992f-istio-token\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 15:37:34.791631 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:37:34.791623 2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cc65bf1-45a7-447e-aeba-a6ab9b43992f-istiod-ca-cert\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 15:37:34.791631 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:37:34.791632 2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cc65bf1-45a7-447e-aeba-a6ab9b43992f-credential-socket\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 15:37:34.791631 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:37:34.791642 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmkq4\" (UniqueName: \"kubernetes.io/projected/0cc65bf1-45a7-447e-aeba-a6ab9b43992f-kube-api-access-gmkq4\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 15:37:34.792035 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:37:34.791651 2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cc65bf1-45a7-447e-aeba-a6ab9b43992f-istio-podinfo\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 15:37:34.792035 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:37:34.791659 2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cc65bf1-45a7-447e-aeba-a6ab9b43992f-istio-envoy\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 15:37:34.792035 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:37:34.791667 2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cc65bf1-45a7-447e-aeba-a6ab9b43992f-workload-socket\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 15:37:34.799903 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:37:34.799830 2563 generic.go:356] "Generic (PLEG): container finished" podID="0cc65bf1-45a7-447e-aeba-a6ab9b43992f" containerID="0bd6538d3fae3eeba42597eaddf3e5754e902dcc209ae28def7b33d30da84698" exitCode=0 Jul 21 15:37:34.799903 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:37:34.799884 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-719155ca-gw-openshift-default-8578dbf467-lcfrg" Jul 21 15:37:34.800070 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:37:34.799910 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-719155ca-gw-openshift-default-8578dbf467-lcfrg" event={"ID":"0cc65bf1-45a7-447e-aeba-a6ab9b43992f","Type":"ContainerDied","Data":"0bd6538d3fae3eeba42597eaddf3e5754e902dcc209ae28def7b33d30da84698"} Jul 21 15:37:34.800070 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:37:34.799962 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-719155ca-gw-openshift-default-8578dbf467-lcfrg" event={"ID":"0cc65bf1-45a7-447e-aeba-a6ab9b43992f","Type":"ContainerDied","Data":"e733784a5e4da7c6a2c733d44197f53545d01ce8ba77b033d05ca0095251449c"} Jul 21 15:37:34.800070 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:37:34.799982 2563 scope.go:117] "RemoveContainer" containerID="0bd6538d3fae3eeba42597eaddf3e5754e902dcc209ae28def7b33d30da84698" Jul 21 15:37:34.810513 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:37:34.810497 2563 scope.go:117] "RemoveContainer" containerID="0efe42190699b2a26d862ab36e46ef0a42059194f73a7b9bb212296b3f4efa8d" Jul 21 15:37:34.818907 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:37:34.818890 2563 scope.go:117] "RemoveContainer" containerID="0bd6538d3fae3eeba42597eaddf3e5754e902dcc209ae28def7b33d30da84698" Jul 21 15:37:34.819137 ip-10-0-135-176 kubenswrapper[2563]: E0721 15:37:34.819121 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0bd6538d3fae3eeba42597eaddf3e5754e902dcc209ae28def7b33d30da84698\": container with ID starting with 0bd6538d3fae3eeba42597eaddf3e5754e902dcc209ae28def7b33d30da84698 not found: ID does not exist" containerID="0bd6538d3fae3eeba42597eaddf3e5754e902dcc209ae28def7b33d30da84698" Jul 21 15:37:34.819181 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:37:34.819144 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bd6538d3fae3eeba42597eaddf3e5754e902dcc209ae28def7b33d30da84698"} err="failed to get container status \"0bd6538d3fae3eeba42597eaddf3e5754e902dcc209ae28def7b33d30da84698\": rpc error: code = NotFound desc = could not find container \"0bd6538d3fae3eeba42597eaddf3e5754e902dcc209ae28def7b33d30da84698\": container with ID starting with 0bd6538d3fae3eeba42597eaddf3e5754e902dcc209ae28def7b33d30da84698 not found: ID does not exist" Jul 21 15:37:34.819181 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:37:34.819162 2563 scope.go:117] "RemoveContainer" containerID="0efe42190699b2a26d862ab36e46ef0a42059194f73a7b9bb212296b3f4efa8d" Jul 21 15:37:34.819358 ip-10-0-135-176 kubenswrapper[2563]: E0721 15:37:34.819343 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0efe42190699b2a26d862ab36e46ef0a42059194f73a7b9bb212296b3f4efa8d\": container with ID starting with 0efe42190699b2a26d862ab36e46ef0a42059194f73a7b9bb212296b3f4efa8d not found: ID does not exist" containerID="0efe42190699b2a26d862ab36e46ef0a42059194f73a7b9bb212296b3f4efa8d" Jul 21 15:37:34.819404 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:37:34.819361 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0efe42190699b2a26d862ab36e46ef0a42059194f73a7b9bb212296b3f4efa8d"} err="failed to get container status \"0efe42190699b2a26d862ab36e46ef0a42059194f73a7b9bb212296b3f4efa8d\": rpc error: code = NotFound desc = could not find container \"0efe42190699b2a26d862ab36e46ef0a42059194f73a7b9bb212296b3f4efa8d\": container with ID starting with 0efe42190699b2a26d862ab36e46ef0a42059194f73a7b9bb212296b3f4efa8d not found: ID does not exist" Jul 21 15:37:34.823304 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:37:34.823281 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-719155ca-gw-openshift-default-8578dbf467-lcfrg"] Jul 21 15:37:34.834526 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:37:34.834496 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-719155ca-gw-openshift-default-8578dbf467-lcfrg"] Jul 21 15:37:36.135783 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:37:36.135752 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-j4c8p"] Jul 21 15:37:36.481210 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:37:36.481119 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0cc65bf1-45a7-447e-aeba-a6ab9b43992f" path="/var/lib/kubelet/pods/0cc65bf1-45a7-447e-aeba-a6ab9b43992f/volumes" Jul 21 15:37:45.862245 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:37:45.862213 2563 generic.go:356] "Generic (PLEG): container finished" podID="c5c452e5-24ee-4782-8098-41d4922d0548" containerID="baced6578f936d2562bed8185edf1654f44f04f13aafc965c9d94d222e4516af" exitCode=0 Jul 21 15:37:45.862658 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:37:45.862272 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ed4302cf-openshift-default-7784d75b45-5zct2" event={"ID":"c5c452e5-24ee-4782-8098-41d4922d0548","Type":"ContainerDied","Data":"baced6578f936d2562bed8185edf1654f44f04f13aafc965c9d94d222e4516af"} Jul 21 15:37:45.862701 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:37:45.862673 2563 scope.go:117] "RemoveContainer" containerID="baced6578f936d2562bed8185edf1654f44f04f13aafc965c9d94d222e4516af" Jul 21 15:37:45.863632 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:37:45.863603 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 21 15:37:45.863699 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:37:45.863672 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 21 15:37:45.863752 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:37:45.863702 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 21 15:37:46.782927 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:37:46.782894 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-ed4302cf-openshift-default-7784d75b45-5zct2" Jul 21 15:37:46.869457 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:37:46.869419 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ed4302cf-openshift-default-7784d75b45-5zct2" event={"ID":"c5c452e5-24ee-4782-8098-41d4922d0548","Type":"ContainerStarted","Data":"d922367fd36c0e8f722ed274e65f40d0ec08ba743edfd6c77a0db56272cde6af"} Jul 21 15:37:47.422118 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:37:47.422080 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-ed4302cf-openshift-default-7784d75b45-5zct2" Jul 21 15:37:47.423144 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:37:47.423117 2563 patch_prober.go:28] interesting pod/e2e-ait-ed4302cf-openshift-default-7784d75b45-5zct2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Jul 21 15:37:47.423247 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:37:47.423167 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ed4302cf-openshift-default-7784d75b45-5zct2" podUID="c5c452e5-24ee-4782-8098-41d4922d0548" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Jul 21 15:37:48.422911 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:37:48.422879 2563 patch_prober.go:28] interesting pod/e2e-ait-ed4302cf-openshift-default-7784d75b45-5zct2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Jul 21 15:37:48.423309 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:37:48.422933 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ed4302cf-openshift-default-7784d75b45-5zct2" podUID="c5c452e5-24ee-4782-8098-41d4922d0548" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Jul 21 15:37:49.422656 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:37:49.422620 2563 patch_prober.go:28] interesting pod/e2e-ait-ed4302cf-openshift-default-7784d75b45-5zct2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Jul 21 15:37:49.422838 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:37:49.422678 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ed4302cf-openshift-default-7784d75b45-5zct2" podUID="c5c452e5-24ee-4782-8098-41d4922d0548" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Jul 21 15:37:50.423023 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:37:50.422984 2563 patch_prober.go:28] interesting pod/e2e-ait-ed4302cf-openshift-default-7784d75b45-5zct2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Jul 21 15:37:50.423456 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:37:50.423071 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ed4302cf-openshift-default-7784d75b45-5zct2" podUID="c5c452e5-24ee-4782-8098-41d4922d0548" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Jul 21 15:37:51.422297 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:37:51.422263 2563 patch_prober.go:28] interesting pod/e2e-ait-ed4302cf-openshift-default-7784d75b45-5zct2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Jul 21 15:37:51.422490 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:37:51.422317 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ed4302cf-openshift-default-7784d75b45-5zct2" podUID="c5c452e5-24ee-4782-8098-41d4922d0548" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Jul 21 15:37:52.426006 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:37:52.425978 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-ed4302cf-openshift-default-7784d75b45-5zct2" Jul 21 15:37:52.426416 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:37:52.426271 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-ed4302cf-openshift-default-7784d75b45-5zct2" Jul 21 15:37:52.426969 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:37:52.426950 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-ed4302cf-openshift-default-7784d75b45-5zct2" Jul 21 15:38:10.640875 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:38:10.640830 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-j4c8p"] Jul 21 15:39:28.583172 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:39:28.583143 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kn9fn_02f3d0c6-4116-4134-ba51-1c02c73bef05/ovn-acl-logging/0.log" Jul 21 15:39:28.594970 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:39:28.594946 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kn9fn_02f3d0c6-4116-4134-ba51-1c02c73bef05/ovn-acl-logging/0.log" Jul 21 15:43:58.284310 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:43:58.284269 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-ed4302cf-openshift-default-7784d75b45-5zct2"] Jul 21 15:43:58.286988 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:43:58.284557 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-ed4302cf-openshift-default-7784d75b45-5zct2" podUID="c5c452e5-24ee-4782-8098-41d4922d0548" containerName="istio-proxy" containerID="cri-o://d922367fd36c0e8f722ed274e65f40d0ec08ba743edfd6c77a0db56272cde6af" gracePeriod=30 Jul 21 15:43:59.134169 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:43:59.134131 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-5195cc17-openshift-default-7b878b664c-88mgd"] Jul 21 15:43:59.134563 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:43:59.134549 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0cc65bf1-45a7-447e-aeba-a6ab9b43992f" containerName="istio-proxy" Jul 21 15:43:59.134563 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:43:59.134563 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0cc65bf1-45a7-447e-aeba-a6ab9b43992f" containerName="istio-proxy" Jul 21 15:43:59.134717 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:43:59.134586 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0cc65bf1-45a7-447e-aeba-a6ab9b43992f" containerName="istio-proxy" Jul 21 15:43:59.134717 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:43:59.134592 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0cc65bf1-45a7-447e-aeba-a6ab9b43992f" containerName="istio-proxy" Jul 21 15:43:59.134717 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:43:59.134661 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="96fd54f9-c100-467b-b848-dcf6fbb4a403" containerName="istio-proxy" Jul 21 15:43:59.134717 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:43:59.134672 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="0cc65bf1-45a7-447e-aeba-a6ab9b43992f" containerName="istio-proxy" Jul 21 15:43:59.134717 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:43:59.134678 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="0cc65bf1-45a7-447e-aeba-a6ab9b43992f" containerName="istio-proxy" Jul 21 15:43:59.137780 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:43:59.137760 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-5195cc17-openshift-default-7b878b664c-88mgd" Jul 21 15:43:59.140479 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:43:59.140456 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-5195cc17-openshift-default-dockercfg-t4lfc\"" Jul 21 15:43:59.150359 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:43:59.150312 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-5195cc17-openshift-default-7b878b664c-88mgd"] Jul 21 15:43:59.199165 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:43:59.199125 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08a6053f-5451-4537-98e8-e6016d5af8cd-workload-socket\") pod \"e2e-derive-5195cc17-openshift-default-7b878b664c-88mgd\" (UID: \"08a6053f-5451-4537-98e8-e6016d5af8cd\") " pod="openshift-ingress/e2e-derive-5195cc17-openshift-default-7b878b664c-88mgd" Jul 21 15:43:59.199165 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:43:59.199175 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08a6053f-5451-4537-98e8-e6016d5af8cd-istio-data\") pod \"e2e-derive-5195cc17-openshift-default-7b878b664c-88mgd\" (UID: \"08a6053f-5451-4537-98e8-e6016d5af8cd\") " pod="openshift-ingress/e2e-derive-5195cc17-openshift-default-7b878b664c-88mgd" Jul 21 15:43:59.199468 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:43:59.199204 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08a6053f-5451-4537-98e8-e6016d5af8cd-credential-socket\") pod \"e2e-derive-5195cc17-openshift-default-7b878b664c-88mgd\" (UID: \"08a6053f-5451-4537-98e8-e6016d5af8cd\") " pod="openshift-ingress/e2e-derive-5195cc17-openshift-default-7b878b664c-88mgd" Jul 21 15:43:59.199468 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:43:59.199229 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08a6053f-5451-4537-98e8-e6016d5af8cd-istio-podinfo\") pod \"e2e-derive-5195cc17-openshift-default-7b878b664c-88mgd\" (UID: \"08a6053f-5451-4537-98e8-e6016d5af8cd\") " pod="openshift-ingress/e2e-derive-5195cc17-openshift-default-7b878b664c-88mgd" Jul 21 15:43:59.199468 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:43:59.199353 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08a6053f-5451-4537-98e8-e6016d5af8cd-workload-certs\") pod \"e2e-derive-5195cc17-openshift-default-7b878b664c-88mgd\" (UID: \"08a6053f-5451-4537-98e8-e6016d5af8cd\") " pod="openshift-ingress/e2e-derive-5195cc17-openshift-default-7b878b664c-88mgd" Jul 21 15:43:59.199468 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:43:59.199385 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08a6053f-5451-4537-98e8-e6016d5af8cd-istio-token\") pod \"e2e-derive-5195cc17-openshift-default-7b878b664c-88mgd\" (UID: \"08a6053f-5451-4537-98e8-e6016d5af8cd\") " pod="openshift-ingress/e2e-derive-5195cc17-openshift-default-7b878b664c-88mgd" Jul 21 15:43:59.199468 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:43:59.199401 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08a6053f-5451-4537-98e8-e6016d5af8cd-istio-envoy\") pod \"e2e-derive-5195cc17-openshift-default-7b878b664c-88mgd\" (UID: \"08a6053f-5451-4537-98e8-e6016d5af8cd\") " pod="openshift-ingress/e2e-derive-5195cc17-openshift-default-7b878b664c-88mgd" Jul 21 15:43:59.199671 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:43:59.199485 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlj8s\" (UniqueName: \"kubernetes.io/projected/08a6053f-5451-4537-98e8-e6016d5af8cd-kube-api-access-jlj8s\") pod \"e2e-derive-5195cc17-openshift-default-7b878b664c-88mgd\" (UID: \"08a6053f-5451-4537-98e8-e6016d5af8cd\") " pod="openshift-ingress/e2e-derive-5195cc17-openshift-default-7b878b664c-88mgd" Jul 21 15:43:59.199671 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:43:59.199516 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08a6053f-5451-4537-98e8-e6016d5af8cd-istiod-ca-cert\") pod \"e2e-derive-5195cc17-openshift-default-7b878b664c-88mgd\" (UID: \"08a6053f-5451-4537-98e8-e6016d5af8cd\") " pod="openshift-ingress/e2e-derive-5195cc17-openshift-default-7b878b664c-88mgd" Jul 21 15:43:59.300155 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:43:59.300103 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08a6053f-5451-4537-98e8-e6016d5af8cd-workload-socket\") pod \"e2e-derive-5195cc17-openshift-default-7b878b664c-88mgd\" (UID: \"08a6053f-5451-4537-98e8-e6016d5af8cd\") " pod="openshift-ingress/e2e-derive-5195cc17-openshift-default-7b878b664c-88mgd" Jul 21 15:43:59.300155 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:43:59.300155 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08a6053f-5451-4537-98e8-e6016d5af8cd-istio-data\") pod \"e2e-derive-5195cc17-openshift-default-7b878b664c-88mgd\" (UID: \"08a6053f-5451-4537-98e8-e6016d5af8cd\") " pod="openshift-ingress/e2e-derive-5195cc17-openshift-default-7b878b664c-88mgd" Jul 21 15:43:59.300750 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:43:59.300280 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08a6053f-5451-4537-98e8-e6016d5af8cd-credential-socket\") pod \"e2e-derive-5195cc17-openshift-default-7b878b664c-88mgd\" (UID: \"08a6053f-5451-4537-98e8-e6016d5af8cd\") " pod="openshift-ingress/e2e-derive-5195cc17-openshift-default-7b878b664c-88mgd" Jul 21 15:43:59.300750 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:43:59.300352 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08a6053f-5451-4537-98e8-e6016d5af8cd-istio-podinfo\") pod \"e2e-derive-5195cc17-openshift-default-7b878b664c-88mgd\" (UID: \"08a6053f-5451-4537-98e8-e6016d5af8cd\") " pod="openshift-ingress/e2e-derive-5195cc17-openshift-default-7b878b664c-88mgd" Jul 21 15:43:59.300750 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:43:59.300438 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08a6053f-5451-4537-98e8-e6016d5af8cd-workload-certs\") pod \"e2e-derive-5195cc17-openshift-default-7b878b664c-88mgd\" (UID: \"08a6053f-5451-4537-98e8-e6016d5af8cd\") " pod="openshift-ingress/e2e-derive-5195cc17-openshift-default-7b878b664c-88mgd" Jul 21 15:43:59.300750 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:43:59.300465 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08a6053f-5451-4537-98e8-e6016d5af8cd-istio-token\") pod \"e2e-derive-5195cc17-openshift-default-7b878b664c-88mgd\" (UID: \"08a6053f-5451-4537-98e8-e6016d5af8cd\") " pod="openshift-ingress/e2e-derive-5195cc17-openshift-default-7b878b664c-88mgd" Jul 21 15:43:59.300750 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:43:59.300492 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08a6053f-5451-4537-98e8-e6016d5af8cd-istio-envoy\") pod \"e2e-derive-5195cc17-openshift-default-7b878b664c-88mgd\" (UID: \"08a6053f-5451-4537-98e8-e6016d5af8cd\") " pod="openshift-ingress/e2e-derive-5195cc17-openshift-default-7b878b664c-88mgd" Jul 21 15:43:59.300750 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:43:59.300573 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlj8s\" (UniqueName: \"kubernetes.io/projected/08a6053f-5451-4537-98e8-e6016d5af8cd-kube-api-access-jlj8s\") pod \"e2e-derive-5195cc17-openshift-default-7b878b664c-88mgd\" (UID: \"08a6053f-5451-4537-98e8-e6016d5af8cd\") " pod="openshift-ingress/e2e-derive-5195cc17-openshift-default-7b878b664c-88mgd" Jul 21 15:43:59.300750 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:43:59.300579 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08a6053f-5451-4537-98e8-e6016d5af8cd-workload-socket\") pod \"e2e-derive-5195cc17-openshift-default-7b878b664c-88mgd\" (UID: \"08a6053f-5451-4537-98e8-e6016d5af8cd\") " pod="openshift-ingress/e2e-derive-5195cc17-openshift-default-7b878b664c-88mgd" Jul 21 15:43:59.300750 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:43:59.300605 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08a6053f-5451-4537-98e8-e6016d5af8cd-istiod-ca-cert\") pod \"e2e-derive-5195cc17-openshift-default-7b878b664c-88mgd\" (UID: \"08a6053f-5451-4537-98e8-e6016d5af8cd\") " pod="openshift-ingress/e2e-derive-5195cc17-openshift-default-7b878b664c-88mgd" Jul 21 15:43:59.300750 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:43:59.300672 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08a6053f-5451-4537-98e8-e6016d5af8cd-istio-data\") pod \"e2e-derive-5195cc17-openshift-default-7b878b664c-88mgd\" (UID: \"08a6053f-5451-4537-98e8-e6016d5af8cd\") " pod="openshift-ingress/e2e-derive-5195cc17-openshift-default-7b878b664c-88mgd" Jul 21 15:43:59.301238 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:43:59.300817 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08a6053f-5451-4537-98e8-e6016d5af8cd-workload-certs\") pod \"e2e-derive-5195cc17-openshift-default-7b878b664c-88mgd\" (UID: \"08a6053f-5451-4537-98e8-e6016d5af8cd\") " pod="openshift-ingress/e2e-derive-5195cc17-openshift-default-7b878b664c-88mgd" Jul 21 15:43:59.301238 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:43:59.300856 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08a6053f-5451-4537-98e8-e6016d5af8cd-credential-socket\") pod \"e2e-derive-5195cc17-openshift-default-7b878b664c-88mgd\" (UID: \"08a6053f-5451-4537-98e8-e6016d5af8cd\") " pod="openshift-ingress/e2e-derive-5195cc17-openshift-default-7b878b664c-88mgd" Jul 21 15:43:59.301483 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:43:59.301463 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08a6053f-5451-4537-98e8-e6016d5af8cd-istiod-ca-cert\") pod \"e2e-derive-5195cc17-openshift-default-7b878b664c-88mgd\" (UID: \"08a6053f-5451-4537-98e8-e6016d5af8cd\") " pod="openshift-ingress/e2e-derive-5195cc17-openshift-default-7b878b664c-88mgd" Jul 21 15:43:59.302848 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:43:59.302831 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08a6053f-5451-4537-98e8-e6016d5af8cd-istio-envoy\") pod \"e2e-derive-5195cc17-openshift-default-7b878b664c-88mgd\" (UID: \"08a6053f-5451-4537-98e8-e6016d5af8cd\") " pod="openshift-ingress/e2e-derive-5195cc17-openshift-default-7b878b664c-88mgd" Jul 21 15:43:59.303130 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:43:59.303111 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08a6053f-5451-4537-98e8-e6016d5af8cd-istio-podinfo\") pod \"e2e-derive-5195cc17-openshift-default-7b878b664c-88mgd\" (UID: \"08a6053f-5451-4537-98e8-e6016d5af8cd\") " pod="openshift-ingress/e2e-derive-5195cc17-openshift-default-7b878b664c-88mgd" Jul 21 15:43:59.308614 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:43:59.308581 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08a6053f-5451-4537-98e8-e6016d5af8cd-istio-token\") pod \"e2e-derive-5195cc17-openshift-default-7b878b664c-88mgd\" (UID: \"08a6053f-5451-4537-98e8-e6016d5af8cd\") " pod="openshift-ingress/e2e-derive-5195cc17-openshift-default-7b878b664c-88mgd" Jul 21 15:43:59.308744 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:43:59.308709 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlj8s\" (UniqueName: \"kubernetes.io/projected/08a6053f-5451-4537-98e8-e6016d5af8cd-kube-api-access-jlj8s\") pod \"e2e-derive-5195cc17-openshift-default-7b878b664c-88mgd\" (UID: \"08a6053f-5451-4537-98e8-e6016d5af8cd\") " pod="openshift-ingress/e2e-derive-5195cc17-openshift-default-7b878b664c-88mgd" Jul 21 15:43:59.449312 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:43:59.449219 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-5195cc17-openshift-default-7b878b664c-88mgd" Jul 21 15:43:59.594820 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:43:59.594795 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-5195cc17-openshift-default-7b878b664c-88mgd"] Jul 21 15:43:59.596458 ip-10-0-135-176 kubenswrapper[2563]: W0721 15:43:59.596427 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08a6053f_5451_4537_98e8_e6016d5af8cd.slice/crio-da367e4dd2864e0f34ffc48103fbd18c7fcd1976d1142d57fc76eb9614cc6bd8 WatchSource:0}: Error finding container da367e4dd2864e0f34ffc48103fbd18c7fcd1976d1142d57fc76eb9614cc6bd8: Status 404 returned error can't find the container with id da367e4dd2864e0f34ffc48103fbd18c7fcd1976d1142d57fc76eb9614cc6bd8 Jul 21 15:43:59.599091 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:43:59.599058 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 21 15:43:59.599185 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:43:59.599125 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 21 15:43:59.599185 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:43:59.599155 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 21 15:43:59.928676 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:43:59.928634 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-5195cc17-openshift-default-7b878b664c-88mgd" event={"ID":"08a6053f-5451-4537-98e8-e6016d5af8cd","Type":"ContainerStarted","Data":"1a513436a28643a52adc9b221d4c4aaf93a0b2a7843841a2e8096bbca6219e3f"} Jul 21 15:43:59.928871 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:43:59.928693 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-5195cc17-openshift-default-7b878b664c-88mgd" event={"ID":"08a6053f-5451-4537-98e8-e6016d5af8cd","Type":"ContainerStarted","Data":"da367e4dd2864e0f34ffc48103fbd18c7fcd1976d1142d57fc76eb9614cc6bd8"} Jul 21 15:43:59.956696 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:43:59.950705 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-5195cc17-openshift-default-7b878b664c-88mgd" podStartSLOduration=0.950673926 podStartE2EDuration="950.673926ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 15:43:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 15:43:59.948616989 +0000 UTC m=+2671.944402408" watchObservedRunningTime="2026-07-21 15:43:59.950673926 +0000 UTC m=+2671.946459345" Jul 21 15:44:00.449438 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:44:00.449402 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-5195cc17-openshift-default-7b878b664c-88mgd" Jul 21 15:44:00.454753 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:44:00.454724 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-5195cc17-openshift-default-7b878b664c-88mgd" Jul 21 15:44:00.935865 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:44:00.935798 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-5195cc17-openshift-default-7b878b664c-88mgd" Jul 21 15:44:00.936992 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:44:00.936973 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-5195cc17-openshift-default-7b878b664c-88mgd" Jul 21 15:44:03.549178 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:44:03.549105 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ed4302cf-openshift-default-7784d75b45-5zct2" Jul 21 15:44:03.641485 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:44:03.641450 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5c452e5-24ee-4782-8098-41d4922d0548-istio-token\") pod \"c5c452e5-24ee-4782-8098-41d4922d0548\" (UID: \"c5c452e5-24ee-4782-8098-41d4922d0548\") " Jul 21 15:44:03.641485 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:44:03.641488 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5c452e5-24ee-4782-8098-41d4922d0548-istio-podinfo\") pod \"c5c452e5-24ee-4782-8098-41d4922d0548\" (UID: \"c5c452e5-24ee-4782-8098-41d4922d0548\") " Jul 21 15:44:03.641756 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:44:03.641510 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5c452e5-24ee-4782-8098-41d4922d0548-workload-certs\") pod \"c5c452e5-24ee-4782-8098-41d4922d0548\" (UID: \"c5c452e5-24ee-4782-8098-41d4922d0548\") " Jul 21 15:44:03.641756 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:44:03.641538 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5c452e5-24ee-4782-8098-41d4922d0548-credential-socket\") pod \"c5c452e5-24ee-4782-8098-41d4922d0548\" (UID: \"c5c452e5-24ee-4782-8098-41d4922d0548\") " Jul 21 15:44:03.641756 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:44:03.641663 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s9zq9\" (UniqueName: \"kubernetes.io/projected/c5c452e5-24ee-4782-8098-41d4922d0548-kube-api-access-s9zq9\") pod \"c5c452e5-24ee-4782-8098-41d4922d0548\" (UID: \"c5c452e5-24ee-4782-8098-41d4922d0548\") " Jul 21 15:44:03.641756 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:44:03.641720 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5c452e5-24ee-4782-8098-41d4922d0548-istiod-ca-cert\") pod \"c5c452e5-24ee-4782-8098-41d4922d0548\" (UID: \"c5c452e5-24ee-4782-8098-41d4922d0548\") " Jul 21 15:44:03.641756 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:44:03.641753 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5c452e5-24ee-4782-8098-41d4922d0548-istio-data\") pod \"c5c452e5-24ee-4782-8098-41d4922d0548\" (UID: \"c5c452e5-24ee-4782-8098-41d4922d0548\") " Jul 21 15:44:03.642006 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:44:03.641777 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5c452e5-24ee-4782-8098-41d4922d0548-istio-envoy\") pod \"c5c452e5-24ee-4782-8098-41d4922d0548\" (UID: \"c5c452e5-24ee-4782-8098-41d4922d0548\") " Jul 21 15:44:03.642006 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:44:03.641811 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5c452e5-24ee-4782-8098-41d4922d0548-workload-socket\") pod \"c5c452e5-24ee-4782-8098-41d4922d0548\" (UID: \"c5c452e5-24ee-4782-8098-41d4922d0548\") " Jul 21 15:44:03.642006 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:44:03.641837 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5c452e5-24ee-4782-8098-41d4922d0548-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c5c452e5-24ee-4782-8098-41d4922d0548" (UID: "c5c452e5-24ee-4782-8098-41d4922d0548"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 15:44:03.642006 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:44:03.641848 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5c452e5-24ee-4782-8098-41d4922d0548-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c5c452e5-24ee-4782-8098-41d4922d0548" (UID: "c5c452e5-24ee-4782-8098-41d4922d0548"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 15:44:03.642188 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:44:03.642077 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5c452e5-24ee-4782-8098-41d4922d0548-istio-data" (OuterVolumeSpecName: "istio-data") pod "c5c452e5-24ee-4782-8098-41d4922d0548" (UID: "c5c452e5-24ee-4782-8098-41d4922d0548"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 15:44:03.642257 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:44:03.642245 2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5c452e5-24ee-4782-8098-41d4922d0548-credential-socket\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 15:44:03.642316 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:44:03.642265 2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5c452e5-24ee-4782-8098-41d4922d0548-istio-data\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 15:44:03.642316 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:44:03.642274 2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5c452e5-24ee-4782-8098-41d4922d0548-workload-certs\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 15:44:03.642456 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:44:03.642383 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5c452e5-24ee-4782-8098-41d4922d0548-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c5c452e5-24ee-4782-8098-41d4922d0548" (UID: "c5c452e5-24ee-4782-8098-41d4922d0548"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 15:44:03.642504 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:44:03.642460 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5c452e5-24ee-4782-8098-41d4922d0548-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c5c452e5-24ee-4782-8098-41d4922d0548" (UID: "c5c452e5-24ee-4782-8098-41d4922d0548"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 15:44:03.643911 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:44:03.643890 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5c452e5-24ee-4782-8098-41d4922d0548-istio-token" (OuterVolumeSpecName: "istio-token") pod "c5c452e5-24ee-4782-8098-41d4922d0548" (UID: "c5c452e5-24ee-4782-8098-41d4922d0548"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 15:44:03.644036 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:44:03.644021 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5c452e5-24ee-4782-8098-41d4922d0548-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c5c452e5-24ee-4782-8098-41d4922d0548" (UID: "c5c452e5-24ee-4782-8098-41d4922d0548"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 15:44:03.644387 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:44:03.644368 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c5c452e5-24ee-4782-8098-41d4922d0548-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c5c452e5-24ee-4782-8098-41d4922d0548" (UID: "c5c452e5-24ee-4782-8098-41d4922d0548"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 21 15:44:03.644456 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:44:03.644417 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5c452e5-24ee-4782-8098-41d4922d0548-kube-api-access-s9zq9" (OuterVolumeSpecName: "kube-api-access-s9zq9") pod "c5c452e5-24ee-4782-8098-41d4922d0548" (UID: "c5c452e5-24ee-4782-8098-41d4922d0548"). InnerVolumeSpecName "kube-api-access-s9zq9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 15:44:03.743137 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:44:03.743095 2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5c452e5-24ee-4782-8098-41d4922d0548-istiod-ca-cert\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 15:44:03.743137 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:44:03.743129 2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5c452e5-24ee-4782-8098-41d4922d0548-istio-envoy\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 15:44:03.743137 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:44:03.743139 2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5c452e5-24ee-4782-8098-41d4922d0548-workload-socket\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 15:44:03.743431 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:44:03.743159 2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5c452e5-24ee-4782-8098-41d4922d0548-istio-token\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 15:44:03.743431 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:44:03.743167 2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5c452e5-24ee-4782-8098-41d4922d0548-istio-podinfo\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 15:44:03.743431 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:44:03.743176 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9zq9\" (UniqueName: \"kubernetes.io/projected/c5c452e5-24ee-4782-8098-41d4922d0548-kube-api-access-s9zq9\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 15:44:03.954426 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:44:03.954393 2563 generic.go:356] "Generic (PLEG): container finished" podID="c5c452e5-24ee-4782-8098-41d4922d0548" containerID="d922367fd36c0e8f722ed274e65f40d0ec08ba743edfd6c77a0db56272cde6af" exitCode=0 Jul 21 15:44:03.954602 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:44:03.954449 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ed4302cf-openshift-default-7784d75b45-5zct2" Jul 21 15:44:03.954602 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:44:03.954454 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ed4302cf-openshift-default-7784d75b45-5zct2" event={"ID":"c5c452e5-24ee-4782-8098-41d4922d0548","Type":"ContainerDied","Data":"d922367fd36c0e8f722ed274e65f40d0ec08ba743edfd6c77a0db56272cde6af"} Jul 21 15:44:03.954602 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:44:03.954568 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ed4302cf-openshift-default-7784d75b45-5zct2" event={"ID":"c5c452e5-24ee-4782-8098-41d4922d0548","Type":"ContainerDied","Data":"a24b6fe27243f560de4419ff2513bc543ed5e5fb1b2c81c09d05c5e4cc8536a1"} Jul 21 15:44:03.954602 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:44:03.954588 2563 scope.go:117] "RemoveContainer" containerID="d922367fd36c0e8f722ed274e65f40d0ec08ba743edfd6c77a0db56272cde6af" Jul 21 15:44:03.965223 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:44:03.965205 2563 scope.go:117] "RemoveContainer" containerID="baced6578f936d2562bed8185edf1654f44f04f13aafc965c9d94d222e4516af" Jul 21 15:44:03.974719 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:44:03.974697 2563 scope.go:117] "RemoveContainer" containerID="d922367fd36c0e8f722ed274e65f40d0ec08ba743edfd6c77a0db56272cde6af" Jul 21 15:44:03.975072 ip-10-0-135-176 kubenswrapper[2563]: E0721 15:44:03.975052 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d922367fd36c0e8f722ed274e65f40d0ec08ba743edfd6c77a0db56272cde6af\": container with ID starting with d922367fd36c0e8f722ed274e65f40d0ec08ba743edfd6c77a0db56272cde6af not found: ID does not exist" containerID="d922367fd36c0e8f722ed274e65f40d0ec08ba743edfd6c77a0db56272cde6af" Jul 21 15:44:03.975112 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:44:03.975083 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d922367fd36c0e8f722ed274e65f40d0ec08ba743edfd6c77a0db56272cde6af"} err="failed to get container status \"d922367fd36c0e8f722ed274e65f40d0ec08ba743edfd6c77a0db56272cde6af\": rpc error: code = NotFound desc = could not find container \"d922367fd36c0e8f722ed274e65f40d0ec08ba743edfd6c77a0db56272cde6af\": container with ID starting with d922367fd36c0e8f722ed274e65f40d0ec08ba743edfd6c77a0db56272cde6af not found: ID does not exist" Jul 21 15:44:03.975112 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:44:03.975104 2563 scope.go:117] "RemoveContainer" containerID="baced6578f936d2562bed8185edf1654f44f04f13aafc965c9d94d222e4516af" Jul 21 15:44:03.975413 ip-10-0-135-176 kubenswrapper[2563]: E0721 15:44:03.975394 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"baced6578f936d2562bed8185edf1654f44f04f13aafc965c9d94d222e4516af\": container with ID starting with baced6578f936d2562bed8185edf1654f44f04f13aafc965c9d94d222e4516af not found: ID does not exist" containerID="baced6578f936d2562bed8185edf1654f44f04f13aafc965c9d94d222e4516af" Jul 21 15:44:03.975465 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:44:03.975419 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"baced6578f936d2562bed8185edf1654f44f04f13aafc965c9d94d222e4516af"} err="failed to get container status \"baced6578f936d2562bed8185edf1654f44f04f13aafc965c9d94d222e4516af\": rpc error: code = NotFound desc = could not find container \"baced6578f936d2562bed8185edf1654f44f04f13aafc965c9d94d222e4516af\": container with ID starting with baced6578f936d2562bed8185edf1654f44f04f13aafc965c9d94d222e4516af not found: ID does not exist" Jul 21 15:44:03.979164 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:44:03.979118 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-ed4302cf-openshift-default-7784d75b45-5zct2"] Jul 21 15:44:03.982508 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:44:03.982480 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-ed4302cf-openshift-default-7784d75b45-5zct2"] Jul 21 15:44:04.481257 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:44:04.481215 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c5c452e5-24ee-4782-8098-41d4922d0548" path="/var/lib/kubelet/pods/c5c452e5-24ee-4782-8098-41d4922d0548/volumes" Jul 21 15:44:04.742883 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:44:04.742795 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-j4c8p"] Jul 21 15:44:15.029964 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:44:15.029926 2563 generic.go:356] "Generic (PLEG): container finished" podID="08a6053f-5451-4537-98e8-e6016d5af8cd" containerID="1a513436a28643a52adc9b221d4c4aaf93a0b2a7843841a2e8096bbca6219e3f" exitCode=0 Jul 21 15:44:15.030543 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:44:15.030006 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-5195cc17-openshift-default-7b878b664c-88mgd" event={"ID":"08a6053f-5451-4537-98e8-e6016d5af8cd","Type":"ContainerDied","Data":"1a513436a28643a52adc9b221d4c4aaf93a0b2a7843841a2e8096bbca6219e3f"} Jul 21 15:44:15.030648 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:44:15.030559 2563 scope.go:117] "RemoveContainer" containerID="1a513436a28643a52adc9b221d4c4aaf93a0b2a7843841a2e8096bbca6219e3f" Jul 21 15:44:15.031672 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:44:15.031629 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 21 15:44:15.031798 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:44:15.031716 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 21 15:44:15.031798 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:44:15.031759 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 21 15:44:15.936243 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:44:15.936211 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-5195cc17-openshift-default-7b878b664c-88mgd" Jul 21 15:44:16.037895 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:44:16.037855 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-5195cc17-openshift-default-7b878b664c-88mgd" event={"ID":"08a6053f-5451-4537-98e8-e6016d5af8cd","Type":"ContainerStarted","Data":"e9fb9172fbf4009d19686c16ff7fad6f2dc7728023f3cc76edb4a17d9e2e203c"} Jul 21 15:44:16.449883 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:44:16.449847 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-5195cc17-openshift-default-7b878b664c-88mgd" Jul 21 15:44:16.451211 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:44:16.451184 2563 patch_prober.go:28] interesting pod/e2e-derive-5195cc17-openshift-default-7b878b664c-88mgd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Jul 21 15:44:16.451360 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:44:16.451231 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-5195cc17-openshift-default-7b878b664c-88mgd" podUID="08a6053f-5451-4537-98e8-e6016d5af8cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Jul 21 15:44:17.450600 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:44:17.450555 2563 patch_prober.go:28] interesting pod/e2e-derive-5195cc17-openshift-default-7b878b664c-88mgd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Jul 21 15:44:17.451079 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:44:17.450636 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-5195cc17-openshift-default-7b878b664c-88mgd" podUID="08a6053f-5451-4537-98e8-e6016d5af8cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Jul 21 15:44:18.449985 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:44:18.449949 2563 patch_prober.go:28] interesting pod/e2e-derive-5195cc17-openshift-default-7b878b664c-88mgd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Jul 21 15:44:18.450164 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:44:18.450019 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-5195cc17-openshift-default-7b878b664c-88mgd" podUID="08a6053f-5451-4537-98e8-e6016d5af8cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Jul 21 15:44:19.450280 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:44:19.450194 2563 patch_prober.go:28] interesting pod/e2e-derive-5195cc17-openshift-default-7b878b664c-88mgd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Jul 21 15:44:19.450280 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:44:19.450261 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-5195cc17-openshift-default-7b878b664c-88mgd" podUID="08a6053f-5451-4537-98e8-e6016d5af8cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Jul 21 15:44:20.450340 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:44:20.450281 2563 patch_prober.go:28] interesting pod/e2e-derive-5195cc17-openshift-default-7b878b664c-88mgd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Jul 21 15:44:20.450737 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:44:20.450359 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-5195cc17-openshift-default-7b878b664c-88mgd" podUID="08a6053f-5451-4537-98e8-e6016d5af8cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Jul 21 15:44:21.453970 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:44:21.453937 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-5195cc17-openshift-default-7b878b664c-88mgd" Jul 21 15:44:21.454444 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:44:21.454361 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-5195cc17-openshift-default-7b878b664c-88mgd" Jul 21 15:44:21.455206 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:44:21.455186 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-5195cc17-openshift-default-7b878b664c-88mgd" Jul 21 15:44:28.619443 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:44:28.619416 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kn9fn_02f3d0c6-4116-4134-ba51-1c02c73bef05/ovn-acl-logging/0.log" Jul 21 15:44:28.630925 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:44:28.630904 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kn9fn_02f3d0c6-4116-4134-ba51-1c02c73bef05/ovn-acl-logging/0.log" Jul 21 15:44:38.837336 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:44:38.837279 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-j4c8p"] Jul 21 15:47:26.376685 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:47:26.376598 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-5195cc17-openshift-default-7b878b664c-88mgd"] Jul 21 15:47:26.377228 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:47:26.376862 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-5195cc17-openshift-default-7b878b664c-88mgd" podUID="08a6053f-5451-4537-98e8-e6016d5af8cd" containerName="istio-proxy" containerID="cri-o://e9fb9172fbf4009d19686c16ff7fad6f2dc7728023f3cc76edb4a17d9e2e203c" gracePeriod=30 Jul 21 15:47:31.638731 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:47:31.638705 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-5195cc17-openshift-default-7b878b664c-88mgd" Jul 21 15:47:31.753371 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:47:31.753244 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08a6053f-5451-4537-98e8-e6016d5af8cd-istio-token\") pod \"08a6053f-5451-4537-98e8-e6016d5af8cd\" (UID: \"08a6053f-5451-4537-98e8-e6016d5af8cd\") " Jul 21 15:47:31.753371 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:47:31.753288 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08a6053f-5451-4537-98e8-e6016d5af8cd-istiod-ca-cert\") pod \"08a6053f-5451-4537-98e8-e6016d5af8cd\" (UID: \"08a6053f-5451-4537-98e8-e6016d5af8cd\") " Jul 21 15:47:31.753371 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:47:31.753346 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jlj8s\" (UniqueName: \"kubernetes.io/projected/08a6053f-5451-4537-98e8-e6016d5af8cd-kube-api-access-jlj8s\") pod \"08a6053f-5451-4537-98e8-e6016d5af8cd\" (UID: \"08a6053f-5451-4537-98e8-e6016d5af8cd\") " Jul 21 15:47:31.753651 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:47:31.753425 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08a6053f-5451-4537-98e8-e6016d5af8cd-workload-socket\") pod \"08a6053f-5451-4537-98e8-e6016d5af8cd\" (UID: \"08a6053f-5451-4537-98e8-e6016d5af8cd\") " Jul 21 15:47:31.753651 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:47:31.753443 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08a6053f-5451-4537-98e8-e6016d5af8cd-istio-podinfo\") pod \"08a6053f-5451-4537-98e8-e6016d5af8cd\" (UID: \"08a6053f-5451-4537-98e8-e6016d5af8cd\") " Jul 21 15:47:31.753651 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:47:31.753461 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08a6053f-5451-4537-98e8-e6016d5af8cd-istio-data\") pod \"08a6053f-5451-4537-98e8-e6016d5af8cd\" (UID: \"08a6053f-5451-4537-98e8-e6016d5af8cd\") " Jul 21 15:47:31.753651 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:47:31.753489 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08a6053f-5451-4537-98e8-e6016d5af8cd-workload-certs\") pod \"08a6053f-5451-4537-98e8-e6016d5af8cd\" (UID: \"08a6053f-5451-4537-98e8-e6016d5af8cd\") " Jul 21 15:47:31.753651 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:47:31.753524 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08a6053f-5451-4537-98e8-e6016d5af8cd-credential-socket\") pod \"08a6053f-5451-4537-98e8-e6016d5af8cd\" (UID: \"08a6053f-5451-4537-98e8-e6016d5af8cd\") " Jul 21 15:47:31.753651 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:47:31.753557 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08a6053f-5451-4537-98e8-e6016d5af8cd-istio-envoy\") pod \"08a6053f-5451-4537-98e8-e6016d5af8cd\" (UID: \"08a6053f-5451-4537-98e8-e6016d5af8cd\") " Jul 21 15:47:31.753961 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:47:31.753754 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08a6053f-5451-4537-98e8-e6016d5af8cd-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "08a6053f-5451-4537-98e8-e6016d5af8cd" (UID: "08a6053f-5451-4537-98e8-e6016d5af8cd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 15:47:31.753961 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:47:31.753786 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08a6053f-5451-4537-98e8-e6016d5af8cd-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "08a6053f-5451-4537-98e8-e6016d5af8cd" (UID: "08a6053f-5451-4537-98e8-e6016d5af8cd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 15:47:31.753961 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:47:31.753886 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08a6053f-5451-4537-98e8-e6016d5af8cd-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "08a6053f-5451-4537-98e8-e6016d5af8cd" (UID: "08a6053f-5451-4537-98e8-e6016d5af8cd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 15:47:31.754292 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:47:31.754005 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08a6053f-5451-4537-98e8-e6016d5af8cd-istio-data" (OuterVolumeSpecName: "istio-data") pod "08a6053f-5451-4537-98e8-e6016d5af8cd" (UID: "08a6053f-5451-4537-98e8-e6016d5af8cd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 15:47:31.754292 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:47:31.753997 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08a6053f-5451-4537-98e8-e6016d5af8cd-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "08a6053f-5451-4537-98e8-e6016d5af8cd" (UID: "08a6053f-5451-4537-98e8-e6016d5af8cd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 15:47:31.754292 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:47:31.754021 2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08a6053f-5451-4537-98e8-e6016d5af8cd-workload-socket\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 15:47:31.754292 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:47:31.754044 2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08a6053f-5451-4537-98e8-e6016d5af8cd-workload-certs\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 15:47:31.754292 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:47:31.754057 2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08a6053f-5451-4537-98e8-e6016d5af8cd-istiod-ca-cert\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 15:47:31.755842 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:47:31.755812 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08a6053f-5451-4537-98e8-e6016d5af8cd-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "08a6053f-5451-4537-98e8-e6016d5af8cd" (UID: "08a6053f-5451-4537-98e8-e6016d5af8cd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 15:47:31.755962 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:47:31.755898 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08a6053f-5451-4537-98e8-e6016d5af8cd-kube-api-access-jlj8s" (OuterVolumeSpecName: "kube-api-access-jlj8s") pod "08a6053f-5451-4537-98e8-e6016d5af8cd" (UID: "08a6053f-5451-4537-98e8-e6016d5af8cd"). InnerVolumeSpecName "kube-api-access-jlj8s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 15:47:31.756021 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:47:31.755963 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08a6053f-5451-4537-98e8-e6016d5af8cd-istio-token" (OuterVolumeSpecName: "istio-token") pod "08a6053f-5451-4537-98e8-e6016d5af8cd" (UID: "08a6053f-5451-4537-98e8-e6016d5af8cd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 15:47:31.756021 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:47:31.755957 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/08a6053f-5451-4537-98e8-e6016d5af8cd-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "08a6053f-5451-4537-98e8-e6016d5af8cd" (UID: "08a6053f-5451-4537-98e8-e6016d5af8cd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 21 15:47:31.854638 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:47:31.854594 2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08a6053f-5451-4537-98e8-e6016d5af8cd-istio-podinfo\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 15:47:31.854638 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:47:31.854630 2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08a6053f-5451-4537-98e8-e6016d5af8cd-istio-data\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 15:47:31.854638 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:47:31.854640 2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08a6053f-5451-4537-98e8-e6016d5af8cd-credential-socket\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 15:47:31.854638 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:47:31.854649 2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08a6053f-5451-4537-98e8-e6016d5af8cd-istio-envoy\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 15:47:31.854921 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:47:31.854658 2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08a6053f-5451-4537-98e8-e6016d5af8cd-istio-token\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 15:47:31.854921 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:47:31.854666 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jlj8s\" (UniqueName: \"kubernetes.io/projected/08a6053f-5451-4537-98e8-e6016d5af8cd-kube-api-access-jlj8s\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 15:47:32.140676 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:47:32.140637 2563 generic.go:356] "Generic (PLEG): container finished" podID="08a6053f-5451-4537-98e8-e6016d5af8cd" containerID="e9fb9172fbf4009d19686c16ff7fad6f2dc7728023f3cc76edb4a17d9e2e203c" exitCode=0 Jul 21 15:47:32.140866 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:47:32.140694 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-5195cc17-openshift-default-7b878b664c-88mgd" Jul 21 15:47:32.140866 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:47:32.140724 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-5195cc17-openshift-default-7b878b664c-88mgd" event={"ID":"08a6053f-5451-4537-98e8-e6016d5af8cd","Type":"ContainerDied","Data":"e9fb9172fbf4009d19686c16ff7fad6f2dc7728023f3cc76edb4a17d9e2e203c"} Jul 21 15:47:32.140866 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:47:32.140785 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-5195cc17-openshift-default-7b878b664c-88mgd" event={"ID":"08a6053f-5451-4537-98e8-e6016d5af8cd","Type":"ContainerDied","Data":"da367e4dd2864e0f34ffc48103fbd18c7fcd1976d1142d57fc76eb9614cc6bd8"} Jul 21 15:47:32.140866 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:47:32.140810 2563 scope.go:117] "RemoveContainer" containerID="e9fb9172fbf4009d19686c16ff7fad6f2dc7728023f3cc76edb4a17d9e2e203c" Jul 21 15:47:32.150986 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:47:32.150966 2563 scope.go:117] "RemoveContainer" containerID="1a513436a28643a52adc9b221d4c4aaf93a0b2a7843841a2e8096bbca6219e3f" Jul 21 15:47:32.159854 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:47:32.159827 2563 scope.go:117] "RemoveContainer" containerID="e9fb9172fbf4009d19686c16ff7fad6f2dc7728023f3cc76edb4a17d9e2e203c" Jul 21 15:47:32.160146 ip-10-0-135-176 kubenswrapper[2563]: E0721 15:47:32.160125 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9fb9172fbf4009d19686c16ff7fad6f2dc7728023f3cc76edb4a17d9e2e203c\": container with ID starting with e9fb9172fbf4009d19686c16ff7fad6f2dc7728023f3cc76edb4a17d9e2e203c not found: ID does not exist" containerID="e9fb9172fbf4009d19686c16ff7fad6f2dc7728023f3cc76edb4a17d9e2e203c" Jul 21 15:47:32.160202 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:47:32.160157 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9fb9172fbf4009d19686c16ff7fad6f2dc7728023f3cc76edb4a17d9e2e203c"} err="failed to get container status \"e9fb9172fbf4009d19686c16ff7fad6f2dc7728023f3cc76edb4a17d9e2e203c\": rpc error: code = NotFound desc = could not find container \"e9fb9172fbf4009d19686c16ff7fad6f2dc7728023f3cc76edb4a17d9e2e203c\": container with ID starting with e9fb9172fbf4009d19686c16ff7fad6f2dc7728023f3cc76edb4a17d9e2e203c not found: ID does not exist" Jul 21 15:47:32.160202 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:47:32.160178 2563 scope.go:117] "RemoveContainer" containerID="1a513436a28643a52adc9b221d4c4aaf93a0b2a7843841a2e8096bbca6219e3f" Jul 21 15:47:32.160485 ip-10-0-135-176 kubenswrapper[2563]: E0721 15:47:32.160465 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a513436a28643a52adc9b221d4c4aaf93a0b2a7843841a2e8096bbca6219e3f\": container with ID starting with 1a513436a28643a52adc9b221d4c4aaf93a0b2a7843841a2e8096bbca6219e3f not found: ID does not exist" containerID="1a513436a28643a52adc9b221d4c4aaf93a0b2a7843841a2e8096bbca6219e3f" Jul 21 15:47:32.160543 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:47:32.160491 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a513436a28643a52adc9b221d4c4aaf93a0b2a7843841a2e8096bbca6219e3f"} err="failed to get container status \"1a513436a28643a52adc9b221d4c4aaf93a0b2a7843841a2e8096bbca6219e3f\": rpc error: code = NotFound desc = could not find container \"1a513436a28643a52adc9b221d4c4aaf93a0b2a7843841a2e8096bbca6219e3f\": container with ID starting with 1a513436a28643a52adc9b221d4c4aaf93a0b2a7843841a2e8096bbca6219e3f not found: ID does not exist" Jul 21 15:47:32.170735 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:47:32.170706 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-5195cc17-openshift-default-7b878b664c-88mgd"] Jul 21 15:47:32.177565 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:47:32.177534 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-5195cc17-openshift-default-7b878b664c-88mgd"] Jul 21 15:47:32.480787 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:47:32.480690 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="08a6053f-5451-4537-98e8-e6016d5af8cd" path="/var/lib/kubelet/pods/08a6053f-5451-4537-98e8-e6016d5af8cd/volumes" Jul 21 15:48:48.741818 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:48:48.741784 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-j4c8p"] Jul 21 15:48:54.646506 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:48:54.646470 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-j4c8p"] Jul 21 15:49:01.141526 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:49:01.141483 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-j4c8p"] Jul 21 15:49:02.755146 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:49:02.755114 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-bfbfdffb-openshift-default-5f485cf6dc-h5lws"] Jul 21 15:49:02.755539 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:49:02.755503 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08a6053f-5451-4537-98e8-e6016d5af8cd" containerName="istio-proxy" Jul 21 15:49:02.755539 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:49:02.755514 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="08a6053f-5451-4537-98e8-e6016d5af8cd" containerName="istio-proxy" Jul 21 15:49:02.755539 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:49:02.755524 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5c452e5-24ee-4782-8098-41d4922d0548" containerName="istio-proxy" Jul 21 15:49:02.755539 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:49:02.755529 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5c452e5-24ee-4782-8098-41d4922d0548" containerName="istio-proxy" Jul 21 15:49:02.755668 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:49:02.755543 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5c452e5-24ee-4782-8098-41d4922d0548" containerName="istio-proxy" Jul 21 15:49:02.755668 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:49:02.755550 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5c452e5-24ee-4782-8098-41d4922d0548" containerName="istio-proxy" Jul 21 15:49:02.755668 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:49:02.755561 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08a6053f-5451-4537-98e8-e6016d5af8cd" containerName="istio-proxy" Jul 21 15:49:02.755668 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:49:02.755566 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="08a6053f-5451-4537-98e8-e6016d5af8cd" containerName="istio-proxy" Jul 21 15:49:02.755668 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:49:02.755636 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="08a6053f-5451-4537-98e8-e6016d5af8cd" containerName="istio-proxy" Jul 21 15:49:02.755668 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:49:02.755645 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5c452e5-24ee-4782-8098-41d4922d0548" containerName="istio-proxy" Jul 21 15:49:02.755668 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:49:02.755653 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5c452e5-24ee-4782-8098-41d4922d0548" containerName="istio-proxy" Jul 21 15:49:02.755668 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:49:02.755659 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="08a6053f-5451-4537-98e8-e6016d5af8cd" containerName="istio-proxy" Jul 21 15:49:02.758938 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:49:02.758920 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-bfbfdffb-openshift-default-5f485cf6dc-h5lws" Jul 21 15:49:02.762410 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:49:02.762391 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-bfbfdffb-openshift-default-dockercfg-z7j97\"" Jul 21 15:49:02.784604 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:49:02.784576 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-bfbfdffb-openshift-default-5f485cf6dc-h5lws"] Jul 21 15:49:02.834913 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:49:02.834883 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bec5993-0ec8-47c2-87da-b0d5f27bd306-workload-certs\") pod \"e2e-mt-bfbfdffb-openshift-default-5f485cf6dc-h5lws\" (UID: \"8bec5993-0ec8-47c2-87da-b0d5f27bd306\") " pod="openshift-ingress/e2e-mt-bfbfdffb-openshift-default-5f485cf6dc-h5lws" Jul 21 15:49:02.835076 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:49:02.834916 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqgwg\" (UniqueName: \"kubernetes.io/projected/8bec5993-0ec8-47c2-87da-b0d5f27bd306-kube-api-access-sqgwg\") pod \"e2e-mt-bfbfdffb-openshift-default-5f485cf6dc-h5lws\" (UID: \"8bec5993-0ec8-47c2-87da-b0d5f27bd306\") " pod="openshift-ingress/e2e-mt-bfbfdffb-openshift-default-5f485cf6dc-h5lws" Jul 21 15:49:02.835076 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:49:02.834943 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bec5993-0ec8-47c2-87da-b0d5f27bd306-istio-envoy\") pod \"e2e-mt-bfbfdffb-openshift-default-5f485cf6dc-h5lws\" (UID: \"8bec5993-0ec8-47c2-87da-b0d5f27bd306\") " pod="openshift-ingress/e2e-mt-bfbfdffb-openshift-default-5f485cf6dc-h5lws" Jul 21 15:49:02.835076 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:49:02.834966 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bec5993-0ec8-47c2-87da-b0d5f27bd306-istiod-ca-cert\") pod \"e2e-mt-bfbfdffb-openshift-default-5f485cf6dc-h5lws\" (UID: \"8bec5993-0ec8-47c2-87da-b0d5f27bd306\") " pod="openshift-ingress/e2e-mt-bfbfdffb-openshift-default-5f485cf6dc-h5lws" Jul 21 15:49:02.835076 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:49:02.834998 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bec5993-0ec8-47c2-87da-b0d5f27bd306-istio-data\") pod \"e2e-mt-bfbfdffb-openshift-default-5f485cf6dc-h5lws\" (UID: \"8bec5993-0ec8-47c2-87da-b0d5f27bd306\") " pod="openshift-ingress/e2e-mt-bfbfdffb-openshift-default-5f485cf6dc-h5lws" Jul 21 15:49:02.835076 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:49:02.835026 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bec5993-0ec8-47c2-87da-b0d5f27bd306-credential-socket\") pod \"e2e-mt-bfbfdffb-openshift-default-5f485cf6dc-h5lws\" (UID: \"8bec5993-0ec8-47c2-87da-b0d5f27bd306\") " pod="openshift-ingress/e2e-mt-bfbfdffb-openshift-default-5f485cf6dc-h5lws" Jul 21 15:49:02.835076 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:49:02.835062 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bec5993-0ec8-47c2-87da-b0d5f27bd306-istio-podinfo\") pod \"e2e-mt-bfbfdffb-openshift-default-5f485cf6dc-h5lws\" (UID: \"8bec5993-0ec8-47c2-87da-b0d5f27bd306\") " pod="openshift-ingress/e2e-mt-bfbfdffb-openshift-default-5f485cf6dc-h5lws" Jul 21 15:49:02.835343 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:49:02.835142 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bec5993-0ec8-47c2-87da-b0d5f27bd306-workload-socket\") pod \"e2e-mt-bfbfdffb-openshift-default-5f485cf6dc-h5lws\" (UID: \"8bec5993-0ec8-47c2-87da-b0d5f27bd306\") " pod="openshift-ingress/e2e-mt-bfbfdffb-openshift-default-5f485cf6dc-h5lws" Jul 21 15:49:02.835343 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:49:02.835173 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bec5993-0ec8-47c2-87da-b0d5f27bd306-istio-token\") pod \"e2e-mt-bfbfdffb-openshift-default-5f485cf6dc-h5lws\" (UID: \"8bec5993-0ec8-47c2-87da-b0d5f27bd306\") " pod="openshift-ingress/e2e-mt-bfbfdffb-openshift-default-5f485cf6dc-h5lws" Jul 21 15:49:02.936290 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:49:02.936260 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bec5993-0ec8-47c2-87da-b0d5f27bd306-workload-certs\") pod \"e2e-mt-bfbfdffb-openshift-default-5f485cf6dc-h5lws\" (UID: \"8bec5993-0ec8-47c2-87da-b0d5f27bd306\") " pod="openshift-ingress/e2e-mt-bfbfdffb-openshift-default-5f485cf6dc-h5lws" Jul 21 15:49:02.936290 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:49:02.936293 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqgwg\" (UniqueName: \"kubernetes.io/projected/8bec5993-0ec8-47c2-87da-b0d5f27bd306-kube-api-access-sqgwg\") pod \"e2e-mt-bfbfdffb-openshift-default-5f485cf6dc-h5lws\" (UID: \"8bec5993-0ec8-47c2-87da-b0d5f27bd306\") " pod="openshift-ingress/e2e-mt-bfbfdffb-openshift-default-5f485cf6dc-h5lws" Jul 21 15:49:02.936590 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:49:02.936314 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bec5993-0ec8-47c2-87da-b0d5f27bd306-istio-envoy\") pod \"e2e-mt-bfbfdffb-openshift-default-5f485cf6dc-h5lws\" (UID: \"8bec5993-0ec8-47c2-87da-b0d5f27bd306\") " pod="openshift-ingress/e2e-mt-bfbfdffb-openshift-default-5f485cf6dc-h5lws" Jul 21 15:49:02.936590 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:49:02.936351 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bec5993-0ec8-47c2-87da-b0d5f27bd306-istiod-ca-cert\") pod \"e2e-mt-bfbfdffb-openshift-default-5f485cf6dc-h5lws\" (UID: \"8bec5993-0ec8-47c2-87da-b0d5f27bd306\") " pod="openshift-ingress/e2e-mt-bfbfdffb-openshift-default-5f485cf6dc-h5lws" Jul 21 15:49:02.936590 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:49:02.936396 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bec5993-0ec8-47c2-87da-b0d5f27bd306-istio-data\") pod \"e2e-mt-bfbfdffb-openshift-default-5f485cf6dc-h5lws\" (UID: \"8bec5993-0ec8-47c2-87da-b0d5f27bd306\") " pod="openshift-ingress/e2e-mt-bfbfdffb-openshift-default-5f485cf6dc-h5lws" Jul 21 15:49:02.936590 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:49:02.936443 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bec5993-0ec8-47c2-87da-b0d5f27bd306-credential-socket\") pod \"e2e-mt-bfbfdffb-openshift-default-5f485cf6dc-h5lws\" (UID: \"8bec5993-0ec8-47c2-87da-b0d5f27bd306\") " pod="openshift-ingress/e2e-mt-bfbfdffb-openshift-default-5f485cf6dc-h5lws" Jul 21 15:49:02.936590 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:49:02.936465 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bec5993-0ec8-47c2-87da-b0d5f27bd306-istio-podinfo\") pod \"e2e-mt-bfbfdffb-openshift-default-5f485cf6dc-h5lws\" (UID: \"8bec5993-0ec8-47c2-87da-b0d5f27bd306\") " pod="openshift-ingress/e2e-mt-bfbfdffb-openshift-default-5f485cf6dc-h5lws" Jul 21 15:49:02.936590 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:49:02.936521 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bec5993-0ec8-47c2-87da-b0d5f27bd306-workload-socket\") pod \"e2e-mt-bfbfdffb-openshift-default-5f485cf6dc-h5lws\" (UID: \"8bec5993-0ec8-47c2-87da-b0d5f27bd306\") " pod="openshift-ingress/e2e-mt-bfbfdffb-openshift-default-5f485cf6dc-h5lws" Jul 21 15:49:02.936590 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:49:02.936559 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bec5993-0ec8-47c2-87da-b0d5f27bd306-istio-token\") pod \"e2e-mt-bfbfdffb-openshift-default-5f485cf6dc-h5lws\" (UID: \"8bec5993-0ec8-47c2-87da-b0d5f27bd306\") " pod="openshift-ingress/e2e-mt-bfbfdffb-openshift-default-5f485cf6dc-h5lws" Jul 21 15:49:02.936979 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:49:02.936739 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bec5993-0ec8-47c2-87da-b0d5f27bd306-workload-certs\") pod \"e2e-mt-bfbfdffb-openshift-default-5f485cf6dc-h5lws\" (UID: \"8bec5993-0ec8-47c2-87da-b0d5f27bd306\") " pod="openshift-ingress/e2e-mt-bfbfdffb-openshift-default-5f485cf6dc-h5lws" Jul 21 15:49:02.936979 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:49:02.936831 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bec5993-0ec8-47c2-87da-b0d5f27bd306-credential-socket\") pod \"e2e-mt-bfbfdffb-openshift-default-5f485cf6dc-h5lws\" (UID: \"8bec5993-0ec8-47c2-87da-b0d5f27bd306\") " pod="openshift-ingress/e2e-mt-bfbfdffb-openshift-default-5f485cf6dc-h5lws" Jul 21 15:49:02.936979 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:49:02.936957 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bec5993-0ec8-47c2-87da-b0d5f27bd306-istio-data\") pod \"e2e-mt-bfbfdffb-openshift-default-5f485cf6dc-h5lws\" (UID: \"8bec5993-0ec8-47c2-87da-b0d5f27bd306\") " pod="openshift-ingress/e2e-mt-bfbfdffb-openshift-default-5f485cf6dc-h5lws" Jul 21 15:49:02.937108 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:49:02.936997 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bec5993-0ec8-47c2-87da-b0d5f27bd306-workload-socket\") pod \"e2e-mt-bfbfdffb-openshift-default-5f485cf6dc-h5lws\" (UID: \"8bec5993-0ec8-47c2-87da-b0d5f27bd306\") " pod="openshift-ingress/e2e-mt-bfbfdffb-openshift-default-5f485cf6dc-h5lws" Jul 21 15:49:02.937350 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:49:02.937301 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bec5993-0ec8-47c2-87da-b0d5f27bd306-istiod-ca-cert\") pod \"e2e-mt-bfbfdffb-openshift-default-5f485cf6dc-h5lws\" (UID: \"8bec5993-0ec8-47c2-87da-b0d5f27bd306\") " pod="openshift-ingress/e2e-mt-bfbfdffb-openshift-default-5f485cf6dc-h5lws" Jul 21 15:49:02.938830 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:49:02.938804 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bec5993-0ec8-47c2-87da-b0d5f27bd306-istio-envoy\") pod \"e2e-mt-bfbfdffb-openshift-default-5f485cf6dc-h5lws\" (UID: \"8bec5993-0ec8-47c2-87da-b0d5f27bd306\") " pod="openshift-ingress/e2e-mt-bfbfdffb-openshift-default-5f485cf6dc-h5lws" Jul 21 15:49:02.939222 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:49:02.939199 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bec5993-0ec8-47c2-87da-b0d5f27bd306-istio-podinfo\") pod \"e2e-mt-bfbfdffb-openshift-default-5f485cf6dc-h5lws\" (UID: \"8bec5993-0ec8-47c2-87da-b0d5f27bd306\") " pod="openshift-ingress/e2e-mt-bfbfdffb-openshift-default-5f485cf6dc-h5lws" Jul 21 15:49:02.945117 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:49:02.945094 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bec5993-0ec8-47c2-87da-b0d5f27bd306-istio-token\") pod \"e2e-mt-bfbfdffb-openshift-default-5f485cf6dc-h5lws\" (UID: \"8bec5993-0ec8-47c2-87da-b0d5f27bd306\") " pod="openshift-ingress/e2e-mt-bfbfdffb-openshift-default-5f485cf6dc-h5lws" Jul 21 15:49:02.945416 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:49:02.945394 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqgwg\" (UniqueName: \"kubernetes.io/projected/8bec5993-0ec8-47c2-87da-b0d5f27bd306-kube-api-access-sqgwg\") pod \"e2e-mt-bfbfdffb-openshift-default-5f485cf6dc-h5lws\" (UID: \"8bec5993-0ec8-47c2-87da-b0d5f27bd306\") " pod="openshift-ingress/e2e-mt-bfbfdffb-openshift-default-5f485cf6dc-h5lws" Jul 21 15:49:03.069350 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:49:03.069234 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-bfbfdffb-openshift-default-5f485cf6dc-h5lws" Jul 21 15:49:03.217682 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:49:03.217655 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-bfbfdffb-openshift-default-5f485cf6dc-h5lws"] Jul 21 15:49:03.219943 ip-10-0-135-176 kubenswrapper[2563]: W0721 15:49:03.219916 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bec5993_0ec8_47c2_87da_b0d5f27bd306.slice/crio-033a734219f4062047bea13033492beff9384e5b0520e49be98d6b9c8b4acea1 WatchSource:0}: Error finding container 033a734219f4062047bea13033492beff9384e5b0520e49be98d6b9c8b4acea1: Status 404 returned error can't find the container with id 033a734219f4062047bea13033492beff9384e5b0520e49be98d6b9c8b4acea1 Jul 21 15:49:03.222096 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:49:03.222065 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 21 15:49:03.222193 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:49:03.222133 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 21 15:49:03.222193 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:49:03.222164 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 21 15:49:03.642317 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:49:03.642276 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-bfbfdffb-openshift-default-5f485cf6dc-h5lws" event={"ID":"8bec5993-0ec8-47c2-87da-b0d5f27bd306","Type":"ContainerStarted","Data":"18d76585ff339b4c8bd86925b91ff29b1cbf6a25525abfd947b4c6cbbb66fcc4"} Jul 21 15:49:03.642549 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:49:03.642353 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-bfbfdffb-openshift-default-5f485cf6dc-h5lws" event={"ID":"8bec5993-0ec8-47c2-87da-b0d5f27bd306","Type":"ContainerStarted","Data":"033a734219f4062047bea13033492beff9384e5b0520e49be98d6b9c8b4acea1"} Jul 21 15:49:03.663846 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:49:03.663796 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-bfbfdffb-openshift-default-5f485cf6dc-h5lws" podStartSLOduration=1.663778965 podStartE2EDuration="1.663778965s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 15:49:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 15:49:03.662533669 +0000 UTC m=+2975.658319086" watchObservedRunningTime="2026-07-21 15:49:03.663778965 +0000 UTC m=+2975.659564443" Jul 21 15:49:04.070240 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:49:04.070151 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-bfbfdffb-openshift-default-5f485cf6dc-h5lws" Jul 21 15:49:04.074911 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:49:04.074887 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-bfbfdffb-openshift-default-5f485cf6dc-h5lws" Jul 21 15:49:04.648023 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:49:04.647991 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-bfbfdffb-openshift-default-5f485cf6dc-h5lws" Jul 21 15:49:04.648901 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:49:04.648883 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-bfbfdffb-openshift-default-5f485cf6dc-h5lws" Jul 21 15:49:08.540333 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:49:08.540296 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-j4c8p"] Jul 21 15:49:18.742657 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:49:18.742627 2563 generic.go:356] "Generic (PLEG): container finished" podID="8bec5993-0ec8-47c2-87da-b0d5f27bd306" containerID="18d76585ff339b4c8bd86925b91ff29b1cbf6a25525abfd947b4c6cbbb66fcc4" exitCode=0 Jul 21 15:49:18.743082 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:49:18.742701 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-bfbfdffb-openshift-default-5f485cf6dc-h5lws" event={"ID":"8bec5993-0ec8-47c2-87da-b0d5f27bd306","Type":"ContainerDied","Data":"18d76585ff339b4c8bd86925b91ff29b1cbf6a25525abfd947b4c6cbbb66fcc4"} Jul 21 15:49:18.743126 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:49:18.743088 2563 scope.go:117] "RemoveContainer" containerID="18d76585ff339b4c8bd86925b91ff29b1cbf6a25525abfd947b4c6cbbb66fcc4" Jul 21 15:49:18.744088 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:49:18.744062 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 21 15:49:18.744141 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:49:18.744130 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 21 15:49:18.744177 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:49:18.744166 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 21 15:49:19.648403 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:49:19.648370 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-bfbfdffb-openshift-default-5f485cf6dc-h5lws" Jul 21 15:49:19.750456 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:49:19.750418 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-bfbfdffb-openshift-default-5f485cf6dc-h5lws" event={"ID":"8bec5993-0ec8-47c2-87da-b0d5f27bd306","Type":"ContainerStarted","Data":"ffb8448a503a8fa7cdecf2571af7faed7153afd56db39faca7d9b7a1413da640"} Jul 21 15:49:20.070017 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:49:20.069927 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-bfbfdffb-openshift-default-5f485cf6dc-h5lws" Jul 21 15:49:20.071064 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:49:20.071038 2563 patch_prober.go:28] interesting pod/e2e-mt-bfbfdffb-openshift-default-5f485cf6dc-h5lws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Jul 21 15:49:20.071186 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:49:20.071091 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-bfbfdffb-openshift-default-5f485cf6dc-h5lws" podUID="8bec5993-0ec8-47c2-87da-b0d5f27bd306" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Jul 21 15:49:21.069973 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:49:21.069938 2563 patch_prober.go:28] interesting pod/e2e-mt-bfbfdffb-openshift-default-5f485cf6dc-h5lws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Jul 21 15:49:21.070402 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:49:21.069992 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-bfbfdffb-openshift-default-5f485cf6dc-h5lws" podUID="8bec5993-0ec8-47c2-87da-b0d5f27bd306" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Jul 21 15:49:22.070616 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:49:22.070572 2563 patch_prober.go:28] interesting pod/e2e-mt-bfbfdffb-openshift-default-5f485cf6dc-h5lws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Jul 21 15:49:22.071098 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:49:22.070635 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-bfbfdffb-openshift-default-5f485cf6dc-h5lws" podUID="8bec5993-0ec8-47c2-87da-b0d5f27bd306" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Jul 21 15:49:23.069567 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:49:23.069531 2563 patch_prober.go:28] interesting pod/e2e-mt-bfbfdffb-openshift-default-5f485cf6dc-h5lws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Jul 21 15:49:23.069749 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:49:23.069594 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-bfbfdffb-openshift-default-5f485cf6dc-h5lws" podUID="8bec5993-0ec8-47c2-87da-b0d5f27bd306" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Jul 21 15:49:24.069900 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:49:24.069860 2563 patch_prober.go:28] interesting pod/e2e-mt-bfbfdffb-openshift-default-5f485cf6dc-h5lws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Jul 21 15:49:24.070352 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:49:24.069925 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-bfbfdffb-openshift-default-5f485cf6dc-h5lws" podUID="8bec5993-0ec8-47c2-87da-b0d5f27bd306" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Jul 21 15:49:25.073427 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:49:25.073396 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-bfbfdffb-openshift-default-5f485cf6dc-h5lws" Jul 21 15:49:25.073800 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:49:25.073668 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-bfbfdffb-openshift-default-5f485cf6dc-h5lws" Jul 21 15:49:25.074462 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:49:25.074442 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-bfbfdffb-openshift-default-5f485cf6dc-h5lws" Jul 21 15:49:28.662957 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:49:28.662924 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kn9fn_02f3d0c6-4116-4134-ba51-1c02c73bef05/ovn-acl-logging/0.log" Jul 21 15:49:28.674203 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:49:28.674181 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kn9fn_02f3d0c6-4116-4134-ba51-1c02c73bef05/ovn-acl-logging/0.log" Jul 21 15:49:29.946938 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:49:29.946902 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-j4c8p"] Jul 21 15:52:24.120347 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:52:24.120284 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-bfbfdffb-openshift-default-5f485cf6dc-h5lws"] Jul 21 15:52:24.120904 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:52:24.120584 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-bfbfdffb-openshift-default-5f485cf6dc-h5lws" podUID="8bec5993-0ec8-47c2-87da-b0d5f27bd306" containerName="istio-proxy" containerID="cri-o://ffb8448a503a8fa7cdecf2571af7faed7153afd56db39faca7d9b7a1413da640" gracePeriod=30 Jul 21 15:52:25.076318 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:52:25.076279 2563 patch_prober.go:28] interesting pod/e2e-mt-bfbfdffb-openshift-default-5f485cf6dc-h5lws container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 21 15:52:25.076525 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:52:25.076372 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-bfbfdffb-openshift-default-5f485cf6dc-h5lws" podUID="8bec5993-0ec8-47c2-87da-b0d5f27bd306" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 21 15:52:29.402593 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:52:29.402565 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-bfbfdffb-openshift-default-5f485cf6dc-h5lws" Jul 21 15:52:29.535488 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:52:29.535456 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bec5993-0ec8-47c2-87da-b0d5f27bd306-workload-certs\") pod \"8bec5993-0ec8-47c2-87da-b0d5f27bd306\" (UID: \"8bec5993-0ec8-47c2-87da-b0d5f27bd306\") " Jul 21 15:52:29.535488 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:52:29.535490 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bec5993-0ec8-47c2-87da-b0d5f27bd306-istio-envoy\") pod \"8bec5993-0ec8-47c2-87da-b0d5f27bd306\" (UID: \"8bec5993-0ec8-47c2-87da-b0d5f27bd306\") " Jul 21 15:52:29.535746 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:52:29.535580 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bec5993-0ec8-47c2-87da-b0d5f27bd306-istiod-ca-cert\") pod \"8bec5993-0ec8-47c2-87da-b0d5f27bd306\" (UID: \"8bec5993-0ec8-47c2-87da-b0d5f27bd306\") " Jul 21 15:52:29.535746 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:52:29.535605 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bec5993-0ec8-47c2-87da-b0d5f27bd306-istio-token\") pod \"8bec5993-0ec8-47c2-87da-b0d5f27bd306\" (UID: \"8bec5993-0ec8-47c2-87da-b0d5f27bd306\") " Jul 21 15:52:29.535746 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:52:29.535660 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sqgwg\" (UniqueName: \"kubernetes.io/projected/8bec5993-0ec8-47c2-87da-b0d5f27bd306-kube-api-access-sqgwg\") pod \"8bec5993-0ec8-47c2-87da-b0d5f27bd306\" (UID: \"8bec5993-0ec8-47c2-87da-b0d5f27bd306\") " Jul 21 15:52:29.535746 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:52:29.535719 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bec5993-0ec8-47c2-87da-b0d5f27bd306-istio-data\") pod \"8bec5993-0ec8-47c2-87da-b0d5f27bd306\" (UID: \"8bec5993-0ec8-47c2-87da-b0d5f27bd306\") " Jul 21 15:52:29.535959 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:52:29.535761 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bec5993-0ec8-47c2-87da-b0d5f27bd306-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8bec5993-0ec8-47c2-87da-b0d5f27bd306" (UID: "8bec5993-0ec8-47c2-87da-b0d5f27bd306"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 15:52:29.535959 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:52:29.535775 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bec5993-0ec8-47c2-87da-b0d5f27bd306-istio-podinfo\") pod \"8bec5993-0ec8-47c2-87da-b0d5f27bd306\" (UID: \"8bec5993-0ec8-47c2-87da-b0d5f27bd306\") " Jul 21 15:52:29.535959 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:52:29.535841 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bec5993-0ec8-47c2-87da-b0d5f27bd306-credential-socket\") pod \"8bec5993-0ec8-47c2-87da-b0d5f27bd306\" (UID: \"8bec5993-0ec8-47c2-87da-b0d5f27bd306\") " Jul 21 15:52:29.535959 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:52:29.535871 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bec5993-0ec8-47c2-87da-b0d5f27bd306-workload-socket\") pod \"8bec5993-0ec8-47c2-87da-b0d5f27bd306\" (UID: \"8bec5993-0ec8-47c2-87da-b0d5f27bd306\") " Jul 21 15:52:29.536204 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:52:29.536161 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bec5993-0ec8-47c2-87da-b0d5f27bd306-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8bec5993-0ec8-47c2-87da-b0d5f27bd306" (UID: "8bec5993-0ec8-47c2-87da-b0d5f27bd306"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 15:52:29.536264 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:52:29.536204 2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bec5993-0ec8-47c2-87da-b0d5f27bd306-workload-certs\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 15:52:29.536264 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:52:29.536122 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bec5993-0ec8-47c2-87da-b0d5f27bd306-istio-data" (OuterVolumeSpecName: "istio-data") pod "8bec5993-0ec8-47c2-87da-b0d5f27bd306" (UID: "8bec5993-0ec8-47c2-87da-b0d5f27bd306"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 15:52:29.536424 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:52:29.536400 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bec5993-0ec8-47c2-87da-b0d5f27bd306-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8bec5993-0ec8-47c2-87da-b0d5f27bd306" (UID: "8bec5993-0ec8-47c2-87da-b0d5f27bd306"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 15:52:29.536424 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:52:29.536408 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bec5993-0ec8-47c2-87da-b0d5f27bd306-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8bec5993-0ec8-47c2-87da-b0d5f27bd306" (UID: "8bec5993-0ec8-47c2-87da-b0d5f27bd306"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 15:52:29.538164 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:52:29.538135 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bec5993-0ec8-47c2-87da-b0d5f27bd306-istio-token" (OuterVolumeSpecName: "istio-token") pod "8bec5993-0ec8-47c2-87da-b0d5f27bd306" (UID: "8bec5993-0ec8-47c2-87da-b0d5f27bd306"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 15:52:29.538164 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:52:29.538157 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bec5993-0ec8-47c2-87da-b0d5f27bd306-kube-api-access-sqgwg" (OuterVolumeSpecName: "kube-api-access-sqgwg") pod "8bec5993-0ec8-47c2-87da-b0d5f27bd306" (UID: "8bec5993-0ec8-47c2-87da-b0d5f27bd306"). InnerVolumeSpecName "kube-api-access-sqgwg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 15:52:29.538546 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:52:29.538516 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8bec5993-0ec8-47c2-87da-b0d5f27bd306-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8bec5993-0ec8-47c2-87da-b0d5f27bd306" (UID: "8bec5993-0ec8-47c2-87da-b0d5f27bd306"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 21 15:52:29.538546 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:52:29.538515 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bec5993-0ec8-47c2-87da-b0d5f27bd306-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8bec5993-0ec8-47c2-87da-b0d5f27bd306" (UID: "8bec5993-0ec8-47c2-87da-b0d5f27bd306"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 15:52:29.637213 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:52:29.637124 2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bec5993-0ec8-47c2-87da-b0d5f27bd306-istiod-ca-cert\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 15:52:29.637213 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:52:29.637154 2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bec5993-0ec8-47c2-87da-b0d5f27bd306-istio-token\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 15:52:29.637213 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:52:29.637165 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqgwg\" (UniqueName: \"kubernetes.io/projected/8bec5993-0ec8-47c2-87da-b0d5f27bd306-kube-api-access-sqgwg\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 15:52:29.637213 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:52:29.637175 2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bec5993-0ec8-47c2-87da-b0d5f27bd306-istio-data\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 15:52:29.637213 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:52:29.637184 2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bec5993-0ec8-47c2-87da-b0d5f27bd306-istio-podinfo\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 15:52:29.637213 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:52:29.637192 2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bec5993-0ec8-47c2-87da-b0d5f27bd306-credential-socket\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 15:52:29.637213 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:52:29.637200 2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bec5993-0ec8-47c2-87da-b0d5f27bd306-workload-socket\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 15:52:29.637213 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:52:29.637208 2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bec5993-0ec8-47c2-87da-b0d5f27bd306-istio-envoy\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 15:52:29.814551 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:52:29.814507 2563 generic.go:356] "Generic (PLEG): container finished" podID="8bec5993-0ec8-47c2-87da-b0d5f27bd306" containerID="ffb8448a503a8fa7cdecf2571af7faed7153afd56db39faca7d9b7a1413da640" exitCode=0 Jul 21 15:52:29.814746 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:52:29.814565 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-bfbfdffb-openshift-default-5f485cf6dc-h5lws" Jul 21 15:52:29.814746 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:52:29.814594 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-bfbfdffb-openshift-default-5f485cf6dc-h5lws" event={"ID":"8bec5993-0ec8-47c2-87da-b0d5f27bd306","Type":"ContainerDied","Data":"ffb8448a503a8fa7cdecf2571af7faed7153afd56db39faca7d9b7a1413da640"} Jul 21 15:52:29.814746 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:52:29.814659 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-bfbfdffb-openshift-default-5f485cf6dc-h5lws" event={"ID":"8bec5993-0ec8-47c2-87da-b0d5f27bd306","Type":"ContainerDied","Data":"033a734219f4062047bea13033492beff9384e5b0520e49be98d6b9c8b4acea1"} Jul 21 15:52:29.814746 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:52:29.814682 2563 scope.go:117] "RemoveContainer" containerID="ffb8448a503a8fa7cdecf2571af7faed7153afd56db39faca7d9b7a1413da640" Jul 21 15:52:29.825060 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:52:29.825033 2563 scope.go:117] "RemoveContainer" containerID="18d76585ff339b4c8bd86925b91ff29b1cbf6a25525abfd947b4c6cbbb66fcc4" Jul 21 15:52:29.835210 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:52:29.835185 2563 scope.go:117] "RemoveContainer" containerID="ffb8448a503a8fa7cdecf2571af7faed7153afd56db39faca7d9b7a1413da640" Jul 21 15:52:29.835618 ip-10-0-135-176 kubenswrapper[2563]: E0721 15:52:29.835598 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ffb8448a503a8fa7cdecf2571af7faed7153afd56db39faca7d9b7a1413da640\": container with ID starting with ffb8448a503a8fa7cdecf2571af7faed7153afd56db39faca7d9b7a1413da640 not found: ID does not exist" containerID="ffb8448a503a8fa7cdecf2571af7faed7153afd56db39faca7d9b7a1413da640" Jul 21 15:52:29.835715 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:52:29.835631 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffb8448a503a8fa7cdecf2571af7faed7153afd56db39faca7d9b7a1413da640"} err="failed to get container status \"ffb8448a503a8fa7cdecf2571af7faed7153afd56db39faca7d9b7a1413da640\": rpc error: code = NotFound desc = could not find container \"ffb8448a503a8fa7cdecf2571af7faed7153afd56db39faca7d9b7a1413da640\": container with ID starting with ffb8448a503a8fa7cdecf2571af7faed7153afd56db39faca7d9b7a1413da640 not found: ID does not exist" Jul 21 15:52:29.835715 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:52:29.835656 2563 scope.go:117] "RemoveContainer" containerID="18d76585ff339b4c8bd86925b91ff29b1cbf6a25525abfd947b4c6cbbb66fcc4" Jul 21 15:52:29.836010 ip-10-0-135-176 kubenswrapper[2563]: E0721 15:52:29.835989 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18d76585ff339b4c8bd86925b91ff29b1cbf6a25525abfd947b4c6cbbb66fcc4\": container with ID starting with 18d76585ff339b4c8bd86925b91ff29b1cbf6a25525abfd947b4c6cbbb66fcc4 not found: ID does not exist" containerID="18d76585ff339b4c8bd86925b91ff29b1cbf6a25525abfd947b4c6cbbb66fcc4" Jul 21 15:52:29.836049 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:52:29.836018 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18d76585ff339b4c8bd86925b91ff29b1cbf6a25525abfd947b4c6cbbb66fcc4"} err="failed to get container status \"18d76585ff339b4c8bd86925b91ff29b1cbf6a25525abfd947b4c6cbbb66fcc4\": rpc error: code = NotFound desc = could not find container \"18d76585ff339b4c8bd86925b91ff29b1cbf6a25525abfd947b4c6cbbb66fcc4\": container with ID starting with 18d76585ff339b4c8bd86925b91ff29b1cbf6a25525abfd947b4c6cbbb66fcc4 not found: ID does not exist" Jul 21 15:52:29.838089 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:52:29.838064 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-bfbfdffb-openshift-default-5f485cf6dc-h5lws"] Jul 21 15:52:29.842802 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:52:29.842316 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-bfbfdffb-openshift-default-5f485cf6dc-h5lws"] Jul 21 15:52:30.480907 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:52:30.480852 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8bec5993-0ec8-47c2-87da-b0d5f27bd306" path="/var/lib/kubelet/pods/8bec5993-0ec8-47c2-87da-b0d5f27bd306/volumes" Jul 21 15:52:43.893621 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:52:43.893589 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-534ea2a9-openshift-default-548ff8f88c-d78hp"] Jul 21 15:52:43.894224 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:52:43.894155 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8bec5993-0ec8-47c2-87da-b0d5f27bd306" containerName="istio-proxy" Jul 21 15:52:43.894224 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:52:43.894178 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8bec5993-0ec8-47c2-87da-b0d5f27bd306" containerName="istio-proxy" Jul 21 15:52:43.894367 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:52:43.894293 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="8bec5993-0ec8-47c2-87da-b0d5f27bd306" containerName="istio-proxy" Jul 21 15:52:43.894367 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:52:43.894306 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="8bec5993-0ec8-47c2-87da-b0d5f27bd306" containerName="istio-proxy" Jul 21 15:52:43.897601 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:52:43.897575 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-534ea2a9-openshift-default-548ff8f88c-d78hp" Jul 21 15:52:43.900391 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:52:43.900315 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-534ea2a9-openshift-default-dockercfg-4s54v\"" Jul 21 15:52:43.911503 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:52:43.911468 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-534ea2a9-openshift-default-548ff8f88c-d78hp"] Jul 21 15:52:43.967012 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:52:43.966976 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrvh9\" (UniqueName: \"kubernetes.io/projected/29934221-c862-4f86-bdcc-e9e5b87f084a-kube-api-access-vrvh9\") pod \"e2e-mt-534ea2a9-openshift-default-548ff8f88c-d78hp\" (UID: \"29934221-c862-4f86-bdcc-e9e5b87f084a\") " pod="openshift-ingress/e2e-mt-534ea2a9-openshift-default-548ff8f88c-d78hp" Jul 21 15:52:43.967185 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:52:43.967022 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29934221-c862-4f86-bdcc-e9e5b87f084a-istio-envoy\") pod \"e2e-mt-534ea2a9-openshift-default-548ff8f88c-d78hp\" (UID: \"29934221-c862-4f86-bdcc-e9e5b87f084a\") " pod="openshift-ingress/e2e-mt-534ea2a9-openshift-default-548ff8f88c-d78hp" Jul 21 15:52:43.967185 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:52:43.967046 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29934221-c862-4f86-bdcc-e9e5b87f084a-credential-socket\") pod \"e2e-mt-534ea2a9-openshift-default-548ff8f88c-d78hp\" (UID: \"29934221-c862-4f86-bdcc-e9e5b87f084a\") " pod="openshift-ingress/e2e-mt-534ea2a9-openshift-default-548ff8f88c-d78hp" Jul 21 15:52:43.967185 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:52:43.967094 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29934221-c862-4f86-bdcc-e9e5b87f084a-workload-certs\") pod \"e2e-mt-534ea2a9-openshift-default-548ff8f88c-d78hp\" (UID: \"29934221-c862-4f86-bdcc-e9e5b87f084a\") " pod="openshift-ingress/e2e-mt-534ea2a9-openshift-default-548ff8f88c-d78hp" Jul 21 15:52:43.967185 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:52:43.967123 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29934221-c862-4f86-bdcc-e9e5b87f084a-istiod-ca-cert\") pod \"e2e-mt-534ea2a9-openshift-default-548ff8f88c-d78hp\" (UID: \"29934221-c862-4f86-bdcc-e9e5b87f084a\") " pod="openshift-ingress/e2e-mt-534ea2a9-openshift-default-548ff8f88c-d78hp" Jul 21 15:52:43.967185 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:52:43.967153 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29934221-c862-4f86-bdcc-e9e5b87f084a-istio-podinfo\") pod \"e2e-mt-534ea2a9-openshift-default-548ff8f88c-d78hp\" (UID: \"29934221-c862-4f86-bdcc-e9e5b87f084a\") " pod="openshift-ingress/e2e-mt-534ea2a9-openshift-default-548ff8f88c-d78hp" Jul 21 15:52:43.967474 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:52:43.967264 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29934221-c862-4f86-bdcc-e9e5b87f084a-istio-data\") pod \"e2e-mt-534ea2a9-openshift-default-548ff8f88c-d78hp\" (UID: \"29934221-c862-4f86-bdcc-e9e5b87f084a\") " pod="openshift-ingress/e2e-mt-534ea2a9-openshift-default-548ff8f88c-d78hp" Jul 21 15:52:43.967474 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:52:43.967315 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29934221-c862-4f86-bdcc-e9e5b87f084a-workload-socket\") pod \"e2e-mt-534ea2a9-openshift-default-548ff8f88c-d78hp\" (UID: \"29934221-c862-4f86-bdcc-e9e5b87f084a\") " pod="openshift-ingress/e2e-mt-534ea2a9-openshift-default-548ff8f88c-d78hp" Jul 21 15:52:43.967474 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:52:43.967380 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29934221-c862-4f86-bdcc-e9e5b87f084a-istio-token\") pod \"e2e-mt-534ea2a9-openshift-default-548ff8f88c-d78hp\" (UID: \"29934221-c862-4f86-bdcc-e9e5b87f084a\") " pod="openshift-ingress/e2e-mt-534ea2a9-openshift-default-548ff8f88c-d78hp" Jul 21 15:52:44.068012 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:52:44.067969 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29934221-c862-4f86-bdcc-e9e5b87f084a-istio-envoy\") pod \"e2e-mt-534ea2a9-openshift-default-548ff8f88c-d78hp\" (UID: \"29934221-c862-4f86-bdcc-e9e5b87f084a\") " pod="openshift-ingress/e2e-mt-534ea2a9-openshift-default-548ff8f88c-d78hp" Jul 21 15:52:44.068224 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:52:44.068017 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29934221-c862-4f86-bdcc-e9e5b87f084a-credential-socket\") pod \"e2e-mt-534ea2a9-openshift-default-548ff8f88c-d78hp\" (UID: \"29934221-c862-4f86-bdcc-e9e5b87f084a\") " pod="openshift-ingress/e2e-mt-534ea2a9-openshift-default-548ff8f88c-d78hp" Jul 21 15:52:44.068224 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:52:44.068051 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29934221-c862-4f86-bdcc-e9e5b87f084a-workload-certs\") pod \"e2e-mt-534ea2a9-openshift-default-548ff8f88c-d78hp\" (UID: \"29934221-c862-4f86-bdcc-e9e5b87f084a\") " pod="openshift-ingress/e2e-mt-534ea2a9-openshift-default-548ff8f88c-d78hp" Jul 21 15:52:44.068224 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:52:44.068078 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29934221-c862-4f86-bdcc-e9e5b87f084a-istiod-ca-cert\") pod \"e2e-mt-534ea2a9-openshift-default-548ff8f88c-d78hp\" (UID: \"29934221-c862-4f86-bdcc-e9e5b87f084a\") " pod="openshift-ingress/e2e-mt-534ea2a9-openshift-default-548ff8f88c-d78hp" Jul 21 15:52:44.068224 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:52:44.068119 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29934221-c862-4f86-bdcc-e9e5b87f084a-istio-podinfo\") pod \"e2e-mt-534ea2a9-openshift-default-548ff8f88c-d78hp\" (UID: \"29934221-c862-4f86-bdcc-e9e5b87f084a\") " pod="openshift-ingress/e2e-mt-534ea2a9-openshift-default-548ff8f88c-d78hp" Jul 21 15:52:44.068224 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:52:44.068174 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29934221-c862-4f86-bdcc-e9e5b87f084a-istio-data\") pod \"e2e-mt-534ea2a9-openshift-default-548ff8f88c-d78hp\" (UID: \"29934221-c862-4f86-bdcc-e9e5b87f084a\") " pod="openshift-ingress/e2e-mt-534ea2a9-openshift-default-548ff8f88c-d78hp" Jul 21 15:52:44.068224 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:52:44.068214 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29934221-c862-4f86-bdcc-e9e5b87f084a-workload-socket\") pod \"e2e-mt-534ea2a9-openshift-default-548ff8f88c-d78hp\" (UID: \"29934221-c862-4f86-bdcc-e9e5b87f084a\") " pod="openshift-ingress/e2e-mt-534ea2a9-openshift-default-548ff8f88c-d78hp" Jul 21 15:52:44.068590 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:52:44.068257 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29934221-c862-4f86-bdcc-e9e5b87f084a-istio-token\") pod \"e2e-mt-534ea2a9-openshift-default-548ff8f88c-d78hp\" (UID: \"29934221-c862-4f86-bdcc-e9e5b87f084a\") " pod="openshift-ingress/e2e-mt-534ea2a9-openshift-default-548ff8f88c-d78hp" Jul 21 15:52:44.068590 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:52:44.068313 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrvh9\" (UniqueName: \"kubernetes.io/projected/29934221-c862-4f86-bdcc-e9e5b87f084a-kube-api-access-vrvh9\") pod \"e2e-mt-534ea2a9-openshift-default-548ff8f88c-d78hp\" (UID: \"29934221-c862-4f86-bdcc-e9e5b87f084a\") " pod="openshift-ingress/e2e-mt-534ea2a9-openshift-default-548ff8f88c-d78hp" Jul 21 15:52:44.068590 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:52:44.068548 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29934221-c862-4f86-bdcc-e9e5b87f084a-workload-certs\") pod \"e2e-mt-534ea2a9-openshift-default-548ff8f88c-d78hp\" (UID: \"29934221-c862-4f86-bdcc-e9e5b87f084a\") " pod="openshift-ingress/e2e-mt-534ea2a9-openshift-default-548ff8f88c-d78hp" Jul 21 15:52:44.068773 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:52:44.068588 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29934221-c862-4f86-bdcc-e9e5b87f084a-credential-socket\") pod \"e2e-mt-534ea2a9-openshift-default-548ff8f88c-d78hp\" (UID: \"29934221-c862-4f86-bdcc-e9e5b87f084a\") " pod="openshift-ingress/e2e-mt-534ea2a9-openshift-default-548ff8f88c-d78hp" Jul 21 15:52:44.068773 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:52:44.068650 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29934221-c862-4f86-bdcc-e9e5b87f084a-istio-data\") pod \"e2e-mt-534ea2a9-openshift-default-548ff8f88c-d78hp\" (UID: \"29934221-c862-4f86-bdcc-e9e5b87f084a\") " pod="openshift-ingress/e2e-mt-534ea2a9-openshift-default-548ff8f88c-d78hp" Jul 21 15:52:44.068773 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:52:44.068703 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29934221-c862-4f86-bdcc-e9e5b87f084a-workload-socket\") pod \"e2e-mt-534ea2a9-openshift-default-548ff8f88c-d78hp\" (UID: \"29934221-c862-4f86-bdcc-e9e5b87f084a\") " pod="openshift-ingress/e2e-mt-534ea2a9-openshift-default-548ff8f88c-d78hp" Jul 21 15:52:44.069154 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:52:44.069099 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29934221-c862-4f86-bdcc-e9e5b87f084a-istiod-ca-cert\") pod \"e2e-mt-534ea2a9-openshift-default-548ff8f88c-d78hp\" (UID: \"29934221-c862-4f86-bdcc-e9e5b87f084a\") " pod="openshift-ingress/e2e-mt-534ea2a9-openshift-default-548ff8f88c-d78hp" Jul 21 15:52:44.070759 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:52:44.070735 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29934221-c862-4f86-bdcc-e9e5b87f084a-istio-envoy\") pod \"e2e-mt-534ea2a9-openshift-default-548ff8f88c-d78hp\" (UID: \"29934221-c862-4f86-bdcc-e9e5b87f084a\") " pod="openshift-ingress/e2e-mt-534ea2a9-openshift-default-548ff8f88c-d78hp" Jul 21 15:52:44.070942 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:52:44.070921 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29934221-c862-4f86-bdcc-e9e5b87f084a-istio-podinfo\") pod \"e2e-mt-534ea2a9-openshift-default-548ff8f88c-d78hp\" (UID: \"29934221-c862-4f86-bdcc-e9e5b87f084a\") " pod="openshift-ingress/e2e-mt-534ea2a9-openshift-default-548ff8f88c-d78hp" Jul 21 15:52:44.078256 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:52:44.078228 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29934221-c862-4f86-bdcc-e9e5b87f084a-istio-token\") pod \"e2e-mt-534ea2a9-openshift-default-548ff8f88c-d78hp\" (UID: \"29934221-c862-4f86-bdcc-e9e5b87f084a\") " pod="openshift-ingress/e2e-mt-534ea2a9-openshift-default-548ff8f88c-d78hp" Jul 21 15:52:44.078568 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:52:44.078541 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrvh9\" (UniqueName: \"kubernetes.io/projected/29934221-c862-4f86-bdcc-e9e5b87f084a-kube-api-access-vrvh9\") pod \"e2e-mt-534ea2a9-openshift-default-548ff8f88c-d78hp\" (UID: \"29934221-c862-4f86-bdcc-e9e5b87f084a\") " pod="openshift-ingress/e2e-mt-534ea2a9-openshift-default-548ff8f88c-d78hp" Jul 21 15:52:44.208080 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:52:44.207984 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-534ea2a9-openshift-default-548ff8f88c-d78hp" Jul 21 15:52:44.377403 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:52:44.377365 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-534ea2a9-openshift-default-548ff8f88c-d78hp"] Jul 21 15:52:44.378997 ip-10-0-135-176 kubenswrapper[2563]: W0721 15:52:44.378971 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29934221_c862_4f86_bdcc_e9e5b87f084a.slice/crio-bfc6975c721ea3089c2b6badad012e4f731d34506938ba1d96102c5d7435af70 WatchSource:0}: Error finding container bfc6975c721ea3089c2b6badad012e4f731d34506938ba1d96102c5d7435af70: Status 404 returned error can't find the container with id bfc6975c721ea3089c2b6badad012e4f731d34506938ba1d96102c5d7435af70 Jul 21 15:52:44.381497 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:52:44.381448 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 21 15:52:44.381610 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:52:44.381549 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 21 15:52:44.381610 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:52:44.381594 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 21 15:52:44.909522 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:52:44.909481 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-534ea2a9-openshift-default-548ff8f88c-d78hp" event={"ID":"29934221-c862-4f86-bdcc-e9e5b87f084a","Type":"ContainerStarted","Data":"5e5bf90fd713fd590ac3399cdf7918b20503c430a49747f61d6e45030cce2e36"} Jul 21 15:52:44.910014 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:52:44.909581 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-534ea2a9-openshift-default-548ff8f88c-d78hp" event={"ID":"29934221-c862-4f86-bdcc-e9e5b87f084a","Type":"ContainerStarted","Data":"bfc6975c721ea3089c2b6badad012e4f731d34506938ba1d96102c5d7435af70"} Jul 21 15:52:44.947184 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:52:44.947127 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-534ea2a9-openshift-default-548ff8f88c-d78hp" podStartSLOduration=1.94710783 podStartE2EDuration="1.94710783s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 15:52:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 15:52:44.94624389 +0000 UTC m=+3196.942029307" watchObservedRunningTime="2026-07-21 15:52:44.94710783 +0000 UTC m=+3196.942893318" Jul 21 15:52:45.208407 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:52:45.208284 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-534ea2a9-openshift-default-548ff8f88c-d78hp" Jul 21 15:52:45.213652 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:52:45.213626 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-534ea2a9-openshift-default-548ff8f88c-d78hp" Jul 21 15:52:45.916465 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:52:45.916442 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-534ea2a9-openshift-default-548ff8f88c-d78hp" Jul 21 15:52:45.917435 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:52:45.917415 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-534ea2a9-openshift-default-548ff8f88c-d78hp" Jul 21 15:52:50.347042 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:52:50.347004 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-j4c8p"] Jul 21 15:52:56.968593 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:52:56.968553 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-534ea2a9/e2e-lifecycle-model-534ea2a9-kserve-84b774c4f5-bz49d"] Jul 21 15:52:56.969151 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:52:56.969134 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8bec5993-0ec8-47c2-87da-b0d5f27bd306" containerName="istio-proxy" Jul 21 15:52:56.969200 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:52:56.969155 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8bec5993-0ec8-47c2-87da-b0d5f27bd306" containerName="istio-proxy" Jul 21 15:52:56.973919 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:52:56.973894 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-534ea2a9/e2e-lifecycle-model-534ea2a9-kserve-84b774c4f5-bz49d" Jul 21 15:52:56.978686 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:52:56.978659 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-534ea2a9\"/\"e2e-lifecycle-model-534ea2a9-kserve-self-signed-certs\"" Jul 21 15:52:56.979001 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:52:56.978714 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-534ea2a9\"/\"openshift-service-ca.crt\"" Jul 21 15:52:56.979103 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:52:56.978714 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-534ea2a9\"/\"kube-root-ca.crt\"" Jul 21 15:52:56.979188 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:52:56.978794 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-534ea2a9\"/\"default-dockercfg-bc5wz\"" Jul 21 15:52:56.983532 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:52:56.983112 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-534ea2a9/e2e-lifecycle-model-534ea2a9-kserve-84b774c4f5-bz49d"] Jul 21 15:52:57.090423 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:52:57.090381 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5a8bf5f-1bb5-4379-92d7-ef414993bc66-tls-certs\") pod \"e2e-lifecycle-model-534ea2a9-kserve-84b774c4f5-bz49d\" (UID: \"c5a8bf5f-1bb5-4379-92d7-ef414993bc66\") " pod="ai-tenant-e2e-mt-534ea2a9/e2e-lifecycle-model-534ea2a9-kserve-84b774c4f5-bz49d" Jul 21 15:52:57.090632 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:52:57.090439 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5a8bf5f-1bb5-4379-92d7-ef414993bc66-model-cache\") pod \"e2e-lifecycle-model-534ea2a9-kserve-84b774c4f5-bz49d\" (UID: \"c5a8bf5f-1bb5-4379-92d7-ef414993bc66\") " pod="ai-tenant-e2e-mt-534ea2a9/e2e-lifecycle-model-534ea2a9-kserve-84b774c4f5-bz49d" Jul 21 15:52:57.090632 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:52:57.090541 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c5a8bf5f-1bb5-4379-92d7-ef414993bc66-home\") pod \"e2e-lifecycle-model-534ea2a9-kserve-84b774c4f5-bz49d\" (UID: \"c5a8bf5f-1bb5-4379-92d7-ef414993bc66\") " pod="ai-tenant-e2e-mt-534ea2a9/e2e-lifecycle-model-534ea2a9-kserve-84b774c4f5-bz49d" Jul 21 15:52:57.090632 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:52:57.090596 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5a8bf5f-1bb5-4379-92d7-ef414993bc66-dshm\") pod \"e2e-lifecycle-model-534ea2a9-kserve-84b774c4f5-bz49d\" (UID: \"c5a8bf5f-1bb5-4379-92d7-ef414993bc66\") " pod="ai-tenant-e2e-mt-534ea2a9/e2e-lifecycle-model-534ea2a9-kserve-84b774c4f5-bz49d" Jul 21 15:52:57.090632 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:52:57.090624 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgjbc\" (UniqueName: \"kubernetes.io/projected/c5a8bf5f-1bb5-4379-92d7-ef414993bc66-kube-api-access-bgjbc\") pod \"e2e-lifecycle-model-534ea2a9-kserve-84b774c4f5-bz49d\" (UID: \"c5a8bf5f-1bb5-4379-92d7-ef414993bc66\") " pod="ai-tenant-e2e-mt-534ea2a9/e2e-lifecycle-model-534ea2a9-kserve-84b774c4f5-bz49d" Jul 21 15:52:57.090886 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:52:57.090686 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5a8bf5f-1bb5-4379-92d7-ef414993bc66-tmp-dir\") pod \"e2e-lifecycle-model-534ea2a9-kserve-84b774c4f5-bz49d\" (UID: \"c5a8bf5f-1bb5-4379-92d7-ef414993bc66\") " pod="ai-tenant-e2e-mt-534ea2a9/e2e-lifecycle-model-534ea2a9-kserve-84b774c4f5-bz49d" Jul 21 15:52:57.192240 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:52:57.192147 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5a8bf5f-1bb5-4379-92d7-ef414993bc66-tls-certs\") pod \"e2e-lifecycle-model-534ea2a9-kserve-84b774c4f5-bz49d\" (UID: \"c5a8bf5f-1bb5-4379-92d7-ef414993bc66\") " pod="ai-tenant-e2e-mt-534ea2a9/e2e-lifecycle-model-534ea2a9-kserve-84b774c4f5-bz49d" Jul 21 15:52:57.192240 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:52:57.192198 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5a8bf5f-1bb5-4379-92d7-ef414993bc66-model-cache\") pod \"e2e-lifecycle-model-534ea2a9-kserve-84b774c4f5-bz49d\" (UID: \"c5a8bf5f-1bb5-4379-92d7-ef414993bc66\") " pod="ai-tenant-e2e-mt-534ea2a9/e2e-lifecycle-model-534ea2a9-kserve-84b774c4f5-bz49d" Jul 21 15:52:57.192240 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:52:57.192233 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c5a8bf5f-1bb5-4379-92d7-ef414993bc66-home\") pod \"e2e-lifecycle-model-534ea2a9-kserve-84b774c4f5-bz49d\" (UID: \"c5a8bf5f-1bb5-4379-92d7-ef414993bc66\") " pod="ai-tenant-e2e-mt-534ea2a9/e2e-lifecycle-model-534ea2a9-kserve-84b774c4f5-bz49d" Jul 21 15:52:57.192626 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:52:57.192259 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5a8bf5f-1bb5-4379-92d7-ef414993bc66-dshm\") pod \"e2e-lifecycle-model-534ea2a9-kserve-84b774c4f5-bz49d\" (UID: \"c5a8bf5f-1bb5-4379-92d7-ef414993bc66\") " pod="ai-tenant-e2e-mt-534ea2a9/e2e-lifecycle-model-534ea2a9-kserve-84b774c4f5-bz49d" Jul 21 15:52:57.192626 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:52:57.192365 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgjbc\" (UniqueName: \"kubernetes.io/projected/c5a8bf5f-1bb5-4379-92d7-ef414993bc66-kube-api-access-bgjbc\") pod \"e2e-lifecycle-model-534ea2a9-kserve-84b774c4f5-bz49d\" (UID: \"c5a8bf5f-1bb5-4379-92d7-ef414993bc66\") " pod="ai-tenant-e2e-mt-534ea2a9/e2e-lifecycle-model-534ea2a9-kserve-84b774c4f5-bz49d" Jul 21 15:52:57.192626 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:52:57.192448 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5a8bf5f-1bb5-4379-92d7-ef414993bc66-tmp-dir\") pod \"e2e-lifecycle-model-534ea2a9-kserve-84b774c4f5-bz49d\" (UID: \"c5a8bf5f-1bb5-4379-92d7-ef414993bc66\") " pod="ai-tenant-e2e-mt-534ea2a9/e2e-lifecycle-model-534ea2a9-kserve-84b774c4f5-bz49d" Jul 21 15:52:57.192804 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:52:57.192649 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c5a8bf5f-1bb5-4379-92d7-ef414993bc66-home\") pod \"e2e-lifecycle-model-534ea2a9-kserve-84b774c4f5-bz49d\" (UID: \"c5a8bf5f-1bb5-4379-92d7-ef414993bc66\") " pod="ai-tenant-e2e-mt-534ea2a9/e2e-lifecycle-model-534ea2a9-kserve-84b774c4f5-bz49d" Jul 21 15:52:57.192804 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:52:57.192738 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5a8bf5f-1bb5-4379-92d7-ef414993bc66-model-cache\") pod \"e2e-lifecycle-model-534ea2a9-kserve-84b774c4f5-bz49d\" (UID: \"c5a8bf5f-1bb5-4379-92d7-ef414993bc66\") " pod="ai-tenant-e2e-mt-534ea2a9/e2e-lifecycle-model-534ea2a9-kserve-84b774c4f5-bz49d" Jul 21 15:52:57.192903 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:52:57.192808 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5a8bf5f-1bb5-4379-92d7-ef414993bc66-tmp-dir\") pod \"e2e-lifecycle-model-534ea2a9-kserve-84b774c4f5-bz49d\" (UID: \"c5a8bf5f-1bb5-4379-92d7-ef414993bc66\") " pod="ai-tenant-e2e-mt-534ea2a9/e2e-lifecycle-model-534ea2a9-kserve-84b774c4f5-bz49d" Jul 21 15:52:57.195589 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:52:57.195557 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5a8bf5f-1bb5-4379-92d7-ef414993bc66-tls-certs\") pod \"e2e-lifecycle-model-534ea2a9-kserve-84b774c4f5-bz49d\" (UID: \"c5a8bf5f-1bb5-4379-92d7-ef414993bc66\") " pod="ai-tenant-e2e-mt-534ea2a9/e2e-lifecycle-model-534ea2a9-kserve-84b774c4f5-bz49d" Jul 21 15:52:57.195759 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:52:57.195569 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5a8bf5f-1bb5-4379-92d7-ef414993bc66-dshm\") pod \"e2e-lifecycle-model-534ea2a9-kserve-84b774c4f5-bz49d\" (UID: \"c5a8bf5f-1bb5-4379-92d7-ef414993bc66\") " pod="ai-tenant-e2e-mt-534ea2a9/e2e-lifecycle-model-534ea2a9-kserve-84b774c4f5-bz49d" Jul 21 15:52:57.200896 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:52:57.200867 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgjbc\" (UniqueName: \"kubernetes.io/projected/c5a8bf5f-1bb5-4379-92d7-ef414993bc66-kube-api-access-bgjbc\") pod \"e2e-lifecycle-model-534ea2a9-kserve-84b774c4f5-bz49d\" (UID: \"c5a8bf5f-1bb5-4379-92d7-ef414993bc66\") " pod="ai-tenant-e2e-mt-534ea2a9/e2e-lifecycle-model-534ea2a9-kserve-84b774c4f5-bz49d" Jul 21 15:52:57.287769 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:52:57.287720 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-534ea2a9/e2e-lifecycle-model-534ea2a9-kserve-84b774c4f5-bz49d" Jul 21 15:52:57.447123 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:52:57.447095 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-534ea2a9/e2e-lifecycle-model-534ea2a9-kserve-84b774c4f5-bz49d"] Jul 21 15:52:57.449927 ip-10-0-135-176 kubenswrapper[2563]: W0721 15:52:57.449898 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc5a8bf5f_1bb5_4379_92d7_ef414993bc66.slice/crio-c04b2cc12409cfadd8d82d79da1016ea190c74cc8e5254c2596be1af57fef45a WatchSource:0}: Error finding container c04b2cc12409cfadd8d82d79da1016ea190c74cc8e5254c2596be1af57fef45a: Status 404 returned error can't find the container with id c04b2cc12409cfadd8d82d79da1016ea190c74cc8e5254c2596be1af57fef45a Jul 21 15:52:57.990819 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:52:57.990768 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-534ea2a9/e2e-lifecycle-model-534ea2a9-kserve-84b774c4f5-bz49d" event={"ID":"c5a8bf5f-1bb5-4379-92d7-ef414993bc66","Type":"ContainerStarted","Data":"601eb43d2b4cb843e468d95d372c3db8d9265f523e2603197517c33ef79d6909"} Jul 21 15:52:57.991243 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:52:57.990832 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-534ea2a9/e2e-lifecycle-model-534ea2a9-kserve-84b774c4f5-bz49d" event={"ID":"c5a8bf5f-1bb5-4379-92d7-ef414993bc66","Type":"ContainerStarted","Data":"c04b2cc12409cfadd8d82d79da1016ea190c74cc8e5254c2596be1af57fef45a"} Jul 21 15:52:58.011943 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:52:58.011862 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-534ea2a9/e2e-lifecycle-model-534ea2a9-kserve-84b774c4f5-bz49d" podStartSLOduration=2.011835227 podStartE2EDuration="2.011835227s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 15:52:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 15:52:58.008825842 +0000 UTC m=+3210.004611442" watchObservedRunningTime="2026-07-21 15:52:58.011835227 +0000 UTC m=+3210.007620645" Jul 21 15:52:59.050496 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:52:59.050457 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-j4c8p"] Jul 21 15:53:00.918080 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:53:00.918041 2563 patch_prober.go:28] interesting pod/e2e-mt-534ea2a9-openshift-default-548ff8f88c-d78hp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Jul 21 15:53:00.918552 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:53:00.918106 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-534ea2a9-openshift-default-548ff8f88c-d78hp" podUID="29934221-c862-4f86-bdcc-e9e5b87f084a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Jul 21 15:53:01.009672 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:53:01.009639 2563 generic.go:356] "Generic (PLEG): container finished" podID="29934221-c862-4f86-bdcc-e9e5b87f084a" containerID="5e5bf90fd713fd590ac3399cdf7918b20503c430a49747f61d6e45030cce2e36" exitCode=0 Jul 21 15:53:01.009858 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:53:01.009713 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-534ea2a9-openshift-default-548ff8f88c-d78hp" event={"ID":"29934221-c862-4f86-bdcc-e9e5b87f084a","Type":"ContainerDied","Data":"5e5bf90fd713fd590ac3399cdf7918b20503c430a49747f61d6e45030cce2e36"} Jul 21 15:53:01.010140 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:53:01.010126 2563 scope.go:117] "RemoveContainer" containerID="5e5bf90fd713fd590ac3399cdf7918b20503c430a49747f61d6e45030cce2e36" Jul 21 15:53:01.011181 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:53:01.011151 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 21 15:53:01.011279 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:53:01.011241 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 21 15:53:01.011369 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:53:01.011290 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 21 15:53:02.017727 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:53:02.017686 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-534ea2a9-openshift-default-548ff8f88c-d78hp" event={"ID":"29934221-c862-4f86-bdcc-e9e5b87f084a","Type":"ContainerStarted","Data":"3637d217b64999afeb4c6146240388d8f6ff92d71d0116754d76d00f09974f70"} Jul 21 15:53:02.209047 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:53:02.209011 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-534ea2a9-openshift-default-548ff8f88c-d78hp" Jul 21 15:53:02.209306 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:53:02.209277 2563 patch_prober.go:28] interesting pod/e2e-mt-534ea2a9-openshift-default-548ff8f88c-d78hp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Jul 21 15:53:02.209480 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:53:02.209376 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-534ea2a9-openshift-default-548ff8f88c-d78hp" podUID="29934221-c862-4f86-bdcc-e9e5b87f084a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Jul 21 15:53:03.209593 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:53:03.209543 2563 patch_prober.go:28] interesting pod/e2e-mt-534ea2a9-openshift-default-548ff8f88c-d78hp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Jul 21 15:53:03.210084 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:53:03.209616 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-534ea2a9-openshift-default-548ff8f88c-d78hp" podUID="29934221-c862-4f86-bdcc-e9e5b87f084a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Jul 21 15:53:04.209526 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:53:04.209485 2563 patch_prober.go:28] interesting pod/e2e-mt-534ea2a9-openshift-default-548ff8f88c-d78hp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Jul 21 15:53:04.209752 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:53:04.209552 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-534ea2a9-openshift-default-548ff8f88c-d78hp" podUID="29934221-c862-4f86-bdcc-e9e5b87f084a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Jul 21 15:53:05.209485 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:53:05.209438 2563 patch_prober.go:28] interesting pod/e2e-mt-534ea2a9-openshift-default-548ff8f88c-d78hp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Jul 21 15:53:05.209712 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:53:05.209508 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-534ea2a9-openshift-default-548ff8f88c-d78hp" podUID="29934221-c862-4f86-bdcc-e9e5b87f084a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Jul 21 15:53:06.208989 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:53:06.208936 2563 patch_prober.go:28] interesting pod/e2e-mt-534ea2a9-openshift-default-548ff8f88c-d78hp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Jul 21 15:53:06.209595 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:53:06.209048 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-534ea2a9-openshift-default-548ff8f88c-d78hp" podUID="29934221-c862-4f86-bdcc-e9e5b87f084a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Jul 21 15:53:07.208918 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:53:07.208872 2563 patch_prober.go:28] interesting pod/e2e-mt-534ea2a9-openshift-default-548ff8f88c-d78hp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Jul 21 15:53:07.209137 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:53:07.208958 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-534ea2a9-openshift-default-548ff8f88c-d78hp" podUID="29934221-c862-4f86-bdcc-e9e5b87f084a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Jul 21 15:53:07.287855 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:53:07.287815 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-534ea2a9/e2e-lifecycle-model-534ea2a9-kserve-84b774c4f5-bz49d" Jul 21 15:53:07.288190 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:53:07.288173 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-534ea2a9/e2e-lifecycle-model-534ea2a9-kserve-84b774c4f5-bz49d" Jul 21 15:53:07.307563 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:53:07.307528 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-534ea2a9/e2e-lifecycle-model-534ea2a9-kserve-84b774c4f5-bz49d" Jul 21 15:53:08.063355 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:53:08.063307 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-534ea2a9/e2e-lifecycle-model-534ea2a9-kserve-84b774c4f5-bz49d" Jul 21 15:53:08.212957 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:53:08.212921 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-534ea2a9-openshift-default-548ff8f88c-d78hp" Jul 21 15:53:08.213488 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:53:08.213362 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-534ea2a9-openshift-default-548ff8f88c-d78hp" Jul 21 15:53:08.214021 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:53:08.214002 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-534ea2a9-openshift-default-548ff8f88c-d78hp" Jul 21 15:53:20.446305 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:53:20.446219 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-j4c8p"] Jul 21 15:53:38.544675 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:53:38.544633 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-j4c8p"] Jul 21 15:54:28.702367 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:54:28.702335 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kn9fn_02f3d0c6-4116-4134-ba51-1c02c73bef05/ovn-acl-logging/0.log" Jul 21 15:54:28.713696 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:54:28.713672 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kn9fn_02f3d0c6-4116-4134-ba51-1c02c73bef05/ovn-acl-logging/0.log" Jul 21 15:59:28.738008 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:59:28.737953 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kn9fn_02f3d0c6-4116-4134-ba51-1c02c73bef05/ovn-acl-logging/0.log" Jul 21 15:59:28.758312 ip-10-0-135-176 kubenswrapper[2563]: I0721 15:59:28.758279 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kn9fn_02f3d0c6-4116-4134-ba51-1c02c73bef05/ovn-acl-logging/0.log" Jul 21 16:02:35.279011 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:02:35.278922 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-534ea2a9/e2e-lifecycle-model-534ea2a9-kserve-84b774c4f5-bz49d"] Jul 21 16:02:50.300789 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:02:50.300748 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-534ea2a9/e2e-lifecycle-model-534ea2a9-kserve-84b774c4f5-bz49d" podUID="c5a8bf5f-1bb5-4379-92d7-ef414993bc66" containerName="main" containerID="cri-o://601eb43d2b4cb843e468d95d372c3db8d9265f523e2603197517c33ef79d6909" gracePeriod=45 Jul 21 16:02:50.556640 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:02:50.556574 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-534ea2a9/e2e-lifecycle-model-534ea2a9-kserve-84b774c4f5-bz49d" Jul 21 16:02:50.632502 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:02:50.632468 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5a8bf5f-1bb5-4379-92d7-ef414993bc66-tls-certs\") pod \"c5a8bf5f-1bb5-4379-92d7-ef414993bc66\" (UID: \"c5a8bf5f-1bb5-4379-92d7-ef414993bc66\") " Jul 21 16:02:50.632689 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:02:50.632558 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5a8bf5f-1bb5-4379-92d7-ef414993bc66-model-cache\") pod \"c5a8bf5f-1bb5-4379-92d7-ef414993bc66\" (UID: \"c5a8bf5f-1bb5-4379-92d7-ef414993bc66\") " Jul 21 16:02:50.632689 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:02:50.632587 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bgjbc\" (UniqueName: \"kubernetes.io/projected/c5a8bf5f-1bb5-4379-92d7-ef414993bc66-kube-api-access-bgjbc\") pod \"c5a8bf5f-1bb5-4379-92d7-ef414993bc66\" (UID: \"c5a8bf5f-1bb5-4379-92d7-ef414993bc66\") " Jul 21 16:02:50.632689 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:02:50.632618 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5a8bf5f-1bb5-4379-92d7-ef414993bc66-dshm\") pod \"c5a8bf5f-1bb5-4379-92d7-ef414993bc66\" (UID: \"c5a8bf5f-1bb5-4379-92d7-ef414993bc66\") " Jul 21 16:02:50.632689 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:02:50.632642 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5a8bf5f-1bb5-4379-92d7-ef414993bc66-tmp-dir\") pod \"c5a8bf5f-1bb5-4379-92d7-ef414993bc66\" (UID: \"c5a8bf5f-1bb5-4379-92d7-ef414993bc66\") " Jul 21 16:02:50.632901 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:02:50.632802 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c5a8bf5f-1bb5-4379-92d7-ef414993bc66-home\") pod \"c5a8bf5f-1bb5-4379-92d7-ef414993bc66\" (UID: \"c5a8bf5f-1bb5-4379-92d7-ef414993bc66\") " Jul 21 16:02:50.632901 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:02:50.632883 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5a8bf5f-1bb5-4379-92d7-ef414993bc66-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c5a8bf5f-1bb5-4379-92d7-ef414993bc66" (UID: "c5a8bf5f-1bb5-4379-92d7-ef414993bc66"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 16:02:50.632996 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:02:50.632902 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5a8bf5f-1bb5-4379-92d7-ef414993bc66-model-cache" (OuterVolumeSpecName: "model-cache") pod "c5a8bf5f-1bb5-4379-92d7-ef414993bc66" (UID: "c5a8bf5f-1bb5-4379-92d7-ef414993bc66"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 16:02:50.633046 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:02:50.632996 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5a8bf5f-1bb5-4379-92d7-ef414993bc66-home" (OuterVolumeSpecName: "home") pod "c5a8bf5f-1bb5-4379-92d7-ef414993bc66" (UID: "c5a8bf5f-1bb5-4379-92d7-ef414993bc66"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 16:02:50.633228 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:02:50.633200 2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5a8bf5f-1bb5-4379-92d7-ef414993bc66-model-cache\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 16:02:50.633373 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:02:50.633231 2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5a8bf5f-1bb5-4379-92d7-ef414993bc66-tmp-dir\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 16:02:50.633373 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:02:50.633246 2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c5a8bf5f-1bb5-4379-92d7-ef414993bc66-home\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 16:02:50.635051 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:02:50.635015 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5a8bf5f-1bb5-4379-92d7-ef414993bc66-kube-api-access-bgjbc" (OuterVolumeSpecName: "kube-api-access-bgjbc") pod "c5a8bf5f-1bb5-4379-92d7-ef414993bc66" (UID: "c5a8bf5f-1bb5-4379-92d7-ef414993bc66"). InnerVolumeSpecName "kube-api-access-bgjbc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 16:02:50.635051 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:02:50.635028 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5a8bf5f-1bb5-4379-92d7-ef414993bc66-dshm" (OuterVolumeSpecName: "dshm") pod "c5a8bf5f-1bb5-4379-92d7-ef414993bc66" (UID: "c5a8bf5f-1bb5-4379-92d7-ef414993bc66"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 16:02:50.635051 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:02:50.635031 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5a8bf5f-1bb5-4379-92d7-ef414993bc66-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c5a8bf5f-1bb5-4379-92d7-ef414993bc66" (UID: "c5a8bf5f-1bb5-4379-92d7-ef414993bc66"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 16:02:50.734093 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:02:50.734053 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgjbc\" (UniqueName: \"kubernetes.io/projected/c5a8bf5f-1bb5-4379-92d7-ef414993bc66-kube-api-access-bgjbc\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 16:02:50.734093 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:02:50.734086 2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5a8bf5f-1bb5-4379-92d7-ef414993bc66-dshm\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 16:02:50.734093 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:02:50.734098 2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5a8bf5f-1bb5-4379-92d7-ef414993bc66-tls-certs\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 16:02:51.409818 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:02:51.409783 2563 generic.go:356] "Generic (PLEG): container finished" podID="c5a8bf5f-1bb5-4379-92d7-ef414993bc66" containerID="601eb43d2b4cb843e468d95d372c3db8d9265f523e2603197517c33ef79d6909" exitCode=0 Jul 21 16:02:51.410231 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:02:51.409832 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-534ea2a9/e2e-lifecycle-model-534ea2a9-kserve-84b774c4f5-bz49d" Jul 21 16:02:51.410231 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:02:51.409871 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-534ea2a9/e2e-lifecycle-model-534ea2a9-kserve-84b774c4f5-bz49d" event={"ID":"c5a8bf5f-1bb5-4379-92d7-ef414993bc66","Type":"ContainerDied","Data":"601eb43d2b4cb843e468d95d372c3db8d9265f523e2603197517c33ef79d6909"} Jul 21 16:02:51.410231 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:02:51.409915 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-534ea2a9/e2e-lifecycle-model-534ea2a9-kserve-84b774c4f5-bz49d" event={"ID":"c5a8bf5f-1bb5-4379-92d7-ef414993bc66","Type":"ContainerDied","Data":"c04b2cc12409cfadd8d82d79da1016ea190c74cc8e5254c2596be1af57fef45a"} Jul 21 16:02:51.410231 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:02:51.409938 2563 scope.go:117] "RemoveContainer" containerID="601eb43d2b4cb843e468d95d372c3db8d9265f523e2603197517c33ef79d6909" Jul 21 16:02:51.422919 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:02:51.422894 2563 scope.go:117] "RemoveContainer" containerID="601eb43d2b4cb843e468d95d372c3db8d9265f523e2603197517c33ef79d6909" Jul 21 16:02:51.423236 ip-10-0-135-176 kubenswrapper[2563]: E0721 16:02:51.423213 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"601eb43d2b4cb843e468d95d372c3db8d9265f523e2603197517c33ef79d6909\": container with ID starting with 601eb43d2b4cb843e468d95d372c3db8d9265f523e2603197517c33ef79d6909 not found: ID does not exist" containerID="601eb43d2b4cb843e468d95d372c3db8d9265f523e2603197517c33ef79d6909" Jul 21 16:02:51.423287 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:02:51.423252 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"601eb43d2b4cb843e468d95d372c3db8d9265f523e2603197517c33ef79d6909"} err="failed to get container status \"601eb43d2b4cb843e468d95d372c3db8d9265f523e2603197517c33ef79d6909\": rpc error: code = NotFound desc = could not find container \"601eb43d2b4cb843e468d95d372c3db8d9265f523e2603197517c33ef79d6909\": container with ID starting with 601eb43d2b4cb843e468d95d372c3db8d9265f523e2603197517c33ef79d6909 not found: ID does not exist" Jul 21 16:02:51.434697 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:02:51.434662 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-534ea2a9/e2e-lifecycle-model-534ea2a9-kserve-84b774c4f5-bz49d"] Jul 21 16:02:51.440683 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:02:51.440650 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-534ea2a9/e2e-lifecycle-model-534ea2a9-kserve-84b774c4f5-bz49d"] Jul 21 16:02:52.481859 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:02:52.481825 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c5a8bf5f-1bb5-4379-92d7-ef414993bc66" path="/var/lib/kubelet/pods/c5a8bf5f-1bb5-4379-92d7-ef414993bc66/volumes" Jul 21 16:03:18.327813 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:03:18.327772 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-534ea2a9-openshift-default-548ff8f88c-d78hp"] Jul 21 16:03:18.328470 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:03:18.328004 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-534ea2a9-openshift-default-548ff8f88c-d78hp" podUID="29934221-c862-4f86-bdcc-e9e5b87f084a" containerName="istio-proxy" containerID="cri-o://3637d217b64999afeb4c6146240388d8f6ff92d71d0116754d76d00f09974f70" gracePeriod=30 Jul 21 16:03:21.448504 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:03:21.448459 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-j4c8p"] Jul 21 16:03:23.216425 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:03:23.216381 2563 patch_prober.go:28] interesting pod/e2e-mt-534ea2a9-openshift-default-548ff8f88c-d78hp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 21 16:03:23.216847 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:03:23.216453 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-534ea2a9-openshift-default-548ff8f88c-d78hp" podUID="29934221-c862-4f86-bdcc-e9e5b87f084a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 21 16:03:23.594097 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:03:23.594064 2563 generic.go:356] "Generic (PLEG): container finished" podID="29934221-c862-4f86-bdcc-e9e5b87f084a" containerID="3637d217b64999afeb4c6146240388d8f6ff92d71d0116754d76d00f09974f70" exitCode=0 Jul 21 16:03:23.594276 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:03:23.594136 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-534ea2a9-openshift-default-548ff8f88c-d78hp" event={"ID":"29934221-c862-4f86-bdcc-e9e5b87f084a","Type":"ContainerDied","Data":"3637d217b64999afeb4c6146240388d8f6ff92d71d0116754d76d00f09974f70"} Jul 21 16:03:23.594276 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:03:23.594196 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-534ea2a9-openshift-default-548ff8f88c-d78hp" event={"ID":"29934221-c862-4f86-bdcc-e9e5b87f084a","Type":"ContainerDied","Data":"bfc6975c721ea3089c2b6badad012e4f731d34506938ba1d96102c5d7435af70"} Jul 21 16:03:23.594276 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:03:23.594217 2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bfc6975c721ea3089c2b6badad012e4f731d34506938ba1d96102c5d7435af70" Jul 21 16:03:23.594276 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:03:23.594231 2563 scope.go:117] "RemoveContainer" containerID="5e5bf90fd713fd590ac3399cdf7918b20503c430a49747f61d6e45030cce2e36" Jul 21 16:03:23.599073 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:03:23.599053 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-534ea2a9-openshift-default-548ff8f88c-d78hp" Jul 21 16:03:23.749987 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:03:23.749933 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29934221-c862-4f86-bdcc-e9e5b87f084a-workload-certs\") pod \"29934221-c862-4f86-bdcc-e9e5b87f084a\" (UID: \"29934221-c862-4f86-bdcc-e9e5b87f084a\") " Jul 21 16:03:23.749987 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:03:23.749989 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29934221-c862-4f86-bdcc-e9e5b87f084a-workload-socket\") pod \"29934221-c862-4f86-bdcc-e9e5b87f084a\" (UID: \"29934221-c862-4f86-bdcc-e9e5b87f084a\") " Jul 21 16:03:23.750255 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:03:23.750018 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29934221-c862-4f86-bdcc-e9e5b87f084a-istio-envoy\") pod \"29934221-c862-4f86-bdcc-e9e5b87f084a\" (UID: \"29934221-c862-4f86-bdcc-e9e5b87f084a\") " Jul 21 16:03:23.750255 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:03:23.750062 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29934221-c862-4f86-bdcc-e9e5b87f084a-istio-podinfo\") pod \"29934221-c862-4f86-bdcc-e9e5b87f084a\" (UID: \"29934221-c862-4f86-bdcc-e9e5b87f084a\") " Jul 21 16:03:23.750255 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:03:23.750139 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29934221-c862-4f86-bdcc-e9e5b87f084a-credential-socket\") pod \"29934221-c862-4f86-bdcc-e9e5b87f084a\" (UID: \"29934221-c862-4f86-bdcc-e9e5b87f084a\") " Jul 21 16:03:23.750255 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:03:23.750165 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29934221-c862-4f86-bdcc-e9e5b87f084a-istiod-ca-cert\") pod \"29934221-c862-4f86-bdcc-e9e5b87f084a\" (UID: \"29934221-c862-4f86-bdcc-e9e5b87f084a\") " Jul 21 16:03:23.750255 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:03:23.750194 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29934221-c862-4f86-bdcc-e9e5b87f084a-istio-token\") pod \"29934221-c862-4f86-bdcc-e9e5b87f084a\" (UID: \"29934221-c862-4f86-bdcc-e9e5b87f084a\") " Jul 21 16:03:23.750647 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:03:23.750369 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vrvh9\" (UniqueName: \"kubernetes.io/projected/29934221-c862-4f86-bdcc-e9e5b87f084a-kube-api-access-vrvh9\") pod \"29934221-c862-4f86-bdcc-e9e5b87f084a\" (UID: \"29934221-c862-4f86-bdcc-e9e5b87f084a\") " Jul 21 16:03:23.750647 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:03:23.750376 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29934221-c862-4f86-bdcc-e9e5b87f084a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "29934221-c862-4f86-bdcc-e9e5b87f084a" (UID: "29934221-c862-4f86-bdcc-e9e5b87f084a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 16:03:23.750647 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:03:23.750397 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29934221-c862-4f86-bdcc-e9e5b87f084a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "29934221-c862-4f86-bdcc-e9e5b87f084a" (UID: "29934221-c862-4f86-bdcc-e9e5b87f084a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 16:03:23.750647 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:03:23.750413 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29934221-c862-4f86-bdcc-e9e5b87f084a-istio-data\") pod \"29934221-c862-4f86-bdcc-e9e5b87f084a\" (UID: \"29934221-c862-4f86-bdcc-e9e5b87f084a\") " Jul 21 16:03:23.750647 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:03:23.750518 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29934221-c862-4f86-bdcc-e9e5b87f084a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "29934221-c862-4f86-bdcc-e9e5b87f084a" (UID: "29934221-c862-4f86-bdcc-e9e5b87f084a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 16:03:23.750928 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:03:23.750616 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29934221-c862-4f86-bdcc-e9e5b87f084a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "29934221-c862-4f86-bdcc-e9e5b87f084a" (UID: "29934221-c862-4f86-bdcc-e9e5b87f084a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 16:03:23.750928 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:03:23.750705 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29934221-c862-4f86-bdcc-e9e5b87f084a-istio-data" (OuterVolumeSpecName: "istio-data") pod "29934221-c862-4f86-bdcc-e9e5b87f084a" (UID: "29934221-c862-4f86-bdcc-e9e5b87f084a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 16:03:23.750928 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:03:23.750799 2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29934221-c862-4f86-bdcc-e9e5b87f084a-workload-certs\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 16:03:23.750928 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:03:23.750817 2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29934221-c862-4f86-bdcc-e9e5b87f084a-workload-socket\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 16:03:23.750928 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:03:23.750830 2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29934221-c862-4f86-bdcc-e9e5b87f084a-credential-socket\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 16:03:23.750928 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:03:23.750843 2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29934221-c862-4f86-bdcc-e9e5b87f084a-istiod-ca-cert\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 16:03:23.750928 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:03:23.750856 2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29934221-c862-4f86-bdcc-e9e5b87f084a-istio-data\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 16:03:23.752629 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:03:23.752563 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29934221-c862-4f86-bdcc-e9e5b87f084a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "29934221-c862-4f86-bdcc-e9e5b87f084a" (UID: "29934221-c862-4f86-bdcc-e9e5b87f084a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 16:03:23.752629 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:03:23.752569 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/29934221-c862-4f86-bdcc-e9e5b87f084a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "29934221-c862-4f86-bdcc-e9e5b87f084a" (UID: "29934221-c862-4f86-bdcc-e9e5b87f084a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 21 16:03:23.752777 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:03:23.752651 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29934221-c862-4f86-bdcc-e9e5b87f084a-kube-api-access-vrvh9" (OuterVolumeSpecName: "kube-api-access-vrvh9") pod "29934221-c862-4f86-bdcc-e9e5b87f084a" (UID: "29934221-c862-4f86-bdcc-e9e5b87f084a"). InnerVolumeSpecName "kube-api-access-vrvh9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 16:03:23.752963 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:03:23.752943 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29934221-c862-4f86-bdcc-e9e5b87f084a-istio-token" (OuterVolumeSpecName: "istio-token") pod "29934221-c862-4f86-bdcc-e9e5b87f084a" (UID: "29934221-c862-4f86-bdcc-e9e5b87f084a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 16:03:23.851291 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:03:23.851258 2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29934221-c862-4f86-bdcc-e9e5b87f084a-istio-token\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 16:03:23.851291 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:03:23.851287 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrvh9\" (UniqueName: \"kubernetes.io/projected/29934221-c862-4f86-bdcc-e9e5b87f084a-kube-api-access-vrvh9\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 16:03:23.851291 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:03:23.851298 2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29934221-c862-4f86-bdcc-e9e5b87f084a-istio-envoy\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 16:03:23.851576 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:03:23.851309 2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29934221-c862-4f86-bdcc-e9e5b87f084a-istio-podinfo\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 16:03:24.601579 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:03:24.601544 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-534ea2a9-openshift-default-548ff8f88c-d78hp" Jul 21 16:03:24.660345 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:03:24.660297 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-534ea2a9-openshift-default-548ff8f88c-d78hp"] Jul 21 16:03:24.663689 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:03:24.663657 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-534ea2a9-openshift-default-548ff8f88c-d78hp"] Jul 21 16:03:26.480433 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:03:26.480396 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="29934221-c862-4f86-bdcc-e9e5b87f084a" path="/var/lib/kubelet/pods/29934221-c862-4f86-bdcc-e9e5b87f084a/volumes" Jul 21 16:03:29.083064 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:03:29.083029 2563 scope.go:117] "RemoveContainer" containerID="3637d217b64999afeb4c6146240388d8f6ff92d71d0116754d76d00f09974f70" Jul 21 16:03:30.842384 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:03:30.842341 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-j4c8p"] Jul 21 16:04:13.364076 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:13.364036 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-682cb4-openshift-default-788d8bdd7-dt5s9"] Jul 21 16:04:13.364557 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:13.364542 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5a8bf5f-1bb5-4379-92d7-ef414993bc66" containerName="main" Jul 21 16:04:13.364603 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:13.364560 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5a8bf5f-1bb5-4379-92d7-ef414993bc66" containerName="main" Jul 21 16:04:13.364603 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:13.364576 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29934221-c862-4f86-bdcc-e9e5b87f084a" containerName="istio-proxy" Jul 21 16:04:13.364603 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:13.364584 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="29934221-c862-4f86-bdcc-e9e5b87f084a" containerName="istio-proxy" Jul 21 16:04:13.364701 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:13.364662 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="29934221-c862-4f86-bdcc-e9e5b87f084a" containerName="istio-proxy" Jul 21 16:04:13.364701 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:13.364678 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="29934221-c862-4f86-bdcc-e9e5b87f084a" containerName="istio-proxy" Jul 21 16:04:13.364701 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:13.364688 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5a8bf5f-1bb5-4379-92d7-ef414993bc66" containerName="main" Jul 21 16:04:13.367913 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:13.367894 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-682cb4-openshift-default-788d8bdd7-dt5s9" Jul 21 16:04:13.370500 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:13.370478 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-682cb4-openshift-default-dockercfg-8p9qz\"" Jul 21 16:04:13.378559 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:13.378527 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-682cb4-openshift-default-788d8bdd7-dt5s9"] Jul 21 16:04:13.502862 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:13.502820 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbbade83-f99d-4539-bb0e-4f2fc9d711ff-istio-token\") pod \"e2e-inf-a-682cb4-openshift-default-788d8bdd7-dt5s9\" (UID: \"dbbade83-f99d-4539-bb0e-4f2fc9d711ff\") " pod="openshift-ingress/e2e-inf-a-682cb4-openshift-default-788d8bdd7-dt5s9" Jul 21 16:04:13.503049 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:13.502865 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbbade83-f99d-4539-bb0e-4f2fc9d711ff-workload-socket\") pod \"e2e-inf-a-682cb4-openshift-default-788d8bdd7-dt5s9\" (UID: \"dbbade83-f99d-4539-bb0e-4f2fc9d711ff\") " pod="openshift-ingress/e2e-inf-a-682cb4-openshift-default-788d8bdd7-dt5s9" Jul 21 16:04:13.503049 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:13.502896 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbbade83-f99d-4539-bb0e-4f2fc9d711ff-workload-certs\") pod \"e2e-inf-a-682cb4-openshift-default-788d8bdd7-dt5s9\" (UID: \"dbbade83-f99d-4539-bb0e-4f2fc9d711ff\") " pod="openshift-ingress/e2e-inf-a-682cb4-openshift-default-788d8bdd7-dt5s9" Jul 21 16:04:13.503049 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:13.502983 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mp94\" (UniqueName: \"kubernetes.io/projected/dbbade83-f99d-4539-bb0e-4f2fc9d711ff-kube-api-access-9mp94\") pod \"e2e-inf-a-682cb4-openshift-default-788d8bdd7-dt5s9\" (UID: \"dbbade83-f99d-4539-bb0e-4f2fc9d711ff\") " pod="openshift-ingress/e2e-inf-a-682cb4-openshift-default-788d8bdd7-dt5s9" Jul 21 16:04:13.503049 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:13.503033 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbbade83-f99d-4539-bb0e-4f2fc9d711ff-credential-socket\") pod \"e2e-inf-a-682cb4-openshift-default-788d8bdd7-dt5s9\" (UID: \"dbbade83-f99d-4539-bb0e-4f2fc9d711ff\") " pod="openshift-ingress/e2e-inf-a-682cb4-openshift-default-788d8bdd7-dt5s9" Jul 21 16:04:13.503049 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:13.503050 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbbade83-f99d-4539-bb0e-4f2fc9d711ff-istiod-ca-cert\") pod \"e2e-inf-a-682cb4-openshift-default-788d8bdd7-dt5s9\" (UID: \"dbbade83-f99d-4539-bb0e-4f2fc9d711ff\") " pod="openshift-ingress/e2e-inf-a-682cb4-openshift-default-788d8bdd7-dt5s9" Jul 21 16:04:13.503228 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:13.503068 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbbade83-f99d-4539-bb0e-4f2fc9d711ff-istio-data\") pod \"e2e-inf-a-682cb4-openshift-default-788d8bdd7-dt5s9\" (UID: \"dbbade83-f99d-4539-bb0e-4f2fc9d711ff\") " pod="openshift-ingress/e2e-inf-a-682cb4-openshift-default-788d8bdd7-dt5s9" Jul 21 16:04:13.503228 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:13.503114 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbbade83-f99d-4539-bb0e-4f2fc9d711ff-istio-envoy\") pod \"e2e-inf-a-682cb4-openshift-default-788d8bdd7-dt5s9\" (UID: \"dbbade83-f99d-4539-bb0e-4f2fc9d711ff\") " pod="openshift-ingress/e2e-inf-a-682cb4-openshift-default-788d8bdd7-dt5s9" Jul 21 16:04:13.503228 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:13.503147 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbbade83-f99d-4539-bb0e-4f2fc9d711ff-istio-podinfo\") pod \"e2e-inf-a-682cb4-openshift-default-788d8bdd7-dt5s9\" (UID: \"dbbade83-f99d-4539-bb0e-4f2fc9d711ff\") " pod="openshift-ingress/e2e-inf-a-682cb4-openshift-default-788d8bdd7-dt5s9" Jul 21 16:04:13.604133 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:13.604098 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbbade83-f99d-4539-bb0e-4f2fc9d711ff-istio-envoy\") pod \"e2e-inf-a-682cb4-openshift-default-788d8bdd7-dt5s9\" (UID: \"dbbade83-f99d-4539-bb0e-4f2fc9d711ff\") " pod="openshift-ingress/e2e-inf-a-682cb4-openshift-default-788d8bdd7-dt5s9" Jul 21 16:04:13.604380 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:13.604148 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbbade83-f99d-4539-bb0e-4f2fc9d711ff-istio-podinfo\") pod \"e2e-inf-a-682cb4-openshift-default-788d8bdd7-dt5s9\" (UID: \"dbbade83-f99d-4539-bb0e-4f2fc9d711ff\") " pod="openshift-ingress/e2e-inf-a-682cb4-openshift-default-788d8bdd7-dt5s9" Jul 21 16:04:13.604380 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:13.604170 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbbade83-f99d-4539-bb0e-4f2fc9d711ff-istio-token\") pod \"e2e-inf-a-682cb4-openshift-default-788d8bdd7-dt5s9\" (UID: \"dbbade83-f99d-4539-bb0e-4f2fc9d711ff\") " pod="openshift-ingress/e2e-inf-a-682cb4-openshift-default-788d8bdd7-dt5s9" Jul 21 16:04:13.604380 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:13.604195 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbbade83-f99d-4539-bb0e-4f2fc9d711ff-workload-socket\") pod \"e2e-inf-a-682cb4-openshift-default-788d8bdd7-dt5s9\" (UID: \"dbbade83-f99d-4539-bb0e-4f2fc9d711ff\") " pod="openshift-ingress/e2e-inf-a-682cb4-openshift-default-788d8bdd7-dt5s9" Jul 21 16:04:13.604380 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:13.604211 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbbade83-f99d-4539-bb0e-4f2fc9d711ff-workload-certs\") pod \"e2e-inf-a-682cb4-openshift-default-788d8bdd7-dt5s9\" (UID: \"dbbade83-f99d-4539-bb0e-4f2fc9d711ff\") " pod="openshift-ingress/e2e-inf-a-682cb4-openshift-default-788d8bdd7-dt5s9" Jul 21 16:04:13.604380 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:13.604260 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mp94\" (UniqueName: \"kubernetes.io/projected/dbbade83-f99d-4539-bb0e-4f2fc9d711ff-kube-api-access-9mp94\") pod \"e2e-inf-a-682cb4-openshift-default-788d8bdd7-dt5s9\" (UID: \"dbbade83-f99d-4539-bb0e-4f2fc9d711ff\") " pod="openshift-ingress/e2e-inf-a-682cb4-openshift-default-788d8bdd7-dt5s9" Jul 21 16:04:13.604380 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:13.604353 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbbade83-f99d-4539-bb0e-4f2fc9d711ff-credential-socket\") pod \"e2e-inf-a-682cb4-openshift-default-788d8bdd7-dt5s9\" (UID: \"dbbade83-f99d-4539-bb0e-4f2fc9d711ff\") " pod="openshift-ingress/e2e-inf-a-682cb4-openshift-default-788d8bdd7-dt5s9" Jul 21 16:04:13.604380 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:13.604380 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbbade83-f99d-4539-bb0e-4f2fc9d711ff-istiod-ca-cert\") pod \"e2e-inf-a-682cb4-openshift-default-788d8bdd7-dt5s9\" (UID: \"dbbade83-f99d-4539-bb0e-4f2fc9d711ff\") " pod="openshift-ingress/e2e-inf-a-682cb4-openshift-default-788d8bdd7-dt5s9" Jul 21 16:04:13.604812 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:13.604403 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbbade83-f99d-4539-bb0e-4f2fc9d711ff-istio-data\") pod \"e2e-inf-a-682cb4-openshift-default-788d8bdd7-dt5s9\" (UID: \"dbbade83-f99d-4539-bb0e-4f2fc9d711ff\") " pod="openshift-ingress/e2e-inf-a-682cb4-openshift-default-788d8bdd7-dt5s9" Jul 21 16:04:13.604812 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:13.604691 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbbade83-f99d-4539-bb0e-4f2fc9d711ff-workload-socket\") pod \"e2e-inf-a-682cb4-openshift-default-788d8bdd7-dt5s9\" (UID: \"dbbade83-f99d-4539-bb0e-4f2fc9d711ff\") " pod="openshift-ingress/e2e-inf-a-682cb4-openshift-default-788d8bdd7-dt5s9" Jul 21 16:04:13.604812 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:13.604719 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbbade83-f99d-4539-bb0e-4f2fc9d711ff-istio-data\") pod \"e2e-inf-a-682cb4-openshift-default-788d8bdd7-dt5s9\" (UID: \"dbbade83-f99d-4539-bb0e-4f2fc9d711ff\") " pod="openshift-ingress/e2e-inf-a-682cb4-openshift-default-788d8bdd7-dt5s9" Jul 21 16:04:13.604812 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:13.604789 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbbade83-f99d-4539-bb0e-4f2fc9d711ff-credential-socket\") pod \"e2e-inf-a-682cb4-openshift-default-788d8bdd7-dt5s9\" (UID: \"dbbade83-f99d-4539-bb0e-4f2fc9d711ff\") " pod="openshift-ingress/e2e-inf-a-682cb4-openshift-default-788d8bdd7-dt5s9" Jul 21 16:04:13.604978 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:13.604815 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbbade83-f99d-4539-bb0e-4f2fc9d711ff-workload-certs\") pod \"e2e-inf-a-682cb4-openshift-default-788d8bdd7-dt5s9\" (UID: \"dbbade83-f99d-4539-bb0e-4f2fc9d711ff\") " pod="openshift-ingress/e2e-inf-a-682cb4-openshift-default-788d8bdd7-dt5s9" Jul 21 16:04:13.605198 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:13.605171 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbbade83-f99d-4539-bb0e-4f2fc9d711ff-istiod-ca-cert\") pod \"e2e-inf-a-682cb4-openshift-default-788d8bdd7-dt5s9\" (UID: \"dbbade83-f99d-4539-bb0e-4f2fc9d711ff\") " pod="openshift-ingress/e2e-inf-a-682cb4-openshift-default-788d8bdd7-dt5s9" Jul 21 16:04:13.606660 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:13.606634 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbbade83-f99d-4539-bb0e-4f2fc9d711ff-istio-envoy\") pod \"e2e-inf-a-682cb4-openshift-default-788d8bdd7-dt5s9\" (UID: \"dbbade83-f99d-4539-bb0e-4f2fc9d711ff\") " pod="openshift-ingress/e2e-inf-a-682cb4-openshift-default-788d8bdd7-dt5s9" Jul 21 16:04:13.607220 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:13.607203 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbbade83-f99d-4539-bb0e-4f2fc9d711ff-istio-podinfo\") pod \"e2e-inf-a-682cb4-openshift-default-788d8bdd7-dt5s9\" (UID: \"dbbade83-f99d-4539-bb0e-4f2fc9d711ff\") " pod="openshift-ingress/e2e-inf-a-682cb4-openshift-default-788d8bdd7-dt5s9" Jul 21 16:04:13.617370 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:13.617279 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbbade83-f99d-4539-bb0e-4f2fc9d711ff-istio-token\") pod \"e2e-inf-a-682cb4-openshift-default-788d8bdd7-dt5s9\" (UID: \"dbbade83-f99d-4539-bb0e-4f2fc9d711ff\") " pod="openshift-ingress/e2e-inf-a-682cb4-openshift-default-788d8bdd7-dt5s9" Jul 21 16:04:13.617370 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:13.617343 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mp94\" (UniqueName: \"kubernetes.io/projected/dbbade83-f99d-4539-bb0e-4f2fc9d711ff-kube-api-access-9mp94\") pod \"e2e-inf-a-682cb4-openshift-default-788d8bdd7-dt5s9\" (UID: \"dbbade83-f99d-4539-bb0e-4f2fc9d711ff\") " pod="openshift-ingress/e2e-inf-a-682cb4-openshift-default-788d8bdd7-dt5s9" Jul 21 16:04:13.680343 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:13.680298 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-682cb4-openshift-default-788d8bdd7-dt5s9" Jul 21 16:04:13.830891 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:13.830865 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-682cb4-openshift-default-788d8bdd7-dt5s9"] Jul 21 16:04:13.833031 ip-10-0-135-176 kubenswrapper[2563]: W0721 16:04:13.833009 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbbade83_f99d_4539_bb0e_4f2fc9d711ff.slice/crio-d40699b4ba38d9471c5f7033cd72e5e17fe9947523d8d36f5e3621e0bcbb9e5b WatchSource:0}: Error finding container d40699b4ba38d9471c5f7033cd72e5e17fe9947523d8d36f5e3621e0bcbb9e5b: Status 404 returned error can't find the container with id d40699b4ba38d9471c5f7033cd72e5e17fe9947523d8d36f5e3621e0bcbb9e5b Jul 21 16:04:13.835131 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:13.835098 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 21 16:04:13.835213 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:13.835162 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 21 16:04:13.835213 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:13.835191 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 21 16:04:13.894769 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:13.894733 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-682cb4-openshift-default-788d8bdd7-dt5s9" event={"ID":"dbbade83-f99d-4539-bb0e-4f2fc9d711ff","Type":"ContainerStarted","Data":"d40699b4ba38d9471c5f7033cd72e5e17fe9947523d8d36f5e3621e0bcbb9e5b"} Jul 21 16:04:14.902158 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:14.902126 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-682cb4-openshift-default-788d8bdd7-dt5s9" event={"ID":"dbbade83-f99d-4539-bb0e-4f2fc9d711ff","Type":"ContainerStarted","Data":"74b8be63f4bc36c37ce8dc06069b08088813453542ff2b23aa71acada6379ab6"} Jul 21 16:04:14.923405 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:14.923356 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-682cb4-openshift-default-788d8bdd7-dt5s9" podStartSLOduration=1.923338625 podStartE2EDuration="1.923338625s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 16:04:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 16:04:14.921069857 +0000 UTC m=+3886.916855304" watchObservedRunningTime="2026-07-21 16:04:14.923338625 +0000 UTC m=+3886.919124036" Jul 21 16:04:15.681439 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:15.681389 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-682cb4-openshift-default-788d8bdd7-dt5s9" Jul 21 16:04:15.686846 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:15.686818 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-682cb4-openshift-default-788d8bdd7-dt5s9" Jul 21 16:04:15.907976 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:15.907945 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-682cb4-openshift-default-788d8bdd7-dt5s9" Jul 21 16:04:15.909084 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:15.909065 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-682cb4-openshift-default-788d8bdd7-dt5s9" Jul 21 16:04:20.245982 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:20.245945 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-j4c8p"] Jul 21 16:04:22.099741 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:22.099680 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-26bb92-openshift-default-7b4d9d44f7-6kbfr"] Jul 21 16:04:22.118357 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:22.113210 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29934221-c862-4f86-bdcc-e9e5b87f084a" containerName="istio-proxy" Jul 21 16:04:22.118629 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:22.118602 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="29934221-c862-4f86-bdcc-e9e5b87f084a" containerName="istio-proxy" Jul 21 16:04:22.123925 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:22.123898 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-26bb92-openshift-default-7b4d9d44f7-6kbfr"] Jul 21 16:04:22.124081 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:22.123999 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-26bb92-openshift-default-7b4d9d44f7-6kbfr" Jul 21 16:04:22.126921 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:22.126879 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-26bb92-openshift-default-dockercfg-h9kzb\"" Jul 21 16:04:22.303433 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:22.303381 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3707250-ffeb-4dc2-a97d-a20d0fe845a4-istio-envoy\") pod \"e2e-inf-b-26bb92-openshift-default-7b4d9d44f7-6kbfr\" (UID: \"d3707250-ffeb-4dc2-a97d-a20d0fe845a4\") " pod="openshift-ingress/e2e-inf-b-26bb92-openshift-default-7b4d9d44f7-6kbfr" Jul 21 16:04:22.304008 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:22.303978 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dt6qc\" (UniqueName: \"kubernetes.io/projected/d3707250-ffeb-4dc2-a97d-a20d0fe845a4-kube-api-access-dt6qc\") pod \"e2e-inf-b-26bb92-openshift-default-7b4d9d44f7-6kbfr\" (UID: \"d3707250-ffeb-4dc2-a97d-a20d0fe845a4\") " pod="openshift-ingress/e2e-inf-b-26bb92-openshift-default-7b4d9d44f7-6kbfr" Jul 21 16:04:22.304221 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:22.304204 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3707250-ffeb-4dc2-a97d-a20d0fe845a4-credential-socket\") pod \"e2e-inf-b-26bb92-openshift-default-7b4d9d44f7-6kbfr\" (UID: \"d3707250-ffeb-4dc2-a97d-a20d0fe845a4\") " pod="openshift-ingress/e2e-inf-b-26bb92-openshift-default-7b4d9d44f7-6kbfr" Jul 21 16:04:22.304437 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:22.304419 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3707250-ffeb-4dc2-a97d-a20d0fe845a4-workload-certs\") pod \"e2e-inf-b-26bb92-openshift-default-7b4d9d44f7-6kbfr\" (UID: \"d3707250-ffeb-4dc2-a97d-a20d0fe845a4\") " pod="openshift-ingress/e2e-inf-b-26bb92-openshift-default-7b4d9d44f7-6kbfr" Jul 21 16:04:22.304595 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:22.304580 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3707250-ffeb-4dc2-a97d-a20d0fe845a4-istio-token\") pod \"e2e-inf-b-26bb92-openshift-default-7b4d9d44f7-6kbfr\" (UID: \"d3707250-ffeb-4dc2-a97d-a20d0fe845a4\") " pod="openshift-ingress/e2e-inf-b-26bb92-openshift-default-7b4d9d44f7-6kbfr" Jul 21 16:04:22.304741 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:22.304724 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3707250-ffeb-4dc2-a97d-a20d0fe845a4-istiod-ca-cert\") pod \"e2e-inf-b-26bb92-openshift-default-7b4d9d44f7-6kbfr\" (UID: \"d3707250-ffeb-4dc2-a97d-a20d0fe845a4\") " pod="openshift-ingress/e2e-inf-b-26bb92-openshift-default-7b4d9d44f7-6kbfr" Jul 21 16:04:22.304910 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:22.304894 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3707250-ffeb-4dc2-a97d-a20d0fe845a4-istio-data\") pod \"e2e-inf-b-26bb92-openshift-default-7b4d9d44f7-6kbfr\" (UID: \"d3707250-ffeb-4dc2-a97d-a20d0fe845a4\") " pod="openshift-ingress/e2e-inf-b-26bb92-openshift-default-7b4d9d44f7-6kbfr" Jul 21 16:04:22.305076 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:22.305058 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3707250-ffeb-4dc2-a97d-a20d0fe845a4-istio-podinfo\") pod \"e2e-inf-b-26bb92-openshift-default-7b4d9d44f7-6kbfr\" (UID: \"d3707250-ffeb-4dc2-a97d-a20d0fe845a4\") " pod="openshift-ingress/e2e-inf-b-26bb92-openshift-default-7b4d9d44f7-6kbfr" Jul 21 16:04:22.305267 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:22.305248 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3707250-ffeb-4dc2-a97d-a20d0fe845a4-workload-socket\") pod \"e2e-inf-b-26bb92-openshift-default-7b4d9d44f7-6kbfr\" (UID: \"d3707250-ffeb-4dc2-a97d-a20d0fe845a4\") " pod="openshift-ingress/e2e-inf-b-26bb92-openshift-default-7b4d9d44f7-6kbfr" Jul 21 16:04:22.406430 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:22.406292 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3707250-ffeb-4dc2-a97d-a20d0fe845a4-istio-data\") pod \"e2e-inf-b-26bb92-openshift-default-7b4d9d44f7-6kbfr\" (UID: \"d3707250-ffeb-4dc2-a97d-a20d0fe845a4\") " pod="openshift-ingress/e2e-inf-b-26bb92-openshift-default-7b4d9d44f7-6kbfr" Jul 21 16:04:22.407304 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:22.407175 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3707250-ffeb-4dc2-a97d-a20d0fe845a4-istio-data\") pod \"e2e-inf-b-26bb92-openshift-default-7b4d9d44f7-6kbfr\" (UID: \"d3707250-ffeb-4dc2-a97d-a20d0fe845a4\") " pod="openshift-ingress/e2e-inf-b-26bb92-openshift-default-7b4d9d44f7-6kbfr" Jul 21 16:04:22.407527 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:22.407262 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3707250-ffeb-4dc2-a97d-a20d0fe845a4-istio-podinfo\") pod \"e2e-inf-b-26bb92-openshift-default-7b4d9d44f7-6kbfr\" (UID: \"d3707250-ffeb-4dc2-a97d-a20d0fe845a4\") " pod="openshift-ingress/e2e-inf-b-26bb92-openshift-default-7b4d9d44f7-6kbfr" Jul 21 16:04:22.408295 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:22.408273 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3707250-ffeb-4dc2-a97d-a20d0fe845a4-workload-socket\") pod \"e2e-inf-b-26bb92-openshift-default-7b4d9d44f7-6kbfr\" (UID: \"d3707250-ffeb-4dc2-a97d-a20d0fe845a4\") " pod="openshift-ingress/e2e-inf-b-26bb92-openshift-default-7b4d9d44f7-6kbfr" Jul 21 16:04:22.408801 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:22.408777 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3707250-ffeb-4dc2-a97d-a20d0fe845a4-istio-envoy\") pod \"e2e-inf-b-26bb92-openshift-default-7b4d9d44f7-6kbfr\" (UID: \"d3707250-ffeb-4dc2-a97d-a20d0fe845a4\") " pod="openshift-ingress/e2e-inf-b-26bb92-openshift-default-7b4d9d44f7-6kbfr" Jul 21 16:04:22.408933 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:22.408838 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dt6qc\" (UniqueName: \"kubernetes.io/projected/d3707250-ffeb-4dc2-a97d-a20d0fe845a4-kube-api-access-dt6qc\") pod \"e2e-inf-b-26bb92-openshift-default-7b4d9d44f7-6kbfr\" (UID: \"d3707250-ffeb-4dc2-a97d-a20d0fe845a4\") " pod="openshift-ingress/e2e-inf-b-26bb92-openshift-default-7b4d9d44f7-6kbfr" Jul 21 16:04:22.408933 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:22.408884 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3707250-ffeb-4dc2-a97d-a20d0fe845a4-credential-socket\") pod \"e2e-inf-b-26bb92-openshift-default-7b4d9d44f7-6kbfr\" (UID: \"d3707250-ffeb-4dc2-a97d-a20d0fe845a4\") " pod="openshift-ingress/e2e-inf-b-26bb92-openshift-default-7b4d9d44f7-6kbfr" Jul 21 16:04:22.409058 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:22.408943 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3707250-ffeb-4dc2-a97d-a20d0fe845a4-workload-certs\") pod \"e2e-inf-b-26bb92-openshift-default-7b4d9d44f7-6kbfr\" (UID: \"d3707250-ffeb-4dc2-a97d-a20d0fe845a4\") " pod="openshift-ingress/e2e-inf-b-26bb92-openshift-default-7b4d9d44f7-6kbfr" Jul 21 16:04:22.409058 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:22.408967 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3707250-ffeb-4dc2-a97d-a20d0fe845a4-istio-token\") pod \"e2e-inf-b-26bb92-openshift-default-7b4d9d44f7-6kbfr\" (UID: \"d3707250-ffeb-4dc2-a97d-a20d0fe845a4\") " pod="openshift-ingress/e2e-inf-b-26bb92-openshift-default-7b4d9d44f7-6kbfr" Jul 21 16:04:22.409058 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:22.408991 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3707250-ffeb-4dc2-a97d-a20d0fe845a4-istiod-ca-cert\") pod \"e2e-inf-b-26bb92-openshift-default-7b4d9d44f7-6kbfr\" (UID: \"d3707250-ffeb-4dc2-a97d-a20d0fe845a4\") " pod="openshift-ingress/e2e-inf-b-26bb92-openshift-default-7b4d9d44f7-6kbfr" Jul 21 16:04:22.409732 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:22.408721 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3707250-ffeb-4dc2-a97d-a20d0fe845a4-workload-socket\") pod \"e2e-inf-b-26bb92-openshift-default-7b4d9d44f7-6kbfr\" (UID: \"d3707250-ffeb-4dc2-a97d-a20d0fe845a4\") " pod="openshift-ingress/e2e-inf-b-26bb92-openshift-default-7b4d9d44f7-6kbfr" Jul 21 16:04:22.409732 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:22.409723 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3707250-ffeb-4dc2-a97d-a20d0fe845a4-istiod-ca-cert\") pod \"e2e-inf-b-26bb92-openshift-default-7b4d9d44f7-6kbfr\" (UID: \"d3707250-ffeb-4dc2-a97d-a20d0fe845a4\") " pod="openshift-ingress/e2e-inf-b-26bb92-openshift-default-7b4d9d44f7-6kbfr" Jul 21 16:04:22.410053 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:22.410031 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3707250-ffeb-4dc2-a97d-a20d0fe845a4-credential-socket\") pod \"e2e-inf-b-26bb92-openshift-default-7b4d9d44f7-6kbfr\" (UID: \"d3707250-ffeb-4dc2-a97d-a20d0fe845a4\") " pod="openshift-ingress/e2e-inf-b-26bb92-openshift-default-7b4d9d44f7-6kbfr" Jul 21 16:04:22.410228 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:22.410198 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3707250-ffeb-4dc2-a97d-a20d0fe845a4-workload-certs\") pod \"e2e-inf-b-26bb92-openshift-default-7b4d9d44f7-6kbfr\" (UID: \"d3707250-ffeb-4dc2-a97d-a20d0fe845a4\") " pod="openshift-ingress/e2e-inf-b-26bb92-openshift-default-7b4d9d44f7-6kbfr" Jul 21 16:04:22.410841 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:22.410819 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3707250-ffeb-4dc2-a97d-a20d0fe845a4-istio-podinfo\") pod \"e2e-inf-b-26bb92-openshift-default-7b4d9d44f7-6kbfr\" (UID: \"d3707250-ffeb-4dc2-a97d-a20d0fe845a4\") " pod="openshift-ingress/e2e-inf-b-26bb92-openshift-default-7b4d9d44f7-6kbfr" Jul 21 16:04:22.412592 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:22.412570 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3707250-ffeb-4dc2-a97d-a20d0fe845a4-istio-envoy\") pod \"e2e-inf-b-26bb92-openshift-default-7b4d9d44f7-6kbfr\" (UID: \"d3707250-ffeb-4dc2-a97d-a20d0fe845a4\") " pod="openshift-ingress/e2e-inf-b-26bb92-openshift-default-7b4d9d44f7-6kbfr" Jul 21 16:04:22.417693 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:22.417670 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3707250-ffeb-4dc2-a97d-a20d0fe845a4-istio-token\") pod \"e2e-inf-b-26bb92-openshift-default-7b4d9d44f7-6kbfr\" (UID: \"d3707250-ffeb-4dc2-a97d-a20d0fe845a4\") " pod="openshift-ingress/e2e-inf-b-26bb92-openshift-default-7b4d9d44f7-6kbfr" Jul 21 16:04:22.417788 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:22.417773 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dt6qc\" (UniqueName: \"kubernetes.io/projected/d3707250-ffeb-4dc2-a97d-a20d0fe845a4-kube-api-access-dt6qc\") pod \"e2e-inf-b-26bb92-openshift-default-7b4d9d44f7-6kbfr\" (UID: \"d3707250-ffeb-4dc2-a97d-a20d0fe845a4\") " pod="openshift-ingress/e2e-inf-b-26bb92-openshift-default-7b4d9d44f7-6kbfr" Jul 21 16:04:22.435824 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:22.435792 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-26bb92-openshift-default-7b4d9d44f7-6kbfr" Jul 21 16:04:22.643409 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:22.643372 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-26bb92-openshift-default-7b4d9d44f7-6kbfr"] Jul 21 16:04:22.645370 ip-10-0-135-176 kubenswrapper[2563]: W0721 16:04:22.645342 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3707250_ffeb_4dc2_a97d_a20d0fe845a4.slice/crio-5a7a0526cfdfcf02c2ce632ae5b347e00277cfd6264de146036fa4e4bd8bfff2 WatchSource:0}: Error finding container 5a7a0526cfdfcf02c2ce632ae5b347e00277cfd6264de146036fa4e4bd8bfff2: Status 404 returned error can't find the container with id 5a7a0526cfdfcf02c2ce632ae5b347e00277cfd6264de146036fa4e4bd8bfff2 Jul 21 16:04:22.647902 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:22.647869 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 21 16:04:22.648004 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:22.647937 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 21 16:04:22.648004 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:22.647967 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 21 16:04:22.952460 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:22.952349 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-26bb92-openshift-default-7b4d9d44f7-6kbfr" event={"ID":"d3707250-ffeb-4dc2-a97d-a20d0fe845a4","Type":"ContainerStarted","Data":"4655ba2b77090ed4186da9a15e539c4086039112671ce70892b7205d03eaa1ef"} Jul 21 16:04:22.952460 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:22.952447 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-26bb92-openshift-default-7b4d9d44f7-6kbfr" event={"ID":"d3707250-ffeb-4dc2-a97d-a20d0fe845a4","Type":"ContainerStarted","Data":"5a7a0526cfdfcf02c2ce632ae5b347e00277cfd6264de146036fa4e4bd8bfff2"} Jul 21 16:04:22.973288 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:22.973221 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-26bb92-openshift-default-7b4d9d44f7-6kbfr" podStartSLOduration=0.973201746 podStartE2EDuration="973.201746ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 16:04:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 16:04:22.971676533 +0000 UTC m=+3894.967461976" watchObservedRunningTime="2026-07-21 16:04:22.973201746 +0000 UTC m=+3894.968987162" Jul 21 16:04:23.436246 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:23.436208 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-26bb92-openshift-default-7b4d9d44f7-6kbfr" Jul 21 16:04:23.442792 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:23.442765 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-26bb92-openshift-default-7b4d9d44f7-6kbfr" Jul 21 16:04:23.959298 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:23.959264 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-26bb92-openshift-default-7b4d9d44f7-6kbfr" Jul 21 16:04:23.960378 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:23.960356 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-26bb92-openshift-default-7b4d9d44f7-6kbfr" Jul 21 16:04:28.777553 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:28.777521 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kn9fn_02f3d0c6-4116-4134-ba51-1c02c73bef05/ovn-acl-logging/0.log" Jul 21 16:04:28.799673 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:28.799646 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kn9fn_02f3d0c6-4116-4134-ba51-1c02c73bef05/ovn-acl-logging/0.log" Jul 21 16:04:29.136586 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:29.136554 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-j4c8p"] Jul 21 16:04:30.001095 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:30.001046 2563 generic.go:356] "Generic (PLEG): container finished" podID="dbbade83-f99d-4539-bb0e-4f2fc9d711ff" containerID="74b8be63f4bc36c37ce8dc06069b08088813453542ff2b23aa71acada6379ab6" exitCode=0 Jul 21 16:04:30.001678 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:30.001146 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-682cb4-openshift-default-788d8bdd7-dt5s9" event={"ID":"dbbade83-f99d-4539-bb0e-4f2fc9d711ff","Type":"ContainerDied","Data":"74b8be63f4bc36c37ce8dc06069b08088813453542ff2b23aa71acada6379ab6"} Jul 21 16:04:30.001678 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:30.001657 2563 scope.go:117] "RemoveContainer" containerID="74b8be63f4bc36c37ce8dc06069b08088813453542ff2b23aa71acada6379ab6" Jul 21 16:04:30.003276 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:30.003228 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 21 16:04:30.003419 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:30.003350 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 21 16:04:30.003419 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:30.003399 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 21 16:04:30.908557 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:30.908510 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-682cb4-openshift-default-788d8bdd7-dt5s9" Jul 21 16:04:31.018158 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:31.018113 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-682cb4-openshift-default-788d8bdd7-dt5s9" event={"ID":"dbbade83-f99d-4539-bb0e-4f2fc9d711ff","Type":"ContainerStarted","Data":"30387aa10cd47aacd7a38f0d0f90bde7708597613c1de16857b02fbcef968a5f"} Jul 21 16:04:31.681200 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:31.681163 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-682cb4-openshift-default-788d8bdd7-dt5s9" Jul 21 16:04:31.682490 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:31.682455 2563 patch_prober.go:28] interesting pod/e2e-inf-a-682cb4-openshift-default-788d8bdd7-dt5s9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Jul 21 16:04:31.682719 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:31.682692 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-682cb4-openshift-default-788d8bdd7-dt5s9" podUID="dbbade83-f99d-4539-bb0e-4f2fc9d711ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Jul 21 16:04:32.688352 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:32.681838 2563 patch_prober.go:28] interesting pod/e2e-inf-a-682cb4-openshift-default-788d8bdd7-dt5s9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Jul 21 16:04:32.688352 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:32.681915 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-682cb4-openshift-default-788d8bdd7-dt5s9" podUID="dbbade83-f99d-4539-bb0e-4f2fc9d711ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Jul 21 16:04:33.681052 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:33.681007 2563 patch_prober.go:28] interesting pod/e2e-inf-a-682cb4-openshift-default-788d8bdd7-dt5s9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Jul 21 16:04:33.681435 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:33.681402 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-682cb4-openshift-default-788d8bdd7-dt5s9" podUID="dbbade83-f99d-4539-bb0e-4f2fc9d711ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Jul 21 16:04:34.681224 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:34.681171 2563 patch_prober.go:28] interesting pod/e2e-inf-a-682cb4-openshift-default-788d8bdd7-dt5s9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Jul 21 16:04:34.681787 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:34.681260 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-682cb4-openshift-default-788d8bdd7-dt5s9" podUID="dbbade83-f99d-4539-bb0e-4f2fc9d711ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Jul 21 16:04:35.681836 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:35.681796 2563 patch_prober.go:28] interesting pod/e2e-inf-a-682cb4-openshift-default-788d8bdd7-dt5s9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Jul 21 16:04:35.682482 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:35.682446 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-682cb4-openshift-default-788d8bdd7-dt5s9" podUID="dbbade83-f99d-4539-bb0e-4f2fc9d711ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Jul 21 16:04:36.686474 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:36.686437 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-682cb4-openshift-default-788d8bdd7-dt5s9" Jul 21 16:04:36.686952 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:36.686802 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-682cb4-openshift-default-788d8bdd7-dt5s9" Jul 21 16:04:36.687780 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:36.687759 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-682cb4-openshift-default-788d8bdd7-dt5s9" Jul 21 16:04:38.960871 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:38.960782 2563 patch_prober.go:28] interesting pod/e2e-inf-b-26bb92-openshift-default-7b4d9d44f7-6kbfr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Jul 21 16:04:38.960871 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:38.960838 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-26bb92-openshift-default-7b4d9d44f7-6kbfr" podUID="d3707250-ffeb-4dc2-a97d-a20d0fe845a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Jul 21 16:04:39.092688 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:39.092651 2563 generic.go:356] "Generic (PLEG): container finished" podID="d3707250-ffeb-4dc2-a97d-a20d0fe845a4" containerID="4655ba2b77090ed4186da9a15e539c4086039112671ce70892b7205d03eaa1ef" exitCode=0 Jul 21 16:04:39.092890 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:39.092730 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-26bb92-openshift-default-7b4d9d44f7-6kbfr" event={"ID":"d3707250-ffeb-4dc2-a97d-a20d0fe845a4","Type":"ContainerDied","Data":"4655ba2b77090ed4186da9a15e539c4086039112671ce70892b7205d03eaa1ef"} Jul 21 16:04:39.093169 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:39.093154 2563 scope.go:117] "RemoveContainer" containerID="4655ba2b77090ed4186da9a15e539c4086039112671ce70892b7205d03eaa1ef" Jul 21 16:04:39.094238 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:39.094210 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 21 16:04:39.094297 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:39.094279 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 21 16:04:39.094367 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:39.094353 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 21 16:04:40.108734 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:40.108688 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-26bb92-openshift-default-7b4d9d44f7-6kbfr" event={"ID":"d3707250-ffeb-4dc2-a97d-a20d0fe845a4","Type":"ContainerStarted","Data":"940bf0784f3a206f955f8a0fa9437575cef94eb657ecedf60a7a8ef3803e419d"} Jul 21 16:04:40.436247 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:40.436144 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-26bb92-openshift-default-7b4d9d44f7-6kbfr" Jul 21 16:04:40.436526 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:40.436507 2563 patch_prober.go:28] interesting pod/e2e-inf-b-26bb92-openshift-default-7b4d9d44f7-6kbfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Jul 21 16:04:40.436577 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:40.436550 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-26bb92-openshift-default-7b4d9d44f7-6kbfr" podUID="d3707250-ffeb-4dc2-a97d-a20d0fe845a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Jul 21 16:04:41.437128 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:41.437090 2563 patch_prober.go:28] interesting pod/e2e-inf-b-26bb92-openshift-default-7b4d9d44f7-6kbfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Jul 21 16:04:41.437547 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:41.437153 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-26bb92-openshift-default-7b4d9d44f7-6kbfr" podUID="d3707250-ffeb-4dc2-a97d-a20d0fe845a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Jul 21 16:04:42.436448 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:42.436404 2563 patch_prober.go:28] interesting pod/e2e-inf-b-26bb92-openshift-default-7b4d9d44f7-6kbfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Jul 21 16:04:42.436643 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:42.436485 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-26bb92-openshift-default-7b4d9d44f7-6kbfr" podUID="d3707250-ffeb-4dc2-a97d-a20d0fe845a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Jul 21 16:04:43.436942 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:43.436900 2563 patch_prober.go:28] interesting pod/e2e-inf-b-26bb92-openshift-default-7b4d9d44f7-6kbfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Jul 21 16:04:43.437413 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:43.436961 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-26bb92-openshift-default-7b4d9d44f7-6kbfr" podUID="d3707250-ffeb-4dc2-a97d-a20d0fe845a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Jul 21 16:04:44.437219 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:44.437174 2563 patch_prober.go:28] interesting pod/e2e-inf-b-26bb92-openshift-default-7b4d9d44f7-6kbfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Jul 21 16:04:44.437695 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:44.437260 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-26bb92-openshift-default-7b4d9d44f7-6kbfr" podUID="d3707250-ffeb-4dc2-a97d-a20d0fe845a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Jul 21 16:04:45.441047 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:45.441017 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-26bb92-openshift-default-7b4d9d44f7-6kbfr" Jul 21 16:04:45.441530 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:45.441295 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-26bb92-openshift-default-7b4d9d44f7-6kbfr" Jul 21 16:04:45.442146 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:45.442130 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-26bb92-openshift-default-7b4d9d44f7-6kbfr" Jul 21 16:04:54.842742 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:54.842695 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-682cb4/test-model-682cb4-kserve-5d467b49f-7l7h6"] Jul 21 16:04:54.846569 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:54.846543 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-682cb4/test-model-682cb4-kserve-5d467b49f-7l7h6" Jul 21 16:04:54.850559 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:54.850504 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-682cb4\"/\"kube-root-ca.crt\"" Jul 21 16:04:54.850851 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:54.850578 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-682cb4\"/\"openshift-service-ca.crt\"" Jul 21 16:04:54.850851 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:54.850513 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-682cb4\"/\"test-model-682cb4-kserve-self-signed-certs\"" Jul 21 16:04:54.850851 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:54.850581 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-682cb4\"/\"default-dockercfg-nsk26\"" Jul 21 16:04:54.854721 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:54.854353 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-682cb4/test-model-682cb4-kserve-5d467b49f-7l7h6"] Jul 21 16:04:54.943158 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:54.943109 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/147847d9-249f-4372-a72a-5d84418680d8-dshm\") pod \"test-model-682cb4-kserve-5d467b49f-7l7h6\" (UID: \"147847d9-249f-4372-a72a-5d84418680d8\") " pod="ai-tenant-e2e-inf-a-682cb4/test-model-682cb4-kserve-5d467b49f-7l7h6" Jul 21 16:04:54.943379 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:54.943262 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/147847d9-249f-4372-a72a-5d84418680d8-model-cache\") pod \"test-model-682cb4-kserve-5d467b49f-7l7h6\" (UID: \"147847d9-249f-4372-a72a-5d84418680d8\") " pod="ai-tenant-e2e-inf-a-682cb4/test-model-682cb4-kserve-5d467b49f-7l7h6" Jul 21 16:04:54.943379 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:54.943312 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/147847d9-249f-4372-a72a-5d84418680d8-home\") pod \"test-model-682cb4-kserve-5d467b49f-7l7h6\" (UID: \"147847d9-249f-4372-a72a-5d84418680d8\") " pod="ai-tenant-e2e-inf-a-682cb4/test-model-682cb4-kserve-5d467b49f-7l7h6" Jul 21 16:04:54.943379 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:54.943369 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dwxg\" (UniqueName: \"kubernetes.io/projected/147847d9-249f-4372-a72a-5d84418680d8-kube-api-access-8dwxg\") pod \"test-model-682cb4-kserve-5d467b49f-7l7h6\" (UID: \"147847d9-249f-4372-a72a-5d84418680d8\") " pod="ai-tenant-e2e-inf-a-682cb4/test-model-682cb4-kserve-5d467b49f-7l7h6" Jul 21 16:04:54.943501 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:54.943398 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/147847d9-249f-4372-a72a-5d84418680d8-tmp-dir\") pod \"test-model-682cb4-kserve-5d467b49f-7l7h6\" (UID: \"147847d9-249f-4372-a72a-5d84418680d8\") " pod="ai-tenant-e2e-inf-a-682cb4/test-model-682cb4-kserve-5d467b49f-7l7h6" Jul 21 16:04:54.943501 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:54.943416 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/147847d9-249f-4372-a72a-5d84418680d8-tls-certs\") pod \"test-model-682cb4-kserve-5d467b49f-7l7h6\" (UID: \"147847d9-249f-4372-a72a-5d84418680d8\") " pod="ai-tenant-e2e-inf-a-682cb4/test-model-682cb4-kserve-5d467b49f-7l7h6" Jul 21 16:04:55.044270 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:55.044237 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/147847d9-249f-4372-a72a-5d84418680d8-model-cache\") pod \"test-model-682cb4-kserve-5d467b49f-7l7h6\" (UID: \"147847d9-249f-4372-a72a-5d84418680d8\") " pod="ai-tenant-e2e-inf-a-682cb4/test-model-682cb4-kserve-5d467b49f-7l7h6" Jul 21 16:04:55.044503 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:55.044297 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/147847d9-249f-4372-a72a-5d84418680d8-home\") pod \"test-model-682cb4-kserve-5d467b49f-7l7h6\" (UID: \"147847d9-249f-4372-a72a-5d84418680d8\") " pod="ai-tenant-e2e-inf-a-682cb4/test-model-682cb4-kserve-5d467b49f-7l7h6" Jul 21 16:04:55.044503 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:55.044358 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dwxg\" (UniqueName: \"kubernetes.io/projected/147847d9-249f-4372-a72a-5d84418680d8-kube-api-access-8dwxg\") pod \"test-model-682cb4-kserve-5d467b49f-7l7h6\" (UID: \"147847d9-249f-4372-a72a-5d84418680d8\") " pod="ai-tenant-e2e-inf-a-682cb4/test-model-682cb4-kserve-5d467b49f-7l7h6" Jul 21 16:04:55.044503 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:55.044391 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/147847d9-249f-4372-a72a-5d84418680d8-tmp-dir\") pod \"test-model-682cb4-kserve-5d467b49f-7l7h6\" (UID: \"147847d9-249f-4372-a72a-5d84418680d8\") " pod="ai-tenant-e2e-inf-a-682cb4/test-model-682cb4-kserve-5d467b49f-7l7h6" Jul 21 16:04:55.044503 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:55.044415 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/147847d9-249f-4372-a72a-5d84418680d8-tls-certs\") pod \"test-model-682cb4-kserve-5d467b49f-7l7h6\" (UID: \"147847d9-249f-4372-a72a-5d84418680d8\") " pod="ai-tenant-e2e-inf-a-682cb4/test-model-682cb4-kserve-5d467b49f-7l7h6" Jul 21 16:04:55.044503 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:55.044481 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/147847d9-249f-4372-a72a-5d84418680d8-dshm\") pod \"test-model-682cb4-kserve-5d467b49f-7l7h6\" (UID: \"147847d9-249f-4372-a72a-5d84418680d8\") " pod="ai-tenant-e2e-inf-a-682cb4/test-model-682cb4-kserve-5d467b49f-7l7h6" Jul 21 16:04:55.044765 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:55.044732 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/147847d9-249f-4372-a72a-5d84418680d8-model-cache\") pod \"test-model-682cb4-kserve-5d467b49f-7l7h6\" (UID: \"147847d9-249f-4372-a72a-5d84418680d8\") " pod="ai-tenant-e2e-inf-a-682cb4/test-model-682cb4-kserve-5d467b49f-7l7h6" Jul 21 16:04:55.044826 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:55.044740 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/147847d9-249f-4372-a72a-5d84418680d8-tmp-dir\") pod \"test-model-682cb4-kserve-5d467b49f-7l7h6\" (UID: \"147847d9-249f-4372-a72a-5d84418680d8\") " pod="ai-tenant-e2e-inf-a-682cb4/test-model-682cb4-kserve-5d467b49f-7l7h6" Jul 21 16:04:55.044826 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:55.044785 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/147847d9-249f-4372-a72a-5d84418680d8-home\") pod \"test-model-682cb4-kserve-5d467b49f-7l7h6\" (UID: \"147847d9-249f-4372-a72a-5d84418680d8\") " pod="ai-tenant-e2e-inf-a-682cb4/test-model-682cb4-kserve-5d467b49f-7l7h6" Jul 21 16:04:55.046923 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:55.046904 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/147847d9-249f-4372-a72a-5d84418680d8-dshm\") pod \"test-model-682cb4-kserve-5d467b49f-7l7h6\" (UID: \"147847d9-249f-4372-a72a-5d84418680d8\") " pod="ai-tenant-e2e-inf-a-682cb4/test-model-682cb4-kserve-5d467b49f-7l7h6" Jul 21 16:04:55.047180 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:55.047162 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/147847d9-249f-4372-a72a-5d84418680d8-tls-certs\") pod \"test-model-682cb4-kserve-5d467b49f-7l7h6\" (UID: \"147847d9-249f-4372-a72a-5d84418680d8\") " pod="ai-tenant-e2e-inf-a-682cb4/test-model-682cb4-kserve-5d467b49f-7l7h6" Jul 21 16:04:55.053793 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:55.053767 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dwxg\" (UniqueName: \"kubernetes.io/projected/147847d9-249f-4372-a72a-5d84418680d8-kube-api-access-8dwxg\") pod \"test-model-682cb4-kserve-5d467b49f-7l7h6\" (UID: \"147847d9-249f-4372-a72a-5d84418680d8\") " pod="ai-tenant-e2e-inf-a-682cb4/test-model-682cb4-kserve-5d467b49f-7l7h6" Jul 21 16:04:55.159068 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:55.158957 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-682cb4/test-model-682cb4-kserve-5d467b49f-7l7h6" Jul 21 16:04:55.312344 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:55.312111 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-682cb4/test-model-682cb4-kserve-5d467b49f-7l7h6"] Jul 21 16:04:56.203879 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:56.203827 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-682cb4/test-model-682cb4-kserve-5d467b49f-7l7h6" event={"ID":"147847d9-249f-4372-a72a-5d84418680d8","Type":"ContainerStarted","Data":"a8198d937a93df0d88d079d9288ff54c09bfb6c7456ec1159fbf3b34472b1bfa"} Jul 21 16:04:56.203879 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:56.203881 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-682cb4/test-model-682cb4-kserve-5d467b49f-7l7h6" event={"ID":"147847d9-249f-4372-a72a-5d84418680d8","Type":"ContainerStarted","Data":"84c3b3846bf4c208b95809d690b73a0c8551a1cee49eac4f121eb9360b89986d"} Jul 21 16:04:56.220660 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:56.220604 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-682cb4/test-model-682cb4-kserve-5d467b49f-7l7h6" podStartSLOduration=2.220585299 podStartE2EDuration="2.220585299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 16:04:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 16:04:56.219681011 +0000 UTC m=+3928.215466491" watchObservedRunningTime="2026-07-21 16:04:56.220585299 +0000 UTC m=+3928.216370759" Jul 21 16:04:58.438618 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:04:58.438579 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-j4c8p"] Jul 21 16:05:05.159527 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:05:05.159480 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-682cb4/test-model-682cb4-kserve-5d467b49f-7l7h6" Jul 21 16:05:05.160013 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:05:05.159550 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-682cb4/test-model-682cb4-kserve-5d467b49f-7l7h6" Jul 21 16:05:05.172396 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:05:05.172366 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-682cb4/test-model-682cb4-kserve-5d467b49f-7l7h6" Jul 21 16:05:05.265031 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:05:05.264997 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-682cb4/test-model-682cb4-kserve-5d467b49f-7l7h6" Jul 21 16:05:16.341101 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:05:16.341054 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-j4c8p"] Jul 21 16:05:27.944250 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:05:27.944159 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-j4c8p"] Jul 21 16:06:05.619471 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:06:05.619436 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-682cb4/test-model-682cb4-kserve-5d467b49f-7l7h6"] Jul 21 16:06:09.340936 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:06:09.340901 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-j4c8p"] Jul 21 16:06:19.737127 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:06:19.737082 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-j4c8p"] Jul 21 16:06:20.641868 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:06:20.641806 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-682cb4/test-model-682cb4-kserve-5d467b49f-7l7h6" podUID="147847d9-249f-4372-a72a-5d84418680d8" containerName="main" containerID="cri-o://a8198d937a93df0d88d079d9288ff54c09bfb6c7456ec1159fbf3b34472b1bfa" gracePeriod=45 Jul 21 16:06:20.901356 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:06:20.901273 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-682cb4/test-model-682cb4-kserve-5d467b49f-7l7h6" Jul 21 16:06:20.982066 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:06:20.982030 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/147847d9-249f-4372-a72a-5d84418680d8-tmp-dir\") pod \"147847d9-249f-4372-a72a-5d84418680d8\" (UID: \"147847d9-249f-4372-a72a-5d84418680d8\") " Jul 21 16:06:20.982243 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:06:20.982086 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/147847d9-249f-4372-a72a-5d84418680d8-model-cache\") pod \"147847d9-249f-4372-a72a-5d84418680d8\" (UID: \"147847d9-249f-4372-a72a-5d84418680d8\") " Jul 21 16:06:20.982243 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:06:20.982169 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8dwxg\" (UniqueName: \"kubernetes.io/projected/147847d9-249f-4372-a72a-5d84418680d8-kube-api-access-8dwxg\") pod \"147847d9-249f-4372-a72a-5d84418680d8\" (UID: \"147847d9-249f-4372-a72a-5d84418680d8\") " Jul 21 16:06:20.982243 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:06:20.982193 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/147847d9-249f-4372-a72a-5d84418680d8-tls-certs\") pod \"147847d9-249f-4372-a72a-5d84418680d8\" (UID: \"147847d9-249f-4372-a72a-5d84418680d8\") " Jul 21 16:06:20.982243 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:06:20.982217 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/147847d9-249f-4372-a72a-5d84418680d8-home\") pod \"147847d9-249f-4372-a72a-5d84418680d8\" (UID: \"147847d9-249f-4372-a72a-5d84418680d8\") " Jul 21 16:06:20.982571 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:06:20.982391 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/147847d9-249f-4372-a72a-5d84418680d8-dshm\") pod \"147847d9-249f-4372-a72a-5d84418680d8\" (UID: \"147847d9-249f-4372-a72a-5d84418680d8\") " Jul 21 16:06:20.982571 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:06:20.982413 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/147847d9-249f-4372-a72a-5d84418680d8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "147847d9-249f-4372-a72a-5d84418680d8" (UID: "147847d9-249f-4372-a72a-5d84418680d8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 16:06:20.982571 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:06:20.982499 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/147847d9-249f-4372-a72a-5d84418680d8-home" (OuterVolumeSpecName: "home") pod "147847d9-249f-4372-a72a-5d84418680d8" (UID: "147847d9-249f-4372-a72a-5d84418680d8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 16:06:20.982571 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:06:20.982430 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/147847d9-249f-4372-a72a-5d84418680d8-model-cache" (OuterVolumeSpecName: "model-cache") pod "147847d9-249f-4372-a72a-5d84418680d8" (UID: "147847d9-249f-4372-a72a-5d84418680d8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 16:06:20.982809 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:06:20.982788 2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/147847d9-249f-4372-a72a-5d84418680d8-tmp-dir\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 16:06:20.982872 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:06:20.982813 2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/147847d9-249f-4372-a72a-5d84418680d8-model-cache\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 16:06:20.982872 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:06:20.982822 2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/147847d9-249f-4372-a72a-5d84418680d8-home\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 16:06:20.984822 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:06:20.984799 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/147847d9-249f-4372-a72a-5d84418680d8-kube-api-access-8dwxg" (OuterVolumeSpecName: "kube-api-access-8dwxg") pod "147847d9-249f-4372-a72a-5d84418680d8" (UID: "147847d9-249f-4372-a72a-5d84418680d8"). InnerVolumeSpecName "kube-api-access-8dwxg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 16:06:20.984822 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:06:20.984805 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/147847d9-249f-4372-a72a-5d84418680d8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "147847d9-249f-4372-a72a-5d84418680d8" (UID: "147847d9-249f-4372-a72a-5d84418680d8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 16:06:20.984822 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:06:20.984819 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/147847d9-249f-4372-a72a-5d84418680d8-dshm" (OuterVolumeSpecName: "dshm") pod "147847d9-249f-4372-a72a-5d84418680d8" (UID: "147847d9-249f-4372-a72a-5d84418680d8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 16:06:21.083771 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:06:21.083726 2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/147847d9-249f-4372-a72a-5d84418680d8-dshm\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 16:06:21.083771 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:06:21.083769 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8dwxg\" (UniqueName: \"kubernetes.io/projected/147847d9-249f-4372-a72a-5d84418680d8-kube-api-access-8dwxg\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 16:06:21.083771 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:06:21.083782 2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/147847d9-249f-4372-a72a-5d84418680d8-tls-certs\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 16:06:21.704045 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:06:21.704006 2563 generic.go:356] "Generic (PLEG): container finished" podID="147847d9-249f-4372-a72a-5d84418680d8" containerID="a8198d937a93df0d88d079d9288ff54c09bfb6c7456ec1159fbf3b34472b1bfa" exitCode=0 Jul 21 16:06:21.704202 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:06:21.704057 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-682cb4/test-model-682cb4-kserve-5d467b49f-7l7h6" Jul 21 16:06:21.704202 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:06:21.704093 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-682cb4/test-model-682cb4-kserve-5d467b49f-7l7h6" event={"ID":"147847d9-249f-4372-a72a-5d84418680d8","Type":"ContainerDied","Data":"a8198d937a93df0d88d079d9288ff54c09bfb6c7456ec1159fbf3b34472b1bfa"} Jul 21 16:06:21.704202 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:06:21.704141 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-682cb4/test-model-682cb4-kserve-5d467b49f-7l7h6" event={"ID":"147847d9-249f-4372-a72a-5d84418680d8","Type":"ContainerDied","Data":"84c3b3846bf4c208b95809d690b73a0c8551a1cee49eac4f121eb9360b89986d"} Jul 21 16:06:21.704202 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:06:21.704158 2563 scope.go:117] "RemoveContainer" containerID="a8198d937a93df0d88d079d9288ff54c09bfb6c7456ec1159fbf3b34472b1bfa" Jul 21 16:06:21.714843 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:06:21.714818 2563 scope.go:117] "RemoveContainer" containerID="a8198d937a93df0d88d079d9288ff54c09bfb6c7456ec1159fbf3b34472b1bfa" Jul 21 16:06:21.715159 ip-10-0-135-176 kubenswrapper[2563]: E0721 16:06:21.715139 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8198d937a93df0d88d079d9288ff54c09bfb6c7456ec1159fbf3b34472b1bfa\": container with ID starting with a8198d937a93df0d88d079d9288ff54c09bfb6c7456ec1159fbf3b34472b1bfa not found: ID does not exist" containerID="a8198d937a93df0d88d079d9288ff54c09bfb6c7456ec1159fbf3b34472b1bfa" Jul 21 16:06:21.715211 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:06:21.715169 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8198d937a93df0d88d079d9288ff54c09bfb6c7456ec1159fbf3b34472b1bfa"} err="failed to get container status \"a8198d937a93df0d88d079d9288ff54c09bfb6c7456ec1159fbf3b34472b1bfa\": rpc error: code = NotFound desc = could not find container \"a8198d937a93df0d88d079d9288ff54c09bfb6c7456ec1159fbf3b34472b1bfa\": container with ID starting with a8198d937a93df0d88d079d9288ff54c09bfb6c7456ec1159fbf3b34472b1bfa not found: ID does not exist" Jul 21 16:06:21.725891 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:06:21.725856 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-682cb4/test-model-682cb4-kserve-5d467b49f-7l7h6"] Jul 21 16:06:21.728348 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:06:21.728313 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-682cb4/test-model-682cb4-kserve-5d467b49f-7l7h6"] Jul 21 16:06:22.481078 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:06:22.481035 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="147847d9-249f-4372-a72a-5d84418680d8" path="/var/lib/kubelet/pods/147847d9-249f-4372-a72a-5d84418680d8/volumes" Jul 21 16:09:13.169141 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:09:13.169100 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-682cb4-openshift-default-788d8bdd7-dt5s9"] Jul 21 16:09:13.169818 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:09:13.169356 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-682cb4-openshift-default-788d8bdd7-dt5s9" podUID="dbbade83-f99d-4539-bb0e-4f2fc9d711ff" containerName="istio-proxy" containerID="cri-o://30387aa10cd47aacd7a38f0d0f90bde7708597613c1de16857b02fbcef968a5f" gracePeriod=30 Jul 21 16:09:18.480297 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:09:18.480267 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-682cb4-openshift-default-788d8bdd7-dt5s9" Jul 21 16:09:18.594494 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:09:18.594407 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbbade83-f99d-4539-bb0e-4f2fc9d711ff-workload-certs\") pod \"dbbade83-f99d-4539-bb0e-4f2fc9d711ff\" (UID: \"dbbade83-f99d-4539-bb0e-4f2fc9d711ff\") " Jul 21 16:09:18.594847 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:09:18.594530 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbbade83-f99d-4539-bb0e-4f2fc9d711ff-credential-socket\") pod \"dbbade83-f99d-4539-bb0e-4f2fc9d711ff\" (UID: \"dbbade83-f99d-4539-bb0e-4f2fc9d711ff\") " Jul 21 16:09:18.594847 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:09:18.594564 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbbade83-f99d-4539-bb0e-4f2fc9d711ff-istiod-ca-cert\") pod \"dbbade83-f99d-4539-bb0e-4f2fc9d711ff\" (UID: \"dbbade83-f99d-4539-bb0e-4f2fc9d711ff\") " Jul 21 16:09:18.594847 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:09:18.594628 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbbade83-f99d-4539-bb0e-4f2fc9d711ff-istio-data\") pod \"dbbade83-f99d-4539-bb0e-4f2fc9d711ff\" (UID: \"dbbade83-f99d-4539-bb0e-4f2fc9d711ff\") " Jul 21 16:09:18.594847 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:09:18.594693 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbbade83-f99d-4539-bb0e-4f2fc9d711ff-istio-token\") pod \"dbbade83-f99d-4539-bb0e-4f2fc9d711ff\" (UID: \"dbbade83-f99d-4539-bb0e-4f2fc9d711ff\") " Jul 21 16:09:18.594847 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:09:18.594832 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbbade83-f99d-4539-bb0e-4f2fc9d711ff-istio-envoy\") pod \"dbbade83-f99d-4539-bb0e-4f2fc9d711ff\" (UID: \"dbbade83-f99d-4539-bb0e-4f2fc9d711ff\") " Jul 21 16:09:18.595144 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:09:18.594865 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbbade83-f99d-4539-bb0e-4f2fc9d711ff-workload-socket\") pod \"dbbade83-f99d-4539-bb0e-4f2fc9d711ff\" (UID: \"dbbade83-f99d-4539-bb0e-4f2fc9d711ff\") " Jul 21 16:09:18.595144 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:09:18.594897 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbbade83-f99d-4539-bb0e-4f2fc9d711ff-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "dbbade83-f99d-4539-bb0e-4f2fc9d711ff" (UID: "dbbade83-f99d-4539-bb0e-4f2fc9d711ff"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 16:09:18.595144 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:09:18.594917 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9mp94\" (UniqueName: \"kubernetes.io/projected/dbbade83-f99d-4539-bb0e-4f2fc9d711ff-kube-api-access-9mp94\") pod \"dbbade83-f99d-4539-bb0e-4f2fc9d711ff\" (UID: \"dbbade83-f99d-4539-bb0e-4f2fc9d711ff\") " Jul 21 16:09:18.595144 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:09:18.594924 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbbade83-f99d-4539-bb0e-4f2fc9d711ff-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "dbbade83-f99d-4539-bb0e-4f2fc9d711ff" (UID: "dbbade83-f99d-4539-bb0e-4f2fc9d711ff"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 16:09:18.595144 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:09:18.594963 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbbade83-f99d-4539-bb0e-4f2fc9d711ff-istio-podinfo\") pod \"dbbade83-f99d-4539-bb0e-4f2fc9d711ff\" (UID: \"dbbade83-f99d-4539-bb0e-4f2fc9d711ff\") " Jul 21 16:09:18.595144 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:09:18.595037 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbbade83-f99d-4539-bb0e-4f2fc9d711ff-istio-data" (OuterVolumeSpecName: "istio-data") pod "dbbade83-f99d-4539-bb0e-4f2fc9d711ff" (UID: "dbbade83-f99d-4539-bb0e-4f2fc9d711ff"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 16:09:18.595636 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:09:18.595197 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbbade83-f99d-4539-bb0e-4f2fc9d711ff-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "dbbade83-f99d-4539-bb0e-4f2fc9d711ff" (UID: "dbbade83-f99d-4539-bb0e-4f2fc9d711ff"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 16:09:18.595636 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:09:18.595288 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbbade83-f99d-4539-bb0e-4f2fc9d711ff-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "dbbade83-f99d-4539-bb0e-4f2fc9d711ff" (UID: "dbbade83-f99d-4539-bb0e-4f2fc9d711ff"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 16:09:18.595636 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:09:18.595544 2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbbade83-f99d-4539-bb0e-4f2fc9d711ff-workload-socket\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 16:09:18.595636 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:09:18.595567 2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbbade83-f99d-4539-bb0e-4f2fc9d711ff-workload-certs\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 16:09:18.595636 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:09:18.595582 2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbbade83-f99d-4539-bb0e-4f2fc9d711ff-credential-socket\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 16:09:18.595636 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:09:18.595596 2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbbade83-f99d-4539-bb0e-4f2fc9d711ff-istiod-ca-cert\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 16:09:18.595636 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:09:18.595612 2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbbade83-f99d-4539-bb0e-4f2fc9d711ff-istio-data\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 16:09:18.597913 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:09:18.597869 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dbbade83-f99d-4539-bb0e-4f2fc9d711ff-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "dbbade83-f99d-4539-bb0e-4f2fc9d711ff" (UID: "dbbade83-f99d-4539-bb0e-4f2fc9d711ff"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 21 16:09:18.598166 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:09:18.598142 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbbade83-f99d-4539-bb0e-4f2fc9d711ff-istio-token" (OuterVolumeSpecName: "istio-token") pod "dbbade83-f99d-4539-bb0e-4f2fc9d711ff" (UID: "dbbade83-f99d-4539-bb0e-4f2fc9d711ff"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 16:09:18.598255 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:09:18.598144 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbbade83-f99d-4539-bb0e-4f2fc9d711ff-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "dbbade83-f99d-4539-bb0e-4f2fc9d711ff" (UID: "dbbade83-f99d-4539-bb0e-4f2fc9d711ff"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 16:09:18.598255 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:09:18.598193 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbbade83-f99d-4539-bb0e-4f2fc9d711ff-kube-api-access-9mp94" (OuterVolumeSpecName: "kube-api-access-9mp94") pod "dbbade83-f99d-4539-bb0e-4f2fc9d711ff" (UID: "dbbade83-f99d-4539-bb0e-4f2fc9d711ff"). InnerVolumeSpecName "kube-api-access-9mp94". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 16:09:18.697063 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:09:18.697007 2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbbade83-f99d-4539-bb0e-4f2fc9d711ff-istio-token\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 16:09:18.697063 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:09:18.697053 2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbbade83-f99d-4539-bb0e-4f2fc9d711ff-istio-envoy\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 16:09:18.697063 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:09:18.697064 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mp94\" (UniqueName: \"kubernetes.io/projected/dbbade83-f99d-4539-bb0e-4f2fc9d711ff-kube-api-access-9mp94\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 16:09:18.697063 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:09:18.697078 2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbbade83-f99d-4539-bb0e-4f2fc9d711ff-istio-podinfo\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 16:09:18.739278 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:09:18.739091 2563 generic.go:356] "Generic (PLEG): container finished" podID="dbbade83-f99d-4539-bb0e-4f2fc9d711ff" containerID="30387aa10cd47aacd7a38f0d0f90bde7708597613c1de16857b02fbcef968a5f" exitCode=0 Jul 21 16:09:18.739278 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:09:18.739181 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-682cb4-openshift-default-788d8bdd7-dt5s9" event={"ID":"dbbade83-f99d-4539-bb0e-4f2fc9d711ff","Type":"ContainerDied","Data":"30387aa10cd47aacd7a38f0d0f90bde7708597613c1de16857b02fbcef968a5f"} Jul 21 16:09:18.739278 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:09:18.739235 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-682cb4-openshift-default-788d8bdd7-dt5s9" Jul 21 16:09:18.739698 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:09:18.739300 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-682cb4-openshift-default-788d8bdd7-dt5s9" event={"ID":"dbbade83-f99d-4539-bb0e-4f2fc9d711ff","Type":"ContainerDied","Data":"d40699b4ba38d9471c5f7033cd72e5e17fe9947523d8d36f5e3621e0bcbb9e5b"} Jul 21 16:09:18.739698 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:09:18.739376 2563 scope.go:117] "RemoveContainer" containerID="30387aa10cd47aacd7a38f0d0f90bde7708597613c1de16857b02fbcef968a5f" Jul 21 16:09:18.754293 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:09:18.754266 2563 scope.go:117] "RemoveContainer" containerID="74b8be63f4bc36c37ce8dc06069b08088813453542ff2b23aa71acada6379ab6" Jul 21 16:09:18.769926 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:09:18.769487 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-682cb4-openshift-default-788d8bdd7-dt5s9"] Jul 21 16:09:18.770044 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:09:18.770020 2563 scope.go:117] "RemoveContainer" containerID="30387aa10cd47aacd7a38f0d0f90bde7708597613c1de16857b02fbcef968a5f" Jul 21 16:09:18.774996 ip-10-0-135-176 kubenswrapper[2563]: E0721 16:09:18.770629 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30387aa10cd47aacd7a38f0d0f90bde7708597613c1de16857b02fbcef968a5f\": container with ID starting with 30387aa10cd47aacd7a38f0d0f90bde7708597613c1de16857b02fbcef968a5f not found: ID does not exist" containerID="30387aa10cd47aacd7a38f0d0f90bde7708597613c1de16857b02fbcef968a5f" Jul 21 16:09:18.774996 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:09:18.770682 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30387aa10cd47aacd7a38f0d0f90bde7708597613c1de16857b02fbcef968a5f"} err="failed to get container status \"30387aa10cd47aacd7a38f0d0f90bde7708597613c1de16857b02fbcef968a5f\": rpc error: code = NotFound desc = could not find container \"30387aa10cd47aacd7a38f0d0f90bde7708597613c1de16857b02fbcef968a5f\": container with ID starting with 30387aa10cd47aacd7a38f0d0f90bde7708597613c1de16857b02fbcef968a5f not found: ID does not exist" Jul 21 16:09:18.774996 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:09:18.770729 2563 scope.go:117] "RemoveContainer" containerID="74b8be63f4bc36c37ce8dc06069b08088813453542ff2b23aa71acada6379ab6" Jul 21 16:09:18.774996 ip-10-0-135-176 kubenswrapper[2563]: E0721 16:09:18.771216 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74b8be63f4bc36c37ce8dc06069b08088813453542ff2b23aa71acada6379ab6\": container with ID starting with 74b8be63f4bc36c37ce8dc06069b08088813453542ff2b23aa71acada6379ab6 not found: ID does not exist" containerID="74b8be63f4bc36c37ce8dc06069b08088813453542ff2b23aa71acada6379ab6" Jul 21 16:09:18.774996 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:09:18.771266 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74b8be63f4bc36c37ce8dc06069b08088813453542ff2b23aa71acada6379ab6"} err="failed to get container status \"74b8be63f4bc36c37ce8dc06069b08088813453542ff2b23aa71acada6379ab6\": rpc error: code = NotFound desc = could not find container \"74b8be63f4bc36c37ce8dc06069b08088813453542ff2b23aa71acada6379ab6\": container with ID starting with 74b8be63f4bc36c37ce8dc06069b08088813453542ff2b23aa71acada6379ab6 not found: ID does not exist" Jul 21 16:09:18.776850 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:09:18.776802 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-682cb4-openshift-default-788d8bdd7-dt5s9"] Jul 21 16:09:20.482269 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:09:20.482223 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dbbade83-f99d-4539-bb0e-4f2fc9d711ff" path="/var/lib/kubelet/pods/dbbade83-f99d-4539-bb0e-4f2fc9d711ff/volumes" Jul 21 16:09:26.535789 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:09:26.535733 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-j4c8p"] Jul 21 16:09:28.816174 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:09:28.816144 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kn9fn_02f3d0c6-4116-4134-ba51-1c02c73bef05/ovn-acl-logging/0.log" Jul 21 16:09:28.841132 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:09:28.841101 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kn9fn_02f3d0c6-4116-4134-ba51-1c02c73bef05/ovn-acl-logging/0.log" Jul 21 16:12:19.879992 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:19.879951 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-26bb92-openshift-default-7b4d9d44f7-6kbfr"] Jul 21 16:12:19.880567 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:19.880255 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-26bb92-openshift-default-7b4d9d44f7-6kbfr" podUID="d3707250-ffeb-4dc2-a97d-a20d0fe845a4" containerName="istio-proxy" containerID="cri-o://940bf0784f3a206f955f8a0fa9437575cef94eb657ecedf60a7a8ef3803e419d" gracePeriod=30 Jul 21 16:12:20.791432 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:20.791395 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-40a33d-openshift-default-b67d4c6b9-7bdzf"] Jul 21 16:12:20.791867 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:20.791852 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbbade83-f99d-4539-bb0e-4f2fc9d711ff" containerName="istio-proxy" Jul 21 16:12:20.791921 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:20.791869 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbbade83-f99d-4539-bb0e-4f2fc9d711ff" containerName="istio-proxy" Jul 21 16:12:20.791921 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:20.791885 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbbade83-f99d-4539-bb0e-4f2fc9d711ff" containerName="istio-proxy" Jul 21 16:12:20.791921 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:20.791891 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbbade83-f99d-4539-bb0e-4f2fc9d711ff" containerName="istio-proxy" Jul 21 16:12:20.791921 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:20.791912 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="147847d9-249f-4372-a72a-5d84418680d8" containerName="main" Jul 21 16:12:20.791921 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:20.791918 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="147847d9-249f-4372-a72a-5d84418680d8" containerName="main" Jul 21 16:12:20.792085 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:20.791977 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="147847d9-249f-4372-a72a-5d84418680d8" containerName="main" Jul 21 16:12:20.792085 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:20.791991 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="dbbade83-f99d-4539-bb0e-4f2fc9d711ff" containerName="istio-proxy" Jul 21 16:12:20.792085 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:20.791999 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="dbbade83-f99d-4539-bb0e-4f2fc9d711ff" containerName="istio-proxy" Jul 21 16:12:20.795156 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:20.795135 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-40a33d-openshift-default-b67d4c6b9-7bdzf" Jul 21 16:12:20.797923 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:20.797895 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-40a33d-openshift-default-dockercfg-7psqq\"" Jul 21 16:12:20.806090 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:20.806067 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-40a33d-openshift-default-b67d4c6b9-7bdzf"] Jul 21 16:12:20.861562 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:20.861526 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e983a18-2bf8-4c06-8ace-120597ed0f8e-credential-socket\") pod \"e2e-api-a-40a33d-openshift-default-b67d4c6b9-7bdzf\" (UID: \"0e983a18-2bf8-4c06-8ace-120597ed0f8e\") " pod="openshift-ingress/e2e-api-a-40a33d-openshift-default-b67d4c6b9-7bdzf" Jul 21 16:12:20.861562 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:20.861564 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e983a18-2bf8-4c06-8ace-120597ed0f8e-istio-token\") pod \"e2e-api-a-40a33d-openshift-default-b67d4c6b9-7bdzf\" (UID: \"0e983a18-2bf8-4c06-8ace-120597ed0f8e\") " pod="openshift-ingress/e2e-api-a-40a33d-openshift-default-b67d4c6b9-7bdzf" Jul 21 16:12:20.861805 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:20.861661 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e983a18-2bf8-4c06-8ace-120597ed0f8e-workload-certs\") pod \"e2e-api-a-40a33d-openshift-default-b67d4c6b9-7bdzf\" (UID: \"0e983a18-2bf8-4c06-8ace-120597ed0f8e\") " pod="openshift-ingress/e2e-api-a-40a33d-openshift-default-b67d4c6b9-7bdzf" Jul 21 16:12:20.861805 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:20.861708 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e983a18-2bf8-4c06-8ace-120597ed0f8e-istio-envoy\") pod \"e2e-api-a-40a33d-openshift-default-b67d4c6b9-7bdzf\" (UID: \"0e983a18-2bf8-4c06-8ace-120597ed0f8e\") " pod="openshift-ingress/e2e-api-a-40a33d-openshift-default-b67d4c6b9-7bdzf" Jul 21 16:12:20.861805 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:20.861743 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e983a18-2bf8-4c06-8ace-120597ed0f8e-istio-podinfo\") pod \"e2e-api-a-40a33d-openshift-default-b67d4c6b9-7bdzf\" (UID: \"0e983a18-2bf8-4c06-8ace-120597ed0f8e\") " pod="openshift-ingress/e2e-api-a-40a33d-openshift-default-b67d4c6b9-7bdzf" Jul 21 16:12:20.861805 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:20.861762 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e983a18-2bf8-4c06-8ace-120597ed0f8e-istio-data\") pod \"e2e-api-a-40a33d-openshift-default-b67d4c6b9-7bdzf\" (UID: \"0e983a18-2bf8-4c06-8ace-120597ed0f8e\") " pod="openshift-ingress/e2e-api-a-40a33d-openshift-default-b67d4c6b9-7bdzf" Jul 21 16:12:20.861805 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:20.861779 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nmdb\" (UniqueName: \"kubernetes.io/projected/0e983a18-2bf8-4c06-8ace-120597ed0f8e-kube-api-access-8nmdb\") pod \"e2e-api-a-40a33d-openshift-default-b67d4c6b9-7bdzf\" (UID: \"0e983a18-2bf8-4c06-8ace-120597ed0f8e\") " pod="openshift-ingress/e2e-api-a-40a33d-openshift-default-b67d4c6b9-7bdzf" Jul 21 16:12:20.862043 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:20.861843 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e983a18-2bf8-4c06-8ace-120597ed0f8e-istiod-ca-cert\") pod \"e2e-api-a-40a33d-openshift-default-b67d4c6b9-7bdzf\" (UID: \"0e983a18-2bf8-4c06-8ace-120597ed0f8e\") " pod="openshift-ingress/e2e-api-a-40a33d-openshift-default-b67d4c6b9-7bdzf" Jul 21 16:12:20.862043 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:20.861919 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e983a18-2bf8-4c06-8ace-120597ed0f8e-workload-socket\") pod \"e2e-api-a-40a33d-openshift-default-b67d4c6b9-7bdzf\" (UID: \"0e983a18-2bf8-4c06-8ace-120597ed0f8e\") " pod="openshift-ingress/e2e-api-a-40a33d-openshift-default-b67d4c6b9-7bdzf" Jul 21 16:12:20.962793 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:20.962730 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e983a18-2bf8-4c06-8ace-120597ed0f8e-istio-podinfo\") pod \"e2e-api-a-40a33d-openshift-default-b67d4c6b9-7bdzf\" (UID: \"0e983a18-2bf8-4c06-8ace-120597ed0f8e\") " pod="openshift-ingress/e2e-api-a-40a33d-openshift-default-b67d4c6b9-7bdzf" Jul 21 16:12:20.963294 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:20.962815 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e983a18-2bf8-4c06-8ace-120597ed0f8e-istio-data\") pod \"e2e-api-a-40a33d-openshift-default-b67d4c6b9-7bdzf\" (UID: \"0e983a18-2bf8-4c06-8ace-120597ed0f8e\") " pod="openshift-ingress/e2e-api-a-40a33d-openshift-default-b67d4c6b9-7bdzf" Jul 21 16:12:20.963294 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:20.962870 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nmdb\" (UniqueName: \"kubernetes.io/projected/0e983a18-2bf8-4c06-8ace-120597ed0f8e-kube-api-access-8nmdb\") pod \"e2e-api-a-40a33d-openshift-default-b67d4c6b9-7bdzf\" (UID: \"0e983a18-2bf8-4c06-8ace-120597ed0f8e\") " pod="openshift-ingress/e2e-api-a-40a33d-openshift-default-b67d4c6b9-7bdzf" Jul 21 16:12:20.963294 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:20.962897 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e983a18-2bf8-4c06-8ace-120597ed0f8e-istiod-ca-cert\") pod \"e2e-api-a-40a33d-openshift-default-b67d4c6b9-7bdzf\" (UID: \"0e983a18-2bf8-4c06-8ace-120597ed0f8e\") " pod="openshift-ingress/e2e-api-a-40a33d-openshift-default-b67d4c6b9-7bdzf" Jul 21 16:12:20.963294 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:20.962957 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e983a18-2bf8-4c06-8ace-120597ed0f8e-workload-socket\") pod \"e2e-api-a-40a33d-openshift-default-b67d4c6b9-7bdzf\" (UID: \"0e983a18-2bf8-4c06-8ace-120597ed0f8e\") " pod="openshift-ingress/e2e-api-a-40a33d-openshift-default-b67d4c6b9-7bdzf" Jul 21 16:12:20.963294 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:20.963109 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e983a18-2bf8-4c06-8ace-120597ed0f8e-credential-socket\") pod \"e2e-api-a-40a33d-openshift-default-b67d4c6b9-7bdzf\" (UID: \"0e983a18-2bf8-4c06-8ace-120597ed0f8e\") " pod="openshift-ingress/e2e-api-a-40a33d-openshift-default-b67d4c6b9-7bdzf" Jul 21 16:12:20.963294 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:20.963134 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e983a18-2bf8-4c06-8ace-120597ed0f8e-istio-token\") pod \"e2e-api-a-40a33d-openshift-default-b67d4c6b9-7bdzf\" (UID: \"0e983a18-2bf8-4c06-8ace-120597ed0f8e\") " pod="openshift-ingress/e2e-api-a-40a33d-openshift-default-b67d4c6b9-7bdzf" Jul 21 16:12:20.963294 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:20.963194 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e983a18-2bf8-4c06-8ace-120597ed0f8e-workload-certs\") pod \"e2e-api-a-40a33d-openshift-default-b67d4c6b9-7bdzf\" (UID: \"0e983a18-2bf8-4c06-8ace-120597ed0f8e\") " pod="openshift-ingress/e2e-api-a-40a33d-openshift-default-b67d4c6b9-7bdzf" Jul 21 16:12:20.963294 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:20.963219 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e983a18-2bf8-4c06-8ace-120597ed0f8e-istio-envoy\") pod \"e2e-api-a-40a33d-openshift-default-b67d4c6b9-7bdzf\" (UID: \"0e983a18-2bf8-4c06-8ace-120597ed0f8e\") " pod="openshift-ingress/e2e-api-a-40a33d-openshift-default-b67d4c6b9-7bdzf" Jul 21 16:12:20.963788 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:20.963361 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e983a18-2bf8-4c06-8ace-120597ed0f8e-istio-data\") pod \"e2e-api-a-40a33d-openshift-default-b67d4c6b9-7bdzf\" (UID: \"0e983a18-2bf8-4c06-8ace-120597ed0f8e\") " pod="openshift-ingress/e2e-api-a-40a33d-openshift-default-b67d4c6b9-7bdzf" Jul 21 16:12:20.963788 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:20.963692 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e983a18-2bf8-4c06-8ace-120597ed0f8e-workload-socket\") pod \"e2e-api-a-40a33d-openshift-default-b67d4c6b9-7bdzf\" (UID: \"0e983a18-2bf8-4c06-8ace-120597ed0f8e\") " pod="openshift-ingress/e2e-api-a-40a33d-openshift-default-b67d4c6b9-7bdzf" Jul 21 16:12:20.963914 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:20.963804 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e983a18-2bf8-4c06-8ace-120597ed0f8e-credential-socket\") pod \"e2e-api-a-40a33d-openshift-default-b67d4c6b9-7bdzf\" (UID: \"0e983a18-2bf8-4c06-8ace-120597ed0f8e\") " pod="openshift-ingress/e2e-api-a-40a33d-openshift-default-b67d4c6b9-7bdzf" Jul 21 16:12:20.964167 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:20.964108 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e983a18-2bf8-4c06-8ace-120597ed0f8e-workload-certs\") pod \"e2e-api-a-40a33d-openshift-default-b67d4c6b9-7bdzf\" (UID: \"0e983a18-2bf8-4c06-8ace-120597ed0f8e\") " pod="openshift-ingress/e2e-api-a-40a33d-openshift-default-b67d4c6b9-7bdzf" Jul 21 16:12:20.964167 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:20.964165 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e983a18-2bf8-4c06-8ace-120597ed0f8e-istiod-ca-cert\") pod \"e2e-api-a-40a33d-openshift-default-b67d4c6b9-7bdzf\" (UID: \"0e983a18-2bf8-4c06-8ace-120597ed0f8e\") " pod="openshift-ingress/e2e-api-a-40a33d-openshift-default-b67d4c6b9-7bdzf" Jul 21 16:12:20.965901 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:20.965869 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e983a18-2bf8-4c06-8ace-120597ed0f8e-istio-envoy\") pod \"e2e-api-a-40a33d-openshift-default-b67d4c6b9-7bdzf\" (UID: \"0e983a18-2bf8-4c06-8ace-120597ed0f8e\") " pod="openshift-ingress/e2e-api-a-40a33d-openshift-default-b67d4c6b9-7bdzf" Jul 21 16:12:20.966262 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:20.966244 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e983a18-2bf8-4c06-8ace-120597ed0f8e-istio-podinfo\") pod \"e2e-api-a-40a33d-openshift-default-b67d4c6b9-7bdzf\" (UID: \"0e983a18-2bf8-4c06-8ace-120597ed0f8e\") " pod="openshift-ingress/e2e-api-a-40a33d-openshift-default-b67d4c6b9-7bdzf" Jul 21 16:12:20.972774 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:20.972745 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e983a18-2bf8-4c06-8ace-120597ed0f8e-istio-token\") pod \"e2e-api-a-40a33d-openshift-default-b67d4c6b9-7bdzf\" (UID: \"0e983a18-2bf8-4c06-8ace-120597ed0f8e\") " pod="openshift-ingress/e2e-api-a-40a33d-openshift-default-b67d4c6b9-7bdzf" Jul 21 16:12:20.972957 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:20.972936 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nmdb\" (UniqueName: \"kubernetes.io/projected/0e983a18-2bf8-4c06-8ace-120597ed0f8e-kube-api-access-8nmdb\") pod \"e2e-api-a-40a33d-openshift-default-b67d4c6b9-7bdzf\" (UID: \"0e983a18-2bf8-4c06-8ace-120597ed0f8e\") " pod="openshift-ingress/e2e-api-a-40a33d-openshift-default-b67d4c6b9-7bdzf" Jul 21 16:12:21.106465 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:21.106425 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-40a33d-openshift-default-b67d4c6b9-7bdzf" Jul 21 16:12:21.252383 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:21.252354 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-40a33d-openshift-default-b67d4c6b9-7bdzf"] Jul 21 16:12:21.253503 ip-10-0-135-176 kubenswrapper[2563]: W0721 16:12:21.253472 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e983a18_2bf8_4c06_8ace_120597ed0f8e.slice/crio-f58db88728d9ff3c850c293359fdf70b3cb4c3b72f1f7d223f3a50052bdff36b WatchSource:0}: Error finding container f58db88728d9ff3c850c293359fdf70b3cb4c3b72f1f7d223f3a50052bdff36b: Status 404 returned error can't find the container with id f58db88728d9ff3c850c293359fdf70b3cb4c3b72f1f7d223f3a50052bdff36b Jul 21 16:12:21.255616 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:21.255580 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 21 16:12:21.255729 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:21.255664 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 21 16:12:21.255729 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:21.255713 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 21 16:12:21.795857 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:21.795819 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-40a33d-openshift-default-b67d4c6b9-7bdzf" event={"ID":"0e983a18-2bf8-4c06-8ace-120597ed0f8e","Type":"ContainerStarted","Data":"c818066e741e0cd984113d9aaacf8bfa03b9fc456f0a5ba13aab78706999a679"} Jul 21 16:12:21.796054 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:21.795884 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-40a33d-openshift-default-b67d4c6b9-7bdzf" event={"ID":"0e983a18-2bf8-4c06-8ace-120597ed0f8e","Type":"ContainerStarted","Data":"f58db88728d9ff3c850c293359fdf70b3cb4c3b72f1f7d223f3a50052bdff36b"} Jul 21 16:12:21.816692 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:21.816635 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-40a33d-openshift-default-b67d4c6b9-7bdzf" podStartSLOduration=1.816614116 podStartE2EDuration="1.816614116s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 16:12:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 16:12:21.815102889 +0000 UTC m=+4373.810888506" watchObservedRunningTime="2026-07-21 16:12:21.816614116 +0000 UTC m=+4373.812399533" Jul 21 16:12:22.106648 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:22.106608 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-40a33d-openshift-default-b67d4c6b9-7bdzf" Jul 21 16:12:22.111979 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:22.111951 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-40a33d-openshift-default-b67d4c6b9-7bdzf" Jul 21 16:12:22.801407 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:22.801368 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-40a33d-openshift-default-b67d4c6b9-7bdzf" Jul 21 16:12:22.802395 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:22.802374 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-40a33d-openshift-default-b67d4c6b9-7bdzf" Jul 21 16:12:25.156459 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:25.156385 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-26bb92-openshift-default-7b4d9d44f7-6kbfr" Jul 21 16:12:25.200837 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:25.200803 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3707250-ffeb-4dc2-a97d-a20d0fe845a4-istio-envoy\") pod \"d3707250-ffeb-4dc2-a97d-a20d0fe845a4\" (UID: \"d3707250-ffeb-4dc2-a97d-a20d0fe845a4\") " Jul 21 16:12:25.200837 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:25.200849 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dt6qc\" (UniqueName: \"kubernetes.io/projected/d3707250-ffeb-4dc2-a97d-a20d0fe845a4-kube-api-access-dt6qc\") pod \"d3707250-ffeb-4dc2-a97d-a20d0fe845a4\" (UID: \"d3707250-ffeb-4dc2-a97d-a20d0fe845a4\") " Jul 21 16:12:25.201120 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:25.200869 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3707250-ffeb-4dc2-a97d-a20d0fe845a4-credential-socket\") pod \"d3707250-ffeb-4dc2-a97d-a20d0fe845a4\" (UID: \"d3707250-ffeb-4dc2-a97d-a20d0fe845a4\") " Jul 21 16:12:25.201120 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:25.200887 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3707250-ffeb-4dc2-a97d-a20d0fe845a4-istio-token\") pod \"d3707250-ffeb-4dc2-a97d-a20d0fe845a4\" (UID: \"d3707250-ffeb-4dc2-a97d-a20d0fe845a4\") " Jul 21 16:12:25.201120 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:25.200904 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3707250-ffeb-4dc2-a97d-a20d0fe845a4-workload-socket\") pod \"d3707250-ffeb-4dc2-a97d-a20d0fe845a4\" (UID: \"d3707250-ffeb-4dc2-a97d-a20d0fe845a4\") " Jul 21 16:12:25.201120 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:25.200937 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3707250-ffeb-4dc2-a97d-a20d0fe845a4-workload-certs\") pod \"d3707250-ffeb-4dc2-a97d-a20d0fe845a4\" (UID: \"d3707250-ffeb-4dc2-a97d-a20d0fe845a4\") " Jul 21 16:12:25.201120 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:25.200990 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3707250-ffeb-4dc2-a97d-a20d0fe845a4-istio-podinfo\") pod \"d3707250-ffeb-4dc2-a97d-a20d0fe845a4\" (UID: \"d3707250-ffeb-4dc2-a97d-a20d0fe845a4\") " Jul 21 16:12:25.201120 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:25.201016 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3707250-ffeb-4dc2-a97d-a20d0fe845a4-istiod-ca-cert\") pod \"d3707250-ffeb-4dc2-a97d-a20d0fe845a4\" (UID: \"d3707250-ffeb-4dc2-a97d-a20d0fe845a4\") " Jul 21 16:12:25.201120 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:25.201058 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3707250-ffeb-4dc2-a97d-a20d0fe845a4-istio-data\") pod \"d3707250-ffeb-4dc2-a97d-a20d0fe845a4\" (UID: \"d3707250-ffeb-4dc2-a97d-a20d0fe845a4\") " Jul 21 16:12:25.201551 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:25.201164 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3707250-ffeb-4dc2-a97d-a20d0fe845a4-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d3707250-ffeb-4dc2-a97d-a20d0fe845a4" (UID: "d3707250-ffeb-4dc2-a97d-a20d0fe845a4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 16:12:25.201551 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:25.201195 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3707250-ffeb-4dc2-a97d-a20d0fe845a4-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d3707250-ffeb-4dc2-a97d-a20d0fe845a4" (UID: "d3707250-ffeb-4dc2-a97d-a20d0fe845a4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 16:12:25.201551 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:25.201392 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3707250-ffeb-4dc2-a97d-a20d0fe845a4-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d3707250-ffeb-4dc2-a97d-a20d0fe845a4" (UID: "d3707250-ffeb-4dc2-a97d-a20d0fe845a4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 16:12:25.201551 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:25.201484 2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3707250-ffeb-4dc2-a97d-a20d0fe845a4-credential-socket\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 16:12:25.201551 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:25.201503 2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3707250-ffeb-4dc2-a97d-a20d0fe845a4-workload-socket\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 16:12:25.201551 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:25.201518 2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3707250-ffeb-4dc2-a97d-a20d0fe845a4-workload-certs\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 16:12:25.201813 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:25.201551 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3707250-ffeb-4dc2-a97d-a20d0fe845a4-istio-data" (OuterVolumeSpecName: "istio-data") pod "d3707250-ffeb-4dc2-a97d-a20d0fe845a4" (UID: "d3707250-ffeb-4dc2-a97d-a20d0fe845a4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 16:12:25.201813 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:25.201571 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3707250-ffeb-4dc2-a97d-a20d0fe845a4-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d3707250-ffeb-4dc2-a97d-a20d0fe845a4" (UID: "d3707250-ffeb-4dc2-a97d-a20d0fe845a4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 16:12:25.203241 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:25.203218 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3707250-ffeb-4dc2-a97d-a20d0fe845a4-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d3707250-ffeb-4dc2-a97d-a20d0fe845a4" (UID: "d3707250-ffeb-4dc2-a97d-a20d0fe845a4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 16:12:25.203691 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:25.203674 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3707250-ffeb-4dc2-a97d-a20d0fe845a4-kube-api-access-dt6qc" (OuterVolumeSpecName: "kube-api-access-dt6qc") pod "d3707250-ffeb-4dc2-a97d-a20d0fe845a4" (UID: "d3707250-ffeb-4dc2-a97d-a20d0fe845a4"). InnerVolumeSpecName "kube-api-access-dt6qc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 16:12:25.203947 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:25.203917 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3707250-ffeb-4dc2-a97d-a20d0fe845a4-istio-token" (OuterVolumeSpecName: "istio-token") pod "d3707250-ffeb-4dc2-a97d-a20d0fe845a4" (UID: "d3707250-ffeb-4dc2-a97d-a20d0fe845a4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 16:12:25.204044 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:25.203995 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d3707250-ffeb-4dc2-a97d-a20d0fe845a4-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d3707250-ffeb-4dc2-a97d-a20d0fe845a4" (UID: "d3707250-ffeb-4dc2-a97d-a20d0fe845a4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 21 16:12:25.302113 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:25.302075 2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3707250-ffeb-4dc2-a97d-a20d0fe845a4-istio-podinfo\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 16:12:25.302113 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:25.302109 2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3707250-ffeb-4dc2-a97d-a20d0fe845a4-istiod-ca-cert\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 16:12:25.302113 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:25.302118 2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3707250-ffeb-4dc2-a97d-a20d0fe845a4-istio-data\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 16:12:25.302113 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:25.302126 2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3707250-ffeb-4dc2-a97d-a20d0fe845a4-istio-envoy\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 16:12:25.302491 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:25.302134 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dt6qc\" (UniqueName: \"kubernetes.io/projected/d3707250-ffeb-4dc2-a97d-a20d0fe845a4-kube-api-access-dt6qc\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 16:12:25.302491 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:25.302143 2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3707250-ffeb-4dc2-a97d-a20d0fe845a4-istio-token\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 16:12:25.820667 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:25.820632 2563 generic.go:356] "Generic (PLEG): container finished" podID="d3707250-ffeb-4dc2-a97d-a20d0fe845a4" containerID="940bf0784f3a206f955f8a0fa9437575cef94eb657ecedf60a7a8ef3803e419d" exitCode=0 Jul 21 16:12:25.820877 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:25.820682 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-26bb92-openshift-default-7b4d9d44f7-6kbfr" Jul 21 16:12:25.820877 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:25.820713 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-26bb92-openshift-default-7b4d9d44f7-6kbfr" event={"ID":"d3707250-ffeb-4dc2-a97d-a20d0fe845a4","Type":"ContainerDied","Data":"940bf0784f3a206f955f8a0fa9437575cef94eb657ecedf60a7a8ef3803e419d"} Jul 21 16:12:25.820877 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:25.820777 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-26bb92-openshift-default-7b4d9d44f7-6kbfr" event={"ID":"d3707250-ffeb-4dc2-a97d-a20d0fe845a4","Type":"ContainerDied","Data":"5a7a0526cfdfcf02c2ce632ae5b347e00277cfd6264de146036fa4e4bd8bfff2"} Jul 21 16:12:25.820877 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:25.820808 2563 scope.go:117] "RemoveContainer" containerID="940bf0784f3a206f955f8a0fa9437575cef94eb657ecedf60a7a8ef3803e419d" Jul 21 16:12:25.834212 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:25.834193 2563 scope.go:117] "RemoveContainer" containerID="4655ba2b77090ed4186da9a15e539c4086039112671ce70892b7205d03eaa1ef" Jul 21 16:12:25.844132 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:25.844103 2563 scope.go:117] "RemoveContainer" containerID="940bf0784f3a206f955f8a0fa9437575cef94eb657ecedf60a7a8ef3803e419d" Jul 21 16:12:25.844534 ip-10-0-135-176 kubenswrapper[2563]: E0721 16:12:25.844509 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"940bf0784f3a206f955f8a0fa9437575cef94eb657ecedf60a7a8ef3803e419d\": container with ID starting with 940bf0784f3a206f955f8a0fa9437575cef94eb657ecedf60a7a8ef3803e419d not found: ID does not exist" containerID="940bf0784f3a206f955f8a0fa9437575cef94eb657ecedf60a7a8ef3803e419d" Jul 21 16:12:25.844649 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:25.844548 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"940bf0784f3a206f955f8a0fa9437575cef94eb657ecedf60a7a8ef3803e419d"} err="failed to get container status \"940bf0784f3a206f955f8a0fa9437575cef94eb657ecedf60a7a8ef3803e419d\": rpc error: code = NotFound desc = could not find container \"940bf0784f3a206f955f8a0fa9437575cef94eb657ecedf60a7a8ef3803e419d\": container with ID starting with 940bf0784f3a206f955f8a0fa9437575cef94eb657ecedf60a7a8ef3803e419d not found: ID does not exist" Jul 21 16:12:25.844649 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:25.844576 2563 scope.go:117] "RemoveContainer" containerID="4655ba2b77090ed4186da9a15e539c4086039112671ce70892b7205d03eaa1ef" Jul 21 16:12:25.844751 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:25.844643 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-26bb92-openshift-default-7b4d9d44f7-6kbfr"] Jul 21 16:12:25.844886 ip-10-0-135-176 kubenswrapper[2563]: E0721 16:12:25.844865 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4655ba2b77090ed4186da9a15e539c4086039112671ce70892b7205d03eaa1ef\": container with ID starting with 4655ba2b77090ed4186da9a15e539c4086039112671ce70892b7205d03eaa1ef not found: ID does not exist" containerID="4655ba2b77090ed4186da9a15e539c4086039112671ce70892b7205d03eaa1ef" Jul 21 16:12:25.844938 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:25.844899 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4655ba2b77090ed4186da9a15e539c4086039112671ce70892b7205d03eaa1ef"} err="failed to get container status \"4655ba2b77090ed4186da9a15e539c4086039112671ce70892b7205d03eaa1ef\": rpc error: code = NotFound desc = could not find container \"4655ba2b77090ed4186da9a15e539c4086039112671ce70892b7205d03eaa1ef\": container with ID starting with 4655ba2b77090ed4186da9a15e539c4086039112671ce70892b7205d03eaa1ef not found: ID does not exist" Jul 21 16:12:25.848949 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:25.848920 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-26bb92-openshift-default-7b4d9d44f7-6kbfr"] Jul 21 16:12:26.445772 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:26.445739 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-j4c8p"] Jul 21 16:12:26.481128 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:26.481093 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d3707250-ffeb-4dc2-a97d-a20d0fe845a4" path="/var/lib/kubelet/pods/d3707250-ffeb-4dc2-a97d-a20d0fe845a4/volumes" Jul 21 16:12:29.100237 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:29.100181 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-fb94bd-openshift-default-85b95d89ff-72mhf"] Jul 21 16:12:29.101438 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:29.101410 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3707250-ffeb-4dc2-a97d-a20d0fe845a4" containerName="istio-proxy" Jul 21 16:12:29.101593 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:29.101581 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3707250-ffeb-4dc2-a97d-a20d0fe845a4" containerName="istio-proxy" Jul 21 16:12:29.101843 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:29.101826 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3707250-ffeb-4dc2-a97d-a20d0fe845a4" containerName="istio-proxy" Jul 21 16:12:29.109661 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:29.109633 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-fb94bd-openshift-default-85b95d89ff-72mhf" Jul 21 16:12:29.113121 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:29.112914 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-fb94bd-openshift-default-dockercfg-j57pj\"" Jul 21 16:12:29.120791 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:29.120648 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-fb94bd-openshift-default-85b95d89ff-72mhf"] Jul 21 16:12:29.138056 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:29.138017 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3d7cd1c-2ffc-4de0-bcd9-cb555a5b10b1-workload-certs\") pod \"e2e-api-b-fb94bd-openshift-default-85b95d89ff-72mhf\" (UID: \"a3d7cd1c-2ffc-4de0-bcd9-cb555a5b10b1\") " pod="openshift-ingress/e2e-api-b-fb94bd-openshift-default-85b95d89ff-72mhf" Jul 21 16:12:29.138207 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:29.138085 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3d7cd1c-2ffc-4de0-bcd9-cb555a5b10b1-istio-token\") pod \"e2e-api-b-fb94bd-openshift-default-85b95d89ff-72mhf\" (UID: \"a3d7cd1c-2ffc-4de0-bcd9-cb555a5b10b1\") " pod="openshift-ingress/e2e-api-b-fb94bd-openshift-default-85b95d89ff-72mhf" Jul 21 16:12:29.138207 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:29.138127 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3d7cd1c-2ffc-4de0-bcd9-cb555a5b10b1-workload-socket\") pod \"e2e-api-b-fb94bd-openshift-default-85b95d89ff-72mhf\" (UID: \"a3d7cd1c-2ffc-4de0-bcd9-cb555a5b10b1\") " pod="openshift-ingress/e2e-api-b-fb94bd-openshift-default-85b95d89ff-72mhf" Jul 21 16:12:29.138207 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:29.138149 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3d7cd1c-2ffc-4de0-bcd9-cb555a5b10b1-istio-podinfo\") pod \"e2e-api-b-fb94bd-openshift-default-85b95d89ff-72mhf\" (UID: \"a3d7cd1c-2ffc-4de0-bcd9-cb555a5b10b1\") " pod="openshift-ingress/e2e-api-b-fb94bd-openshift-default-85b95d89ff-72mhf" Jul 21 16:12:29.138370 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:29.138318 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3d7cd1c-2ffc-4de0-bcd9-cb555a5b10b1-istio-data\") pod \"e2e-api-b-fb94bd-openshift-default-85b95d89ff-72mhf\" (UID: \"a3d7cd1c-2ffc-4de0-bcd9-cb555a5b10b1\") " pod="openshift-ingress/e2e-api-b-fb94bd-openshift-default-85b95d89ff-72mhf" Jul 21 16:12:29.138479 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:29.138463 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jw7rr\" (UniqueName: \"kubernetes.io/projected/a3d7cd1c-2ffc-4de0-bcd9-cb555a5b10b1-kube-api-access-jw7rr\") pod \"e2e-api-b-fb94bd-openshift-default-85b95d89ff-72mhf\" (UID: \"a3d7cd1c-2ffc-4de0-bcd9-cb555a5b10b1\") " pod="openshift-ingress/e2e-api-b-fb94bd-openshift-default-85b95d89ff-72mhf" Jul 21 16:12:29.138571 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:29.138557 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3d7cd1c-2ffc-4de0-bcd9-cb555a5b10b1-istio-envoy\") pod \"e2e-api-b-fb94bd-openshift-default-85b95d89ff-72mhf\" (UID: \"a3d7cd1c-2ffc-4de0-bcd9-cb555a5b10b1\") " pod="openshift-ingress/e2e-api-b-fb94bd-openshift-default-85b95d89ff-72mhf" Jul 21 16:12:29.138650 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:29.138639 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3d7cd1c-2ffc-4de0-bcd9-cb555a5b10b1-credential-socket\") pod \"e2e-api-b-fb94bd-openshift-default-85b95d89ff-72mhf\" (UID: \"a3d7cd1c-2ffc-4de0-bcd9-cb555a5b10b1\") " pod="openshift-ingress/e2e-api-b-fb94bd-openshift-default-85b95d89ff-72mhf" Jul 21 16:12:29.138713 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:29.138705 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3d7cd1c-2ffc-4de0-bcd9-cb555a5b10b1-istiod-ca-cert\") pod \"e2e-api-b-fb94bd-openshift-default-85b95d89ff-72mhf\" (UID: \"a3d7cd1c-2ffc-4de0-bcd9-cb555a5b10b1\") " pod="openshift-ingress/e2e-api-b-fb94bd-openshift-default-85b95d89ff-72mhf" Jul 21 16:12:29.240248 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:29.240204 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3d7cd1c-2ffc-4de0-bcd9-cb555a5b10b1-istio-token\") pod \"e2e-api-b-fb94bd-openshift-default-85b95d89ff-72mhf\" (UID: \"a3d7cd1c-2ffc-4de0-bcd9-cb555a5b10b1\") " pod="openshift-ingress/e2e-api-b-fb94bd-openshift-default-85b95d89ff-72mhf" Jul 21 16:12:29.240248 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:29.240251 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3d7cd1c-2ffc-4de0-bcd9-cb555a5b10b1-workload-socket\") pod \"e2e-api-b-fb94bd-openshift-default-85b95d89ff-72mhf\" (UID: \"a3d7cd1c-2ffc-4de0-bcd9-cb555a5b10b1\") " pod="openshift-ingress/e2e-api-b-fb94bd-openshift-default-85b95d89ff-72mhf" Jul 21 16:12:29.240502 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:29.240268 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3d7cd1c-2ffc-4de0-bcd9-cb555a5b10b1-istio-podinfo\") pod \"e2e-api-b-fb94bd-openshift-default-85b95d89ff-72mhf\" (UID: \"a3d7cd1c-2ffc-4de0-bcd9-cb555a5b10b1\") " pod="openshift-ingress/e2e-api-b-fb94bd-openshift-default-85b95d89ff-72mhf" Jul 21 16:12:29.240502 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:29.240348 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3d7cd1c-2ffc-4de0-bcd9-cb555a5b10b1-istio-data\") pod \"e2e-api-b-fb94bd-openshift-default-85b95d89ff-72mhf\" (UID: \"a3d7cd1c-2ffc-4de0-bcd9-cb555a5b10b1\") " pod="openshift-ingress/e2e-api-b-fb94bd-openshift-default-85b95d89ff-72mhf" Jul 21 16:12:29.240502 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:29.240372 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jw7rr\" (UniqueName: \"kubernetes.io/projected/a3d7cd1c-2ffc-4de0-bcd9-cb555a5b10b1-kube-api-access-jw7rr\") pod \"e2e-api-b-fb94bd-openshift-default-85b95d89ff-72mhf\" (UID: \"a3d7cd1c-2ffc-4de0-bcd9-cb555a5b10b1\") " pod="openshift-ingress/e2e-api-b-fb94bd-openshift-default-85b95d89ff-72mhf" Jul 21 16:12:29.240502 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:29.240400 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3d7cd1c-2ffc-4de0-bcd9-cb555a5b10b1-istio-envoy\") pod \"e2e-api-b-fb94bd-openshift-default-85b95d89ff-72mhf\" (UID: \"a3d7cd1c-2ffc-4de0-bcd9-cb555a5b10b1\") " pod="openshift-ingress/e2e-api-b-fb94bd-openshift-default-85b95d89ff-72mhf" Jul 21 16:12:29.240502 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:29.240426 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3d7cd1c-2ffc-4de0-bcd9-cb555a5b10b1-credential-socket\") pod \"e2e-api-b-fb94bd-openshift-default-85b95d89ff-72mhf\" (UID: \"a3d7cd1c-2ffc-4de0-bcd9-cb555a5b10b1\") " pod="openshift-ingress/e2e-api-b-fb94bd-openshift-default-85b95d89ff-72mhf" Jul 21 16:12:29.240502 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:29.240448 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3d7cd1c-2ffc-4de0-bcd9-cb555a5b10b1-istiod-ca-cert\") pod \"e2e-api-b-fb94bd-openshift-default-85b95d89ff-72mhf\" (UID: \"a3d7cd1c-2ffc-4de0-bcd9-cb555a5b10b1\") " pod="openshift-ingress/e2e-api-b-fb94bd-openshift-default-85b95d89ff-72mhf" Jul 21 16:12:29.240502 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:29.240490 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3d7cd1c-2ffc-4de0-bcd9-cb555a5b10b1-workload-certs\") pod \"e2e-api-b-fb94bd-openshift-default-85b95d89ff-72mhf\" (UID: \"a3d7cd1c-2ffc-4de0-bcd9-cb555a5b10b1\") " pod="openshift-ingress/e2e-api-b-fb94bd-openshift-default-85b95d89ff-72mhf" Jul 21 16:12:29.240936 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:29.240904 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3d7cd1c-2ffc-4de0-bcd9-cb555a5b10b1-workload-certs\") pod \"e2e-api-b-fb94bd-openshift-default-85b95d89ff-72mhf\" (UID: \"a3d7cd1c-2ffc-4de0-bcd9-cb555a5b10b1\") " pod="openshift-ingress/e2e-api-b-fb94bd-openshift-default-85b95d89ff-72mhf" Jul 21 16:12:29.241205 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:29.241151 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3d7cd1c-2ffc-4de0-bcd9-cb555a5b10b1-istio-data\") pod \"e2e-api-b-fb94bd-openshift-default-85b95d89ff-72mhf\" (UID: \"a3d7cd1c-2ffc-4de0-bcd9-cb555a5b10b1\") " pod="openshift-ingress/e2e-api-b-fb94bd-openshift-default-85b95d89ff-72mhf" Jul 21 16:12:29.241367 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:29.241349 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3d7cd1c-2ffc-4de0-bcd9-cb555a5b10b1-workload-socket\") pod \"e2e-api-b-fb94bd-openshift-default-85b95d89ff-72mhf\" (UID: \"a3d7cd1c-2ffc-4de0-bcd9-cb555a5b10b1\") " pod="openshift-ingress/e2e-api-b-fb94bd-openshift-default-85b95d89ff-72mhf" Jul 21 16:12:29.241509 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:29.241488 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3d7cd1c-2ffc-4de0-bcd9-cb555a5b10b1-istiod-ca-cert\") pod \"e2e-api-b-fb94bd-openshift-default-85b95d89ff-72mhf\" (UID: \"a3d7cd1c-2ffc-4de0-bcd9-cb555a5b10b1\") " pod="openshift-ingress/e2e-api-b-fb94bd-openshift-default-85b95d89ff-72mhf" Jul 21 16:12:29.241605 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:29.241585 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3d7cd1c-2ffc-4de0-bcd9-cb555a5b10b1-credential-socket\") pod \"e2e-api-b-fb94bd-openshift-default-85b95d89ff-72mhf\" (UID: \"a3d7cd1c-2ffc-4de0-bcd9-cb555a5b10b1\") " pod="openshift-ingress/e2e-api-b-fb94bd-openshift-default-85b95d89ff-72mhf" Jul 21 16:12:29.243042 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:29.243021 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3d7cd1c-2ffc-4de0-bcd9-cb555a5b10b1-istio-envoy\") pod \"e2e-api-b-fb94bd-openshift-default-85b95d89ff-72mhf\" (UID: \"a3d7cd1c-2ffc-4de0-bcd9-cb555a5b10b1\") " pod="openshift-ingress/e2e-api-b-fb94bd-openshift-default-85b95d89ff-72mhf" Jul 21 16:12:29.243143 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:29.243128 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3d7cd1c-2ffc-4de0-bcd9-cb555a5b10b1-istio-podinfo\") pod \"e2e-api-b-fb94bd-openshift-default-85b95d89ff-72mhf\" (UID: \"a3d7cd1c-2ffc-4de0-bcd9-cb555a5b10b1\") " pod="openshift-ingress/e2e-api-b-fb94bd-openshift-default-85b95d89ff-72mhf" Jul 21 16:12:29.249031 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:29.248993 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3d7cd1c-2ffc-4de0-bcd9-cb555a5b10b1-istio-token\") pod \"e2e-api-b-fb94bd-openshift-default-85b95d89ff-72mhf\" (UID: \"a3d7cd1c-2ffc-4de0-bcd9-cb555a5b10b1\") " pod="openshift-ingress/e2e-api-b-fb94bd-openshift-default-85b95d89ff-72mhf" Jul 21 16:12:29.249134 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:29.249046 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jw7rr\" (UniqueName: \"kubernetes.io/projected/a3d7cd1c-2ffc-4de0-bcd9-cb555a5b10b1-kube-api-access-jw7rr\") pod \"e2e-api-b-fb94bd-openshift-default-85b95d89ff-72mhf\" (UID: \"a3d7cd1c-2ffc-4de0-bcd9-cb555a5b10b1\") " pod="openshift-ingress/e2e-api-b-fb94bd-openshift-default-85b95d89ff-72mhf" Jul 21 16:12:29.433378 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:29.433237 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-fb94bd-openshift-default-85b95d89ff-72mhf" Jul 21 16:12:29.637084 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:29.637056 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-fb94bd-openshift-default-85b95d89ff-72mhf"] Jul 21 16:12:29.640006 ip-10-0-135-176 kubenswrapper[2563]: W0721 16:12:29.639978 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3d7cd1c_2ffc_4de0_bcd9_cb555a5b10b1.slice/crio-8b10b7516a9d677ffd80eb20e7322bc8923dfbe5930d5c554cc0d260c4f3ff77 WatchSource:0}: Error finding container 8b10b7516a9d677ffd80eb20e7322bc8923dfbe5930d5c554cc0d260c4f3ff77: Status 404 returned error can't find the container with id 8b10b7516a9d677ffd80eb20e7322bc8923dfbe5930d5c554cc0d260c4f3ff77 Jul 21 16:12:29.642377 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:29.642342 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 21 16:12:29.642457 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:29.642413 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 21 16:12:29.642457 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:29.642444 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 21 16:12:29.854101 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:29.854069 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-fb94bd-openshift-default-85b95d89ff-72mhf" event={"ID":"a3d7cd1c-2ffc-4de0-bcd9-cb555a5b10b1","Type":"ContainerStarted","Data":"c7cd13ade321ee50080ed24f9fa742af7bea342880b641d82d9bae1b3320312a"} Jul 21 16:12:29.854275 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:29.854145 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-fb94bd-openshift-default-85b95d89ff-72mhf" event={"ID":"a3d7cd1c-2ffc-4de0-bcd9-cb555a5b10b1","Type":"ContainerStarted","Data":"8b10b7516a9d677ffd80eb20e7322bc8923dfbe5930d5c554cc0d260c4f3ff77"} Jul 21 16:12:29.873535 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:29.873476 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-fb94bd-openshift-default-85b95d89ff-72mhf" podStartSLOduration=0.873454489 podStartE2EDuration="873.454489ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 16:12:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 16:12:29.871569779 +0000 UTC m=+4381.867355197" watchObservedRunningTime="2026-07-21 16:12:29.873454489 +0000 UTC m=+4381.869239905" Jul 21 16:12:30.433626 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:30.433583 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-fb94bd-openshift-default-85b95d89ff-72mhf" Jul 21 16:12:30.434911 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:30.434876 2563 patch_prober.go:28] interesting pod/e2e-api-b-fb94bd-openshift-default-85b95d89ff-72mhf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Jul 21 16:12:30.435060 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:30.434934 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-fb94bd-openshift-default-85b95d89ff-72mhf" podUID="a3d7cd1c-2ffc-4de0-bcd9-cb555a5b10b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Jul 21 16:12:31.438718 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:31.438688 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-fb94bd-openshift-default-85b95d89ff-72mhf" Jul 21 16:12:31.865636 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:31.865598 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-fb94bd-openshift-default-85b95d89ff-72mhf" Jul 21 16:12:31.866624 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:31.866605 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-fb94bd-openshift-default-85b95d89ff-72mhf" Jul 21 16:12:36.896982 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:36.896939 2563 generic.go:356] "Generic (PLEG): container finished" podID="0e983a18-2bf8-4c06-8ace-120597ed0f8e" containerID="c818066e741e0cd984113d9aaacf8bfa03b9fc456f0a5ba13aab78706999a679" exitCode=0 Jul 21 16:12:36.897517 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:36.896996 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-40a33d-openshift-default-b67d4c6b9-7bdzf" event={"ID":"0e983a18-2bf8-4c06-8ace-120597ed0f8e","Type":"ContainerDied","Data":"c818066e741e0cd984113d9aaacf8bfa03b9fc456f0a5ba13aab78706999a679"} Jul 21 16:12:36.897517 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:36.897467 2563 scope.go:117] "RemoveContainer" containerID="c818066e741e0cd984113d9aaacf8bfa03b9fc456f0a5ba13aab78706999a679" Jul 21 16:12:36.898624 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:36.898591 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 21 16:12:36.898709 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:36.898671 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 21 16:12:36.898761 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:36.898718 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 21 16:12:37.801672 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:37.801572 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-40a33d-openshift-default-b67d4c6b9-7bdzf" Jul 21 16:12:37.918974 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:37.918936 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-40a33d-openshift-default-b67d4c6b9-7bdzf" event={"ID":"0e983a18-2bf8-4c06-8ace-120597ed0f8e","Type":"ContainerStarted","Data":"e526d876134314cdbeb1cccee04ff1b917e8b6e12fca2228e785c49276bb8139"} Jul 21 16:12:38.107515 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:38.107480 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-40a33d-openshift-default-b67d4c6b9-7bdzf" Jul 21 16:12:38.108687 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:38.108662 2563 patch_prober.go:28] interesting pod/e2e-api-a-40a33d-openshift-default-b67d4c6b9-7bdzf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Jul 21 16:12:38.108750 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:38.108732 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-40a33d-openshift-default-b67d4c6b9-7bdzf" podUID="0e983a18-2bf8-4c06-8ace-120597ed0f8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Jul 21 16:12:39.107698 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:39.107653 2563 patch_prober.go:28] interesting pod/e2e-api-a-40a33d-openshift-default-b67d4c6b9-7bdzf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Jul 21 16:12:39.108218 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:39.107725 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-40a33d-openshift-default-b67d4c6b9-7bdzf" podUID="0e983a18-2bf8-4c06-8ace-120597ed0f8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Jul 21 16:12:39.648049 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:39.648012 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-j4c8p"] Jul 21 16:12:40.107168 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:40.107123 2563 patch_prober.go:28] interesting pod/e2e-api-a-40a33d-openshift-default-b67d4c6b9-7bdzf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Jul 21 16:12:40.107388 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:40.107186 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-40a33d-openshift-default-b67d4c6b9-7bdzf" podUID="0e983a18-2bf8-4c06-8ace-120597ed0f8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Jul 21 16:12:41.107595 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:41.107537 2563 patch_prober.go:28] interesting pod/e2e-api-a-40a33d-openshift-default-b67d4c6b9-7bdzf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Jul 21 16:12:41.108272 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:41.108221 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-40a33d-openshift-default-b67d4c6b9-7bdzf" podUID="0e983a18-2bf8-4c06-8ace-120597ed0f8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Jul 21 16:12:42.107134 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:42.107093 2563 patch_prober.go:28] interesting pod/e2e-api-a-40a33d-openshift-default-b67d4c6b9-7bdzf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Jul 21 16:12:42.107492 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:42.107451 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-40a33d-openshift-default-b67d4c6b9-7bdzf" podUID="0e983a18-2bf8-4c06-8ace-120597ed0f8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Jul 21 16:12:43.107576 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:43.107535 2563 patch_prober.go:28] interesting pod/e2e-api-a-40a33d-openshift-default-b67d4c6b9-7bdzf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Jul 21 16:12:43.108005 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:43.107599 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-40a33d-openshift-default-b67d4c6b9-7bdzf" podUID="0e983a18-2bf8-4c06-8ace-120597ed0f8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Jul 21 16:12:44.111254 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:44.111225 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-40a33d-openshift-default-b67d4c6b9-7bdzf" Jul 21 16:12:44.111678 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:44.111646 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-40a33d-openshift-default-b67d4c6b9-7bdzf" Jul 21 16:12:44.112520 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:44.112502 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-40a33d-openshift-default-b67d4c6b9-7bdzf" Jul 21 16:12:58.037535 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:12:58.037451 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-j4c8p"] Jul 21 16:14:28.855922 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:14:28.855834 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kn9fn_02f3d0c6-4116-4134-ba51-1c02c73bef05/ovn-acl-logging/0.log" Jul 21 16:14:28.879591 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:14:28.879561 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kn9fn_02f3d0c6-4116-4134-ba51-1c02c73bef05/ovn-acl-logging/0.log" Jul 21 16:15:51.174857 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:15:51.174774 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-40a33d-openshift-default-b67d4c6b9-7bdzf"] Jul 21 16:15:51.175780 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:15:51.175005 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-40a33d-openshift-default-b67d4c6b9-7bdzf" podUID="0e983a18-2bf8-4c06-8ace-120597ed0f8e" containerName="istio-proxy" containerID="cri-o://e526d876134314cdbeb1cccee04ff1b917e8b6e12fca2228e785c49276bb8139" gracePeriod=30 Jul 21 16:15:56.441438 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:15:56.441378 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-40a33d-openshift-default-b67d4c6b9-7bdzf" Jul 21 16:15:56.574187 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:15:56.574156 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e983a18-2bf8-4c06-8ace-120597ed0f8e-istio-data\") pod \"0e983a18-2bf8-4c06-8ace-120597ed0f8e\" (UID: \"0e983a18-2bf8-4c06-8ace-120597ed0f8e\") " Jul 21 16:15:56.574381 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:15:56.574198 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e983a18-2bf8-4c06-8ace-120597ed0f8e-credential-socket\") pod \"0e983a18-2bf8-4c06-8ace-120597ed0f8e\" (UID: \"0e983a18-2bf8-4c06-8ace-120597ed0f8e\") " Jul 21 16:15:56.574381 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:15:56.574219 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e983a18-2bf8-4c06-8ace-120597ed0f8e-istio-podinfo\") pod \"0e983a18-2bf8-4c06-8ace-120597ed0f8e\" (UID: \"0e983a18-2bf8-4c06-8ace-120597ed0f8e\") " Jul 21 16:15:56.574381 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:15:56.574240 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e983a18-2bf8-4c06-8ace-120597ed0f8e-istio-token\") pod \"0e983a18-2bf8-4c06-8ace-120597ed0f8e\" (UID: \"0e983a18-2bf8-4c06-8ace-120597ed0f8e\") " Jul 21 16:15:56.574381 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:15:56.574262 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e983a18-2bf8-4c06-8ace-120597ed0f8e-workload-socket\") pod \"0e983a18-2bf8-4c06-8ace-120597ed0f8e\" (UID: \"0e983a18-2bf8-4c06-8ace-120597ed0f8e\") " Jul 21 16:15:56.574381 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:15:56.574282 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8nmdb\" (UniqueName: \"kubernetes.io/projected/0e983a18-2bf8-4c06-8ace-120597ed0f8e-kube-api-access-8nmdb\") pod \"0e983a18-2bf8-4c06-8ace-120597ed0f8e\" (UID: \"0e983a18-2bf8-4c06-8ace-120597ed0f8e\") " Jul 21 16:15:56.574381 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:15:56.574347 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e983a18-2bf8-4c06-8ace-120597ed0f8e-workload-certs\") pod \"0e983a18-2bf8-4c06-8ace-120597ed0f8e\" (UID: \"0e983a18-2bf8-4c06-8ace-120597ed0f8e\") " Jul 21 16:15:56.574727 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:15:56.574440 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e983a18-2bf8-4c06-8ace-120597ed0f8e-istio-envoy\") pod \"0e983a18-2bf8-4c06-8ace-120597ed0f8e\" (UID: \"0e983a18-2bf8-4c06-8ace-120597ed0f8e\") " Jul 21 16:15:56.574727 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:15:56.574467 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e983a18-2bf8-4c06-8ace-120597ed0f8e-istiod-ca-cert\") pod \"0e983a18-2bf8-4c06-8ace-120597ed0f8e\" (UID: \"0e983a18-2bf8-4c06-8ace-120597ed0f8e\") " Jul 21 16:15:56.574727 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:15:56.574496 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e983a18-2bf8-4c06-8ace-120597ed0f8e-istio-data" (OuterVolumeSpecName: "istio-data") pod "0e983a18-2bf8-4c06-8ace-120597ed0f8e" (UID: "0e983a18-2bf8-4c06-8ace-120597ed0f8e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 16:15:56.574727 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:15:56.574600 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e983a18-2bf8-4c06-8ace-120597ed0f8e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0e983a18-2bf8-4c06-8ace-120597ed0f8e" (UID: "0e983a18-2bf8-4c06-8ace-120597ed0f8e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 16:15:56.574727 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:15:56.574674 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e983a18-2bf8-4c06-8ace-120597ed0f8e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0e983a18-2bf8-4c06-8ace-120597ed0f8e" (UID: "0e983a18-2bf8-4c06-8ace-120597ed0f8e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 16:15:56.574966 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:15:56.574954 2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e983a18-2bf8-4c06-8ace-120597ed0f8e-istio-data\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 16:15:56.575021 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:15:56.574966 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e983a18-2bf8-4c06-8ace-120597ed0f8e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0e983a18-2bf8-4c06-8ace-120597ed0f8e" (UID: "0e983a18-2bf8-4c06-8ace-120597ed0f8e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 16:15:56.575021 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:15:56.574977 2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e983a18-2bf8-4c06-8ace-120597ed0f8e-workload-socket\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 16:15:56.575129 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:15:56.575022 2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e983a18-2bf8-4c06-8ace-120597ed0f8e-workload-certs\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 16:15:56.575281 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:15:56.575249 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e983a18-2bf8-4c06-8ace-120597ed0f8e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0e983a18-2bf8-4c06-8ace-120597ed0f8e" (UID: "0e983a18-2bf8-4c06-8ace-120597ed0f8e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 16:15:56.576713 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:15:56.576693 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e983a18-2bf8-4c06-8ace-120597ed0f8e-istio-token" (OuterVolumeSpecName: "istio-token") pod "0e983a18-2bf8-4c06-8ace-120597ed0f8e" (UID: "0e983a18-2bf8-4c06-8ace-120597ed0f8e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 16:15:56.576834 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:15:56.576753 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e983a18-2bf8-4c06-8ace-120597ed0f8e-kube-api-access-8nmdb" (OuterVolumeSpecName: "kube-api-access-8nmdb") pod "0e983a18-2bf8-4c06-8ace-120597ed0f8e" (UID: "0e983a18-2bf8-4c06-8ace-120597ed0f8e"). InnerVolumeSpecName "kube-api-access-8nmdb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 16:15:56.577101 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:15:56.577079 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0e983a18-2bf8-4c06-8ace-120597ed0f8e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0e983a18-2bf8-4c06-8ace-120597ed0f8e" (UID: "0e983a18-2bf8-4c06-8ace-120597ed0f8e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 21 16:15:56.577101 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:15:56.577089 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e983a18-2bf8-4c06-8ace-120597ed0f8e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0e983a18-2bf8-4c06-8ace-120597ed0f8e" (UID: "0e983a18-2bf8-4c06-8ace-120597ed0f8e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 16:15:56.675956 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:15:56.675923 2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e983a18-2bf8-4c06-8ace-120597ed0f8e-credential-socket\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 16:15:56.675956 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:15:56.675952 2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e983a18-2bf8-4c06-8ace-120597ed0f8e-istio-podinfo\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 16:15:56.675956 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:15:56.675962 2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e983a18-2bf8-4c06-8ace-120597ed0f8e-istio-token\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 16:15:56.676203 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:15:56.675971 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8nmdb\" (UniqueName: \"kubernetes.io/projected/0e983a18-2bf8-4c06-8ace-120597ed0f8e-kube-api-access-8nmdb\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 16:15:56.676203 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:15:56.675979 2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e983a18-2bf8-4c06-8ace-120597ed0f8e-istio-envoy\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 16:15:56.676203 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:15:56.675987 2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e983a18-2bf8-4c06-8ace-120597ed0f8e-istiod-ca-cert\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 16:15:57.044704 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:15:57.044667 2563 generic.go:356] "Generic (PLEG): container finished" podID="0e983a18-2bf8-4c06-8ace-120597ed0f8e" containerID="e526d876134314cdbeb1cccee04ff1b917e8b6e12fca2228e785c49276bb8139" exitCode=0 Jul 21 16:15:57.044935 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:15:57.044710 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-40a33d-openshift-default-b67d4c6b9-7bdzf" Jul 21 16:15:57.044935 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:15:57.044737 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-40a33d-openshift-default-b67d4c6b9-7bdzf" event={"ID":"0e983a18-2bf8-4c06-8ace-120597ed0f8e","Type":"ContainerDied","Data":"e526d876134314cdbeb1cccee04ff1b917e8b6e12fca2228e785c49276bb8139"} Jul 21 16:15:57.044935 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:15:57.044785 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-40a33d-openshift-default-b67d4c6b9-7bdzf" event={"ID":"0e983a18-2bf8-4c06-8ace-120597ed0f8e","Type":"ContainerDied","Data":"f58db88728d9ff3c850c293359fdf70b3cb4c3b72f1f7d223f3a50052bdff36b"} Jul 21 16:15:57.044935 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:15:57.044806 2563 scope.go:117] "RemoveContainer" containerID="e526d876134314cdbeb1cccee04ff1b917e8b6e12fca2228e785c49276bb8139" Jul 21 16:15:57.054820 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:15:57.054801 2563 scope.go:117] "RemoveContainer" containerID="c818066e741e0cd984113d9aaacf8bfa03b9fc456f0a5ba13aab78706999a679" Jul 21 16:15:57.063691 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:15:57.063675 2563 scope.go:117] "RemoveContainer" containerID="e526d876134314cdbeb1cccee04ff1b917e8b6e12fca2228e785c49276bb8139" Jul 21 16:15:57.063958 ip-10-0-135-176 kubenswrapper[2563]: E0721 16:15:57.063939 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e526d876134314cdbeb1cccee04ff1b917e8b6e12fca2228e785c49276bb8139\": container with ID starting with e526d876134314cdbeb1cccee04ff1b917e8b6e12fca2228e785c49276bb8139 not found: ID does not exist" containerID="e526d876134314cdbeb1cccee04ff1b917e8b6e12fca2228e785c49276bb8139" Jul 21 16:15:57.064019 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:15:57.063967 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e526d876134314cdbeb1cccee04ff1b917e8b6e12fca2228e785c49276bb8139"} err="failed to get container status \"e526d876134314cdbeb1cccee04ff1b917e8b6e12fca2228e785c49276bb8139\": rpc error: code = NotFound desc = could not find container \"e526d876134314cdbeb1cccee04ff1b917e8b6e12fca2228e785c49276bb8139\": container with ID starting with e526d876134314cdbeb1cccee04ff1b917e8b6e12fca2228e785c49276bb8139 not found: ID does not exist" Jul 21 16:15:57.064019 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:15:57.063984 2563 scope.go:117] "RemoveContainer" containerID="c818066e741e0cd984113d9aaacf8bfa03b9fc456f0a5ba13aab78706999a679" Jul 21 16:15:57.064198 ip-10-0-135-176 kubenswrapper[2563]: E0721 16:15:57.064176 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c818066e741e0cd984113d9aaacf8bfa03b9fc456f0a5ba13aab78706999a679\": container with ID starting with c818066e741e0cd984113d9aaacf8bfa03b9fc456f0a5ba13aab78706999a679 not found: ID does not exist" containerID="c818066e741e0cd984113d9aaacf8bfa03b9fc456f0a5ba13aab78706999a679" Jul 21 16:15:57.064319 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:15:57.064234 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c818066e741e0cd984113d9aaacf8bfa03b9fc456f0a5ba13aab78706999a679"} err="failed to get container status \"c818066e741e0cd984113d9aaacf8bfa03b9fc456f0a5ba13aab78706999a679\": rpc error: code = NotFound desc = could not find container \"c818066e741e0cd984113d9aaacf8bfa03b9fc456f0a5ba13aab78706999a679\": container with ID starting with c818066e741e0cd984113d9aaacf8bfa03b9fc456f0a5ba13aab78706999a679 not found: ID does not exist" Jul 21 16:15:57.067239 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:15:57.067216 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-40a33d-openshift-default-b67d4c6b9-7bdzf"] Jul 21 16:15:57.071495 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:15:57.071460 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-40a33d-openshift-default-b67d4c6b9-7bdzf"] Jul 21 16:15:58.480512 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:15:58.480479 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e983a18-2bf8-4c06-8ace-120597ed0f8e" path="/var/lib/kubelet/pods/0e983a18-2bf8-4c06-8ace-120597ed0f8e/volumes" Jul 21 16:16:04.335215 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:16:04.335177 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-j4c8p"] Jul 21 16:18:57.898370 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:18:57.898268 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-fb94bd-openshift-default-85b95d89ff-72mhf"] Jul 21 16:18:57.900939 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:18:57.898563 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-fb94bd-openshift-default-85b95d89ff-72mhf" podUID="a3d7cd1c-2ffc-4de0-bcd9-cb555a5b10b1" containerName="istio-proxy" containerID="cri-o://c7cd13ade321ee50080ed24f9fa742af7bea342880b641d82d9bae1b3320312a" gracePeriod=30 Jul 21 16:18:58.843238 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:18:58.843206 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-524b80-openshift-default-9cb6b55d7-lwd66"] Jul 21 16:18:58.843642 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:18:58.843620 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e983a18-2bf8-4c06-8ace-120597ed0f8e" containerName="istio-proxy" Jul 21 16:18:58.843642 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:18:58.843641 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e983a18-2bf8-4c06-8ace-120597ed0f8e" containerName="istio-proxy" Jul 21 16:18:58.843813 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:18:58.843659 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e983a18-2bf8-4c06-8ace-120597ed0f8e" containerName="istio-proxy" Jul 21 16:18:58.843813 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:18:58.843664 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e983a18-2bf8-4c06-8ace-120597ed0f8e" containerName="istio-proxy" Jul 21 16:18:58.843813 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:18:58.843679 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3707250-ffeb-4dc2-a97d-a20d0fe845a4" containerName="istio-proxy" Jul 21 16:18:58.843813 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:18:58.843684 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3707250-ffeb-4dc2-a97d-a20d0fe845a4" containerName="istio-proxy" Jul 21 16:18:58.843813 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:18:58.843768 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3707250-ffeb-4dc2-a97d-a20d0fe845a4" containerName="istio-proxy" Jul 21 16:18:58.843813 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:18:58.843781 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e983a18-2bf8-4c06-8ace-120597ed0f8e" containerName="istio-proxy" Jul 21 16:18:58.843813 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:18:58.843787 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e983a18-2bf8-4c06-8ace-120597ed0f8e" containerName="istio-proxy" Jul 21 16:18:58.846884 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:18:58.846866 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-524b80-openshift-default-9cb6b55d7-lwd66" Jul 21 16:18:58.849607 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:18:58.849585 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-524b80-openshift-default-dockercfg-5zmdq\"" Jul 21 16:18:58.857497 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:18:58.857469 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-524b80-openshift-default-9cb6b55d7-lwd66"] Jul 21 16:18:58.915390 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:18:58.915360 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9707e0e1-e63c-4336-b2c1-aba2929a0b75-workload-socket\") pod \"e2e-shared-a-524b80-openshift-default-9cb6b55d7-lwd66\" (UID: \"9707e0e1-e63c-4336-b2c1-aba2929a0b75\") " pod="openshift-ingress/e2e-shared-a-524b80-openshift-default-9cb6b55d7-lwd66" Jul 21 16:18:58.915390 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:18:58.915390 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9707e0e1-e63c-4336-b2c1-aba2929a0b75-workload-certs\") pod \"e2e-shared-a-524b80-openshift-default-9cb6b55d7-lwd66\" (UID: \"9707e0e1-e63c-4336-b2c1-aba2929a0b75\") " pod="openshift-ingress/e2e-shared-a-524b80-openshift-default-9cb6b55d7-lwd66" Jul 21 16:18:58.915803 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:18:58.915410 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9707e0e1-e63c-4336-b2c1-aba2929a0b75-istio-token\") pod \"e2e-shared-a-524b80-openshift-default-9cb6b55d7-lwd66\" (UID: \"9707e0e1-e63c-4336-b2c1-aba2929a0b75\") " pod="openshift-ingress/e2e-shared-a-524b80-openshift-default-9cb6b55d7-lwd66" Jul 21 16:18:58.915803 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:18:58.915495 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9707e0e1-e63c-4336-b2c1-aba2929a0b75-istiod-ca-cert\") pod \"e2e-shared-a-524b80-openshift-default-9cb6b55d7-lwd66\" (UID: \"9707e0e1-e63c-4336-b2c1-aba2929a0b75\") " pod="openshift-ingress/e2e-shared-a-524b80-openshift-default-9cb6b55d7-lwd66" Jul 21 16:18:58.915803 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:18:58.915544 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9707e0e1-e63c-4336-b2c1-aba2929a0b75-istio-data\") pod \"e2e-shared-a-524b80-openshift-default-9cb6b55d7-lwd66\" (UID: \"9707e0e1-e63c-4336-b2c1-aba2929a0b75\") " pod="openshift-ingress/e2e-shared-a-524b80-openshift-default-9cb6b55d7-lwd66" Jul 21 16:18:58.915803 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:18:58.915574 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9707e0e1-e63c-4336-b2c1-aba2929a0b75-credential-socket\") pod \"e2e-shared-a-524b80-openshift-default-9cb6b55d7-lwd66\" (UID: \"9707e0e1-e63c-4336-b2c1-aba2929a0b75\") " pod="openshift-ingress/e2e-shared-a-524b80-openshift-default-9cb6b55d7-lwd66" Jul 21 16:18:58.915803 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:18:58.915590 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9707e0e1-e63c-4336-b2c1-aba2929a0b75-istio-podinfo\") pod \"e2e-shared-a-524b80-openshift-default-9cb6b55d7-lwd66\" (UID: \"9707e0e1-e63c-4336-b2c1-aba2929a0b75\") " pod="openshift-ingress/e2e-shared-a-524b80-openshift-default-9cb6b55d7-lwd66" Jul 21 16:18:58.915803 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:18:58.915643 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9707e0e1-e63c-4336-b2c1-aba2929a0b75-istio-envoy\") pod \"e2e-shared-a-524b80-openshift-default-9cb6b55d7-lwd66\" (UID: \"9707e0e1-e63c-4336-b2c1-aba2929a0b75\") " pod="openshift-ingress/e2e-shared-a-524b80-openshift-default-9cb6b55d7-lwd66" Jul 21 16:18:58.915803 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:18:58.915710 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxrtl\" (UniqueName: \"kubernetes.io/projected/9707e0e1-e63c-4336-b2c1-aba2929a0b75-kube-api-access-wxrtl\") pod \"e2e-shared-a-524b80-openshift-default-9cb6b55d7-lwd66\" (UID: \"9707e0e1-e63c-4336-b2c1-aba2929a0b75\") " pod="openshift-ingress/e2e-shared-a-524b80-openshift-default-9cb6b55d7-lwd66" Jul 21 16:18:59.016658 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:18:59.016620 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9707e0e1-e63c-4336-b2c1-aba2929a0b75-istiod-ca-cert\") pod \"e2e-shared-a-524b80-openshift-default-9cb6b55d7-lwd66\" (UID: \"9707e0e1-e63c-4336-b2c1-aba2929a0b75\") " pod="openshift-ingress/e2e-shared-a-524b80-openshift-default-9cb6b55d7-lwd66" Jul 21 16:18:59.016858 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:18:59.016665 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9707e0e1-e63c-4336-b2c1-aba2929a0b75-istio-data\") pod \"e2e-shared-a-524b80-openshift-default-9cb6b55d7-lwd66\" (UID: \"9707e0e1-e63c-4336-b2c1-aba2929a0b75\") " pod="openshift-ingress/e2e-shared-a-524b80-openshift-default-9cb6b55d7-lwd66" Jul 21 16:18:59.016858 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:18:59.016690 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9707e0e1-e63c-4336-b2c1-aba2929a0b75-credential-socket\") pod \"e2e-shared-a-524b80-openshift-default-9cb6b55d7-lwd66\" (UID: \"9707e0e1-e63c-4336-b2c1-aba2929a0b75\") " pod="openshift-ingress/e2e-shared-a-524b80-openshift-default-9cb6b55d7-lwd66" Jul 21 16:18:59.016858 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:18:59.016705 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9707e0e1-e63c-4336-b2c1-aba2929a0b75-istio-podinfo\") pod \"e2e-shared-a-524b80-openshift-default-9cb6b55d7-lwd66\" (UID: \"9707e0e1-e63c-4336-b2c1-aba2929a0b75\") " pod="openshift-ingress/e2e-shared-a-524b80-openshift-default-9cb6b55d7-lwd66" Jul 21 16:18:59.016858 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:18:59.016727 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9707e0e1-e63c-4336-b2c1-aba2929a0b75-istio-envoy\") pod \"e2e-shared-a-524b80-openshift-default-9cb6b55d7-lwd66\" (UID: \"9707e0e1-e63c-4336-b2c1-aba2929a0b75\") " pod="openshift-ingress/e2e-shared-a-524b80-openshift-default-9cb6b55d7-lwd66" Jul 21 16:18:59.016858 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:18:59.016771 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxrtl\" (UniqueName: \"kubernetes.io/projected/9707e0e1-e63c-4336-b2c1-aba2929a0b75-kube-api-access-wxrtl\") pod \"e2e-shared-a-524b80-openshift-default-9cb6b55d7-lwd66\" (UID: \"9707e0e1-e63c-4336-b2c1-aba2929a0b75\") " pod="openshift-ingress/e2e-shared-a-524b80-openshift-default-9cb6b55d7-lwd66" Jul 21 16:18:59.016858 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:18:59.016825 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9707e0e1-e63c-4336-b2c1-aba2929a0b75-workload-socket\") pod \"e2e-shared-a-524b80-openshift-default-9cb6b55d7-lwd66\" (UID: \"9707e0e1-e63c-4336-b2c1-aba2929a0b75\") " pod="openshift-ingress/e2e-shared-a-524b80-openshift-default-9cb6b55d7-lwd66" Jul 21 16:18:59.017188 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:18:59.016946 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9707e0e1-e63c-4336-b2c1-aba2929a0b75-workload-certs\") pod \"e2e-shared-a-524b80-openshift-default-9cb6b55d7-lwd66\" (UID: \"9707e0e1-e63c-4336-b2c1-aba2929a0b75\") " pod="openshift-ingress/e2e-shared-a-524b80-openshift-default-9cb6b55d7-lwd66" Jul 21 16:18:59.017188 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:18:59.016994 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9707e0e1-e63c-4336-b2c1-aba2929a0b75-istio-token\") pod \"e2e-shared-a-524b80-openshift-default-9cb6b55d7-lwd66\" (UID: \"9707e0e1-e63c-4336-b2c1-aba2929a0b75\") " pod="openshift-ingress/e2e-shared-a-524b80-openshift-default-9cb6b55d7-lwd66" Jul 21 16:18:59.017188 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:18:59.017101 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9707e0e1-e63c-4336-b2c1-aba2929a0b75-istio-data\") pod \"e2e-shared-a-524b80-openshift-default-9cb6b55d7-lwd66\" (UID: \"9707e0e1-e63c-4336-b2c1-aba2929a0b75\") " pod="openshift-ingress/e2e-shared-a-524b80-openshift-default-9cb6b55d7-lwd66" Jul 21 16:18:59.017188 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:18:59.017177 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9707e0e1-e63c-4336-b2c1-aba2929a0b75-workload-socket\") pod \"e2e-shared-a-524b80-openshift-default-9cb6b55d7-lwd66\" (UID: \"9707e0e1-e63c-4336-b2c1-aba2929a0b75\") " pod="openshift-ingress/e2e-shared-a-524b80-openshift-default-9cb6b55d7-lwd66" Jul 21 16:18:59.017438 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:18:59.017372 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9707e0e1-e63c-4336-b2c1-aba2929a0b75-workload-certs\") pod \"e2e-shared-a-524b80-openshift-default-9cb6b55d7-lwd66\" (UID: \"9707e0e1-e63c-4336-b2c1-aba2929a0b75\") " pod="openshift-ingress/e2e-shared-a-524b80-openshift-default-9cb6b55d7-lwd66" Jul 21 16:18:59.017502 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:18:59.017461 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9707e0e1-e63c-4336-b2c1-aba2929a0b75-istiod-ca-cert\") pod \"e2e-shared-a-524b80-openshift-default-9cb6b55d7-lwd66\" (UID: \"9707e0e1-e63c-4336-b2c1-aba2929a0b75\") " pod="openshift-ingress/e2e-shared-a-524b80-openshift-default-9cb6b55d7-lwd66" Jul 21 16:18:59.017598 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:18:59.017579 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9707e0e1-e63c-4336-b2c1-aba2929a0b75-credential-socket\") pod \"e2e-shared-a-524b80-openshift-default-9cb6b55d7-lwd66\" (UID: \"9707e0e1-e63c-4336-b2c1-aba2929a0b75\") " pod="openshift-ingress/e2e-shared-a-524b80-openshift-default-9cb6b55d7-lwd66" Jul 21 16:18:59.019222 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:18:59.019203 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9707e0e1-e63c-4336-b2c1-aba2929a0b75-istio-envoy\") pod \"e2e-shared-a-524b80-openshift-default-9cb6b55d7-lwd66\" (UID: \"9707e0e1-e63c-4336-b2c1-aba2929a0b75\") " pod="openshift-ingress/e2e-shared-a-524b80-openshift-default-9cb6b55d7-lwd66" Jul 21 16:18:59.019465 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:18:59.019445 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9707e0e1-e63c-4336-b2c1-aba2929a0b75-istio-podinfo\") pod \"e2e-shared-a-524b80-openshift-default-9cb6b55d7-lwd66\" (UID: \"9707e0e1-e63c-4336-b2c1-aba2929a0b75\") " pod="openshift-ingress/e2e-shared-a-524b80-openshift-default-9cb6b55d7-lwd66" Jul 21 16:18:59.025646 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:18:59.025621 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9707e0e1-e63c-4336-b2c1-aba2929a0b75-istio-token\") pod \"e2e-shared-a-524b80-openshift-default-9cb6b55d7-lwd66\" (UID: \"9707e0e1-e63c-4336-b2c1-aba2929a0b75\") " pod="openshift-ingress/e2e-shared-a-524b80-openshift-default-9cb6b55d7-lwd66" Jul 21 16:18:59.025779 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:18:59.025762 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxrtl\" (UniqueName: \"kubernetes.io/projected/9707e0e1-e63c-4336-b2c1-aba2929a0b75-kube-api-access-wxrtl\") pod \"e2e-shared-a-524b80-openshift-default-9cb6b55d7-lwd66\" (UID: \"9707e0e1-e63c-4336-b2c1-aba2929a0b75\") " pod="openshift-ingress/e2e-shared-a-524b80-openshift-default-9cb6b55d7-lwd66" Jul 21 16:18:59.158435 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:18:59.158320 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-524b80-openshift-default-9cb6b55d7-lwd66" Jul 21 16:18:59.311348 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:18:59.311303 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-524b80-openshift-default-9cb6b55d7-lwd66"] Jul 21 16:18:59.312518 ip-10-0-135-176 kubenswrapper[2563]: W0721 16:18:59.312476 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9707e0e1_e63c_4336_b2c1_aba2929a0b75.slice/crio-a2d0924cd9558714d25b66a012292414de58cced9dad79554fa03f7c208ed37d WatchSource:0}: Error finding container a2d0924cd9558714d25b66a012292414de58cced9dad79554fa03f7c208ed37d: Status 404 returned error can't find the container with id a2d0924cd9558714d25b66a012292414de58cced9dad79554fa03f7c208ed37d Jul 21 16:18:59.315125 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:18:59.315083 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 21 16:18:59.315201 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:18:59.315175 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 21 16:18:59.315238 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:18:59.315225 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 21 16:19:00.048060 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:00.048022 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-524b80-openshift-default-9cb6b55d7-lwd66" event={"ID":"9707e0e1-e63c-4336-b2c1-aba2929a0b75","Type":"ContainerStarted","Data":"78cb32b24644a423c3a0dbb614f97ef1b92e588c66106f069e1fc545d7198276"} Jul 21 16:19:00.048478 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:00.048120 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-524b80-openshift-default-9cb6b55d7-lwd66" event={"ID":"9707e0e1-e63c-4336-b2c1-aba2929a0b75","Type":"ContainerStarted","Data":"a2d0924cd9558714d25b66a012292414de58cced9dad79554fa03f7c208ed37d"} Jul 21 16:19:00.069312 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:00.069259 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-524b80-openshift-default-9cb6b55d7-lwd66" podStartSLOduration=2.06924502 podStartE2EDuration="2.06924502s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 16:18:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 16:19:00.06879479 +0000 UTC m=+4772.064580207" watchObservedRunningTime="2026-07-21 16:19:00.06924502 +0000 UTC m=+4772.065030436" Jul 21 16:19:00.159431 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:00.159393 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-524b80-openshift-default-9cb6b55d7-lwd66" Jul 21 16:19:00.164140 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:00.164116 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-524b80-openshift-default-9cb6b55d7-lwd66" Jul 21 16:19:01.054089 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:01.054052 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-524b80-openshift-default-9cb6b55d7-lwd66" Jul 21 16:19:01.055049 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:01.055031 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-524b80-openshift-default-9cb6b55d7-lwd66" Jul 21 16:19:01.868128 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:01.868088 2563 patch_prober.go:28] interesting pod/e2e-api-b-fb94bd-openshift-default-85b95d89ff-72mhf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 21 16:19:01.868315 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:01.868146 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-fb94bd-openshift-default-85b95d89ff-72mhf" podUID="a3d7cd1c-2ffc-4de0-bcd9-cb555a5b10b1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 21 16:19:03.067843 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:03.067807 2563 generic.go:356] "Generic (PLEG): container finished" podID="a3d7cd1c-2ffc-4de0-bcd9-cb555a5b10b1" containerID="c7cd13ade321ee50080ed24f9fa742af7bea342880b641d82d9bae1b3320312a" exitCode=0 Jul 21 16:19:03.068278 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:03.067885 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-fb94bd-openshift-default-85b95d89ff-72mhf" event={"ID":"a3d7cd1c-2ffc-4de0-bcd9-cb555a5b10b1","Type":"ContainerDied","Data":"c7cd13ade321ee50080ed24f9fa742af7bea342880b641d82d9bae1b3320312a"} Jul 21 16:19:03.176378 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:03.176306 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-fb94bd-openshift-default-85b95d89ff-72mhf" Jul 21 16:19:03.260755 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:03.260719 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3d7cd1c-2ffc-4de0-bcd9-cb555a5b10b1-workload-socket\") pod \"a3d7cd1c-2ffc-4de0-bcd9-cb555a5b10b1\" (UID: \"a3d7cd1c-2ffc-4de0-bcd9-cb555a5b10b1\") " Jul 21 16:19:03.260755 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:03.260760 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3d7cd1c-2ffc-4de0-bcd9-cb555a5b10b1-istiod-ca-cert\") pod \"a3d7cd1c-2ffc-4de0-bcd9-cb555a5b10b1\" (UID: \"a3d7cd1c-2ffc-4de0-bcd9-cb555a5b10b1\") " Jul 21 16:19:03.261028 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:03.260784 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3d7cd1c-2ffc-4de0-bcd9-cb555a5b10b1-credential-socket\") pod \"a3d7cd1c-2ffc-4de0-bcd9-cb555a5b10b1\" (UID: \"a3d7cd1c-2ffc-4de0-bcd9-cb555a5b10b1\") " Jul 21 16:19:03.261028 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:03.260827 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3d7cd1c-2ffc-4de0-bcd9-cb555a5b10b1-workload-certs\") pod \"a3d7cd1c-2ffc-4de0-bcd9-cb555a5b10b1\" (UID: \"a3d7cd1c-2ffc-4de0-bcd9-cb555a5b10b1\") " Jul 21 16:19:03.261028 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:03.260876 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3d7cd1c-2ffc-4de0-bcd9-cb555a5b10b1-istio-envoy\") pod \"a3d7cd1c-2ffc-4de0-bcd9-cb555a5b10b1\" (UID: \"a3d7cd1c-2ffc-4de0-bcd9-cb555a5b10b1\") " Jul 21 16:19:03.261028 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:03.260912 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jw7rr\" (UniqueName: \"kubernetes.io/projected/a3d7cd1c-2ffc-4de0-bcd9-cb555a5b10b1-kube-api-access-jw7rr\") pod \"a3d7cd1c-2ffc-4de0-bcd9-cb555a5b10b1\" (UID: \"a3d7cd1c-2ffc-4de0-bcd9-cb555a5b10b1\") " Jul 21 16:19:03.261028 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:03.260964 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3d7cd1c-2ffc-4de0-bcd9-cb555a5b10b1-istio-token\") pod \"a3d7cd1c-2ffc-4de0-bcd9-cb555a5b10b1\" (UID: \"a3d7cd1c-2ffc-4de0-bcd9-cb555a5b10b1\") " Jul 21 16:19:03.261028 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:03.261018 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3d7cd1c-2ffc-4de0-bcd9-cb555a5b10b1-istio-podinfo\") pod \"a3d7cd1c-2ffc-4de0-bcd9-cb555a5b10b1\" (UID: \"a3d7cd1c-2ffc-4de0-bcd9-cb555a5b10b1\") " Jul 21 16:19:03.261369 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:03.261075 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3d7cd1c-2ffc-4de0-bcd9-cb555a5b10b1-istio-data\") pod \"a3d7cd1c-2ffc-4de0-bcd9-cb555a5b10b1\" (UID: \"a3d7cd1c-2ffc-4de0-bcd9-cb555a5b10b1\") " Jul 21 16:19:03.261369 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:03.261073 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3d7cd1c-2ffc-4de0-bcd9-cb555a5b10b1-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a3d7cd1c-2ffc-4de0-bcd9-cb555a5b10b1" (UID: "a3d7cd1c-2ffc-4de0-bcd9-cb555a5b10b1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 16:19:03.261369 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:03.261186 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3d7cd1c-2ffc-4de0-bcd9-cb555a5b10b1-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a3d7cd1c-2ffc-4de0-bcd9-cb555a5b10b1" (UID: "a3d7cd1c-2ffc-4de0-bcd9-cb555a5b10b1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 16:19:03.261369 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:03.261253 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3d7cd1c-2ffc-4de0-bcd9-cb555a5b10b1-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a3d7cd1c-2ffc-4de0-bcd9-cb555a5b10b1" (UID: "a3d7cd1c-2ffc-4de0-bcd9-cb555a5b10b1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 16:19:03.261578 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:03.261462 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3d7cd1c-2ffc-4de0-bcd9-cb555a5b10b1-istio-data" (OuterVolumeSpecName: "istio-data") pod "a3d7cd1c-2ffc-4de0-bcd9-cb555a5b10b1" (UID: "a3d7cd1c-2ffc-4de0-bcd9-cb555a5b10b1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 16:19:03.261578 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:03.261549 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3d7cd1c-2ffc-4de0-bcd9-cb555a5b10b1-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a3d7cd1c-2ffc-4de0-bcd9-cb555a5b10b1" (UID: "a3d7cd1c-2ffc-4de0-bcd9-cb555a5b10b1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 16:19:03.261681 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:03.261581 2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3d7cd1c-2ffc-4de0-bcd9-cb555a5b10b1-istio-data\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 16:19:03.261681 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:03.261605 2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3d7cd1c-2ffc-4de0-bcd9-cb555a5b10b1-workload-socket\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 16:19:03.261681 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:03.261620 2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3d7cd1c-2ffc-4de0-bcd9-cb555a5b10b1-istiod-ca-cert\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 16:19:03.261681 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:03.261634 2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3d7cd1c-2ffc-4de0-bcd9-cb555a5b10b1-credential-socket\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 16:19:03.263294 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:03.263271 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a3d7cd1c-2ffc-4de0-bcd9-cb555a5b10b1-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a3d7cd1c-2ffc-4de0-bcd9-cb555a5b10b1" (UID: "a3d7cd1c-2ffc-4de0-bcd9-cb555a5b10b1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 21 16:19:03.263400 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:03.263388 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3d7cd1c-2ffc-4de0-bcd9-cb555a5b10b1-kube-api-access-jw7rr" (OuterVolumeSpecName: "kube-api-access-jw7rr") pod "a3d7cd1c-2ffc-4de0-bcd9-cb555a5b10b1" (UID: "a3d7cd1c-2ffc-4de0-bcd9-cb555a5b10b1"). InnerVolumeSpecName "kube-api-access-jw7rr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 16:19:03.263795 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:03.263770 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3d7cd1c-2ffc-4de0-bcd9-cb555a5b10b1-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a3d7cd1c-2ffc-4de0-bcd9-cb555a5b10b1" (UID: "a3d7cd1c-2ffc-4de0-bcd9-cb555a5b10b1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 16:19:03.263887 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:03.263798 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3d7cd1c-2ffc-4de0-bcd9-cb555a5b10b1-istio-token" (OuterVolumeSpecName: "istio-token") pod "a3d7cd1c-2ffc-4de0-bcd9-cb555a5b10b1" (UID: "a3d7cd1c-2ffc-4de0-bcd9-cb555a5b10b1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 16:19:03.362539 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:03.362505 2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3d7cd1c-2ffc-4de0-bcd9-cb555a5b10b1-istio-token\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 16:19:03.362539 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:03.362534 2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3d7cd1c-2ffc-4de0-bcd9-cb555a5b10b1-istio-podinfo\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 16:19:03.362539 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:03.362543 2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3d7cd1c-2ffc-4de0-bcd9-cb555a5b10b1-workload-certs\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 16:19:03.362539 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:03.362552 2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3d7cd1c-2ffc-4de0-bcd9-cb555a5b10b1-istio-envoy\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 16:19:03.362815 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:03.362560 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jw7rr\" (UniqueName: \"kubernetes.io/projected/a3d7cd1c-2ffc-4de0-bcd9-cb555a5b10b1-kube-api-access-jw7rr\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 16:19:04.074281 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:04.074244 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-fb94bd-openshift-default-85b95d89ff-72mhf" event={"ID":"a3d7cd1c-2ffc-4de0-bcd9-cb555a5b10b1","Type":"ContainerDied","Data":"8b10b7516a9d677ffd80eb20e7322bc8923dfbe5930d5c554cc0d260c4f3ff77"} Jul 21 16:19:04.074281 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:04.074281 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-fb94bd-openshift-default-85b95d89ff-72mhf" Jul 21 16:19:04.074833 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:04.074342 2563 scope.go:117] "RemoveContainer" containerID="c7cd13ade321ee50080ed24f9fa742af7bea342880b641d82d9bae1b3320312a" Jul 21 16:19:04.100020 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:04.099994 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-fb94bd-openshift-default-85b95d89ff-72mhf"] Jul 21 16:19:04.105404 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:04.105377 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-fb94bd-openshift-default-85b95d89ff-72mhf"] Jul 21 16:19:04.482155 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:04.482118 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a3d7cd1c-2ffc-4de0-bcd9-cb555a5b10b1" path="/var/lib/kubelet/pods/a3d7cd1c-2ffc-4de0-bcd9-cb555a5b10b1/volumes" Jul 21 16:19:04.654841 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:04.654803 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-j4c8p"] Jul 21 16:19:07.179915 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:07.179881 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-bbeb7d-openshift-default-7d4d479875-lgrtw"] Jul 21 16:19:07.190110 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:07.180491 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3d7cd1c-2ffc-4de0-bcd9-cb555a5b10b1" containerName="istio-proxy" Jul 21 16:19:07.190110 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:07.180515 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3d7cd1c-2ffc-4de0-bcd9-cb555a5b10b1" containerName="istio-proxy" Jul 21 16:19:07.190110 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:07.180651 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3d7cd1c-2ffc-4de0-bcd9-cb555a5b10b1" containerName="istio-proxy" Jul 21 16:19:07.191726 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:07.190635 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-bbeb7d-openshift-default-7d4d479875-lgrtw" Jul 21 16:19:07.193927 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:07.193848 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-bbeb7d-openshift-default-dockercfg-b4lwl\"" Jul 21 16:19:07.197683 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:07.197655 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-bbeb7d-openshift-default-7d4d479875-lgrtw"] Jul 21 16:19:07.300360 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:07.300050 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb72b135-46ae-47cf-8927-74fa140deb48-workload-certs\") pod \"e2e-shared-b-bbeb7d-openshift-default-7d4d479875-lgrtw\" (UID: \"cb72b135-46ae-47cf-8927-74fa140deb48\") " pod="openshift-ingress/e2e-shared-b-bbeb7d-openshift-default-7d4d479875-lgrtw" Jul 21 16:19:07.300360 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:07.300164 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb72b135-46ae-47cf-8927-74fa140deb48-istio-podinfo\") pod \"e2e-shared-b-bbeb7d-openshift-default-7d4d479875-lgrtw\" (UID: \"cb72b135-46ae-47cf-8927-74fa140deb48\") " pod="openshift-ingress/e2e-shared-b-bbeb7d-openshift-default-7d4d479875-lgrtw" Jul 21 16:19:07.300360 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:07.300210 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8rgk\" (UniqueName: \"kubernetes.io/projected/cb72b135-46ae-47cf-8927-74fa140deb48-kube-api-access-b8rgk\") pod \"e2e-shared-b-bbeb7d-openshift-default-7d4d479875-lgrtw\" (UID: \"cb72b135-46ae-47cf-8927-74fa140deb48\") " pod="openshift-ingress/e2e-shared-b-bbeb7d-openshift-default-7d4d479875-lgrtw" Jul 21 16:19:07.300360 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:07.300275 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb72b135-46ae-47cf-8927-74fa140deb48-credential-socket\") pod \"e2e-shared-b-bbeb7d-openshift-default-7d4d479875-lgrtw\" (UID: \"cb72b135-46ae-47cf-8927-74fa140deb48\") " pod="openshift-ingress/e2e-shared-b-bbeb7d-openshift-default-7d4d479875-lgrtw" Jul 21 16:19:07.300360 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:07.300305 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb72b135-46ae-47cf-8927-74fa140deb48-istio-data\") pod \"e2e-shared-b-bbeb7d-openshift-default-7d4d479875-lgrtw\" (UID: \"cb72b135-46ae-47cf-8927-74fa140deb48\") " pod="openshift-ingress/e2e-shared-b-bbeb7d-openshift-default-7d4d479875-lgrtw" Jul 21 16:19:07.300360 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:07.300355 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb72b135-46ae-47cf-8927-74fa140deb48-istio-token\") pod \"e2e-shared-b-bbeb7d-openshift-default-7d4d479875-lgrtw\" (UID: \"cb72b135-46ae-47cf-8927-74fa140deb48\") " pod="openshift-ingress/e2e-shared-b-bbeb7d-openshift-default-7d4d479875-lgrtw" Jul 21 16:19:07.300957 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:07.300400 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb72b135-46ae-47cf-8927-74fa140deb48-istio-envoy\") pod \"e2e-shared-b-bbeb7d-openshift-default-7d4d479875-lgrtw\" (UID: \"cb72b135-46ae-47cf-8927-74fa140deb48\") " pod="openshift-ingress/e2e-shared-b-bbeb7d-openshift-default-7d4d479875-lgrtw" Jul 21 16:19:07.300957 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:07.300550 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb72b135-46ae-47cf-8927-74fa140deb48-workload-socket\") pod \"e2e-shared-b-bbeb7d-openshift-default-7d4d479875-lgrtw\" (UID: \"cb72b135-46ae-47cf-8927-74fa140deb48\") " pod="openshift-ingress/e2e-shared-b-bbeb7d-openshift-default-7d4d479875-lgrtw" Jul 21 16:19:07.300957 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:07.300632 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb72b135-46ae-47cf-8927-74fa140deb48-istiod-ca-cert\") pod \"e2e-shared-b-bbeb7d-openshift-default-7d4d479875-lgrtw\" (UID: \"cb72b135-46ae-47cf-8927-74fa140deb48\") " pod="openshift-ingress/e2e-shared-b-bbeb7d-openshift-default-7d4d479875-lgrtw" Jul 21 16:19:07.401549 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:07.401480 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb72b135-46ae-47cf-8927-74fa140deb48-workload-socket\") pod \"e2e-shared-b-bbeb7d-openshift-default-7d4d479875-lgrtw\" (UID: \"cb72b135-46ae-47cf-8927-74fa140deb48\") " pod="openshift-ingress/e2e-shared-b-bbeb7d-openshift-default-7d4d479875-lgrtw" Jul 21 16:19:07.401884 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:07.401575 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb72b135-46ae-47cf-8927-74fa140deb48-istiod-ca-cert\") pod \"e2e-shared-b-bbeb7d-openshift-default-7d4d479875-lgrtw\" (UID: \"cb72b135-46ae-47cf-8927-74fa140deb48\") " pod="openshift-ingress/e2e-shared-b-bbeb7d-openshift-default-7d4d479875-lgrtw" Jul 21 16:19:07.401884 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:07.401692 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb72b135-46ae-47cf-8927-74fa140deb48-workload-certs\") pod \"e2e-shared-b-bbeb7d-openshift-default-7d4d479875-lgrtw\" (UID: \"cb72b135-46ae-47cf-8927-74fa140deb48\") " pod="openshift-ingress/e2e-shared-b-bbeb7d-openshift-default-7d4d479875-lgrtw" Jul 21 16:19:07.401884 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:07.401781 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb72b135-46ae-47cf-8927-74fa140deb48-istio-podinfo\") pod \"e2e-shared-b-bbeb7d-openshift-default-7d4d479875-lgrtw\" (UID: \"cb72b135-46ae-47cf-8927-74fa140deb48\") " pod="openshift-ingress/e2e-shared-b-bbeb7d-openshift-default-7d4d479875-lgrtw" Jul 21 16:19:07.401884 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:07.401824 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8rgk\" (UniqueName: \"kubernetes.io/projected/cb72b135-46ae-47cf-8927-74fa140deb48-kube-api-access-b8rgk\") pod \"e2e-shared-b-bbeb7d-openshift-default-7d4d479875-lgrtw\" (UID: \"cb72b135-46ae-47cf-8927-74fa140deb48\") " pod="openshift-ingress/e2e-shared-b-bbeb7d-openshift-default-7d4d479875-lgrtw" Jul 21 16:19:07.402134 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:07.401887 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb72b135-46ae-47cf-8927-74fa140deb48-credential-socket\") pod \"e2e-shared-b-bbeb7d-openshift-default-7d4d479875-lgrtw\" (UID: \"cb72b135-46ae-47cf-8927-74fa140deb48\") " pod="openshift-ingress/e2e-shared-b-bbeb7d-openshift-default-7d4d479875-lgrtw" Jul 21 16:19:07.402134 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:07.401920 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb72b135-46ae-47cf-8927-74fa140deb48-istio-data\") pod \"e2e-shared-b-bbeb7d-openshift-default-7d4d479875-lgrtw\" (UID: \"cb72b135-46ae-47cf-8927-74fa140deb48\") " pod="openshift-ingress/e2e-shared-b-bbeb7d-openshift-default-7d4d479875-lgrtw" Jul 21 16:19:07.402134 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:07.401947 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb72b135-46ae-47cf-8927-74fa140deb48-istio-token\") pod \"e2e-shared-b-bbeb7d-openshift-default-7d4d479875-lgrtw\" (UID: \"cb72b135-46ae-47cf-8927-74fa140deb48\") " pod="openshift-ingress/e2e-shared-b-bbeb7d-openshift-default-7d4d479875-lgrtw" Jul 21 16:19:07.402134 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:07.401980 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb72b135-46ae-47cf-8927-74fa140deb48-istio-envoy\") pod \"e2e-shared-b-bbeb7d-openshift-default-7d4d479875-lgrtw\" (UID: \"cb72b135-46ae-47cf-8927-74fa140deb48\") " pod="openshift-ingress/e2e-shared-b-bbeb7d-openshift-default-7d4d479875-lgrtw" Jul 21 16:19:07.403224 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:07.403174 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb72b135-46ae-47cf-8927-74fa140deb48-credential-socket\") pod \"e2e-shared-b-bbeb7d-openshift-default-7d4d479875-lgrtw\" (UID: \"cb72b135-46ae-47cf-8927-74fa140deb48\") " pod="openshift-ingress/e2e-shared-b-bbeb7d-openshift-default-7d4d479875-lgrtw" Jul 21 16:19:07.406440 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:07.404371 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb72b135-46ae-47cf-8927-74fa140deb48-istiod-ca-cert\") pod \"e2e-shared-b-bbeb7d-openshift-default-7d4d479875-lgrtw\" (UID: \"cb72b135-46ae-47cf-8927-74fa140deb48\") " pod="openshift-ingress/e2e-shared-b-bbeb7d-openshift-default-7d4d479875-lgrtw" Jul 21 16:19:07.406723 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:07.404682 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb72b135-46ae-47cf-8927-74fa140deb48-workload-socket\") pod \"e2e-shared-b-bbeb7d-openshift-default-7d4d479875-lgrtw\" (UID: \"cb72b135-46ae-47cf-8927-74fa140deb48\") " pod="openshift-ingress/e2e-shared-b-bbeb7d-openshift-default-7d4d479875-lgrtw" Jul 21 16:19:07.406854 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:07.405210 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb72b135-46ae-47cf-8927-74fa140deb48-workload-certs\") pod \"e2e-shared-b-bbeb7d-openshift-default-7d4d479875-lgrtw\" (UID: \"cb72b135-46ae-47cf-8927-74fa140deb48\") " pod="openshift-ingress/e2e-shared-b-bbeb7d-openshift-default-7d4d479875-lgrtw" Jul 21 16:19:07.406978 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:07.406943 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb72b135-46ae-47cf-8927-74fa140deb48-istio-podinfo\") pod \"e2e-shared-b-bbeb7d-openshift-default-7d4d479875-lgrtw\" (UID: \"cb72b135-46ae-47cf-8927-74fa140deb48\") " pod="openshift-ingress/e2e-shared-b-bbeb7d-openshift-default-7d4d479875-lgrtw" Jul 21 16:19:07.407153 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:07.406582 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb72b135-46ae-47cf-8927-74fa140deb48-istio-envoy\") pod \"e2e-shared-b-bbeb7d-openshift-default-7d4d479875-lgrtw\" (UID: \"cb72b135-46ae-47cf-8927-74fa140deb48\") " pod="openshift-ingress/e2e-shared-b-bbeb7d-openshift-default-7d4d479875-lgrtw" Jul 21 16:19:07.407250 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:07.405930 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb72b135-46ae-47cf-8927-74fa140deb48-istio-data\") pod \"e2e-shared-b-bbeb7d-openshift-default-7d4d479875-lgrtw\" (UID: \"cb72b135-46ae-47cf-8927-74fa140deb48\") " pod="openshift-ingress/e2e-shared-b-bbeb7d-openshift-default-7d4d479875-lgrtw" Jul 21 16:19:07.416798 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:07.416743 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb72b135-46ae-47cf-8927-74fa140deb48-istio-token\") pod \"e2e-shared-b-bbeb7d-openshift-default-7d4d479875-lgrtw\" (UID: \"cb72b135-46ae-47cf-8927-74fa140deb48\") " pod="openshift-ingress/e2e-shared-b-bbeb7d-openshift-default-7d4d479875-lgrtw" Jul 21 16:19:07.417021 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:07.416845 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8rgk\" (UniqueName: \"kubernetes.io/projected/cb72b135-46ae-47cf-8927-74fa140deb48-kube-api-access-b8rgk\") pod \"e2e-shared-b-bbeb7d-openshift-default-7d4d479875-lgrtw\" (UID: \"cb72b135-46ae-47cf-8927-74fa140deb48\") " pod="openshift-ingress/e2e-shared-b-bbeb7d-openshift-default-7d4d479875-lgrtw" Jul 21 16:19:07.520144 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:07.519999 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-bbeb7d-openshift-default-7d4d479875-lgrtw" Jul 21 16:19:07.759435 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:07.759387 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-bbeb7d-openshift-default-7d4d479875-lgrtw"] Jul 21 16:19:07.761832 ip-10-0-135-176 kubenswrapper[2563]: W0721 16:19:07.761790 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb72b135_46ae_47cf_8927_74fa140deb48.slice/crio-1ea9bea69a7fce763483b87bdcb1d9e1aa035f055ae808184ad166083ffcb702 WatchSource:0}: Error finding container 1ea9bea69a7fce763483b87bdcb1d9e1aa035f055ae808184ad166083ffcb702: Status 404 returned error can't find the container with id 1ea9bea69a7fce763483b87bdcb1d9e1aa035f055ae808184ad166083ffcb702 Jul 21 16:19:07.765007 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:07.764959 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 21 16:19:07.765103 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:07.765073 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 21 16:19:07.765167 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:07.765116 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 21 16:19:08.111229 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:08.110234 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-bbeb7d-openshift-default-7d4d479875-lgrtw" event={"ID":"cb72b135-46ae-47cf-8927-74fa140deb48","Type":"ContainerStarted","Data":"daeb82b21e0516b4fc3afb9cb6755bd626e3b902cddb3ecb2dc38b7cd5292015"} Jul 21 16:19:08.111229 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:08.110303 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-bbeb7d-openshift-default-7d4d479875-lgrtw" event={"ID":"cb72b135-46ae-47cf-8927-74fa140deb48","Type":"ContainerStarted","Data":"1ea9bea69a7fce763483b87bdcb1d9e1aa035f055ae808184ad166083ffcb702"} Jul 21 16:19:08.132690 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:08.132639 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-bbeb7d-openshift-default-7d4d479875-lgrtw" podStartSLOduration=1.1326222910000001 podStartE2EDuration="1.132622291s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 16:19:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 16:19:08.130679437 +0000 UTC m=+4780.126464855" watchObservedRunningTime="2026-07-21 16:19:08.132622291 +0000 UTC m=+4780.128407707" Jul 21 16:19:08.521421 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:08.520475 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-bbeb7d-openshift-default-7d4d479875-lgrtw" Jul 21 16:19:08.525295 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:08.525260 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-bbeb7d-openshift-default-7d4d479875-lgrtw" Jul 21 16:19:09.116389 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:09.116358 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-bbeb7d-openshift-default-7d4d479875-lgrtw" Jul 21 16:19:09.117396 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:09.117376 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-bbeb7d-openshift-default-7d4d479875-lgrtw" Jul 21 16:19:13.442640 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:13.442601 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-j4c8p"] Jul 21 16:19:15.152604 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:15.152568 2563 generic.go:356] "Generic (PLEG): container finished" podID="9707e0e1-e63c-4336-b2c1-aba2929a0b75" containerID="78cb32b24644a423c3a0dbb614f97ef1b92e588c66106f069e1fc545d7198276" exitCode=0 Jul 21 16:19:15.153026 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:15.152628 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-524b80-openshift-default-9cb6b55d7-lwd66" event={"ID":"9707e0e1-e63c-4336-b2c1-aba2929a0b75","Type":"ContainerDied","Data":"78cb32b24644a423c3a0dbb614f97ef1b92e588c66106f069e1fc545d7198276"} Jul 21 16:19:15.153026 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:15.153015 2563 scope.go:117] "RemoveContainer" containerID="78cb32b24644a423c3a0dbb614f97ef1b92e588c66106f069e1fc545d7198276" Jul 21 16:19:15.154058 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:15.154028 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 21 16:19:15.154159 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:15.154140 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 21 16:19:15.154201 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:15.154183 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 21 16:19:16.054240 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:16.054201 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-524b80-openshift-default-9cb6b55d7-lwd66" Jul 21 16:19:16.161501 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:16.161463 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-524b80-openshift-default-9cb6b55d7-lwd66" event={"ID":"9707e0e1-e63c-4336-b2c1-aba2929a0b75","Type":"ContainerStarted","Data":"1782193e92d36dba60a9a34e155444268fe9761a7758b37499096deef69de542"} Jul 21 16:19:17.159485 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:17.159447 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-524b80-openshift-default-9cb6b55d7-lwd66" Jul 21 16:19:17.160705 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:17.160667 2563 patch_prober.go:28] interesting pod/e2e-shared-a-524b80-openshift-default-9cb6b55d7-lwd66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Jul 21 16:19:17.160837 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:17.160766 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-524b80-openshift-default-9cb6b55d7-lwd66" podUID="9707e0e1-e63c-4336-b2c1-aba2929a0b75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Jul 21 16:19:18.159797 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:18.159748 2563 patch_prober.go:28] interesting pod/e2e-shared-a-524b80-openshift-default-9cb6b55d7-lwd66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Jul 21 16:19:18.160292 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:18.159813 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-524b80-openshift-default-9cb6b55d7-lwd66" podUID="9707e0e1-e63c-4336-b2c1-aba2929a0b75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Jul 21 16:19:19.158923 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:19.158877 2563 patch_prober.go:28] interesting pod/e2e-shared-a-524b80-openshift-default-9cb6b55d7-lwd66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Jul 21 16:19:19.159119 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:19.158949 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-524b80-openshift-default-9cb6b55d7-lwd66" podUID="9707e0e1-e63c-4336-b2c1-aba2929a0b75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Jul 21 16:19:20.159666 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:20.159630 2563 patch_prober.go:28] interesting pod/e2e-shared-a-524b80-openshift-default-9cb6b55d7-lwd66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Jul 21 16:19:20.160094 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:20.159686 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-524b80-openshift-default-9cb6b55d7-lwd66" podUID="9707e0e1-e63c-4336-b2c1-aba2929a0b75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Jul 21 16:19:21.159584 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:21.159549 2563 patch_prober.go:28] interesting pod/e2e-shared-a-524b80-openshift-default-9cb6b55d7-lwd66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Jul 21 16:19:21.159776 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:21.159611 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-524b80-openshift-default-9cb6b55d7-lwd66" podUID="9707e0e1-e63c-4336-b2c1-aba2929a0b75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Jul 21 16:19:22.169791 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:22.169756 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-524b80-openshift-default-9cb6b55d7-lwd66" Jul 21 16:19:22.170274 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:22.170153 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-524b80-openshift-default-9cb6b55d7-lwd66" Jul 21 16:19:22.170903 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:22.170884 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-524b80-openshift-default-9cb6b55d7-lwd66" Jul 21 16:19:24.118253 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:24.118211 2563 patch_prober.go:28] interesting pod/e2e-shared-b-bbeb7d-openshift-default-7d4d479875-lgrtw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Jul 21 16:19:24.118720 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:24.118286 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-bbeb7d-openshift-default-7d4d479875-lgrtw" podUID="cb72b135-46ae-47cf-8927-74fa140deb48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Jul 21 16:19:24.218919 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:24.218885 2563 generic.go:356] "Generic (PLEG): container finished" podID="cb72b135-46ae-47cf-8927-74fa140deb48" containerID="daeb82b21e0516b4fc3afb9cb6755bd626e3b902cddb3ecb2dc38b7cd5292015" exitCode=0 Jul 21 16:19:24.219132 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:24.218966 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-bbeb7d-openshift-default-7d4d479875-lgrtw" event={"ID":"cb72b135-46ae-47cf-8927-74fa140deb48","Type":"ContainerDied","Data":"daeb82b21e0516b4fc3afb9cb6755bd626e3b902cddb3ecb2dc38b7cd5292015"} Jul 21 16:19:24.219461 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:24.219435 2563 scope.go:117] "RemoveContainer" containerID="daeb82b21e0516b4fc3afb9cb6755bd626e3b902cddb3ecb2dc38b7cd5292015" Jul 21 16:19:24.220519 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:24.220490 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 21 16:19:24.220639 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:24.220572 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 21 16:19:24.220639 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:24.220620 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 21 16:19:25.227565 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:25.227533 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-bbeb7d-openshift-default-7d4d479875-lgrtw" event={"ID":"cb72b135-46ae-47cf-8927-74fa140deb48","Type":"ContainerStarted","Data":"b545cdbe78b5246778eadc82d417b9c64968c5839afc626317168e70c0416b5b"} Jul 21 16:19:25.520316 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:25.520217 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-bbeb7d-openshift-default-7d4d479875-lgrtw" Jul 21 16:19:25.520547 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:25.520525 2563 patch_prober.go:28] interesting pod/e2e-shared-b-bbeb7d-openshift-default-7d4d479875-lgrtw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Jul 21 16:19:25.520594 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:25.520577 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-bbeb7d-openshift-default-7d4d479875-lgrtw" podUID="cb72b135-46ae-47cf-8927-74fa140deb48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Jul 21 16:19:26.520948 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:26.520905 2563 patch_prober.go:28] interesting pod/e2e-shared-b-bbeb7d-openshift-default-7d4d479875-lgrtw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Jul 21 16:19:26.521379 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:26.520985 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-bbeb7d-openshift-default-7d4d479875-lgrtw" podUID="cb72b135-46ae-47cf-8927-74fa140deb48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Jul 21 16:19:27.521297 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:27.521256 2563 patch_prober.go:28] interesting pod/e2e-shared-b-bbeb7d-openshift-default-7d4d479875-lgrtw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Jul 21 16:19:27.521719 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:27.521343 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-bbeb7d-openshift-default-7d4d479875-lgrtw" podUID="cb72b135-46ae-47cf-8927-74fa140deb48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Jul 21 16:19:28.520618 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:28.520584 2563 patch_prober.go:28] interesting pod/e2e-shared-b-bbeb7d-openshift-default-7d4d479875-lgrtw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Jul 21 16:19:28.520836 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:28.520655 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-bbeb7d-openshift-default-7d4d479875-lgrtw" podUID="cb72b135-46ae-47cf-8927-74fa140deb48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Jul 21 16:19:28.921460 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:28.921432 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kn9fn_02f3d0c6-4116-4134-ba51-1c02c73bef05/ovn-acl-logging/0.log" Jul 21 16:19:28.933891 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:28.933866 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kn9fn_02f3d0c6-4116-4134-ba51-1c02c73bef05/ovn-acl-logging/0.log" Jul 21 16:19:29.520976 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:29.520933 2563 patch_prober.go:28] interesting pod/e2e-shared-b-bbeb7d-openshift-default-7d4d479875-lgrtw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Jul 21 16:19:29.521161 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:29.520991 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-bbeb7d-openshift-default-7d4d479875-lgrtw" podUID="cb72b135-46ae-47cf-8927-74fa140deb48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Jul 21 16:19:30.524949 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:30.524921 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-bbeb7d-openshift-default-7d4d479875-lgrtw" Jul 21 16:19:30.525473 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:30.525176 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-bbeb7d-openshift-default-7d4d479875-lgrtw" Jul 21 16:19:30.526036 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:30.526018 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-bbeb7d-openshift-default-7d4d479875-lgrtw" Jul 21 16:19:32.961371 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:32.961313 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-524b80/auth-test-model-524b80-kserve-769656479-cj8rv"] Jul 21 16:19:32.966282 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:32.966257 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-524b80/auth-test-model-524b80-kserve-769656479-cj8rv" Jul 21 16:19:32.973292 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:32.973044 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-524b80\"/\"kube-root-ca.crt\"" Jul 21 16:19:32.973292 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:32.973062 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-524b80\"/\"auth-test-model-524b80-kserve-self-signed-certs\"" Jul 21 16:19:32.973292 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:32.973077 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-524b80\"/\"openshift-service-ca.crt\"" Jul 21 16:19:32.973292 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:32.973059 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-524b80\"/\"default-dockercfg-bbl9x\"" Jul 21 16:19:32.978522 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:32.978495 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-524b80/auth-test-model-524b80-kserve-769656479-cj8rv"] Jul 21 16:19:33.077765 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:33.077730 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7efc78e-2a98-4fdf-806c-aca2d06969f3-model-cache\") pod \"auth-test-model-524b80-kserve-769656479-cj8rv\" (UID: \"d7efc78e-2a98-4fdf-806c-aca2d06969f3\") " pod="ai-tenant-e2e-shared-a-524b80/auth-test-model-524b80-kserve-769656479-cj8rv" Jul 21 16:19:33.077942 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:33.077774 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7efc78e-2a98-4fdf-806c-aca2d06969f3-tmp-dir\") pod \"auth-test-model-524b80-kserve-769656479-cj8rv\" (UID: \"d7efc78e-2a98-4fdf-806c-aca2d06969f3\") " pod="ai-tenant-e2e-shared-a-524b80/auth-test-model-524b80-kserve-769656479-cj8rv" Jul 21 16:19:33.077942 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:33.077850 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7efc78e-2a98-4fdf-806c-aca2d06969f3-dshm\") pod \"auth-test-model-524b80-kserve-769656479-cj8rv\" (UID: \"d7efc78e-2a98-4fdf-806c-aca2d06969f3\") " pod="ai-tenant-e2e-shared-a-524b80/auth-test-model-524b80-kserve-769656479-cj8rv" Jul 21 16:19:33.077942 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:33.077901 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5qwl\" (UniqueName: \"kubernetes.io/projected/d7efc78e-2a98-4fdf-806c-aca2d06969f3-kube-api-access-j5qwl\") pod \"auth-test-model-524b80-kserve-769656479-cj8rv\" (UID: \"d7efc78e-2a98-4fdf-806c-aca2d06969f3\") " pod="ai-tenant-e2e-shared-a-524b80/auth-test-model-524b80-kserve-769656479-cj8rv" Jul 21 16:19:33.077942 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:33.077935 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d7efc78e-2a98-4fdf-806c-aca2d06969f3-home\") pod \"auth-test-model-524b80-kserve-769656479-cj8rv\" (UID: \"d7efc78e-2a98-4fdf-806c-aca2d06969f3\") " pod="ai-tenant-e2e-shared-a-524b80/auth-test-model-524b80-kserve-769656479-cj8rv" Jul 21 16:19:33.078087 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:33.078037 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7efc78e-2a98-4fdf-806c-aca2d06969f3-tls-certs\") pod \"auth-test-model-524b80-kserve-769656479-cj8rv\" (UID: \"d7efc78e-2a98-4fdf-806c-aca2d06969f3\") " pod="ai-tenant-e2e-shared-a-524b80/auth-test-model-524b80-kserve-769656479-cj8rv" Jul 21 16:19:33.179058 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:33.179005 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d7efc78e-2a98-4fdf-806c-aca2d06969f3-home\") pod \"auth-test-model-524b80-kserve-769656479-cj8rv\" (UID: \"d7efc78e-2a98-4fdf-806c-aca2d06969f3\") " pod="ai-tenant-e2e-shared-a-524b80/auth-test-model-524b80-kserve-769656479-cj8rv" Jul 21 16:19:33.179284 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:33.179092 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7efc78e-2a98-4fdf-806c-aca2d06969f3-tls-certs\") pod \"auth-test-model-524b80-kserve-769656479-cj8rv\" (UID: \"d7efc78e-2a98-4fdf-806c-aca2d06969f3\") " pod="ai-tenant-e2e-shared-a-524b80/auth-test-model-524b80-kserve-769656479-cj8rv" Jul 21 16:19:33.179284 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:33.179133 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7efc78e-2a98-4fdf-806c-aca2d06969f3-model-cache\") pod \"auth-test-model-524b80-kserve-769656479-cj8rv\" (UID: \"d7efc78e-2a98-4fdf-806c-aca2d06969f3\") " pod="ai-tenant-e2e-shared-a-524b80/auth-test-model-524b80-kserve-769656479-cj8rv" Jul 21 16:19:33.179284 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:33.179178 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7efc78e-2a98-4fdf-806c-aca2d06969f3-tmp-dir\") pod \"auth-test-model-524b80-kserve-769656479-cj8rv\" (UID: \"d7efc78e-2a98-4fdf-806c-aca2d06969f3\") " pod="ai-tenant-e2e-shared-a-524b80/auth-test-model-524b80-kserve-769656479-cj8rv" Jul 21 16:19:33.179284 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:33.179267 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7efc78e-2a98-4fdf-806c-aca2d06969f3-dshm\") pod \"auth-test-model-524b80-kserve-769656479-cj8rv\" (UID: \"d7efc78e-2a98-4fdf-806c-aca2d06969f3\") " pod="ai-tenant-e2e-shared-a-524b80/auth-test-model-524b80-kserve-769656479-cj8rv" Jul 21 16:19:33.179589 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:33.179304 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5qwl\" (UniqueName: \"kubernetes.io/projected/d7efc78e-2a98-4fdf-806c-aca2d06969f3-kube-api-access-j5qwl\") pod \"auth-test-model-524b80-kserve-769656479-cj8rv\" (UID: \"d7efc78e-2a98-4fdf-806c-aca2d06969f3\") " pod="ai-tenant-e2e-shared-a-524b80/auth-test-model-524b80-kserve-769656479-cj8rv" Jul 21 16:19:33.179660 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:33.179597 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d7efc78e-2a98-4fdf-806c-aca2d06969f3-home\") pod \"auth-test-model-524b80-kserve-769656479-cj8rv\" (UID: \"d7efc78e-2a98-4fdf-806c-aca2d06969f3\") " pod="ai-tenant-e2e-shared-a-524b80/auth-test-model-524b80-kserve-769656479-cj8rv" Jul 21 16:19:33.179660 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:33.179615 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7efc78e-2a98-4fdf-806c-aca2d06969f3-model-cache\") pod \"auth-test-model-524b80-kserve-769656479-cj8rv\" (UID: \"d7efc78e-2a98-4fdf-806c-aca2d06969f3\") " pod="ai-tenant-e2e-shared-a-524b80/auth-test-model-524b80-kserve-769656479-cj8rv" Jul 21 16:19:33.179779 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:33.179657 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7efc78e-2a98-4fdf-806c-aca2d06969f3-tmp-dir\") pod \"auth-test-model-524b80-kserve-769656479-cj8rv\" (UID: \"d7efc78e-2a98-4fdf-806c-aca2d06969f3\") " pod="ai-tenant-e2e-shared-a-524b80/auth-test-model-524b80-kserve-769656479-cj8rv" Jul 21 16:19:33.182127 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:33.182081 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7efc78e-2a98-4fdf-806c-aca2d06969f3-dshm\") pod \"auth-test-model-524b80-kserve-769656479-cj8rv\" (UID: \"d7efc78e-2a98-4fdf-806c-aca2d06969f3\") " pod="ai-tenant-e2e-shared-a-524b80/auth-test-model-524b80-kserve-769656479-cj8rv" Jul 21 16:19:33.182534 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:33.182510 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7efc78e-2a98-4fdf-806c-aca2d06969f3-tls-certs\") pod \"auth-test-model-524b80-kserve-769656479-cj8rv\" (UID: \"d7efc78e-2a98-4fdf-806c-aca2d06969f3\") " pod="ai-tenant-e2e-shared-a-524b80/auth-test-model-524b80-kserve-769656479-cj8rv" Jul 21 16:19:33.189848 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:33.189808 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5qwl\" (UniqueName: \"kubernetes.io/projected/d7efc78e-2a98-4fdf-806c-aca2d06969f3-kube-api-access-j5qwl\") pod \"auth-test-model-524b80-kserve-769656479-cj8rv\" (UID: \"d7efc78e-2a98-4fdf-806c-aca2d06969f3\") " pod="ai-tenant-e2e-shared-a-524b80/auth-test-model-524b80-kserve-769656479-cj8rv" Jul 21 16:19:33.275717 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:33.275624 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-524b80/auth-test-model-524b80-kserve-769656479-cj8rv" Jul 21 16:19:33.425465 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:33.425157 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-524b80/auth-test-model-524b80-kserve-769656479-cj8rv"] Jul 21 16:19:33.428098 ip-10-0-135-176 kubenswrapper[2563]: W0721 16:19:33.428069 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd7efc78e_2a98_4fdf_806c_aca2d06969f3.slice/crio-24378ccd42ec3477353b05f53675b780120609db3815f293357a5ec43fc55b9d WatchSource:0}: Error finding container 24378ccd42ec3477353b05f53675b780120609db3815f293357a5ec43fc55b9d: Status 404 returned error can't find the container with id 24378ccd42ec3477353b05f53675b780120609db3815f293357a5ec43fc55b9d Jul 21 16:19:34.281395 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:34.281354 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-524b80/auth-test-model-524b80-kserve-769656479-cj8rv" event={"ID":"d7efc78e-2a98-4fdf-806c-aca2d06969f3","Type":"ContainerStarted","Data":"f9ad6bad2f2bbb145c5a29a5eca6dc55970993ca1037b074292e69465b0ebdef"} Jul 21 16:19:34.281395 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:34.281401 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-524b80/auth-test-model-524b80-kserve-769656479-cj8rv" event={"ID":"d7efc78e-2a98-4fdf-806c-aca2d06969f3","Type":"ContainerStarted","Data":"24378ccd42ec3477353b05f53675b780120609db3815f293357a5ec43fc55b9d"} Jul 21 16:19:34.302210 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:34.302154 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-524b80/auth-test-model-524b80-kserve-769656479-cj8rv" podStartSLOduration=2.302136377 podStartE2EDuration="2.302136377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 16:19:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 16:19:34.298926571 +0000 UTC m=+4806.294712000" watchObservedRunningTime="2026-07-21 16:19:34.302136377 +0000 UTC m=+4806.297921793" Jul 21 16:19:35.753913 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:35.753879 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-j4c8p"] Jul 21 16:19:43.276430 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:43.276378 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-524b80/auth-test-model-524b80-kserve-769656479-cj8rv" Jul 21 16:19:43.277026 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:43.276456 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-524b80/auth-test-model-524b80-kserve-769656479-cj8rv" Jul 21 16:19:43.289244 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:43.289210 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-524b80/auth-test-model-524b80-kserve-769656479-cj8rv" Jul 21 16:19:43.348588 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:19:43.348557 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-524b80/auth-test-model-524b80-kserve-769656479-cj8rv" Jul 21 16:20:14.561186 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:20:14.561141 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-j4c8p"] Jul 21 16:20:30.648113 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:20:30.648028 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-j4c8p"] Jul 21 16:20:42.567167 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:20:42.567133 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-j4c8p"] Jul 21 16:20:51.976409 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:20:51.976368 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-j4c8p"] Jul 21 16:20:55.691516 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:20:55.691481 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-524b80/auth-test-model-524b80-kserve-769656479-cj8rv"] Jul 21 16:21:03.478924 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:21:03.478888 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-j4c8p"] Jul 21 16:21:10.712439 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:21:10.712391 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-524b80/auth-test-model-524b80-kserve-769656479-cj8rv" podUID="d7efc78e-2a98-4fdf-806c-aca2d06969f3" containerName="main" containerID="cri-o://f9ad6bad2f2bbb145c5a29a5eca6dc55970993ca1037b074292e69465b0ebdef" gracePeriod=45 Jul 21 16:21:10.856721 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:21:10.856689 2563 generic.go:356] "Generic (PLEG): container finished" podID="d7efc78e-2a98-4fdf-806c-aca2d06969f3" containerID="f9ad6bad2f2bbb145c5a29a5eca6dc55970993ca1037b074292e69465b0ebdef" exitCode=0 Jul 21 16:21:10.856881 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:21:10.856765 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-524b80/auth-test-model-524b80-kserve-769656479-cj8rv" event={"ID":"d7efc78e-2a98-4fdf-806c-aca2d06969f3","Type":"ContainerDied","Data":"f9ad6bad2f2bbb145c5a29a5eca6dc55970993ca1037b074292e69465b0ebdef"} Jul 21 16:21:10.941220 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:21:10.941185 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-j4c8p"] Jul 21 16:21:10.968957 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:21:10.968902 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-524b80/auth-test-model-524b80-kserve-769656479-cj8rv" Jul 21 16:21:11.129005 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:21:11.128969 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j5qwl\" (UniqueName: \"kubernetes.io/projected/d7efc78e-2a98-4fdf-806c-aca2d06969f3-kube-api-access-j5qwl\") pod \"d7efc78e-2a98-4fdf-806c-aca2d06969f3\" (UID: \"d7efc78e-2a98-4fdf-806c-aca2d06969f3\") " Jul 21 16:21:11.129186 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:21:11.129023 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7efc78e-2a98-4fdf-806c-aca2d06969f3-tmp-dir\") pod \"d7efc78e-2a98-4fdf-806c-aca2d06969f3\" (UID: \"d7efc78e-2a98-4fdf-806c-aca2d06969f3\") " Jul 21 16:21:11.129186 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:21:11.129059 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d7efc78e-2a98-4fdf-806c-aca2d06969f3-home\") pod \"d7efc78e-2a98-4fdf-806c-aca2d06969f3\" (UID: \"d7efc78e-2a98-4fdf-806c-aca2d06969f3\") " Jul 21 16:21:11.129186 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:21:11.129081 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7efc78e-2a98-4fdf-806c-aca2d06969f3-dshm\") pod \"d7efc78e-2a98-4fdf-806c-aca2d06969f3\" (UID: \"d7efc78e-2a98-4fdf-806c-aca2d06969f3\") " Jul 21 16:21:11.129186 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:21:11.129105 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7efc78e-2a98-4fdf-806c-aca2d06969f3-tls-certs\") pod \"d7efc78e-2a98-4fdf-806c-aca2d06969f3\" (UID: \"d7efc78e-2a98-4fdf-806c-aca2d06969f3\") " Jul 21 16:21:11.129381 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:21:11.129188 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7efc78e-2a98-4fdf-806c-aca2d06969f3-model-cache\") pod \"d7efc78e-2a98-4fdf-806c-aca2d06969f3\" (UID: \"d7efc78e-2a98-4fdf-806c-aca2d06969f3\") " Jul 21 16:21:11.129381 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:21:11.129361 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7efc78e-2a98-4fdf-806c-aca2d06969f3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d7efc78e-2a98-4fdf-806c-aca2d06969f3" (UID: "d7efc78e-2a98-4fdf-806c-aca2d06969f3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 16:21:11.129514 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:21:11.129388 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7efc78e-2a98-4fdf-806c-aca2d06969f3-home" (OuterVolumeSpecName: "home") pod "d7efc78e-2a98-4fdf-806c-aca2d06969f3" (UID: "d7efc78e-2a98-4fdf-806c-aca2d06969f3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 16:21:11.129610 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:21:11.129589 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7efc78e-2a98-4fdf-806c-aca2d06969f3-model-cache" (OuterVolumeSpecName: "model-cache") pod "d7efc78e-2a98-4fdf-806c-aca2d06969f3" (UID: "d7efc78e-2a98-4fdf-806c-aca2d06969f3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 16:21:11.129723 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:21:11.129686 2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d7efc78e-2a98-4fdf-806c-aca2d06969f3-home\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 16:21:11.129723 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:21:11.129708 2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7efc78e-2a98-4fdf-806c-aca2d06969f3-model-cache\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 16:21:11.129834 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:21:11.129725 2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7efc78e-2a98-4fdf-806c-aca2d06969f3-tmp-dir\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 16:21:11.131547 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:21:11.131520 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7efc78e-2a98-4fdf-806c-aca2d06969f3-dshm" (OuterVolumeSpecName: "dshm") pod "d7efc78e-2a98-4fdf-806c-aca2d06969f3" (UID: "d7efc78e-2a98-4fdf-806c-aca2d06969f3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 16:21:11.131671 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:21:11.131614 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7efc78e-2a98-4fdf-806c-aca2d06969f3-kube-api-access-j5qwl" (OuterVolumeSpecName: "kube-api-access-j5qwl") pod "d7efc78e-2a98-4fdf-806c-aca2d06969f3" (UID: "d7efc78e-2a98-4fdf-806c-aca2d06969f3"). InnerVolumeSpecName "kube-api-access-j5qwl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 16:21:11.131671 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:21:11.131625 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7efc78e-2a98-4fdf-806c-aca2d06969f3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d7efc78e-2a98-4fdf-806c-aca2d06969f3" (UID: "d7efc78e-2a98-4fdf-806c-aca2d06969f3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 16:21:11.230728 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:21:11.230650 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5qwl\" (UniqueName: \"kubernetes.io/projected/d7efc78e-2a98-4fdf-806c-aca2d06969f3-kube-api-access-j5qwl\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 16:21:11.230728 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:21:11.230677 2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7efc78e-2a98-4fdf-806c-aca2d06969f3-dshm\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 16:21:11.230728 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:21:11.230688 2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7efc78e-2a98-4fdf-806c-aca2d06969f3-tls-certs\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 16:21:11.863447 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:21:11.863417 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-524b80/auth-test-model-524b80-kserve-769656479-cj8rv" Jul 21 16:21:11.863447 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:21:11.863438 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-524b80/auth-test-model-524b80-kserve-769656479-cj8rv" event={"ID":"d7efc78e-2a98-4fdf-806c-aca2d06969f3","Type":"ContainerDied","Data":"24378ccd42ec3477353b05f53675b780120609db3815f293357a5ec43fc55b9d"} Jul 21 16:21:11.863961 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:21:11.863490 2563 scope.go:117] "RemoveContainer" containerID="f9ad6bad2f2bbb145c5a29a5eca6dc55970993ca1037b074292e69465b0ebdef" Jul 21 16:21:11.888523 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:21:11.888492 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-524b80/auth-test-model-524b80-kserve-769656479-cj8rv"] Jul 21 16:21:11.899933 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:21:11.899904 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-524b80/auth-test-model-524b80-kserve-769656479-cj8rv"] Jul 21 16:21:12.483104 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:21:12.483068 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d7efc78e-2a98-4fdf-806c-aca2d06969f3" path="/var/lib/kubelet/pods/d7efc78e-2a98-4fdf-806c-aca2d06969f3/volumes" Jul 21 16:24:03.585724 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:24:03.585690 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-524b80-openshift-default-9cb6b55d7-lwd66"] Jul 21 16:24:03.586237 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:24:03.585905 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-524b80-openshift-default-9cb6b55d7-lwd66" podUID="9707e0e1-e63c-4336-b2c1-aba2929a0b75" containerName="istio-proxy" containerID="cri-o://1782193e92d36dba60a9a34e155444268fe9761a7758b37499096deef69de542" gracePeriod=30 Jul 21 16:24:07.172688 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:24:07.172640 2563 patch_prober.go:28] interesting pod/e2e-shared-a-524b80-openshift-default-9cb6b55d7-lwd66 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 21 16:24:07.173081 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:24:07.172711 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-524b80-openshift-default-9cb6b55d7-lwd66" podUID="9707e0e1-e63c-4336-b2c1-aba2929a0b75" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 21 16:24:08.846551 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:24:08.846494 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-524b80-openshift-default-9cb6b55d7-lwd66" Jul 21 16:24:08.881741 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:24:08.881711 2563 generic.go:356] "Generic (PLEG): container finished" podID="9707e0e1-e63c-4336-b2c1-aba2929a0b75" containerID="1782193e92d36dba60a9a34e155444268fe9761a7758b37499096deef69de542" exitCode=0 Jul 21 16:24:08.881971 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:24:08.881768 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-524b80-openshift-default-9cb6b55d7-lwd66" Jul 21 16:24:08.881971 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:24:08.881775 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-524b80-openshift-default-9cb6b55d7-lwd66" event={"ID":"9707e0e1-e63c-4336-b2c1-aba2929a0b75","Type":"ContainerDied","Data":"1782193e92d36dba60a9a34e155444268fe9761a7758b37499096deef69de542"} Jul 21 16:24:08.881971 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:24:08.881845 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-524b80-openshift-default-9cb6b55d7-lwd66" event={"ID":"9707e0e1-e63c-4336-b2c1-aba2929a0b75","Type":"ContainerDied","Data":"a2d0924cd9558714d25b66a012292414de58cced9dad79554fa03f7c208ed37d"} Jul 21 16:24:08.881971 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:24:08.881876 2563 scope.go:117] "RemoveContainer" containerID="1782193e92d36dba60a9a34e155444268fe9761a7758b37499096deef69de542" Jul 21 16:24:08.893851 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:24:08.893827 2563 scope.go:117] "RemoveContainer" containerID="78cb32b24644a423c3a0dbb614f97ef1b92e588c66106f069e1fc545d7198276" Jul 21 16:24:08.902487 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:24:08.902464 2563 scope.go:117] "RemoveContainer" containerID="1782193e92d36dba60a9a34e155444268fe9761a7758b37499096deef69de542" Jul 21 16:24:08.902782 ip-10-0-135-176 kubenswrapper[2563]: E0721 16:24:08.902763 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1782193e92d36dba60a9a34e155444268fe9761a7758b37499096deef69de542\": container with ID starting with 1782193e92d36dba60a9a34e155444268fe9761a7758b37499096deef69de542 not found: ID does not exist" containerID="1782193e92d36dba60a9a34e155444268fe9761a7758b37499096deef69de542" Jul 21 16:24:08.902861 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:24:08.902794 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1782193e92d36dba60a9a34e155444268fe9761a7758b37499096deef69de542"} err="failed to get container status \"1782193e92d36dba60a9a34e155444268fe9761a7758b37499096deef69de542\": rpc error: code = NotFound desc = could not find container \"1782193e92d36dba60a9a34e155444268fe9761a7758b37499096deef69de542\": container with ID starting with 1782193e92d36dba60a9a34e155444268fe9761a7758b37499096deef69de542 not found: ID does not exist" Jul 21 16:24:08.902861 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:24:08.902819 2563 scope.go:117] "RemoveContainer" containerID="78cb32b24644a423c3a0dbb614f97ef1b92e588c66106f069e1fc545d7198276" Jul 21 16:24:08.903045 ip-10-0-135-176 kubenswrapper[2563]: E0721 16:24:08.903026 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78cb32b24644a423c3a0dbb614f97ef1b92e588c66106f069e1fc545d7198276\": container with ID starting with 78cb32b24644a423c3a0dbb614f97ef1b92e588c66106f069e1fc545d7198276 not found: ID does not exist" containerID="78cb32b24644a423c3a0dbb614f97ef1b92e588c66106f069e1fc545d7198276" Jul 21 16:24:08.903092 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:24:08.903055 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78cb32b24644a423c3a0dbb614f97ef1b92e588c66106f069e1fc545d7198276"} err="failed to get container status \"78cb32b24644a423c3a0dbb614f97ef1b92e588c66106f069e1fc545d7198276\": rpc error: code = NotFound desc = could not find container \"78cb32b24644a423c3a0dbb614f97ef1b92e588c66106f069e1fc545d7198276\": container with ID starting with 78cb32b24644a423c3a0dbb614f97ef1b92e588c66106f069e1fc545d7198276 not found: ID does not exist" Jul 21 16:24:08.979615 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:24:08.979580 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9707e0e1-e63c-4336-b2c1-aba2929a0b75-credential-socket\") pod \"9707e0e1-e63c-4336-b2c1-aba2929a0b75\" (UID: \"9707e0e1-e63c-4336-b2c1-aba2929a0b75\") " Jul 21 16:24:08.979799 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:24:08.979630 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9707e0e1-e63c-4336-b2c1-aba2929a0b75-istiod-ca-cert\") pod \"9707e0e1-e63c-4336-b2c1-aba2929a0b75\" (UID: \"9707e0e1-e63c-4336-b2c1-aba2929a0b75\") " Jul 21 16:24:08.979799 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:24:08.979692 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9707e0e1-e63c-4336-b2c1-aba2929a0b75-workload-certs\") pod \"9707e0e1-e63c-4336-b2c1-aba2929a0b75\" (UID: \"9707e0e1-e63c-4336-b2c1-aba2929a0b75\") " Jul 21 16:24:08.979799 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:24:08.979752 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9707e0e1-e63c-4336-b2c1-aba2929a0b75-istio-envoy\") pod \"9707e0e1-e63c-4336-b2c1-aba2929a0b75\" (UID: \"9707e0e1-e63c-4336-b2c1-aba2929a0b75\") " Jul 21 16:24:08.979975 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:24:08.979807 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wxrtl\" (UniqueName: \"kubernetes.io/projected/9707e0e1-e63c-4336-b2c1-aba2929a0b75-kube-api-access-wxrtl\") pod \"9707e0e1-e63c-4336-b2c1-aba2929a0b75\" (UID: \"9707e0e1-e63c-4336-b2c1-aba2929a0b75\") " Jul 21 16:24:08.979975 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:24:08.979844 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9707e0e1-e63c-4336-b2c1-aba2929a0b75-istio-data\") pod \"9707e0e1-e63c-4336-b2c1-aba2929a0b75\" (UID: \"9707e0e1-e63c-4336-b2c1-aba2929a0b75\") " Jul 21 16:24:08.979975 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:24:08.979878 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9707e0e1-e63c-4336-b2c1-aba2929a0b75-workload-socket\") pod \"9707e0e1-e63c-4336-b2c1-aba2929a0b75\" (UID: \"9707e0e1-e63c-4336-b2c1-aba2929a0b75\") " Jul 21 16:24:08.979975 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:24:08.979920 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9707e0e1-e63c-4336-b2c1-aba2929a0b75-istio-token\") pod \"9707e0e1-e63c-4336-b2c1-aba2929a0b75\" (UID: \"9707e0e1-e63c-4336-b2c1-aba2929a0b75\") " Jul 21 16:24:08.979975 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:24:08.979945 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9707e0e1-e63c-4336-b2c1-aba2929a0b75-istio-podinfo\") pod \"9707e0e1-e63c-4336-b2c1-aba2929a0b75\" (UID: \"9707e0e1-e63c-4336-b2c1-aba2929a0b75\") " Jul 21 16:24:08.979975 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:24:08.979922 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9707e0e1-e63c-4336-b2c1-aba2929a0b75-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9707e0e1-e63c-4336-b2c1-aba2929a0b75" (UID: "9707e0e1-e63c-4336-b2c1-aba2929a0b75"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 16:24:08.980269 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:24:08.979990 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9707e0e1-e63c-4336-b2c1-aba2929a0b75-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9707e0e1-e63c-4336-b2c1-aba2929a0b75" (UID: "9707e0e1-e63c-4336-b2c1-aba2929a0b75"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 16:24:08.980269 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:24:08.980153 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9707e0e1-e63c-4336-b2c1-aba2929a0b75-istio-data" (OuterVolumeSpecName: "istio-data") pod "9707e0e1-e63c-4336-b2c1-aba2929a0b75" (UID: "9707e0e1-e63c-4336-b2c1-aba2929a0b75"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 16:24:08.980269 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:24:08.980171 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9707e0e1-e63c-4336-b2c1-aba2929a0b75-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9707e0e1-e63c-4336-b2c1-aba2929a0b75" (UID: "9707e0e1-e63c-4336-b2c1-aba2929a0b75"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 16:24:08.980269 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:24:08.980078 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9707e0e1-e63c-4336-b2c1-aba2929a0b75-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9707e0e1-e63c-4336-b2c1-aba2929a0b75" (UID: "9707e0e1-e63c-4336-b2c1-aba2929a0b75"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 16:24:08.980513 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:24:08.980290 2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9707e0e1-e63c-4336-b2c1-aba2929a0b75-workload-certs\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 16:24:08.980513 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:24:08.980309 2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9707e0e1-e63c-4336-b2c1-aba2929a0b75-istio-data\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 16:24:08.980513 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:24:08.980347 2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9707e0e1-e63c-4336-b2c1-aba2929a0b75-workload-socket\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 16:24:08.980513 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:24:08.980360 2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9707e0e1-e63c-4336-b2c1-aba2929a0b75-credential-socket\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 16:24:08.980513 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:24:08.980369 2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9707e0e1-e63c-4336-b2c1-aba2929a0b75-istiod-ca-cert\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 16:24:08.982217 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:24:08.982189 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9707e0e1-e63c-4336-b2c1-aba2929a0b75-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9707e0e1-e63c-4336-b2c1-aba2929a0b75" (UID: "9707e0e1-e63c-4336-b2c1-aba2929a0b75"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 16:24:08.982391 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:24:08.982260 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9707e0e1-e63c-4336-b2c1-aba2929a0b75-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9707e0e1-e63c-4336-b2c1-aba2929a0b75" (UID: "9707e0e1-e63c-4336-b2c1-aba2929a0b75"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 21 16:24:08.982591 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:24:08.982566 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9707e0e1-e63c-4336-b2c1-aba2929a0b75-istio-token" (OuterVolumeSpecName: "istio-token") pod "9707e0e1-e63c-4336-b2c1-aba2929a0b75" (UID: "9707e0e1-e63c-4336-b2c1-aba2929a0b75"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 16:24:08.982659 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:24:08.982571 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9707e0e1-e63c-4336-b2c1-aba2929a0b75-kube-api-access-wxrtl" (OuterVolumeSpecName: "kube-api-access-wxrtl") pod "9707e0e1-e63c-4336-b2c1-aba2929a0b75" (UID: "9707e0e1-e63c-4336-b2c1-aba2929a0b75"). InnerVolumeSpecName "kube-api-access-wxrtl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 16:24:09.081077 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:24:09.081048 2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9707e0e1-e63c-4336-b2c1-aba2929a0b75-istio-envoy\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 16:24:09.081077 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:24:09.081075 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxrtl\" (UniqueName: \"kubernetes.io/projected/9707e0e1-e63c-4336-b2c1-aba2929a0b75-kube-api-access-wxrtl\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 16:24:09.081077 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:24:09.081085 2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9707e0e1-e63c-4336-b2c1-aba2929a0b75-istio-token\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 16:24:09.081319 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:24:09.081094 2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9707e0e1-e63c-4336-b2c1-aba2929a0b75-istio-podinfo\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 16:24:09.205344 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:24:09.205294 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-524b80-openshift-default-9cb6b55d7-lwd66"] Jul 21 16:24:09.210869 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:24:09.210846 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-524b80-openshift-default-9cb6b55d7-lwd66"] Jul 21 16:24:10.480887 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:24:10.480852 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9707e0e1-e63c-4336-b2c1-aba2929a0b75" path="/var/lib/kubelet/pods/9707e0e1-e63c-4336-b2c1-aba2929a0b75/volumes" Jul 21 16:24:16.538288 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:24:16.538252 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-j4c8p"] Jul 21 16:24:28.957958 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:24:28.957928 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kn9fn_02f3d0c6-4116-4134-ba51-1c02c73bef05/ovn-acl-logging/0.log" Jul 21 16:24:28.972292 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:24:28.972265 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kn9fn_02f3d0c6-4116-4134-ba51-1c02c73bef05/ovn-acl-logging/0.log" Jul 21 16:27:10.498499 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:10.498454 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-bbeb7d-openshift-default-7d4d479875-lgrtw"] Jul 21 16:27:10.499017 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:10.498670 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-bbeb7d-openshift-default-7d4d479875-lgrtw" podUID="cb72b135-46ae-47cf-8927-74fa140deb48" containerName="istio-proxy" containerID="cri-o://b545cdbe78b5246778eadc82d417b9c64968c5839afc626317168e70c0416b5b" gracePeriod=30 Jul 21 16:27:15.526520 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:15.526491 2563 patch_prober.go:28] interesting pod/e2e-shared-b-bbeb7d-openshift-default-7d4d479875-lgrtw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Jul 21 16:27:15.526819 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:15.526554 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-bbeb7d-openshift-default-7d4d479875-lgrtw" podUID="cb72b135-46ae-47cf-8927-74fa140deb48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Jul 21 16:27:15.759559 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:15.759535 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-bbeb7d-openshift-default-7d4d479875-lgrtw" Jul 21 16:27:15.821868 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:15.821769 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb72b135-46ae-47cf-8927-74fa140deb48-istio-token\") pod \"cb72b135-46ae-47cf-8927-74fa140deb48\" (UID: \"cb72b135-46ae-47cf-8927-74fa140deb48\") " Jul 21 16:27:15.821868 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:15.821816 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b8rgk\" (UniqueName: \"kubernetes.io/projected/cb72b135-46ae-47cf-8927-74fa140deb48-kube-api-access-b8rgk\") pod \"cb72b135-46ae-47cf-8927-74fa140deb48\" (UID: \"cb72b135-46ae-47cf-8927-74fa140deb48\") " Jul 21 16:27:15.821868 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:15.821839 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb72b135-46ae-47cf-8927-74fa140deb48-istio-podinfo\") pod \"cb72b135-46ae-47cf-8927-74fa140deb48\" (UID: \"cb72b135-46ae-47cf-8927-74fa140deb48\") " Jul 21 16:27:15.821868 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:15.821859 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb72b135-46ae-47cf-8927-74fa140deb48-workload-certs\") pod \"cb72b135-46ae-47cf-8927-74fa140deb48\" (UID: \"cb72b135-46ae-47cf-8927-74fa140deb48\") " Jul 21 16:27:15.821868 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:15.821876 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb72b135-46ae-47cf-8927-74fa140deb48-istiod-ca-cert\") pod \"cb72b135-46ae-47cf-8927-74fa140deb48\" (UID: \"cb72b135-46ae-47cf-8927-74fa140deb48\") " Jul 21 16:27:15.822260 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:15.821900 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb72b135-46ae-47cf-8927-74fa140deb48-workload-socket\") pod \"cb72b135-46ae-47cf-8927-74fa140deb48\" (UID: \"cb72b135-46ae-47cf-8927-74fa140deb48\") " Jul 21 16:27:15.822260 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:15.821938 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb72b135-46ae-47cf-8927-74fa140deb48-istio-data\") pod \"cb72b135-46ae-47cf-8927-74fa140deb48\" (UID: \"cb72b135-46ae-47cf-8927-74fa140deb48\") " Jul 21 16:27:15.822260 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:15.822004 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb72b135-46ae-47cf-8927-74fa140deb48-credential-socket\") pod \"cb72b135-46ae-47cf-8927-74fa140deb48\" (UID: \"cb72b135-46ae-47cf-8927-74fa140deb48\") " Jul 21 16:27:15.822260 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:15.822057 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb72b135-46ae-47cf-8927-74fa140deb48-istio-envoy\") pod \"cb72b135-46ae-47cf-8927-74fa140deb48\" (UID: \"cb72b135-46ae-47cf-8927-74fa140deb48\") " Jul 21 16:27:15.822260 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:15.822107 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb72b135-46ae-47cf-8927-74fa140deb48-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "cb72b135-46ae-47cf-8927-74fa140deb48" (UID: "cb72b135-46ae-47cf-8927-74fa140deb48"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 16:27:15.822541 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:15.822279 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb72b135-46ae-47cf-8927-74fa140deb48-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "cb72b135-46ae-47cf-8927-74fa140deb48" (UID: "cb72b135-46ae-47cf-8927-74fa140deb48"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 16:27:15.822541 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:15.822307 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb72b135-46ae-47cf-8927-74fa140deb48-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "cb72b135-46ae-47cf-8927-74fa140deb48" (UID: "cb72b135-46ae-47cf-8927-74fa140deb48"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 16:27:15.822541 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:15.822320 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb72b135-46ae-47cf-8927-74fa140deb48-istio-data" (OuterVolumeSpecName: "istio-data") pod "cb72b135-46ae-47cf-8927-74fa140deb48" (UID: "cb72b135-46ae-47cf-8927-74fa140deb48"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 16:27:15.822541 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:15.822487 2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb72b135-46ae-47cf-8927-74fa140deb48-credential-socket\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 16:27:15.822541 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:15.822508 2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb72b135-46ae-47cf-8927-74fa140deb48-workload-certs\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 16:27:15.822541 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:15.822523 2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb72b135-46ae-47cf-8927-74fa140deb48-workload-socket\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 16:27:15.822541 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:15.822536 2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb72b135-46ae-47cf-8927-74fa140deb48-istio-data\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 16:27:15.822823 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:15.822547 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb72b135-46ae-47cf-8927-74fa140deb48-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "cb72b135-46ae-47cf-8927-74fa140deb48" (UID: "cb72b135-46ae-47cf-8927-74fa140deb48"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 16:27:15.824231 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:15.824210 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb72b135-46ae-47cf-8927-74fa140deb48-istio-token" (OuterVolumeSpecName: "istio-token") pod "cb72b135-46ae-47cf-8927-74fa140deb48" (UID: "cb72b135-46ae-47cf-8927-74fa140deb48"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 16:27:15.824550 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:15.824533 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb72b135-46ae-47cf-8927-74fa140deb48-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "cb72b135-46ae-47cf-8927-74fa140deb48" (UID: "cb72b135-46ae-47cf-8927-74fa140deb48"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 16:27:15.824642 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:15.824626 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cb72b135-46ae-47cf-8927-74fa140deb48-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "cb72b135-46ae-47cf-8927-74fa140deb48" (UID: "cb72b135-46ae-47cf-8927-74fa140deb48"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 21 16:27:15.824725 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:15.824640 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb72b135-46ae-47cf-8927-74fa140deb48-kube-api-access-b8rgk" (OuterVolumeSpecName: "kube-api-access-b8rgk") pod "cb72b135-46ae-47cf-8927-74fa140deb48" (UID: "cb72b135-46ae-47cf-8927-74fa140deb48"). InnerVolumeSpecName "kube-api-access-b8rgk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 16:27:15.923629 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:15.923583 2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb72b135-46ae-47cf-8927-74fa140deb48-istio-envoy\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 16:27:15.923629 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:15.923621 2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb72b135-46ae-47cf-8927-74fa140deb48-istio-token\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 16:27:15.923629 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:15.923630 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8rgk\" (UniqueName: \"kubernetes.io/projected/cb72b135-46ae-47cf-8927-74fa140deb48-kube-api-access-b8rgk\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 16:27:15.923629 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:15.923642 2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb72b135-46ae-47cf-8927-74fa140deb48-istio-podinfo\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 16:27:15.923926 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:15.923651 2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb72b135-46ae-47cf-8927-74fa140deb48-istiod-ca-cert\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\"" Jul 21 16:27:15.957105 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:15.957073 2563 generic.go:356] "Generic (PLEG): container finished" podID="cb72b135-46ae-47cf-8927-74fa140deb48" containerID="b545cdbe78b5246778eadc82d417b9c64968c5839afc626317168e70c0416b5b" exitCode=0 Jul 21 16:27:15.957270 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:15.957115 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-bbeb7d-openshift-default-7d4d479875-lgrtw" Jul 21 16:27:15.957270 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:15.957113 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-bbeb7d-openshift-default-7d4d479875-lgrtw" event={"ID":"cb72b135-46ae-47cf-8927-74fa140deb48","Type":"ContainerDied","Data":"b545cdbe78b5246778eadc82d417b9c64968c5839afc626317168e70c0416b5b"} Jul 21 16:27:15.957270 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:15.957245 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-bbeb7d-openshift-default-7d4d479875-lgrtw" event={"ID":"cb72b135-46ae-47cf-8927-74fa140deb48","Type":"ContainerDied","Data":"1ea9bea69a7fce763483b87bdcb1d9e1aa035f055ae808184ad166083ffcb702"} Jul 21 16:27:15.957403 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:15.957274 2563 scope.go:117] "RemoveContainer" containerID="b545cdbe78b5246778eadc82d417b9c64968c5839afc626317168e70c0416b5b" Jul 21 16:27:15.966955 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:15.966939 2563 scope.go:117] "RemoveContainer" containerID="daeb82b21e0516b4fc3afb9cb6755bd626e3b902cddb3ecb2dc38b7cd5292015" Jul 21 16:27:15.975184 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:15.975167 2563 scope.go:117] "RemoveContainer" containerID="b545cdbe78b5246778eadc82d417b9c64968c5839afc626317168e70c0416b5b" Jul 21 16:27:15.975454 ip-10-0-135-176 kubenswrapper[2563]: E0721 16:27:15.975435 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b545cdbe78b5246778eadc82d417b9c64968c5839afc626317168e70c0416b5b\": container with ID starting with b545cdbe78b5246778eadc82d417b9c64968c5839afc626317168e70c0416b5b not found: ID does not exist" containerID="b545cdbe78b5246778eadc82d417b9c64968c5839afc626317168e70c0416b5b" Jul 21 16:27:15.975523 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:15.975464 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b545cdbe78b5246778eadc82d417b9c64968c5839afc626317168e70c0416b5b"} err="failed to get container status \"b545cdbe78b5246778eadc82d417b9c64968c5839afc626317168e70c0416b5b\": rpc error: code = NotFound desc = could not find container \"b545cdbe78b5246778eadc82d417b9c64968c5839afc626317168e70c0416b5b\": container with ID starting with b545cdbe78b5246778eadc82d417b9c64968c5839afc626317168e70c0416b5b not found: ID does not exist" Jul 21 16:27:15.975523 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:15.975483 2563 scope.go:117] "RemoveContainer" containerID="daeb82b21e0516b4fc3afb9cb6755bd626e3b902cddb3ecb2dc38b7cd5292015" Jul 21 16:27:15.975743 ip-10-0-135-176 kubenswrapper[2563]: E0721 16:27:15.975728 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"daeb82b21e0516b4fc3afb9cb6755bd626e3b902cddb3ecb2dc38b7cd5292015\": container with ID starting with daeb82b21e0516b4fc3afb9cb6755bd626e3b902cddb3ecb2dc38b7cd5292015 not found: ID does not exist" containerID="daeb82b21e0516b4fc3afb9cb6755bd626e3b902cddb3ecb2dc38b7cd5292015" Jul 21 16:27:15.975791 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:15.975745 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"daeb82b21e0516b4fc3afb9cb6755bd626e3b902cddb3ecb2dc38b7cd5292015"} err="failed to get container status \"daeb82b21e0516b4fc3afb9cb6755bd626e3b902cddb3ecb2dc38b7cd5292015\": rpc error: code = NotFound desc = could not find container \"daeb82b21e0516b4fc3afb9cb6755bd626e3b902cddb3ecb2dc38b7cd5292015\": container with ID starting with daeb82b21e0516b4fc3afb9cb6755bd626e3b902cddb3ecb2dc38b7cd5292015 not found: ID does not exist" Jul 21 16:27:15.980039 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:15.980019 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-bbeb7d-openshift-default-7d4d479875-lgrtw"] Jul 21 16:27:15.984098 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:15.984079 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-bbeb7d-openshift-default-7d4d479875-lgrtw"] Jul 21 16:27:16.480424 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:16.480390 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cb72b135-46ae-47cf-8927-74fa140deb48" path="/var/lib/kubelet/pods/cb72b135-46ae-47cf-8927-74fa140deb48/volumes" Jul 21 16:27:16.734647 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:16.734571 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-jrpr5_41551a7a-bb35-4aac-8b24-d5a407c9364d/manager/0.log" Jul 21 16:27:16.856810 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:16.856779 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-mfgpf_027898ef-0c27-4aa0-84d8-71a02817fb7d/manager/0.log" Jul 21 16:27:17.208590 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:17.208560 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-7pcnb_3a2888a0-305c-4d9a-9f53-46bed879f2c3/manager/0.log" Jul 21 16:27:17.318406 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:17.318384 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-85b74964cc-5fl76_bf078948-6939-460b-99f5-12aba0a5e33d/copy-manifests/0.log" Jul 21 16:27:17.325224 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:17.325207 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-85b74964cc-5fl76_bf078948-6939-460b-99f5-12aba0a5e33d/manager/0.log" Jul 21 16:27:20.802588 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:20.802500 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-558c6fb9c4-p288c_e4b9fea5-b9a4-4844-888a-9a4835b73382/manager/0.log" Jul 21 16:27:21.029909 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:21.029882 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-c9z78_1da5d3b2-039e-41b4-9df7-23a127b47a57/kuadrant-console-plugin/0.log" Jul 21 16:27:21.145153 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:21.145121 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-7569bc7c69-z7r6n_2f00f1fa-667b-4345-911f-8dd3c0404da1/manager/0.log" Jul 21 16:27:21.260148 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:21.260120 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-69574b596d-j4c8p_40d18d90-5c0e-48f2-bf99-eed3392a66ee/limitador/0.log" Jul 21 16:27:21.370026 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:21.370000 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-7d6bb9cf99-xw7fm_b7a3e4c2-4cf1-48c9-a69c-5f3ba844253d/manager/0.log" Jul 21 16:27:21.752033 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:21.752001 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7f964fb678qkp8b_72f38b39-1295-435c-b9ef-fc2c03990034/istio-proxy/0.log" Jul 21 16:27:22.121114 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:22.121088 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7db96775b-tckp8_6a378105-bb9e-4ca6-9887-8a0f3d6c6c94/kube-auth-proxy/0.log" Jul 21 16:27:22.242979 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:22.242943 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-klhxd_e2e9e18c-5caf-48ef-82d0-16411ada9674/istio-proxy/1.log" Jul 21 16:27:23.433107 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:23.433073 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-fd497b998-txmbc_5cec8b20-be0d-4f3d-a308-6d6056a96db7/router/0.log" Jul 21 16:27:24.345778 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:24.345748 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-dfzpn_04b26501-4761-44b0-a36a-48caa3a51787/main/0.log" Jul 21 16:27:33.782099 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:33.782063 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-5nz6r/must-gather-dtn2z"] Jul 21 16:27:33.782538 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:33.782442 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9707e0e1-e63c-4336-b2c1-aba2929a0b75" containerName="istio-proxy" Jul 21 16:27:33.782538 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:33.782453 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="9707e0e1-e63c-4336-b2c1-aba2929a0b75" containerName="istio-proxy" Jul 21 16:27:33.782538 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:33.782470 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb72b135-46ae-47cf-8927-74fa140deb48" containerName="istio-proxy" Jul 21 16:27:33.782538 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:33.782475 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb72b135-46ae-47cf-8927-74fa140deb48" containerName="istio-proxy" Jul 21 16:27:33.782538 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:33.782504 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7efc78e-2a98-4fdf-806c-aca2d06969f3" containerName="main" Jul 21 16:27:33.782538 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:33.782509 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7efc78e-2a98-4fdf-806c-aca2d06969f3" containerName="main" Jul 21 16:27:33.782762 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:33.782568 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb72b135-46ae-47cf-8927-74fa140deb48" containerName="istio-proxy" Jul 21 16:27:33.782762 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:33.782575 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="9707e0e1-e63c-4336-b2c1-aba2929a0b75" containerName="istio-proxy" Jul 21 16:27:33.782762 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:33.782581 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d7efc78e-2a98-4fdf-806c-aca2d06969f3" containerName="main" Jul 21 16:27:33.782762 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:33.782588 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb72b135-46ae-47cf-8927-74fa140deb48" containerName="istio-proxy" Jul 21 16:27:33.782762 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:33.782637 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9707e0e1-e63c-4336-b2c1-aba2929a0b75" containerName="istio-proxy" Jul 21 16:27:33.782762 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:33.782643 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="9707e0e1-e63c-4336-b2c1-aba2929a0b75" containerName="istio-proxy" Jul 21 16:27:33.782762 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:33.782652 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb72b135-46ae-47cf-8927-74fa140deb48" containerName="istio-proxy" Jul 21 16:27:33.782762 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:33.782657 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb72b135-46ae-47cf-8927-74fa140deb48" containerName="istio-proxy" Jul 21 16:27:33.782762 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:33.782719 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="9707e0e1-e63c-4336-b2c1-aba2929a0b75" containerName="istio-proxy" Jul 21 16:27:33.787020 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:33.786999 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5nz6r/must-gather-dtn2z" Jul 21 16:27:33.789829 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:33.789812 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-5nz6r\"/\"default-dockercfg-zjl95\"" Jul 21 16:27:33.789964 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:33.789828 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5nz6r\"/\"openshift-service-ca.crt\"" Jul 21 16:27:33.790858 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:33.790840 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5nz6r\"/\"kube-root-ca.crt\"" Jul 21 16:27:33.799281 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:33.799259 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5nz6r/must-gather-dtn2z"] Jul 21 16:27:33.886967 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:33.886931 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/76f5d21d-768b-4b8c-a269-26331212598a-must-gather-output\") pod \"must-gather-dtn2z\" (UID: \"76f5d21d-768b-4b8c-a269-26331212598a\") " pod="openshift-must-gather-5nz6r/must-gather-dtn2z" Jul 21 16:27:33.886967 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:33.886970 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jslpb\" (UniqueName: \"kubernetes.io/projected/76f5d21d-768b-4b8c-a269-26331212598a-kube-api-access-jslpb\") pod \"must-gather-dtn2z\" (UID: \"76f5d21d-768b-4b8c-a269-26331212598a\") " pod="openshift-must-gather-5nz6r/must-gather-dtn2z" Jul 21 16:27:33.988346 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:33.988297 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/76f5d21d-768b-4b8c-a269-26331212598a-must-gather-output\") pod \"must-gather-dtn2z\" (UID: \"76f5d21d-768b-4b8c-a269-26331212598a\") " pod="openshift-must-gather-5nz6r/must-gather-dtn2z" Jul 21 16:27:33.988514 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:33.988358 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jslpb\" (UniqueName: \"kubernetes.io/projected/76f5d21d-768b-4b8c-a269-26331212598a-kube-api-access-jslpb\") pod \"must-gather-dtn2z\" (UID: \"76f5d21d-768b-4b8c-a269-26331212598a\") " pod="openshift-must-gather-5nz6r/must-gather-dtn2z" Jul 21 16:27:33.988637 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:33.988617 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/76f5d21d-768b-4b8c-a269-26331212598a-must-gather-output\") pod \"must-gather-dtn2z\" (UID: \"76f5d21d-768b-4b8c-a269-26331212598a\") " pod="openshift-must-gather-5nz6r/must-gather-dtn2z" Jul 21 16:27:33.999965 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:33.999934 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jslpb\" (UniqueName: \"kubernetes.io/projected/76f5d21d-768b-4b8c-a269-26331212598a-kube-api-access-jslpb\") pod \"must-gather-dtn2z\" (UID: \"76f5d21d-768b-4b8c-a269-26331212598a\") " pod="openshift-must-gather-5nz6r/must-gather-dtn2z" Jul 21 16:27:34.096952 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:34.096912 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5nz6r/must-gather-dtn2z" Jul 21 16:27:34.236757 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:34.236695 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5nz6r/must-gather-dtn2z"] Jul 21 16:27:34.239207 ip-10-0-135-176 kubenswrapper[2563]: W0721 16:27:34.239173 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod76f5d21d_768b_4b8c_a269_26331212598a.slice/crio-2f105e8e16753fa863d4fde5a87a83d8418b759a3323247335262a36c6a66075 WatchSource:0}: Error finding container 2f105e8e16753fa863d4fde5a87a83d8418b759a3323247335262a36c6a66075: Status 404 returned error can't find the container with id 2f105e8e16753fa863d4fde5a87a83d8418b759a3323247335262a36c6a66075 Jul 21 16:27:34.241200 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:34.241184 2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 21 16:27:35.060931 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:35.060896 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5nz6r/must-gather-dtn2z" event={"ID":"76f5d21d-768b-4b8c-a269-26331212598a","Type":"ContainerStarted","Data":"2f105e8e16753fa863d4fde5a87a83d8418b759a3323247335262a36c6a66075"} Jul 21 16:27:36.068657 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:36.068616 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5nz6r/must-gather-dtn2z" event={"ID":"76f5d21d-768b-4b8c-a269-26331212598a","Type":"ContainerStarted","Data":"5c89183171275388515e855dbb63017368ff02ecb23c6f037b3b8d8010ae101f"} Jul 21 16:27:36.068657 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:36.068658 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5nz6r/must-gather-dtn2z" event={"ID":"76f5d21d-768b-4b8c-a269-26331212598a","Type":"ContainerStarted","Data":"6a85c06c9c6b390a181a2eedf3e928ec2d876393b0d316741bafa95c45be4b3a"} Jul 21 16:27:36.085169 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:36.085115 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-5nz6r/must-gather-dtn2z" podStartSLOduration=2.104590434 podStartE2EDuration="3.08509637s" totalImagesPullingTime="980.505936ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 16:27:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 16:27:36.084636278 +0000 UTC m=+5288.080421734" watchObservedRunningTime="2026-07-21 16:27:36.08509637 +0000 UTC m=+5288.080881787" Jul 21 16:27:36.812379 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:36.812348 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-jttxg_e706cd7a-7458-4f77-b02d-3fd5837d735d/global-pull-secret-syncer/0.log" Jul 21 16:27:37.004000 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:37.003962 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-m4pp8_216f52c0-8c33-426f-b2eb-244a5787bdeb/konnectivity-agent/0.log" Jul 21 16:27:37.078744 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:37.078717 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-176.ec2.internal_b953891c06139c6cc3d45f0903be0d66/haproxy/0.log" Jul 21 16:27:41.842037 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:41.841974 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-558c6fb9c4-p288c_e4b9fea5-b9a4-4844-888a-9a4835b73382/manager/0.log" Jul 21 16:27:41.888690 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:41.888599 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-c9z78_1da5d3b2-039e-41b4-9df7-23a127b47a57/kuadrant-console-plugin/0.log" Jul 21 16:27:42.036402 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:42.034046 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-7569bc7c69-z7r6n_2f00f1fa-667b-4345-911f-8dd3c0404da1/manager/0.log" Jul 21 16:27:42.076215 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:42.076184 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-69574b596d-j4c8p_40d18d90-5c0e-48f2-bf99-eed3392a66ee/limitador/0.log" Jul 21 16:27:42.160348 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:42.160256 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-7d6bb9cf99-xw7fm_b7a3e4c2-4cf1-48c9-a69c-5f3ba844253d/manager/0.log" Jul 21 16:27:43.805478 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:43.805443 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-76cf8966f8-6mtbk_0f91a500-cd56-4cc9-8f38-d60a5e8a0562/cluster-monitoring-operator/0.log" Jul 21 16:27:43.973083 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:43.973052 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-49gbt_db2b1697-e840-48cf-bd68-ac5c48dca546/node-exporter/0.log" Jul 21 16:27:43.998393 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:43.998301 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-49gbt_db2b1697-e840-48cf-bd68-ac5c48dca546/kube-rbac-proxy/0.log" Jul 21 16:27:44.020442 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:44.020413 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-49gbt_db2b1697-e840-48cf-bd68-ac5c48dca546/init-textfile/0.log" Jul 21 16:27:44.654127 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:44.654094 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-756c494bbd-kwhhc_74f05f06-466c-4a88-bb75-58634a88d2d5/thanos-query/0.log" Jul 21 16:27:44.679094 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:44.679061 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-756c494bbd-kwhhc_74f05f06-466c-4a88-bb75-58634a88d2d5/kube-rbac-proxy-web/0.log" Jul 21 16:27:44.703828 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:44.703799 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-756c494bbd-kwhhc_74f05f06-466c-4a88-bb75-58634a88d2d5/kube-rbac-proxy/0.log" Jul 21 16:27:44.724171 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:44.724133 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-756c494bbd-kwhhc_74f05f06-466c-4a88-bb75-58634a88d2d5/prom-label-proxy/0.log" Jul 21 16:27:44.744936 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:44.744887 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-756c494bbd-kwhhc_74f05f06-466c-4a88-bb75-58634a88d2d5/kube-rbac-proxy-rules/0.log" Jul 21 16:27:44.763918 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:44.763887 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-756c494bbd-kwhhc_74f05f06-466c-4a88-bb75-58634a88d2d5/kube-rbac-proxy-metrics/0.log" Jul 21 16:27:47.437209 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:47.437168 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-8558d588c-xbtgr_5fc95f8b-070c-4199-9bbd-b184c6222f03/console/0.log" Jul 21 16:27:47.523666 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:47.523629 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-5nz6r/perf-node-gather-daemonset-hkk9h"] Jul 21 16:27:47.530025 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:47.529998 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5nz6r/perf-node-gather-daemonset-hkk9h" Jul 21 16:27:47.539749 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:47.539536 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5nz6r/perf-node-gather-daemonset-hkk9h"] Jul 21 16:27:47.640845 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:47.640810 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2f88eb01-2152-4d38-8efd-4e2d2955280d-sys\") pod \"perf-node-gather-daemonset-hkk9h\" (UID: \"2f88eb01-2152-4d38-8efd-4e2d2955280d\") " pod="openshift-must-gather-5nz6r/perf-node-gather-daemonset-hkk9h" Jul 21 16:27:47.641110 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:47.641094 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2f88eb01-2152-4d38-8efd-4e2d2955280d-proc\") pod \"perf-node-gather-daemonset-hkk9h\" (UID: \"2f88eb01-2152-4d38-8efd-4e2d2955280d\") " pod="openshift-must-gather-5nz6r/perf-node-gather-daemonset-hkk9h" Jul 21 16:27:47.641230 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:47.641215 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2f88eb01-2152-4d38-8efd-4e2d2955280d-podres\") pod \"perf-node-gather-daemonset-hkk9h\" (UID: \"2f88eb01-2152-4d38-8efd-4e2d2955280d\") " pod="openshift-must-gather-5nz6r/perf-node-gather-daemonset-hkk9h" Jul 21 16:27:47.641379 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:47.641364 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2f88eb01-2152-4d38-8efd-4e2d2955280d-lib-modules\") pod \"perf-node-gather-daemonset-hkk9h\" (UID: \"2f88eb01-2152-4d38-8efd-4e2d2955280d\") " pod="openshift-must-gather-5nz6r/perf-node-gather-daemonset-hkk9h" Jul 21 16:27:47.641637 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:47.641616 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k824l\" (UniqueName: \"kubernetes.io/projected/2f88eb01-2152-4d38-8efd-4e2d2955280d-kube-api-access-k824l\") pod \"perf-node-gather-daemonset-hkk9h\" (UID: \"2f88eb01-2152-4d38-8efd-4e2d2955280d\") " pod="openshift-must-gather-5nz6r/perf-node-gather-daemonset-hkk9h" Jul 21 16:27:47.744092 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:47.743136 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2f88eb01-2152-4d38-8efd-4e2d2955280d-sys\") pod \"perf-node-gather-daemonset-hkk9h\" (UID: \"2f88eb01-2152-4d38-8efd-4e2d2955280d\") " pod="openshift-must-gather-5nz6r/perf-node-gather-daemonset-hkk9h" Jul 21 16:27:47.744092 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:47.743177 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2f88eb01-2152-4d38-8efd-4e2d2955280d-proc\") pod \"perf-node-gather-daemonset-hkk9h\" (UID: \"2f88eb01-2152-4d38-8efd-4e2d2955280d\") " pod="openshift-must-gather-5nz6r/perf-node-gather-daemonset-hkk9h" Jul 21 16:27:47.744092 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:47.743196 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2f88eb01-2152-4d38-8efd-4e2d2955280d-podres\") pod \"perf-node-gather-daemonset-hkk9h\" (UID: \"2f88eb01-2152-4d38-8efd-4e2d2955280d\") " pod="openshift-must-gather-5nz6r/perf-node-gather-daemonset-hkk9h" Jul 21 16:27:47.744092 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:47.743214 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2f88eb01-2152-4d38-8efd-4e2d2955280d-lib-modules\") pod \"perf-node-gather-daemonset-hkk9h\" (UID: \"2f88eb01-2152-4d38-8efd-4e2d2955280d\") " pod="openshift-must-gather-5nz6r/perf-node-gather-daemonset-hkk9h" Jul 21 16:27:47.744092 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:47.743268 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k824l\" (UniqueName: \"kubernetes.io/projected/2f88eb01-2152-4d38-8efd-4e2d2955280d-kube-api-access-k824l\") pod \"perf-node-gather-daemonset-hkk9h\" (UID: \"2f88eb01-2152-4d38-8efd-4e2d2955280d\") " pod="openshift-must-gather-5nz6r/perf-node-gather-daemonset-hkk9h" Jul 21 16:27:47.744092 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:47.743748 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2f88eb01-2152-4d38-8efd-4e2d2955280d-sys\") pod \"perf-node-gather-daemonset-hkk9h\" (UID: \"2f88eb01-2152-4d38-8efd-4e2d2955280d\") " pod="openshift-must-gather-5nz6r/perf-node-gather-daemonset-hkk9h" Jul 21 16:27:47.744092 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:47.743818 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2f88eb01-2152-4d38-8efd-4e2d2955280d-proc\") pod \"perf-node-gather-daemonset-hkk9h\" (UID: \"2f88eb01-2152-4d38-8efd-4e2d2955280d\") " pod="openshift-must-gather-5nz6r/perf-node-gather-daemonset-hkk9h" Jul 21 16:27:47.744092 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:47.743928 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2f88eb01-2152-4d38-8efd-4e2d2955280d-podres\") pod \"perf-node-gather-daemonset-hkk9h\" (UID: \"2f88eb01-2152-4d38-8efd-4e2d2955280d\") " pod="openshift-must-gather-5nz6r/perf-node-gather-daemonset-hkk9h" Jul 21 16:27:47.744092 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:47.744011 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2f88eb01-2152-4d38-8efd-4e2d2955280d-lib-modules\") pod \"perf-node-gather-daemonset-hkk9h\" (UID: \"2f88eb01-2152-4d38-8efd-4e2d2955280d\") " pod="openshift-must-gather-5nz6r/perf-node-gather-daemonset-hkk9h" Jul 21 16:27:47.752346 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:47.752291 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k824l\" (UniqueName: \"kubernetes.io/projected/2f88eb01-2152-4d38-8efd-4e2d2955280d-kube-api-access-k824l\") pod \"perf-node-gather-daemonset-hkk9h\" (UID: \"2f88eb01-2152-4d38-8efd-4e2d2955280d\") " pod="openshift-must-gather-5nz6r/perf-node-gather-daemonset-hkk9h" Jul 21 16:27:47.845167 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:47.844913 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5nz6r/perf-node-gather-daemonset-hkk9h" Jul 21 16:27:48.054250 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:48.054083 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5nz6r/perf-node-gather-daemonset-hkk9h"] Jul 21 16:27:48.057494 ip-10-0-135-176 kubenswrapper[2563]: W0721 16:27:48.057451 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod2f88eb01_2152_4d38_8efd_4e2d2955280d.slice/crio-dfdba7c5d4005663799e03d450c08f8b113104790307bb9b20d12034120bbb18 WatchSource:0}: Error finding container dfdba7c5d4005663799e03d450c08f8b113104790307bb9b20d12034120bbb18: Status 404 returned error can't find the container with id dfdba7c5d4005663799e03d450c08f8b113104790307bb9b20d12034120bbb18 Jul 21 16:27:48.173942 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:48.171969 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5nz6r/perf-node-gather-daemonset-hkk9h" event={"ID":"2f88eb01-2152-4d38-8efd-4e2d2955280d","Type":"ContainerStarted","Data":"dfdba7c5d4005663799e03d450c08f8b113104790307bb9b20d12034120bbb18"} Jul 21 16:27:48.254651 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:48.254612 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-78cf4c6f6-jd987_feec54ec-7a62-4ad4-94bf-f28792b7dd4f/volume-data-source-validator/0.log" Jul 21 16:27:49.182210 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:49.182115 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5nz6r/perf-node-gather-daemonset-hkk9h" event={"ID":"2f88eb01-2152-4d38-8efd-4e2d2955280d","Type":"ContainerStarted","Data":"aae60810fc4a5348536001bfda0101de9512cb527332e9406f2c228ac1f158aa"} Jul 21 16:27:49.182656 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:49.182269 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-5nz6r/perf-node-gather-daemonset-hkk9h" Jul 21 16:27:49.201123 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:49.201075 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-5nz6r/perf-node-gather-daemonset-hkk9h" podStartSLOduration=2.201058239 podStartE2EDuration="2.201058239s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 16:27:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 16:27:49.198949573 +0000 UTC m=+5301.194734991" watchObservedRunningTime="2026-07-21 16:27:49.201058239 +0000 UTC m=+5301.196843656" Jul 21 16:27:49.360291 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:49.360262 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-dpzts_28687086-f5b5-4889-afdb-8145abba8b1a/dns/0.log" Jul 21 16:27:49.382275 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:49.382248 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-dpzts_28687086-f5b5-4889-afdb-8145abba8b1a/kube-rbac-proxy/0.log" Jul 21 16:27:49.506807 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:49.506742 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-h2kn2_42b48be4-5000-431e-9d8b-b932609faca0/dns-node-resolver/0.log" Jul 21 16:27:50.249609 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:50.249571 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-9qzhs_4d94d908-6ab5-4463-a878-0b340bb80365/node-ca/0.log" Jul 21 16:27:51.187139 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:51.187103 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7f964fb678qkp8b_72f38b39-1295-435c-b9ef-fc2c03990034/istio-proxy/0.log" Jul 21 16:27:51.457151 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:51.457076 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7db96775b-tckp8_6a378105-bb9e-4ca6-9887-8a0f3d6c6c94/kube-auth-proxy/0.log" Jul 21 16:27:51.487586 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:51.487558 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-klhxd_e2e9e18c-5caf-48ef-82d0-16411ada9674/istio-proxy/0.log" Jul 21 16:27:51.495375 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:51.495347 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-klhxd_e2e9e18c-5caf-48ef-82d0-16411ada9674/istio-proxy/1.log" Jul 21 16:27:51.581385 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:51.581356 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-fd497b998-txmbc_5cec8b20-be0d-4f3d-a308-6d6056a96db7/router/0.log" Jul 21 16:27:52.342057 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:52.342026 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-vnx89_9d1e2d09-f1a3-4fee-a2c3-86647059afe5/serve-healthcheck-canary/0.log" Jul 21 16:27:52.838123 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:52.838095 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-gvst7_8864333e-37a1-4558-ae06-751ef7d6a898/kube-rbac-proxy/0.log" Jul 21 16:27:52.859568 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:52.859540 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-gvst7_8864333e-37a1-4558-ae06-751ef7d6a898/exporter/0.log" Jul 21 16:27:52.882421 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:52.882396 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-gvst7_8864333e-37a1-4558-ae06-751ef7d6a898/extractor/0.log" Jul 21 16:27:55.200311 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:55.200282 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-5nz6r/perf-node-gather-daemonset-hkk9h" Jul 21 16:27:55.239612 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:55.239582 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-jrpr5_41551a7a-bb35-4aac-8b24-d5a407c9364d/manager/0.log" Jul 21 16:27:55.392420 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:55.392391 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-mfgpf_027898ef-0c27-4aa0-84d8-71a02817fb7d/manager/0.log" Jul 21 16:27:55.659488 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:55.659457 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-7pcnb_3a2888a0-305c-4d9a-9f53-46bed879f2c3/manager/0.log" Jul 21 16:27:55.685915 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:55.685882 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-85b74964cc-5fl76_bf078948-6939-460b-99f5-12aba0a5e33d/manager/0.log" Jul 21 16:27:55.715001 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:55.714978 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-85b74964cc-5fl76_bf078948-6939-460b-99f5-12aba0a5e33d/copy-manifests/0.log" Jul 21 16:27:57.874362 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:57.874243 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7f964fb678qkp8b_72f38b39-1295-435c-b9ef-fc2c03990034/istio-proxy/0.log" Jul 21 16:27:58.113439 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:58.113407 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7db96775b-tckp8_6a378105-bb9e-4ca6-9887-8a0f3d6c6c94/kube-auth-proxy/0.log" Jul 21 16:27:58.133487 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:58.133460 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-klhxd_e2e9e18c-5caf-48ef-82d0-16411ada9674/istio-proxy/0.log" Jul 21 16:27:58.141250 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:58.141220 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-klhxd_e2e9e18c-5caf-48ef-82d0-16411ada9674/istio-proxy/1.log" Jul 21 16:27:58.200286 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:58.200258 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-fd497b998-txmbc_5cec8b20-be0d-4f3d-a308-6d6056a96db7/router/0.log" Jul 21 16:27:59.450160 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:59.450091 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7f964fb678qkp8b_72f38b39-1295-435c-b9ef-fc2c03990034/istio-proxy/0.log" Jul 21 16:27:59.698742 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:59.698717 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7db96775b-tckp8_6a378105-bb9e-4ca6-9887-8a0f3d6c6c94/kube-auth-proxy/0.log" Jul 21 16:27:59.722192 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:59.722162 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-klhxd_e2e9e18c-5caf-48ef-82d0-16411ada9674/istio-proxy/0.log" Jul 21 16:27:59.729198 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:59.729169 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-klhxd_e2e9e18c-5caf-48ef-82d0-16411ada9674/istio-proxy/1.log" Jul 21 16:27:59.800450 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:27:59.800420 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-fd497b998-txmbc_5cec8b20-be0d-4f3d-a308-6d6056a96db7/router/0.log" Jul 21 16:28:02.525970 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:28:02.525942 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7cc9c7ffd7-p45k9_5d344c8c-6b9f-44f4-a10a-d75f0d798087/migrator/0.log" Jul 21 16:28:02.549396 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:28:02.549372 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7cc9c7ffd7-p45k9_5d344c8c-6b9f-44f4-a10a-d75f0d798087/graceful-termination/0.log" Jul 21 16:28:02.943214 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:28:02.943186 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7658d9b749-bfv6n_53f7dd9f-230e-4c5c-bc17-bd14332b1923/kube-storage-version-migrator-operator/1.log" Jul 21 16:28:02.944142 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:28:02.944118 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7658d9b749-bfv6n_53f7dd9f-230e-4c5c-bc17-bd14332b1923/kube-storage-version-migrator-operator/0.log" Jul 21 16:28:04.096953 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:28:04.096923 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-8vxjj_8e65c9e9-347b-46b1-a1f5-774fde3b8107/kube-multus/0.log" Jul 21 16:28:04.119525 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:28:04.119501 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8h5jp_327b1f5e-8e27-4572-8025-5c516e59d481/kube-multus-additional-cni-plugins/0.log" Jul 21 16:28:04.142429 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:28:04.142398 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8h5jp_327b1f5e-8e27-4572-8025-5c516e59d481/egress-router-binary-copy/0.log" Jul 21 16:28:04.163599 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:28:04.163576 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8h5jp_327b1f5e-8e27-4572-8025-5c516e59d481/cni-plugins/0.log" Jul 21 16:28:04.185799 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:28:04.185770 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8h5jp_327b1f5e-8e27-4572-8025-5c516e59d481/bond-cni-plugin/0.log" Jul 21 16:28:04.206937 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:28:04.206918 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8h5jp_327b1f5e-8e27-4572-8025-5c516e59d481/routeoverride-cni/0.log" Jul 21 16:28:04.227499 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:28:04.227478 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8h5jp_327b1f5e-8e27-4572-8025-5c516e59d481/whereabouts-cni-bincopy/0.log" Jul 21 16:28:04.245803 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:28:04.245777 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8h5jp_327b1f5e-8e27-4572-8025-5c516e59d481/whereabouts-cni/0.log" Jul 21 16:28:04.671371 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:28:04.671309 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-dh8jm_26de6bb3-eb05-461a-958f-406464dea020/network-metrics-daemon/0.log" Jul 21 16:28:04.700854 ip-10-0-135-176 kubenswrapper[2563]: I0721 16:28:04.700828 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-dh8jm_26de6bb3-eb05-461a-958f-406464dea020/kube-rbac-proxy/0.log"